2 June 2025 - Post source at 🐙

ReactPHP Parallel v2(.1)

With fibers in PHP 8.1 and templates types in Promise v3 the main focus for ReactPHP Parallel v2 is a simpler type safe user facing API. With Promise v3 and Async v4 providing exactly with we need to make this transformation.

Photo by Wolfgang Weiser from Pexels

Early Days

ReactPHP parallel started out as, and still at it’s core is, a Future to Promise converter. (That repository has been archived and the package marked abandoned as the event loop for v2 provides this functionality and is the central place for converting futures and channels.)

That formed the foundation to get results from threads and push all CPU bound operations there. Around the same time I started working on my ReactPHP optimized Docker images. Running threads required a ZTS (Zend Thread Safe) PHP version. And by building my own images it was easy to get ext-parallel and other extensions in the same images matching my needs, it gives me an easy to use set up.

Wrapped it up with a infinite pool of threads and a limited pool, depending on your needs. The former will just run anything while the latter is useful to keep CPU and memory usage under control.

Crazy Ideas

Beyond the initial foundation I wanted to create a way to make promises look sync by creating an object proxy. This proved possible by doing the following things:

• Run that code in a thread
• Create a proxy for any class that returns a promise
• Pass the proxy into the code that runs in the thread
• Run the original object in the main thread and proxy calls and results back and forth using channels

The thing that blew my mind at the time was, aside from threads in PHP, is that it works. It worked very well for small and simple calls. The more complex things got the slower things got. Even wrote a PRS-11 container proxy to be injected in the thread and automate everything.

Created a PSR-15 adapter to use PSR-15 middleware in a ReactPHP HTTP server without blocking the event loop. Unlike the object proxy this ran the blocking code inside a worker thread pool. Also blogged about that a couple of years ago: Building the next generation react/http PSR-15 adapter

Fibers

With PHP 8.1 Fibers turned those ideas upside down. The object proxy lost it’s purpose and has been archived, so are all related packages to it. It also significantly reduced my use for threads to almost 0. Given that green threads (Fibers) are significantly faster than threads this is the way forward for all but CPU-bound use cases.

Personal Usage

My cluster at home however does run some code that runs image conversion and image manipulation (combining a set of images into one bigger image) for WyriMaps. This is where threads shine. The main thread runs Bunny getting messages from RabbitMQ. It uses worker threads to do the heaving image work. The conversion isn’t super time consuming, it most of time takes less then a second per image to convert. Bigger images take longer but that’s as expected. Stitching a couple hundred images together into one however can take minutes.

$this->logger->info($logLine . 'Converting');
$png = $this->threads->run(static function (string $from, string $data): string {
    $tmpFileName = '/tmp/' . md5($from) . '.' . md5($data) . '.blp.png';
    $img         = imagecreatefromblp($data, $from);
    unset($data);
    imagepng($img, $tmpFileName, 0, PNG_NO_FILTER);
    imagedestroy($img);
    $contents = file_get_contents($tmpFileName);
    unlink($tmpFileName);
    return $contents;
}, [
    $from,
    await($sourceFile->getContents()),
]);
$this->logger->info($logLine . 'Writing');
await($destination->putContents($png));
$this->logger->info($logLine . 'Converted');

v2

Version 2 returns the original focus on running CPU-bound operations in threads. But utilizes fibers to have a sync looking API by doing the following two things:

• Anything that returned a promise in v1 new returns the value the future holds
• Anything that returned an observable v1 now returns an iterable

This results in a much simpler API: No need to handle promises or await them, plus Observables can now easily used in a foreach without using external packages.

All of this, however, means you need to run everything inside a fiber. So the examples while a simpler API, are wrapping all those calls inside a fiber, making gain improvement look less. But, that should be extracted away by your HTTP Server, Queue Worker, etc etc. (For sure mine do, but more on that later this year.)

v2.1

v2 was originally planned to be PHP 8.2 and up, but after a support request before release I lowered it to PHP 8.1 with the plan to release v2.1 soon after to bring it back up to PHP 8.2 which turned into PHP 8.3. Mainly due to QA tooling reasons making it easier to maintain it.

Future

There are two features I want to bring to ReactPHP Parallel this year. One is updating the Worker Pool package from v1 to v2. It’s a nice syntactic sugar wrapper around other pools to have dedicated pools/threads to specific types of work. This cleans up the API from:

$threads->run(static function (string $file): string {
    return file_get_contents($file);
}, [__FILE__]);

To:

$worker->perform(new File(__FILE__));

The other is adding support for ReactPHP Streams to the streams package. The goal there is to take a ReactPHP Stream and stream is over a channel to a thread, but also the other way around. This could for example using blocking file parsing in the thread and streaming each bit of data we get into a HTTP response:

new HttpServer(function (ServerRequestInterface $request) {
    $stream = $threads->run(static function (string $file): string {
      foreach ((new Reader($from))->generateRecords() as $row) {
          yield $row + PHP_EOL;
      }
    });
    return new Response(body: $stream);
});

Building ReactPHP Parallel v1 has allowed me to experiment with some really crazy ideas, for that time, that turned out to be working. It thought me how to think about threads and multi core usage on both a single core VPS and Kubernetes, and how that impacts both. And it all started with a need to try and process as many images as possible (which now runs on Serverless using Bref Ironically):

With fibers in PHP 8.1 and templates types in Promise v3 the main focus for ReactPHP Parallel v2 is a simpler type safe user facing API. With Promise v3 and Async v4 providing exactly with we need to make this transformation. Photo by Wolfgang Weiser from Pexels

It’s clearly true that if you add a new parameter to a method on an interface, then that’s a BC break as every concrete implementation of the interface needs to change their signature.

However, Gina commented that you don’t need to use func_get_arg() as concrete implementations can add additional optional arguments.

WHAT?!!!

I didn’t know this and it had never occurred to me to try, so I had to go 3v4l.org and check

<?php
interface Foo
{
    public function bar(int $a): void;
}
class Baz implements Foo
{
    public function bar(int $a, int $b=2): void
    {
        echo "$a: $b";
    }
}
(new Baz)->bar(1,3);

Sure enough, this code works!

I don’t have much of a need for this, but it’s useful to know.

Increasingly, websites are using multi-step login forms. They might first ask for just an email address and only after present users with a password field.

Unfortunately, this doesn’t always work well with password managers. Password managers are such an integral part of the web, I think it’s important as web developers to make this work as best as possible. It’s also an accessibility issue.

Why do websites use multi-step login forms?

The short version is that many companies want to centralize their login / password management systems for all their employees. This lets them change passwords in a single place, and also disable accounts after an employee leaves the company. This is called Single Sign-On (SSO) and is often facilitates using the SAML or OpenID Connect protocols.

When you log in with a web application that supports this, and you enter your email, the first thing the system needs to do is check if a SSO system is in place for your account (or the domainname of your email address), and if so they will redirect out to your sign-on system.

Only if this is not the case, they will present you with a password field.

Another reason is that systems increasingly allow users to auhtenticate with means other than a password, and similarly they first need to know your email address / username before they can know what login flow to present you with.

Password managers

Password managers such as KeepassXC, Bitwarden and 1Password, but also the ‘save password’ feature that’s built into browsers look for HTML forms that are roughly ‘login form shaped’ to do their thing. If something looks like a login form, they will suggest or auto-fill your username and password.

The problem with multi-step login forms that ask for an email address first, is that they no longer look like plain old login forms, which either means that a user has to do more clicks to complete the login, or in the worst case the password manager can’t detect the fields at all and you end up copy-pasting your password. This is pretty annoying and something I see people often yelling about on the internet.

So, how do you fix this?

1. Make sure you have the correct autocomplete attribute

If you have a lone <input> field for the username, it should have the attribute to signal password managers that this is a username field:

<input type="text" autocomplete="username" name="username" required />
or:
<input type="email" autocomplete="username" name="username" required />

2. Add a hidden username field on the password page

The Chrome Wiki recommends that when you collect the email first, and redirect the user after to a password page, to include the username field again (prefilled), and hidden with CSS.

<input
    type="email"
    autocomplete="username"
    name="username"
    required
    value="spam@evertpot.com"
    style="display:none"
/>
<input
    type="password"
    autocomplete="password"
    name="password"
    required
/>

This is particularly great if the user has multiple accounts on your system. The pre-filled username lets a password manager detect which account is being used to log in.

This might also be important for ensuring that password managers offer to save the credentials in the first place.

Many login systems use SMS codes or TOTP (Authenticator App) codes for additional security. Password managers let you store TOTP codes, but in order for your password manager to detect the field, you also need to annotate this correctly:

<input
    type="text"
    autocomplete="one-time-code"
    name="otp"
    required
    pattern="0-9{6}"
    inputmode="numeric"
/>

The key attribute here is ‘autocomplete’. You can even get this auto-filled with SMS codes, if you use a specially formatted SMS, which looks a bit like this:

Your one-time code is 123123
@mydomain #123123

If you want users to be able to auto-login use HTML correctly.

The 1Password site describes that they simulate a ‘click’ on a login button to trigger the login.

But if there’s no <button>, it’s less likely password managers will be able to find it. Using HTML correctly is always a good idea. If you use a component library, check what it generates for buttons and if it’s not <div>’s all the way down.

Password manager developers:

Please also try to follow these recommendations! For better or worse, a lot of applications are going to this multi-step flow. KeepassXC-Browser in particular doesn’t handle these cases well, unless you’re one of of their [blessed sites][11 (feature request).

References and further reading:

• Origin-bound one-time codes delivered via SMS
• Design your website to work best with 1Password
• Password Form Styles that Chromium Understands
• Create Amazing Password Forms (chromium.org)
• <input type="password"> (MDN)
• KeeppassXC-browser rejected feature request to better support multi-step login forms

Thoughts?

If you have further workarounds to make password managers behave well, you can reply to these posts on Mastodon or Bluesky to automatically appear on this website as a comment:

The python-oracledb 3.1 release is now available on PyPI.

This is hot on the heels of last month’s big 3.0 release (see New release of python-oracledb 3.0 Fetches Directly to Data Frames, has Thin Mode AQ, supports Centralized Configuration Providers, and more). We had originally intended just to do a 3.0.x bug fix release, but some enhancements were ready to ship, so we decided to share them with you now as a feature release.

The enhancements are:

• Advanced Queueing in Thin mode now supports JSON payloads, and bulk enqueueing and dequeuing. These changes continue the progress towards Thin mode parity with Thick mode. AQ support was also extended to the python-oracledb async API too.
• The python-oracledb data frame query functionality was extended with support for CLOB, BLOB, and RAW data types.
• Support for scrollable cursors was added to python-oracledb Thin mode. Scrollable cursors are now useable in both modes.
• The other notable change was that we dropped support for Python 3.8, which the upstream maintainers already ceased supporting last year.

There are also plenty of bug fixes. Check the release notes for all the details.

Installing or Upgrading python-oracledb

You can install or upgrade python-oracledb by running:

python3 -m pip install oracledb --upgrade

The pip options --proxy and --user may be useful in some environments. See Installing python-oracledb for details.

Python-oracledb Resources

Python-oracledb is an open source package for the Python Database API specification with many additions to support advanced Oracle Database features. By default, it is a ‘Thin’ driver that is immediately usable without needing any additional install e.g. no Instant Client is required. Python-oracledb is used by frameworks, ORMs, SQL generation libraries, and other projects.

• Installation instructions: python-oracledb.readthedocs.io/en/latest/installation.html
• Documentation: python-oracledb.readthedocs.io/en/latest/index.html
• Discussions: github.com/oracle/python-oracledb/discussions
• Source Code: github.com/oracle/python-oracledb

Replies show up as comments after moderation, and favourites show up as Likes under each of the articles.

When fetching all the likes, my internal API returns the following structure — just like how PDOStatement::fetchAll() returns them:

array(7) {
  [0]=>
  array(8) {
    ["name"]=>
    string(14) "Marcus Bointon"
    ["linked_account"]=>
    string(33) "https://phpc.social/users/Synchro"
  }
  [1]=>
  array(8) {
    ["name"]=>
    string(14) "Derick Rethans"
    ["linked_account"]=>
    string(33) "https://phpc.social/users/derickr"
  }
  [2]=>
  array(8) {
    ["name"]=>
    string(26) "🇵🇸 Álvaro González"
    ["linked_account"]=>
    string(37) "https://mastodon.social/users/kAlvaro"
  }
}

I wanted to sort the likes by name, which turned out harder to be than I had hoped. Mostly because people do odd things with their names sometimes, such as not using an upper-case letter, or by prefixing their names with emojis as you can see in the example above.

If each entry in my array was only a string with a name, and no emoji is prefixed, you can use natcasesort(), but I have both complex objects and an emoji. This means PHP's built-in functions don't cut it.

However, PHP also has the Intl extension, which has a Collator class with a sort method.

To create such a collator in code, use:

$collator = new Collator('root');

The Root Collation is ICU's Default Collation, but as the ICU documentation says:

Not all languages have sorting sequences that correspond with the root collation order because no single sort order can simultaneously encompass the specifics of all the languages. In particular, languages that share a script may sort the same letters differently.

This is good enough for my use case there, where I only want to sort names in a reasonable fashion.

One thing that it does do is to sort lower-case letters before upper-case letters, but this behaviour can be changed by setting CASE_FIRST attribute on the collator to UPPER_FIRST:

$collator->setAttribute(Collator::CASE_FIRST, Collator::UPPER_FIRST);

Letters in non-Latin script will sort after the Latin letters.

However, the Collator->sort() method can only handle arrays of strings, and not complex objects.

To get around this limitation we instead use PHP's built-in usort() function with our own callback. In our callback we can use the Collator::getSortKey() method on each sort element's name array key to create the key that Collator::sort() would have used internally.

For good measure, we also use trim to remove preceding and trailing whitespace. The code to sort the elements now looks like:

$col = new Collator('root');
$col->setAttribute(Collator::CASE_FIRST, Collator::UPPER_FIRST);
usort(
    $allLikes,
    function($a, $b) use ($col) {
        $aName = trim($a['name']);
        $bName = trim($b['name']);
        $aKey = $col->getSortKey($aName);
        $bKey = $col->getSortKey($bName);
        return $aKey <=> $bKey;
    }
);

Now the only problem is that emojis sort before actual letters, so we need to scrub them. Each letter defined in the Unicode standard has properties associated with it. One of those properties is the General Category. If you examine the chart you will fined that for 0061;LATIN SMALL LETTER A the GC is Ll (Letter, Lower-case). The flag emoji from my example is composed of two characters: 1F1F5;REGIONAL INDICATOR SYMBOL LETTER P and 1F1F8;REGIONAL INDICATOR SYMBOL LETTER S. Both of these have as category So (Symbol, Other).

The PCRE library that PHP uses for regular expressions is aware of these categories, and you can select for for them with the \p{…} specifier. The documentation has a page on it.

In our case we want to remove all Symbols (with category S). Changing our trim lines to strip this character class accomplishes this.

When we now combine all of this, we get:

<?php
$allLikes = [
    [ 'name' => 'Marcus Bointon', 'linked_account' => 'https://phpc.social/users/Synchro' ],
    [ 'name' => 'Derick Rethans', 'linked_account' => 'https://phpc.social/users/derickr' ],
    [ 'name' => '🇵🇸 Álvaro González', 'linked_account' => 'https://mastodon.social/users/kAlvaro' ],
];
$col = new Collator('root');
$col->setAttribute(Collator::CASE_FIRST, Collator::UPPER_FIRST);
usort(
    $allLikes,
    function($a, $b) use ($col) {
        $aName = trim(preg_replace('/\p{S}+/u', '', $a['name']));
        $bName = trim(preg_replace('/\p{S}+/u', '', $b['name']));
        $aKey = $col->getSortKey($aName);
        $bKey = $col->getSortKey($bName);
        return $aKey <=> $bKey;
    }
);
var_dump($allLikes);
?>

When we now run the example with the $allLikes array from previously, our result is:

array(3) {
  [0] =>
  array(2) {
    'name' =>
    string(26) "🇵🇸 Álvaro González"
    'linked_account' =>
    string(37) "https://mastodon.social/users/kAlvaro"
  }
  [1] =>
  array(2) {
    'name' =>
    string(14) "Derick Rethans"
    'linked_account' =>
    string(33) "https://phpc.social/users/derickr"
  }
  [2] =>
  array(2) {
    'name' =>
    string(14) "Marcus Bointon"
    'linked_account' =>
    string(33) "https://phpc.social/users/Synchro"
  }
}

And from that, I can then generate the HTML code to render who liked my article, which you can hopefully see below.

More recently, I've decided I really need a proper photo album tool to carry around "wallet photos" of family and such to show people. A few years back I started building my own application for that in Symfony 4, but I ran into some walls and eventually abandoned the effort. This time, I figured I'd see what was available on the market for self-hosted photo albums for me and my family to use.

Strap yourself in, because this is a really depressing story (with a happy ending, at least).

I reviewed 7 self-hosted photo album tools, after checking various review sites for their top-ten lists. Of those 7:

• 3 were in PHP, 2 were in JavaScript or TypeScript, and 2 were in Go.
• 2 used the MIT license, 2 used the GPL, 1 used AGPL, and two had broken non-free licenses.
• I managed to get one working. 1. Uno.
• Most really pushed you to use their Docker Compose setup to install, none of which actually worked.

Let's have a look at the mess directly.

Language: TypeScript License: MIT

PiGallery 2 is intended as a light-weight, directory-based photo album. The recommended way to install it is to use their Docker compose file and nginx conf file... which you have to just manually copy out of Git. (Seriously?) And when I tried to get that to run locally, I could never connect to it successfully. There was something weird with the port configuration, and I wasn't able to quickly figure it out. If I can't get the "easy" install to work, I'm not interested.

Language: PHP/MySQL License: GPLv2

Unlike many on here it doesn't provide a Docker image, which is fine so I set one up using phpdocker.io. Unfortunately, it's net installer crashed when I tried to use it, without useful errors. Trying to install manually resulted in PHP null-value errors from the install script. When I looked at the install script, I found dozens upon dozens of file system operations with the @ operator on them to hide errors.

At that point I gave up on Piwigo.

Language: PHP/MySQL License: GPL, version unspecified

When I first visted the Coppermine website, I got an error that their TLS certificate had expired a week and a half before. How reassuring.

Skipping past that, I was greeted with a website with minuscule text, with a design dating from the Clinton presidency. How reassuring.

Right on the home page, it says Coppermine is compatible all the way down to PHP 4.2, and supposedly up to 8.2. For those not familiar with PHP, 4.2 was released in 2002, only slightly after the Clinton presidency. PHP has evolved, um, a lot in 22 years, and most developers today view PHP 4 as an embarrassment to be forgotten. If their code is still designed to run on 4.2, it means they're ignoring literally 20 years of language improvements, including security improvements. How reassuring.

Oh, and the installation instructions, linked in the menu, are a direct link to some random forum post from 2017. How reassuring.

At this point I was so reassured that I Noped right out and didn't even bother trying to install it.

Language: JavaScript. (Not TypeScript, raw JS as far as I can tell.) License: None specified.

Although this app showed up on a few top-ten lists, its license is not specified, and installation only offers Windows and Mac. (Really?) The "others" section eventually lets you get to an Ubuntu section, where their recommendation is to install it via... an Apt remote. Which is an interesting choice.

It has a GitHub repo, but that has no license listed at all. Which technically means it's not licensed at all, and so downloading it is a felony. (Yes, copyright law is like that.)

Being a good Netizen, I reached out to the company through their Contact form to ask them to clarify. They eventually responded that, despite some parts of the code being in public GitHub repos, none of it is Open Source.

Noping right out of that one.

Language: Go License: It's complicated

I actually managed to get this one to run! This one also "installs" via Docker Compose, but it actually worked. This is the only one of the apps I reviewed that I could get to work. Mind you, as a Go app I cannot fathom why it needs a container to run, since Go compiles to a single binary.

Their system requirements are absurdly high. Quoting from their site, "you should host PhotoPrism on a server with at least 2 cores, 3 GB of physical memory,1 and a 64-bit operating system." What the heck are they doing? It's Go, not the JVM.

In quick experimentation, it seemed decent enough. The interface is snappy and supports uploading directly from the browser.

However, I then ran into a pickle. The GitHub repository says the license is AGPL, which I am fine with. However, in the app itself is a License page that is not even remotely close to Free Software anything, listing mainly all the ways you cannot modify or redistribute the code.

I filed an issue on their repository about it, and got back a rather blunt comment that only the "Community Edition" is AGPL, which is a different download. The supported version is not.

Noping right out of this one, too.

Language: Go, with TypeScript for front-end License: AGPLv3

Another app wants you install via Docker Compose. And when I tried to do so, I got a bunch of errors about undefined environment variables. The install documentation says nothing about setting them, and it's not clear how to do so, so at this point I gave up.

Language: PHP License: MIT

Lychee is built with Laravel, which I don't care for but I have used very good Laravel-based apps in the past so I had high hopes. It talks about using Docker, but unlike the others here doesn't provide a docker-compose file, just some very long Docker run commands.

Their primary instructions are to git-clone the project, then run composer install and npm. Unfortunately, phpdocker.io is still built using Ubuntu 22.04, which has an ancient version of npm in it, and I didn't want to bother trying to figure out how to upgrade it.

Lychee did offer a demo container, which uses SQLite. That I was able to get to run successfully. However, for unclear reasons it wouldn't actually show any images.

At this point, I gave up.

Rather disappointed in the state of the art, I decided to take a different approach. As I mentioned, I use Nextcloud to store all my images. Nextcloud has a photo app, but the last time I used it, it was very basic, and pretty bad. That was a few years ago, though, so I went searching.

Turns out, not only has Nextcloud Photos improved considerably, there's also another extension app on it called Memories. On paper, it looks like it does everything I'm after. A timeline feed, custom albums that don't require duplicating files, you can edit the Exif data of the image to show a title and description, plus some fancy extras like mapping geo information to OpenStreetMap and AI-based tagging, if you have the right additional apps installed. So would it work?

Turns out... yes. The setup was slightly fiddly, but mostly because it took a while to download all the map data and index a half-million photos. Once it did that, though... it just worked. It does almost everything I was looking for. I haven't figured out how to reorder albums or pictures within an album, and it looks like it doesn't support sub-albums. But otherwise, it does what I need. It even has a mobile app (free) that let's me show off selected pictures on my phone, which is what I was ultimately after.

I have always had a love/hate relationship with Nextcloud. In concept, I love it. Self-hosted file server and application hub? Sign me up! Despite being a PHP dev of 25 years, I've never quite understood why PHP made sense for it, though. And upgrades have always been a pain, and frequently break. But its functionality is just so useful. Apps are hit or miss, ranging from first-rate (like Memories) to meh.

But in this case, it ended up being both the cleanest and most capable option, as well as the easiest to get going, provided I already had a Nextcloud server. So, solution found. I am now a Memories user, and will be setting up accounts for the rest of the family, too.

A popular general-purpose scripting language that is especially suited to web development.
Fast, flexible and pragmatic, PHP powers everything from your blog to the most popular websites in the world.

What's new in 8.3 Download

• 8.3.13 · Changelog · Upgrading
• 8.2.25 · Changelog · Upgrading
• 8.1.30 · Changelog · Upgrading

GitHub and Pro/Business supporters will get it earlier, around the first of each month.

On GitHub sponsors, I am currently 31% towards my $2,500 per month goal, which is set to allow continued maintenance of Xdebug. This is again less than last month.

If you are leading a team or company, then it is also possible to support Xdebug through a subscription.

In the last month, I spend around 10 hours on Xdebug, with 19 hours funded. I also spent 16 hours on the Native Path Mapping project.

A popular general-purpose scripting language that is especially suited to web development.
Fast, flexible and pragmatic, PHP powers everything from your blog to the most popular websites in the world.

What's new in 8.3 Download

• 8.3.13 · Changelog · Upgrading
• 8.2.25 · Changelog · Upgrading
• 8.1.30 · Changelog · Upgrading

Submitted by Larry on 22 October 2024 - 11:32pm

This is helpful because ESM and CommonJS (require/module.exports) are kind of different worlds and before this change if you wanted to use a “module” from your CommonJS file, you would need to do something like:

const theThing = await import('some/module/file.mjs');

This is called a dynamic import and it’s extra annoying, because you can’t just put this at the top of your file. It returns a Promise, so you can only import these modules ‘asynchronously’ or in functions. The reason is that you can only await inside functions in CommonJS files. So this syntax prevents importing a “module” and immediately using it. To use a module in CommonJS, you’ll need some kind of initialization logic.

Many Javascript packages resort to shipping both a ‘CommonJS’ and a ‘ESM’ version of their code to reduce this kind of annoyance, but not everyone does.

The Node 23 change

Node 23 makes it possible to load ESM modules transparently via require(). This means that the above example can be rewritten to:

const theThing = require('some/module/file.mjs');

The important part here is not the usage of require, but the absense of await. This allows ESM modules to be loaded without this kind of initialization logic.

But there’s a big caveat:

This only works if the module you loaded in is not using top-level await. “Top level await” means awaiting things outside of (async) functions, which is possible in modules (but not CommonJS).

If a top-level await was used, a ERR_REQUIRE_ASYNC_MODULE error will be thrown. But the important thing to note is that this doesn’t just apply to the file you are directly importing/requiring. It also applies to any files loaded by that file, at any level in either your project or any dependency or sub-dependencies.

Using top-level await is now a BC break

Typically when we think of backwards compatibility breaks that require a major new version in semver, you might think of functions changing, or removing or arguments no longer being supported.

Before this change in Node, if your project was fully switched to ESM you might not think of placing a top-level await anywhere in your code as a backwards compatibility break, but if it’s the first await you might now inadvertently break Node.js users, if they used require() to bring in your module.

This means that the first top-level await in your project (or any of your dependencies) might now constitute a new major version if you follow semver.

1. Tell your users you don’t support require()

You could effectively tell them if they use require() they’re on their own. Chances are that your users won’t read this and do it anyway, but arguably they should read the readme of a package they bring in.

2. Add a dummy await

Do you think you or your dependencies might use a top-level await in the future, but you don’t yet? You could add this line to your source to reserve the right to do it later:

await "Good things come to those that support await"

3. Explictly break CommonJS

Packages can provide both CommonJS and ESM support via the exports key in package.json. It could be an option to export a single CommonJS file that just throws an error to warn CommonJS users they shouldn’t use this package.

Out of these 3 options, I kind of like 2 because if Node.js ever changes behavior and does support loading modules that use top-level await in CommonJS files it will one day just start working.

Regardless. it does feel important to me that package maintainers that only ship ESM, or ship CommonJS and intend to use this new feature have a strategy, because it’s bound to blow up if not.

In Bun you can use import and require in the same file, so there’s fewer trade-offs to make. But if you do use require() to import a module with top-level await it still has the same issue. There’s just fewer reasons in Bun to even want to do this.

Deno doesn’t support CommonJS so it doesn’t have that problem.

Does the issue exists when importing CommonJS files into modules?

Importing CommonJS files into modules was always possible via the standard import syntax, and because CommonJS can’t asynchronously export this problem doesn’t exist there.

What should I do as a user?

Most of this article was about the effects for package maintainers, but not every package will do this well so as a user you’re still potentially exposed to this issue.

In order of preference, consider the following:

1. Stop using CommonJS. Start using ESM everywhere. This issue only exists in CommonJS files using ESM, not the reverse. ESM is the future. You’re working with dead-end technology.
2. Have some testing in place that at the very least loads the entirely of your code-base. If all (top-level, not-dynamic) requires() are ran/resolved, any changes in depndencies should just blow up your CI environment and application.
3. Find out if any of your dependencies are ESM-only, and never require() them. This protects you from issues with direct dependencies. But keep in mind that this can still be an issue in sub-dependencies even if your direct dependencies are CommonJS themselves!

My opinion

For the above reasons I don’t think this should land in a stable Node.js version. The intentions are good but it creates yet another avenue of confusion.

It effectively expands the number of Javascript flavors from 2 to 3:

1. CommonJS
2. ESM
3. ESM but without top-level await.

If a module uses flavor #3, it’s compatible with flavor #1, but if anywhere in the dependency tree something starts using top-level await, suddenly the entire tree ESM dependency tree cascades from flavor #3 to #2 and breaks compatibilty with flavor #1.

Given the already confusing landscape (and reputation) of Javascript modules, this feels like a step in the wrong direction, as it adds a feature to CommonJS (which in my opinion is time to freeze), and makes ESM less reliable as a result.

OPTIONS is the method used for that. You may know this HTTP method from CORS, but it’s general purpose is for clients to passively find out ‘What can I do here?’.

All HTTP clients typically support making OPTIONS request. For example with fetch():

const response = await fetch(
  'https://example.org',
  {method: 'OPTIONS'}
);

A basic OPTIONS response might might look like this:

HTTP/1.1 204 No Content
Date: Mon, 23 Sep 2024 02:57:38 GMT
Server: KKachel/1.2
Allow: GET, PUT, POST, DELETE, OPTIONS

Based on the Allow header you can quickly tell which HTTP methods are available at a given endpoint. Many web frameworks emit this automatically and generate the list of methods dynamically per route, so chances are that you get this one for free.

To find out if your server does, try running the command below (with your URL!):

curl -X OPTIONS http://localhost:3000/some/endpoint/

One nice thing you could do with the Allow header, is that you could also communicate access-control information on a very basic level. For example, you could only include DELETE and PUT if a user has write access to a resource.

Accept and Accept-Encoding

There’s server other standard headers for discovery. Here’s an example showing a few at once:

HTTP/1.1 204 No Content
Date: Mon, 23 Sep 2024 02:57:38 GMT
Server: KKachel/1.2
Allow: GET, PUT, POST, DELETE, OPTIONS
Accept: application/vnd.my-company-api+json, application/json, text/html
Accept-Encoding: gzip,brotli,identity

You may already be familiar with Accept and Accept-Encoding from HTTP requests, but they can also appear in responses. Accept in a response lets you tell the client which kind of mimetypes are available at an endpoint. I like adding text/html to every JSON api endpoint and making sure that API urls can be opened in browsers and shared between devs for easy debugging.

The Accept-Encoding lets a client know in this case that they can compress their request bodies with either gzip or brotli (identity means no compression).

Patching, posting and querying

3 other headers that can be used are Accept-Patch, Accept-Post and Accept-Query. These three headers are used to tell a client what content-types are available for the PATCH, POST and QUERY http methods respectively.

For all of these headers, their values effectively dictate what valid values are for the Content-Type header when making the request.

HTTP/1.1 204 No Content
Date: Mon, 23 Sep 2024 02:57:38 GMT
Server: KKachel/1.2
Allow: OPTIONS, QUERY, POST, PATCH
Accept-Patch: application/json-patch+json, application/merge-patch+json
Accept-Query: application/graphql
Accept-Post: multipart/form-data, application/vnd.custom.rpc+json

In the above response, the server indicates it supports both JSON Patch and JSON Merge Patch content-types in PATCH requests. It also suggests that GraphQL can be used via the QUERY method, and for POST it supports both standard file uploads and some custom JSON-based format.

Typically you wouldn’t find all of these at the same endpoint, but I wanted to show a few examples together.

Where’s PUT?

Oddly, there’s no specific header for PUT requests. Arguably you could say that GET and PUT are symmetrical, so perhaps the Accept header kind of extends to both. But the spec is not clear on this.

I think the actual reality is that Accept-Patch was the first header in this category that really clearly defined this as a means of feature discovery on OPTIONS. Accept-Post and Accept-Query followed suit. I think Accept-Patch in OPTIONS was modelled after in-the-wild usage of Accept in OPTIONS, even though the HTTP specific doesn’t super clearly define this.

If I’m wrong with my interpretation here, I would love to know!

Aside: If you’re wondering about DELETE, DELETE should never have a body, so all a user would need to know is can they delete, which you can see in the Allow header. If this is new to you to, read my other article about GET request bodies. Most of the information there is applicable to DELETE as well.

The OPTIONS response is also a great place to tell users where to find additional documentation. In the below example, I included both a machine-readable link to a documentation site, a link to an OpenAPI definition, and a message intended for humans in the response body:

HTTP/1.1 200 OK
Date: Mon, 23 Sep 2024 04:45:38 GMT
Allow: GET, QUERY, OPTIONS
Link: <https://docs.example.org/api/some-endpoint>; rel="service-doc"
Link: <https://api.example.org/openapi.yml>; rel="service-desc" type="application/openapi+yaml"
Content-Type: text/plain
Hey there!
Thanks for checking out this API. You can find the docs for this
specific endpoint at: https://docs.example.org/api/some-endpoint
Cheers,
The dev team

I recommend keeping the response body as mostly informal and minimal any real information should probably just live on its own URL and be linked to.

I used the service-doc and service-desc link relationships here, but you can of course use any of the IANA link relationship types here or a custom one. Also see the Web linking spec for more info.

Obscure uses

WebDAV usage

WebDAV, CalDAV and CardDAV also use OPTIONS for feature discovery. For example:

HTTP/1.1 204 No Content
Date: Mon, 23 Sep 2024 05:01:50 GMT
Allow: GET, PROPFIND, ACL, PROPPATCH, MKCOL, LOCK, UNLOCK
DAV: 1, 2, 3, access-control, addressbook, calendar-access

The server-wide asterisk request

Normally HTTP requests are made to a path on the server, and the first line looks a bit like the following in HTTP/1.1:

GET /path HTTP/1.1

But, there are a few other “request line” formats that are rarely used. One of them lets you discover features available on an entire server, using the asterisk:

OPTIONS * HTTP/1.1

The asterisk here is not a path. Normally asterisks aren’t even allowed in URIs. Many HTTP clients (including fetch()) don’t even support this request.

Classic webservers like Apache and Nginx should support this. To try it out, use CURL

curl -vX OPTIONS --request-target '*' http://example.org

Final notes

If you have a reason to allow clients to discover features on an endpoint, consider using OPTIONS instead of a proprietary approach! As you can see in many of these examples, it’s especially useful if you use mimetypes well.

If you have questions, other novel uses of OPTIONS or other ideas around feature discovery, you can respond via:

A popular general-purpose scripting language that is especially suited to web development.
Fast, flexible and pragmatic, PHP powers everything from your blog to the most popular websites in the world.

What's new in 8.3 Download

• 8.3.12 · Changelog · Upgrading
• 8.2.24 · Changelog · Upgrading
• 8.1.30 · Changelog · Upgrading

Mon, Oct 7, 2024 12:21 PM

- ReqHeader DN-Varnish-Offer-Group: 
- LostHeader DN-Varnish-Offer-Sort: price 
- Error out of workspace (req) 
- LostHeader DN-Varnish-Sbtab-Sorts: 
- Error out of workspace (req) 
- LostHeader DN-Varnish-Use-View: 
- Error out of workspace (req) 
- LostHeader DN-Varnish-Ux-Variant: classic_site 
- Error out of workspace (req)

http_req_hdr_len = 64k (default is 8k)
http_req_size = 128k (default is 32k)
http_max_hdr=256 (default is 64)

What is it?

HTTP headers have been a bit of a free-for all in terms of how complex values are encoded, with many headers requiring their own mini-parser.

A while back an effort was started to fix this for headers going forward, named ‘Structured Fields’. They’re called Fields and not ‘Headers’ because HTTP has both Headers and Trailers!

Structured fields let you encode things like lists, dictionaries, strings, numbers, booleans and binary data. The original RFC from 2021 is pretty successful and although many existing headers can’t be retrofitted to this format, a lot of new standards are taking advantage.

Some examples:

# Parsed an ASCII string
Header: "foo"
# A simple string, called a 'Token' in the spec
Header: foo
# Parsed as number
Header: 5
Header: -10
Header: 5.01415
# Parsed into boolean
Header: ?1
Header: ?0
# Binaries are base64 encoded
Header: :RE0gbWUgZm9yIGEgZnJlZSBjb29raWU=:
# Items can have parameters
Header: "Hello world"; a="5"
# A simple list
Header: 5, "foo", bar, ?1
# Each element can have parameters
Header: sometoken; param1; param2=hi, 42
# A list can also contain lists itself. These are called 'inner lists' and
# use parenthesis
Header: sometoken, (innerlistitem1 innerlistitem2), (anotherlist)
# A simple dictionary
Header: fn="evert", ln="pot", coffee=?1
# Each item may have parameters too
Header: foo=123; q=1, bar=123, q=0.5
# A dictionary value may be an inner list again
Header: foo=(1 2 3)

The new RFC published last week adds 2 new data types: Dates and ‘Display strings’, which is a Unicode serialization that fits in the HTTP header (and trailer) format.

# Parsed into a Date object
Header: @1686634251
# A Unicode string, called a 'Display String' in the spec. They use
# percent encoding, but encode a different set of characters than
# URLs.
Header %"Frysl%C3%A2n"

Why should you care?

If you encounter these headers in the wild, it’s a really good idea to use a standard parser. One of the reasons is that with using structured-fields, there’s a built-in extension mechanism. You’ll want to make sure that when a new parameter appears your application doesn’t suddenly break.

You may also want to define and use your own HTTP headers. The structured fields format is a very good ‘default choice’ that removes decisions such as ‘How should I encode a key value object’ or ‘how do I encode a UTF-8 string’.

With parsers popping up for every language, you don’t have to worry about writing your own one-off formats.

Javascript package

I’m the maintainer of a Javascript library for Structured Fields, called “structured headers”, which I’ve also updated for this new RFC. I wish I picked the name “structured-fields”, but I picked the name before the original standard changed it’s name.

I’ve just released v2 of this library supporting these new types, and also added ES Modules support.

Reply to one of these:

• Mastodon post
• Bluesky post

subscribe

A popular general-purpose scripting language that is especially suited to web development.
Fast, flexible and pragmatic, PHP powers everything from your blog to the most popular websites in the world.

What's new in 8.3 Download

• 8.3.12 · Changelog · Upgrading
• 8.2.24 · Changelog · Upgrading
• 8.1.30 · Changelog · Upgrading

For example:

"Client error: `GET https://dev.clientproject.com:4444/oauth2/authorize?client_id=983e98d2fab8756a&scope=scope&response_type=code&redirect_uri=%2Fhome&code_challenge=some_code_challenge_here` resulted in a `400 Bad Request` response:
{"error":"invalid_request","error_description":"The request is missing a required parameter, includes an invalid parame (truncated...)

To retrieve the full text of the body, you grab it from the response property of the exception:

$body = $e->getResponse()->getBody()->getContents();

If you want more text in the exception message you need to pass a handler stack to the Client‘s constructor with an httpErrors middleware that has a BodySummarizer with a larger truncation limit.

The easiest way to do this is to create the default stack and then modify it:

use GuzzleHttp\BodySummarizer;
use GuzzleHttp\HandlerStack;
// ...
$stack = HandlerStack::create();
$stack->remove('http_errors');
$stack->unshift(Middleware::httpErrors(new BodySummarizer(1500)), 'http_errors');

Set the number of characters to truncate at in the BodySummarizer‘s constructor. I picked 1500; the default is 120 (as of this time of writing).

After creating the stack, we remove the current httpErrors middleware and add a new one. Note that it needs to be first in the stack as we want it to handle any error from any subsequent middleware too.

We then create our client with our custom stack:

use GuzzleHttp\Client;
// ...
$this->client = new Client([
    'handler'  => $stack,
    'base_uri' => $this->baseUri,
]);

That’s it. There’s now more text in the BadResponseException message.

Di sinilah peran jasa desain interior menjadi sangat penting, jasa profesional tidak hanya membantu Anda mewujudkan impian, tetapi juga memastikan setiap detail terlihat mewah dan harmonis. Mari kita bahas lebih dalam bagaimana jasa desain interior Bandung bisa menciptakan nuansa mewah di hunian Anda.

Cara Jasa Desain Interior Menciptakan Nuansa Mewah

1. Pemilihan Material Berkualitas

Material berkualitas tinggi adalah salah satu kunci utama menciptakan nuansa mewah. Desainer interior akan membantu Anda memilih material yang tahan lama dan elegan, seperti marmer, kayu solid, atau logam berlapis. Bahan-bahan ini tidak hanya meningkatkan tampilan, tetapi juga daya tahan furnitur dan elemen dekoratif di rumah Anda.

Selain itu, pemilihan material juga harus memperhatikan kesesuaian dengan konsep desain. Material yang tepat akan memberikan kesan eksklusif dan elegan tanpa mengabaikan fungsi. Dengan bantuan jasa desain interior, Anda dapat memastikan pilihan material yang sesuai dengan visi dan anggaran Anda.

2. Penataan Pencahayaan yang Tepat

Pencahayaan memainkan peran penting dalam menciptakan suasana mewah. Desainer interior profesional akan membantu Anda merancang sistem pencahayaan yang tidak hanya fungsional, tetapi juga estetis. Pencahayaan yang tepat bisa memperkuat kesan ruangan yang lebih luas dan lebih nyaman.

Pemanfaatan lampu gantung, lampu dinding, atau bahkan pencahayaan tersembunyi bisa membuat perbedaan besar. Jasa desain interior Bandung memastikan bahwa setiap sudut ruangan mendapatkan pencahayaan yang ideal untuk menonjolkan kemewahan desainnya.

3. Kombinasi Warna yang Harmonis

Warna memegang peran penting dalam menciptakan suasana mewah. Desainer interior tahu bagaimana mengombinasikan warna-warna netral seperti putih, krem, dan abu-abu dengan sentuhan warna yang lebih tegas. Kombinasi warna ini akan menghasilkan kesan elegan yang tetap hangat.

Pilihan warna yang tepat juga membantu memberikan dimensi pada ruangan Anda. Dengan menggunakan jasa desain interior, Anda akan mendapatkan panduan dalam memilih palet warna yang tidak hanya cantik, tetapi juga memancarkan kemewahan.

4. Pemanfaatan Ruang Secara Optimal

Salah satu keahlian desainer interior adalah memaksimalkan ruang yang ada. Mereka akan merancang tata letak yang tidak hanya fungsional tetapi juga estetik. Setiap ruang akan dimanfaatkan dengan baik, sehingga tidak ada sudut yang terbuang.

Dengan jasa desain interior, Anda dapat menciptakan ruang yang terorganisir, rapi, dan mewah. Desainer akan memastikan setiap elemen dekoratif dan furnitur ditempatkan pada posisi yang tepat untuk memaksimalkan keindahan dan kenyamanan ruangan Anda.

5. Penempatan Aksen Dekoratif yang Menawan

Aksen dekoratif seperti lukisan, patung, atau hiasan dinding adalah elemen yang tidak boleh diabaikan. Desainer interior akan membantu Anda memilih aksen yang sesuai dengan tema mewah yang ingin Anda hadirkan. Aksen ini bisa menjadi pusat perhatian yang menarik di ruangan Anda.

Aksen dekoratif yang dipilih dengan cermat akan memberikan karakter unik pada ruangan Anda. Dengan jasa desain interior, setiap aksen akan dipadukan secara harmonis untuk menciptakan kesan yang lebih eksklusif.

6. Pilihan Furnitur yang Elegan

Furnitur adalah elemen kunci dalam desain interior. Jasa desain interior membantu Anda memilih furnitur yang tidak hanya nyaman tetapi juga elegan. Penggunaan furnitur dengan desain yang unik dan berkualitas tinggi akan menambah kesan mewah pada hunian Anda.

Pemilihan furnitur juga harus sesuai dengan tema dan gaya desain yang diinginkan. Desainer interior akan membantu Anda menemukan furnitur yang mampu memadukan kenyamanan, fungsi, dan estetika dengan sempurna.

Mewujudkan Hunian Mewah dengan Sentuhan Profesional

Jika Anda ingin menciptakan hunian mewah dengan nuansa yang elegan, bantuan jasa desain interior profesional adalah pilihan yang tepat. Mereka tidak hanya membantu Anda dalam merancang konsep, tetapi juga merealisasikan visi Anda menjadi kenyataan. Setiap detail diperhatikan agar sesuai dengan keinginan dan anggaran Anda.

Dengan bekerja sama dengan jasa desain interior Bandung, Anda bisa mendapatkan hunian yang mencerminkan kemewahan dan kenyamanan. Dari pemilihan material hingga penataan ruang, desainer interior akan memastikan hasil akhir yang memuaskan.

Jika Anda tertarik untuk mewujudkan hunian mewah, Reka Kayu siap membantu Anda dengan sentuhan desain interior yang unik dan eksklusif. Dengan pengalaman dan keahlian, Reka Kayu memastikan setiap proyek dikerjakan dengan teliti dan profesional.

The relevant parts of Makefile look like this:

# Set DEBUG=1 to enable Xdebug
ifeq ($(origin DEBUG),undefined)
    XDEBUG :=
else
    XDEBUG := XDEBUG_SESSION=PHPSTORM
endif
unit: ## Run unit tests
        docker compose exec php bash -c "$(XDEBUG) vendor/bin/phpunit --testsuite=unit"

I can now set a break point and run the unit tests with Xdebug enabled with DEBUG=1 make unit.

I have an environment variable set in compose.yaml: PHP_IDE_CONFIG=serverName=project.com.localhost and I use to set up the mappings in PhpStorm’s settings. As we’re using Docker, I also have xdebug.client_host = host.docker.internal set up in my PHP ini settings.

I noticed that when I didn’t have a breakpoint set, PhpStorm would break with this error in Debug console for a particular test:

That’s a strange error message: “Cannot find a local copy of the file on server /var/www/html/Standard input code”! Clearly I don’t have a file called “Standard input code” and so it’s not surprising that PhpStorm can’t map to it.

Google to the rescue!

I eventually worked out that it’s related to having a test with the @runInSeparateProcess annotation. As a result, phpunit does some magic and Xdebug sends through a file that it has labeled as “Standard input code” – presumably as it was injected via stdin..

To fix this, I disabled these PhpStorm settings in PHP -> Debug:

• “Break at first line in PHP scripts” in the External connections section
• “Force break at first line when no path mapping specified” in the Xdebug section

In PhpStorm 2024, it looks like this:

Of course, you’ll have to manually set up path mappings in PHP -> Servers, but I do that anyway, so not really a problem for me, at least.

Problem solved!

In Pipelined database operations with python-oracledb 2.4 I gave a simple benchmark showing the performance benefit of Oracle Database 23ai Pipelining. We now have results which are even more impressive.

The system testing team (thank you!) ran a performance evaluation with Python 3.12 and python-oracledb 2.4.1 using the same schema they had used for their ODP.NET Pipelining benchmark (discussed here).

The app used 100 threads that repeatedly executed different SELECT statements. The tests measured performance metrics over 20,000 iterations with pipelining enabled versus pipelining disabled:

• Pipelining runs finished 530% faster on average.
• Pipelining runs performed 529% more transactions per second.
• Pipelining runs performed 52% more transactions per second per processor.

I’m not sure I think that last stat is what it says it is, or that it is useful, but I include it for completeness with the ODP.NET results. Regardless the time and TPS figures are very significant!

One interesting, but obvious when you think about it, note is that client CPU usage increases with Pipelining because the client is able to be kept busy and do more work.

Resources

• Blog: Pipelined database operations with python-oracledb 2.4
• Doc: Pipelining Database Operations

I’m noting the relevant parts of my Dockerfile for when I need them again, so this is entirely an aide-mémoire!

FROM ubuntu:22.04
# Install PHP as per https://akrabat.com/7126
## Register the Microsoft package repository
RUN curl https://packages.microsoft.com/keys/microsoft.asc | tee /etc/apt/trusted.gpg.d/microsoft.asc \
    && curl https://packages.microsoft.com/config/debian/11/prod.list | tee /etc/apt/sources.list.d/mssql-release.list \
    && apt-get update
# Install the ODBC libraries and MS command line tools
RUN ACCEPT_EULA=Y apt-get install -y unixodbc-dev msodbcsql18 mssql-tools18
# Install sqlsrv and pdo_sqlsrv PHP extensions
RUN pecl install sqlsrv \
    && pecl install pdo_sqlsrv \
    && echo "extension=sqlsrv.so" > /etc/php/8.2/mods-available/sqlsrv.ini \
    && echo "extension=pdo_sqlsrv.so" > /etc/php/8.2/mods-available/pdo_sqlsrv.ini \
    && phpenmod -v 8.2 sqlsrv pdo_sqlsrv \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
# Other stuff, such as copying over settings files and starting PHP-FPM

It all works remarkably well and makes development that much easier. There are two key MS libraries needed: msodbcsql18 for the PHP extensions and mssql-tools18 installs /opt/mssql-tools18/bin/sqlcmd, the CLI tool for SQL Server.

Configuring SQL Server in Docker Compose

To run SQL Server on an Apple Silicon Mac you need to enable Rosetta in Docker Desktop and then configure a few environment variables. This is how I set the db service in compose.yaml:

  db:
      platform: linux/amd64
      image: mcr.microsoft.com/mssql/server:2022-latest
      environment:
        ACCEPT_EULA: 'Y'
        MSSQL_PID: 'Developer'
        MSSQL_SA_PASSWORD: 'Password123!'
        MSSQL_TCP_PORT: 1433
      ports:
        - "1433:1433"
      volumes:
        - ./opt/mssql:/var/opt/mssql

We set plaform to remove an annoying warning, but as there isn’t an ARM64 container for full SQL Server, the warning doesn’t help.

The environment variables set are:

• ACCEPT_EULA: Must be Y otherwise nothing works :)
• MSSQL_PID: Set to Developer for the developer licence, unless you have a another license.
• MSSQL_SA_PASSWORD: The root password (The sa user in SQL Server’s world)
• MSSQL_TCP_PORT: Not needed as it defaults to 1433, but a useful reminder.

The default MSSQL_DATA_DIR directory on Linux is /var/opt/mssql, so I map that to the opt/mssql directory in my project, which is registered in .gitignore.

GitHub and Pro/Business supporters will get it earlier, around the first of each month.

On GitHub sponsors, I am currently 32% towards my $2,500 per month goal, which is set to allow continued maintenance of Xdebug. This is again less than last month.

If you are leading a team or company, then it is also possible to support Xdebug through a subscription.

In the last month, I spend around 8 hours on Xdebug, with 20 hours funded. I also spent 8.5 hours on the Native Path Mapping project.

remote_prefix: /usr/local/www
local_prefix: /home/derick/project
/projects/example.php:5-17 = /example.php:8
/example.php:5-17 = /example.php:8-20
/example.php:17 = /example.php:20
/projects/php-web/ = /php-web/

AFL_SKIP_CPUFREQ=1 afl-fuzz -b 7 -i fuzz-seeds -o fuzz-output -- ./afl-test @@

remote_prefix: /usr/local/www
local_prefix: /home/derick/project
=/example.php:42-5 = /example.php

A little like this:

FROM ubuntu:22.04
RUN apt-get update && apt-get upgrade -y && apt-get install -y gnupg curl
# Register the Ondrej package repo for PHP
RUN mkdir -p /etc/apt/keyrings \
    curl -sS 'https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x14aa40ec0831756756d7f66c4f4ea0aae5267a6c' | gpg --dearmor | tee /etc/apt/keyrings/ppa_ondrej_php.gpg > /dev/null \
    && echo "deb [signed-by=/etc/apt/keyrings/ppa_ondrej_php.gpg] https://ppa.launchpadcontent.net/ondrej/php/ubuntu jammy main" > /etc/apt/sources.list.d/ppa_ondrej_php.list \
    && apt-get update
# Install PHP
RUN apt-get install -y php8.2-cli php8.2-fpm php8.2-dev \
       php8.2-sqlite3 php8.2-gd php8.2-intl php8.2-imagick \
       php8.2-mbstring php8.2-xml php8.2-zip php8.2-bcmath \
       php8.2-curl php8.2-pdo php8.2-opcache php8.2-gettext \
       php-pear
# other stuff, such as copying over settings files
# Run php-fpm
WORKDIR /var/www/html
EXPOSE 9000
CMD /usr/sbin/php-fpm8.2 -F -R

This works fine, however, when stopping the containers with docker compose down, I noticed that it takes 10 seconds for the PHP container to stop:

$ docker compose down
[+] Running 5/5
 ✔ Container portal-web-1   Removed                            0.1s 
 ✔ Container portal-db-1    Removed                            0.3s 
 ✔ Container portal-mail-1  Removed                            0.1s 
 ✔ Container portal-php-1   Removed                           10.1s 
 ✔ Network portal_default   Removed                            0.0s 

Googling around, it seems that 10 seconds is a Docker timeout, so it seems that the PHP container isn’t shutting down itself, but is being killed by Docker.

Further googling and many tests later, I found that the solution is to use exec form for the CMD:

CMD ["/usr/sbin/php-fpm8.2", "-F", "-R"]

Once I had made this change, stopping the PHP container takes 0.1s as I would hope:

$ docker compose down
[+] Running 5/5
 ✔ Container portal-web-1   Removed                            0.2s 
 ✔ Container portal-mail-1  Removed                            0.1s 
 ✔ Container portal-db-1    Removed                            0.3s 
 ✔ Container portal-php-1   Removed                            0.1s 
 ✔ Network portal_default   Removed                            0.0s 

Much better!

Aside: why does this work?

The underlying reason is the way unix-like operations systems handle termination of processes. When a process needs to terminate via say a SIGTERM signal, it will terminate and become one of those “zombie” processes that you sometimes see when you type ps. It’s parent process then “waits on” (or “reaps”) it for the exit code and then it is really gone. If that process that’s been terminated has children, then they now no longer have a parent and so the init process (PID 1) takes them over (“adopts” them) and all is well with the system.

With Docker however, we usually run our process as PID 1 and we need this process to correctly reap child processes and adopt orphans (i.e. act like init). Specifically, CMD /usr/sbin/php-fpm8.2 -F -R will start Bash as PID 1 and then php-fpm as a child. This seems to be fine as Bash can reap and adopt processes, except that Bash doesn’t handle signals properly!

On shutdown, Docker sends SIGTERM to Bash, which terminates. However, Bash, does not send SIGTERM to its child processes (php-fpm in this case) and so it sits there waiting php-fpm to terminate. As php-fpm doesn’t know that it needs to terminate, we have to wait 10 seconds until Docker gets bored and kills the container via SIGKILL.

Knowing this, the solution becomes obvious: we want php-fpm to be PID 1 as it knows how to handle it’s children. This happens when you specify the CMD using exec form as a shell is not created which leaves php-fpm as PID 1 and the SIGTERM is handled correctly.

This leads naturally on to another way to solve it: use the shell’s exec command:

CMD exec /usr/sbin/php-fpm8.2 -F -R

This has the same effect as bash exec will replace the current shell with the command being executed.

Another solution is to run a real init process as PID 1. There are many options available out there.

What was it? quite simply, the php_admin_value struct can be used to configure php.ini settings for the pool. This is a great alternative to also adding PHP configuration settings via php.ini (or an include file for php.ini), as it allows you to keep the settings specific to that pool. That way, if you must have multiple pools (e.g., to serve multiple applications from the same machine and/or same PHP version), you can still have separate configuration for each.

How does it work? In your pool configuration, add values to that struct:

php_admin_value[memory_limit] = 32M
php_admin_flag[error_reporting] = E_ALL & ~E_NOTICE & ~E_DEPRECATED
php_admin_flat[track_errors] = Off
; etc

With ZendPHP, we just launched some Ansible tooling, which operates on the assumption that you are deploying PHP-FPM — and as part of its operation, it creates a template for the FPM pool configuration, but not for the PHP SAPI. And this is fine! Because you can use the php_admin_value settings to configure the pool for the application you're deploying!

Looking forward to simplifying a few of my deployments with this!

Python-oracledb 2.4 introduces pipelining functionality to improve the performance and scalability of your Python applications. Pipelining is simple in python-oracledb: you add operations such as INSERT and SELECT statements to a “pipeline” object, send that object to the database which processes the statements, and finally all the results will be returned to the application. Since the API is asynchronous, your application can submit the pipeline and continue with other local tasks while the database is doing its processing. This lets the database server and the application be kept busy, giving efficiencies and letting them work concurrently instead of waiting on each other for statements and results to be submitted/fetched.

Pipelining Overview

Pipelining is useful when many small database operations need to be performed in rapid succession. It is supported by various drivers, including JDBC, Oracle Call Interface, ODP.NET, and Python.

The benefits of Oracle Database 23ai Pipelining:

• Your app can do local work at the same time the database is processing statements.
• Your app doesn’t have to wait on one database response before sending a second statement.
• Your app is kept busy.
• After the database finishes one statement, it doesn’t have to wait for your app to fetch results and send a second statement.
• The database is kept busy.
• Fewer round-trips: reduced Oracle network listener wake-ups. Reduced interrupts. More efficient network usage.
• Better overall system scalability.

The reduction in round-trips is a significant contributor to pipelining’s performance improvement in comparison to executing the equivalent SQL statements individually. But, even with high-speed networks, where the performance benefit of pipelining may be lower, the database and network efficiencies of pipelining can still help system scalability.

Pipelining in Python is available via python-oracledb’s Async classes: this means you must use the default Thin mode of python-oracledb, and use it in an asynchronous programming style. Pipelining works with Oracle Database 23ai. (Although you can actually use the new python-oracledb API when connected to older database versions, you won’t get the internal pipelining behavior and benefits — this is recommended only for migration or compatibility reasons). You can get Oracle Database 23ai from Oracle Database Software Downloads.

You can use the following python-oracledb calls to add operations to a pipeline:

• add_callfunc() - calling a stored PL/SQL function
• add_callproc() - calling a stored PL/SQL procedure
• add_commit() - commiting current transaction on the connection
• add_execute() - executing one SQL statement
• add_executemany() - executing one SQL statement with many bind values
• add_fetchall() - executing a query and fetching all the results
• add_fetchmany() - executing a query and fetching a set of the results
• add_fetchone() - executing a query and fetching one row

Note the database processes the pipelined statements sequentially. The concurrency gain is betwen your application’s local work and the database doing its work.

Query results or OUT binds from one operation cannot be passed to subsequent operations in the same python-oracledb pipeline. If you need to use results from a pipeline operation in a subsequent database step, you can use multiple pipelines.

Social Network Example

An example use case is a social networking site. After you log in, your home page needs to show information gathered from various sources. It might show how many of your friends are also logged in. A news feed might show the top news items of the day. Current and forecast temperatures could be shown. Some of this data could be in a database, but require several distinct queries to fetch. Finding the current temperature might require Python calling out to a service to return that data. This is great use case for Pipelining: the distinct queries can be sent in a pipeline for processing, while the remote temperature sensor data is gathered at the same time.

The full code for this simple example web app is in pipeline-blog-quart.py.

The app needs to run asynchronously, so a framework that supports this is needed. I semi-arbitrarily chose the Quart micro-framework, which has a relationship to Flask. Flask’s async support isn’t complete enough for a full async application.

The main body is:

app = Quart(__name__, )# Users home page
@app.route("/homepage")
async def homepage():
    r = await getdata() # Calls the database and remote temperature sensor
    return buildhtml(r) # Puts the data into an HTML page and returns it for display@app.before_serving
async def setup():
    start_pool()             # Start a pool of connections
    await create_schema()    # Create the demo tablesif __name__ == "__main__":
    print(f"Try loading http://127.0.0.1:{PORT}/homepage in a browser")
    app.run(port=PORT)

I chose to show how a connection pool would be used, since connection pooling would be important for any actual multi-user app to do, and is something that is worth taking care to implement and configure correctly.

Then, for demo purposes, the app creates the schema and loads sample data when it starts. (This loading is itself another example use case for pipelining since multiple SQL statements are needed. See create_schema() in the app).

Whenever the /homepage URL is invoked, the getdata() routine is called. This is where Pipelining really stars. The method does the database queries and accesses the temperature sensor concurrently. When all the data has been gathered, it is formatted into an HTML page by buildhtml() which is then displayed by Quart. The getdata() method and the external sensor method get_current_temperature() look like:

async def get_current_temperature():
    return randint(0, 50) # fake what the thermometer returned - in Celsius of courseasync def getdata():
    global pool
    async with pool.acquire() as connection:# Create a pipeline and define the operations
        pipeline = oracledb.create_pipeline()
        pipeline.add_fetchone("select high from weather")
        pipeline.add_fetchall("select name from friends where active = true")
        pipeline.add_fetchall("select story from news order by popularity fetch next 5 rows only")# Run the pipeline and non-database operations concurrently
        current_temp, result_pl = await asyncio.gather(
            get_current_temperature(),
            connection.run_pipeline(pipeline)
        )# Extract the database pipeline query results
        t = result_pl[0].rows[0][0]          # max forecast temperature
        f = [n for n, in result_pl[1].rows]  # list of active friends
        n = [s for s, in result_pl[2].rows]  # top 5 news storiesreturn { "current": current_temp, "forecast": t, "friends": f, "news": n }

This creates a pipeline with three SELECT statements. By using asyncio.gather(), the pipeline is executed in parallel with a call to get_current_temperature() (which fakes reading an external thermometer). When all the data is available, it is returned in a dict.

Starting the app and loading the home page in a browser gives a page like:

The reading of the thermometer and the database work are executed concurrently: both the app and the database are kept busy. The application response time is reduced, making users happier. When the database is doing its work, it is kept busy with fewer interrupts and pauses, making it more efficient. Overall system scalability has improved.

Performance Example

An example pipeline with more operations makes it easier to see performance benefits.

Let’s start with a program that doesn’t use Pipelining (see pipeline-bm-non.py). It uses the existing async functionality of python-oracledb but every statement is executed sequentially:

# No Pipeliningfor i in range(50):
    await connection.execute("insert into TestTempTable values (:1, :2)", [i + 1, f"Value {i + 1}"])await connection.execute("""
    update TestTempTable set Description = Description || ' (modified)'
    where mod(ID, 2) = 1""")for i in range(50):
    await connection.execute("insert into TestTempTable values (:1, :2)", [i + 51, f"Value {i + 51}"])await connection.commit()rows = await connection.fetchall("select * from TestTempTable")

Note the demo executes lots of the same INSERT statement. This is simply to showcase a “larger” pipeline length. In production non-pipelining and pipelining code, you should use executemany() for such a flow.

But, putting aside that comment, each of the statements executed will require a round-trip.

To cut down the number of round-trips, the script can be converted to use Pipelining (see pipeline-bm.py):

# Pipelining with Oracle Database 23aipipeline = oracledb.create_pipeline()for i in range(50):
    pipeline.add_execute("insert into TestTempTable values (:1, :2)", [i + 1, f"Value {i + 1}"])pipeline.add_execute("""
    update TestTempTable set
        Description = Description || ' (modified)'
    where mod(ID, 2) = 1""")for i in range(50):
    pipeline.add_execute("insert into TestTempTable values (:1, :2)", [i + 51, f"Value {i + 51}"])pipeline.add_commit()pipeline.add_fetchall("select * from TestTempTable")result = await connection.run_pipeline(pipeline)

This will require only one round-trip, initiated when run_pipeline() is called.

On my laptop, with the database running locally in an emulated container environment, typical end-to-end timing results I saw were:

$ python3 pipeline-bm.py; python3 pipeline-bm-non.py
pipelining:    0.34 seconds
no pipelining: 0.46 seconds

This particular result is a 26% decrease in end-to-end time for the application’s database workload. Your results will vary. The Pipelined application was much more efficient. I could also have utilized parallelism to allow the app to continue other work while the database was processing statements (as shown in the social networking example), giving further gains to overall app performance.

Conclusion

Pipelining in Oracle Database 23ai is a technique to improve application performance and system scalability by keeping the database busy while allowing applications to concurrently continue local work. Pipelining is available to applications using python-oracledb, JDBC, ODP.Net and Oracle Call Interface.

Pipelining is useful when many small database operations need to be performed in rapid succession. It is most effective when networks costs are high. But even on fast networks, or with a small number of operations, pipelining provides system benefits.

The new Pipeline support in python-oracledb 2.4 builds on the existing support for asyncio. It is available in Thin mode via new Pipeline classes. In this release it is marked ‘experimental’ to give you time to test and suggest small or big changes before we finalize the API. We have tested with asyncio. Your feedback about the behavior with other alternatives like uvloop would be invaluable.

Pipelining References

• Python-oracledb documentation: Pipelining Database Operations
• Blog: Introducing Pipelining and Asynch Programming for Oracle Database 23ai
• Oracle Call Interface documentation: OCI Pipelining
• Blog: What’s in ‘Oracle DB 23c Free Developer Release’ for Java Developers
• JDBC Documentation: Support for Pipelined Database Operations
• Blog: Asynchronous ODP.NET and Pipelining: Improve Performance by Reducing Latency

Installing or Upgrading python-oracledb

You can install or upgrade python-oracledb by running:

python -m pip install oracledb --upgrade

The pip options --proxy and --user may be useful in some environments. See python-oracledb Installation for details.

Python-oracledb References

Home page: oracle.github.io/python-oracledb/index.html

Installation instructions: python-oracledb.readthedocs.io/en/latest/installation.html

Documentation: python-oracledb.readthedocs.io/en/latest/index.html

Release Notes: python-oracledb.readthedocs.io/en/latest/release_notes.html

Discussions: github.com/oracle/python-oracledb/discussions

Issues: github.com/oracle/python-oracledb/issues

Source Code Repository: github.com/oracle/python-oracledb

Python-oracledb 2.4, the extremely popular Oracle Database interface for Python, is now on PyPI.

Python-oracledb is an open source package for the Python Database API specification with many additions to support advanced Oracle Database features. By default, it is a ‘Thin’ driver that is immediately usable without needing any additional install e.g. no Instant Client is required. Python-oracledb is the new name for the cx_Oracle driver.

To get started quickly, use samples/sample_container to create a container image containing Oracle Database and python-oracledb.

Top Features in python-oracledb 2.4

• Support for Oracle Database 23ai Pipelining. This is a great feature. For details, see my companion blog post Pipelined database operations with python-oracledb 2.4.
• A refactored connection string parser to improve support for various connection string syntaxes.
• Added packages for Python 3.13 and dropped support for Python 3.7.

Other enhancements and bug fixes also landed. Check out the release notes for details.

Installing or Upgrading python-oracledb

You can install or upgrade python-oracledb by running:

python -m pip install oracledb --upgrade

The pip options --proxy and --user may be useful in some environments. See python-oracledb Installation for details.

Python-oracledb References

Home page: oracle.github.io/python-oracledb/index.html

Installation instructions: python-oracledb.readthedocs.io/en/latest/installation.html

Documentation: python-oracledb.readthedocs.io/en/latest/index.html

Release Notes: python-oracledb.readthedocs.io/en/latest/release_notes.html

Discussions: github.com/oracle/python-oracledb/discussions

Issues: github.com/oracle/python-oracledb/issues

Source Code Repository: github.com/oracle/python-oracledb

Oracle Database’s built-in “Client Result Cache” is an efficient, integrated, managed cache that can dramatically improve query performance and significantly reduce database load when repeatedly querying mostly-static tables, such as postal codes or part numbers. No application changes are needed. No separate mid-tier cache needs to be installed. CRC is available to any “Thick” client that uses Oracle Client libraries, such as drivers for Python, Node.js, Go, PHP, Rust, Ruby, and Oracle’s C API. It is also available in JDBC. This blog post demos an example in Python.

Benefits of Client Result Caching

• Can be used without needing to modify application code
• Improved query response time
• Statements are not sent to the database to be executed
• Better performance by eliminating server round-trips
• Improved database server scalability by saving server resources
• Automatically managed cache invalidation, keeping the cache consistent with database changes
• No mid-tier cache server required
• Developers don’t need to build, or use, a custom cache

What is Client Result Caching?

The Client Result Cache (CRC) is a memory area inside a client process (i.e. the application process) that caches SELECT query results. Its presence is invisible to application code, which just executes queries and gets results as normal. The cache is internal to the Oracle client libraries used by the application. The client libraries know whether a query’s result set is already cached. If so, the SELECT statement gets the results from the cache immediately without involving the database. If the results aren’t in the cache, then the SELECT is sent to the database for execution — which has obvious network and database costs, and contributes to overall system load. Magically the client libraries know whether the cache is outdated and needs to be refreshed. This makes CRC very attractive because application logic doesn’t need to be changed to give performance benefits.

Oracle Database has supported CRC for many versions. When we first released it back in the Oracle Database 11g timeframe, we ran the ‘Niles benchmark’ and saw these material improvements:

How does Client Result Caching work?

Oracle Client libraries manage a result cache for each client process. It is shared by all sessions (i.e. connections) inside that process. The cache can be enabled, and its size specified, with a database initialization parameter.

Client result caching stores the results of the outermost query, which are the columns defined by application.

Oracle Database transparently keeps the client result cache consistent with session state or database changes that affect it. When any database transaction changes the data or metadata of database objects used to build the cached result, the database sends an invalidation flag to the client as part of its response to the application’s next round-trip. (A round-trip is defined as the travel of a message from the application to the database and back. Calling each driver function, or accessing a driver attribute, will require zero or more round-trips.)

If the application is idle and hasn’t initiated a round-trip to the database in a certain amount of time, then cached values are assumed to be invalid. This invalidation time is configurable.

Oracle recommends using CRC for queries from small, read-only or read-mostly tables, however some customers have used it for relatively large tables.

Enabling Client Result Caching in Oracle Database

Caching is disabled by default. It can be enabled by setting the database parameter CLIENT_RESULT_CACHE_SIZE to 32K or greater, and optionally tuning the cache entry invalidation time parameter CLIENT_RESULT_CACHE_LAG. The defaults for the parameters are 0 bytes and 3000 milliseconds, respectively.

If you are using pluggable databases the parameters can be set per-PDB.

If CRC is enabled (i.e CLIENT_RESULT_CACHE_SIZE is 32K or more), then individual client applications can use different sizes by setting sqlnet.ora or oraaccess.xml parameters, see the Oracle Call Interface documentation.

Once CRC is enabled, caching of query result sets is then activated by one of:

• Setting the table caching mode when creating a table: CREATE TABLE mytable (...) RESULT_CACHE (MODE FORCE);
• Modifying an existing table: ALTER TABLE mytable RESULT_CACHE (MODE FORCE);
• Using an explicit query hint SELECT /*+ RESULT_CACHE */ * FROM mytable;

The application must be using the separate, driver statement caching feature. This is enabled by default for most drivers.

If CRC is activated at the table level, you can disable it for specific queries by using the hint /*+ NO_RESULT_CACHE */

Python Example

In my demo, I enabled caching by setting the database initialization parameters in my pluggable database:

$ sqlplus -l sys@localhost/orclcdb as sysdba
. . .SQL> alter session set container=orclpdb1;Session altered.SQL> alter system set client_result_cache_size=32K scope=spfile;System altered.SQL> alter pluggable database orclpdb1 close;Pluggable database altered.SQL> alter pluggable database orclpdb1 open;Pluggable database altered.SQL> show parameter client_result_cacheNAME         TYPE  VALUE
------------------------------------ ----------- ------------------------------
client_result_cache_lag       big integer 3000
client_result_cache_size      big integer 32K

My demo app (code is here) is a Python Flask web app returning six records from a table crcparts:

create table crcparts
   (id        number primary key,
    category  varchar2(20),
    name      varchar2(20),
    price     number);

The app enables Thick mode, which is currently a pre-requisite for using CRC in python-oracledb:

oracledb.init_oracle_client(lib_dir=ld)

The app has two endpoint routes, /usecrc and /nocrc that respectively execute the SQL queries:

SELECT /*+ RESULT_CACHE */    * FROM CRCPARTSSELECT /*+ NO_RESULT_CACHE */ * FROM CRCPARTS

The queries and application endpoints return the exact same results.

The app also has an endpoint /monitor that queries v$sqlarea to show how many times the database executed the two SELECT statements:

cursor.execute(
    """select executions, sql_text
       from v$sqlarea
       where sql_text = :q1 or sql_text = :q2
       order by 2""",
    [QRY_CRC, QRY_NO_CRC])

I started the application:

$ python3 app.py

And initiated some load against both end points using the Apache benchmark utility. Representative samples are shown below.

With CRC

$ ab -n 1000 http://127.0.0.1:8080/usecrc
. . .
Time taken for tests:   13.457 seconds
Complete requests:      1000
. . .
Requests per second:    74.31 [#/sec] (mean)
Time per request:       13.457 [ms] (mean)
. . .

Without CRC

$ ab -n 1000 http://127.0.0.1:8080/nocrc
. . .
Time taken for tests:   22.359 seconds
Complete requests:      1000
. . .
Requests per second:    44.72 [#/sec] (mean)
Time per request:       22.359 [ms] (mean)
. . .

Results

You can see that the CRC-enabled endpoint was more efficient, serving the 1000 requests in almost half the time of the non-CRC endpoint. This also translates to reduced load on the database. This can be shown with the /monitor endpoint:

Although we know the application executed 1000 queries in each case, the database never even saw 999 of the queries initiated by the /usecrc endpoint. Instead, the application got the results directly from the cache. This let the database handle other load, thus improving system scalability.

Conclusion

Oracle Database’s Client Result Cache is a simple, efficient, integrated, managed cache that can dramatically improve application performance when querying lookup tables, and can significantly reduce database load. No application changes are needed. It is best for repeated queries against relatively small tables containing read-mostly or read-only data.

CRC is a database feature available in all editions. It is a mature technology, having been released back in the 11g days. Enhancements continue to be made.

CRC is available to any “Thick” client that uses Oracle Client libraries, such as drivers for Python, Node.js, Go, PHP, Rust, and Oracle’s C API. It is also available in JDBC. It is currently not available in python-oracledb Thin mode or node-oracledb Thin mode, but support is on the roadmap.

Client Result Caching FAQ

1. What is Client Result Caching? CRC is an Oracle Database feature that stores results of queries (i.e. SELECT and WITH statements) in the client application so subsequent, identical queries don’t need to be sent to the database for processing.
2. How is Client Result Caching enabled? Set the database initialization parameter CLIENT_RESULT_CACHE_SIZE to 32K or higher, and optionally tune CLIENT_RESULT_CACHE_LAG. Use table annotations to mark tables as cacheable, or alternatively explicitly annotate queries on the client with the result cache hint /*+ RESULT_CACHE */
3. Which Oracle Database APIs support CRC? Any “Thick” client that uses Oracle Client libraries, such as drivers for Python, Node.js, Go, PHP, Rust, Ruby, and Oracle’s C API. It is also available in JDBC.
4. How many caches are there? There is one cache per client process. It is shared by all sessions in each process.
5. Can one session see results of other sessions? Sessions logged in as User-A cannot see the results cached by sessions logged in as User-B.
6. What are table annotations? The table annotation RESULT_CACHE (MODE FORCE)withALTER TABLE and CREATE TABLE can enable caching on a table without needing any application changes. All tables used in the query need to have FORCE mode otherwise a query hint is needed for caching. To check, do: select substr(table_name,1,25), result_cache from user_tables
7. Does client result caching need the tables to be read only? No. The cache is sensitive to database and session level changes that can affect the result set. However it is most efficient for mostly static tables.
8. What datatypes are supported? Queries involving basic datatypes from user-owned tables are supported. Queries involving LOBs, LONGs, cursors, database links, complex types, or PL/SQL functions in the SELECT list will not have results cached.
9. How does Oracle Client keep results consistent? Every round-trip from the application to the database comes back with any invalidations for the client cache entries. These changes invalidate the relevant cache results (not the whole cache). The next application query execution will be sent to the database and the new results will be loaded into the client cache. If the application is not making any round-trips then, after a configurable time interval, the client library will always send the next query execution to the database. Hence unnecessary round-trips, e.g. polling to server, is not required to check for updates. The cache is also sensitive to session level changes like NLS settings, roles, etc. that may affect the result set. So the caching and consistency with respect to database and session changes are transparent to the application.
10. How is CRC related to the server result cache? The server result cache caches top-level queries and sub-queries in the SGA on the server. It uses the same SQL statement result cache hints but is a different feature from the Client Result Cache. The CRC is for top-level queries executed from applications. Both these two result cache features can be enabled or disabled independently. Note the server cache will optimize “execute” and “fetch” round-trips when they reach the server. The client result cache will prevent “execute” and “fetch” round-trips from reaching the server by making the query execution local to the application
11. How is CRC related to statement caching? Statement caching is a separate client cache that stores the text of SQL statements and is used to reduce the cost of statement parsing and of metadata transfers from the database. The only relationship is that CRC requires statement caching to be enabled.

References

• Oracle Database Performance Tuning Guide Client Result Cache Concepts
• Database Development Guide Using Client Result Cache
• Database initialization parameter CLIENT_RESULT_CACHE_SIZE
• Database initialization parameter CLIENT_RESULT_CACHE_LAG
• sqlnet.ora and oraaccess.xml parameters Oracle Call Interface
  documentation
• Python-oracledb driver https://pypi.org/project/oracledb/

00000000000 4000000004 00000000004

000000000004 400000000000000 00000000000

000000000000 4000000000000000000 000000000000

0000000000000 000000000000000000000 0000000000000

00000000000000 00000000000000000000004 00000000000004

000000000000004 0000000000 000000000 00000000000000

000000000000000 000000000 000000004 000000000000000

0000000400000000 400000000 000000000 0000004000000000

0000000 000000000 000000000 000000000 40000004 000000004

0000000 000000004 000000000 000000000 4000000 00000000

0000004 00000000 000000000 000000000 0000000 400000000

40000004 400000000 0000000004 000000000 0000000 000000000

4000000 000000000 000000000 000000000 0000000 000000000

0000000 000000000 000000000 000000000 0000000 00000000

0000000 00000000 000000000 0000000004 0000000 400000000

0000000 00000000 000000000 000000000 0000000 000000000

0000000 000000000 000000000 000000000 0000000 000000000

0000000 000000000 000000000 000000000 0000000 00000000

00000000000000000000000000000 000000000 000000000 00000000000000000000000000000

00000000000000000000000000000 000000000 000000000 00000000000000000000000000000

00000000000000000000000000000 000000000 000000000 000000000000000000000000000000

000000000000000000000000000000 000000000 000000000 00000000000000000000000000000

00000000000000000000000000000 400000000 000000000 00000000000000000000000000000

44444444444444440000000044444 000000000 000000000 44444444444444400000000044444

000000000 00000000000000000000000 000000000

000000000 000000000000000000000 000000004

00000000 0000000000000000000 00000000

00000000 4000000000000000 000000000

000000000 0000000000 000000000

GitHub and Pro/Business supporters will get it earlier, around the first of each month.

On GitHub sponsors, I am currently 42% towards my $2,500 per month goal, which is set to allow continued maintenance of Xdebug. This is less than last month.

If you are leading a team or company, then it is also possible to support Xdebug through a subscription.

In the last month, I spend around 15 hours on Xdebug, with 21 hours funded. I also spent 6 hours on the Native Path Mapping project.

In case you missed it, Oracle Instant Client is available for macOS ARM64. See the release announcement post by my colleague Sharad Chandran.

I commonly install the ‘Basic’ package to give me the Oracle Client libraries needed for runtime by tools & Thick drivers, the ‘SQL*Plus’ package to give me a command line SQL query tool, and the ‘SDK’ package to give me header files (because I still build OCI-based programs like the PHP OCI8 driver):

curl -O https://download.oracle.com/otn_software/mac/instantclient/instantclient-basic-macos-arm64.dmg
curl -O https://download.oracle.com/otn_software/mac/instantclient/instantclient-sqlplus-macos-arm64.dmg
curl -O https://download.oracle.com/otn_software/mac/instantclient/instantclient-sdk-macos-arm64.dmghdiutil mount instantclient-basic-macos-arm64.dmg
hdiutil mount instantclient-sqlplus-macos-arm64.dmg
hdiutil mount instantclient-sdk-macos-arm64.dmg/Volumes/instantclient-basic-macos*/install_ic.shhdiutil unmount /Volumes/instantclient-basic-*/
hdiutil unmount /Volumes/instantclient-sqlplus-*/
hdiutil unmount /Volumes/instantclient-sdk-*/

Check the download page for the latest links and other packages.

I’ve just released PHP OCI8 3.4 on PECL. PHP is still active and going strong — and this is reflected in the download numbers for the Oracle Database “OCI8” driver for PHP, which had a 50% growth over the last year.

PHP OCI8 3.4 is a small iteration with one important change to better support external authentication, e.g with Oracle wallets, when using oci_pconnect(). Other changes allow it to build with PHP 8.2 and PHP 8.3, and for the test suite to run cleanly with the improved database errors messages of Oracle Database 23ai.

Install the new release in PHP 8.2 or 8.3 as a shared extension by executing:

$ pecl install oci8

and, when prompted, specify where to find Oracle Client libraries. On macOS ARM64 with the latest Instant Client in $HOME/Downloads/instantclient_23_3, my install log is like:

$ pecl install oci8 
downloading oci8-3.4.0.tgz ...
Starting to download oci8-3.4.0.tgz (194,647 bytes)
.........................................done: 194,647 bytes
13 source files, building
running: phpize
Configuring for:
PHP Api Version:         20230831
Zend Module Api No:      20230831
Zend Extension Api No:   420230831
Please provide the path to the ORACLE_HOME directory. Use 'instantclient,/path/to/instant/client/lib' if you're compiling with Oracle Instant Client [autodetect] : instantclient,/Users/cjones/Downloads/instantclient_23_3. . .Build process completed successfully
Installing '/Users/cjones/php/lib/php/extensions/no-debug-non-zts-20230831/oci8.so'
install ok: channel://pecl.php.net/oci8-3.4.0
configuration option "php_ini" is not set to php.ini location
You should add "extension=oci8.so" to php.ini

(Remember to use an absolute path without environment variables!)

I then typically run php --ini to find the location of the php.ini file, and append extension=oci8.so to that file.

You can now check the driver is available:

$ php --ri oci8oci8OCI8 Support => enabled
OCI8 DTrace Support => disabled
OCI8 Version => 3.4.0
Oracle Run-time Client Library Version => 23.3.0.23.9
Oracle Compile-time Instant Client Version => 23.3Directive => Local Value => Master Value
oci8.max_persistent => -1 => -1
oci8.persistent_timeout => -1 => -1
oci8.ping_interval => 60 => 60
oci8.privileged_connect => Off => Off
oci8.statement_cache_size => 20 => 20
oci8.default_prefetch => 100 => 100
oci8.old_oci_close_semantics => Off => Off
oci8.connection_class => no value => no value
oci8.events => Off => Off
oci8.prefetch_lob_size => 0 => 0Statistics =>  
Active Persistent Connections => 0
Active Connections => 0

The OCI8 extension can be built with Oracle Client libraries from Oracle Database 11.2 or later. The Oracle Client libraries are in the free Oracle Instant Client from www.oracle.com/database/technologies/instant-client.html. They are also included in your database installation.

Oracle’s standard cross-version connectivity applies. For example, PHP OCI8 linked with Oracle Client 19c can connect to Oracle Database 11.2 onward. See Oracle’s note Client / Server Interoperability Support Matrix for Different Oracle Versions (ID 207303.1) for details.

PHP OCI8 3.4 also installs on the current PHP 8.4 Alpha release. If you have an older version of PHP, you will need to install an older driver, e.g.:

• Use pecl install oci8–3.2.1 to install for PHP 8.1.
• Use pecl install oci8–3.0.1 to install for PHP 8.0.

If you need a reference for using PHP OCI8, check the second half of The Underground PHP and Oracle Manual and, of course, the OCI8 Documentation.

python-oracledb 2.3 has been released. My colleague @Veronica Dumitriu has posted the release announcement. There are lots of good improvements, such as support for the new VECTOR binary format that is available in Oracle Database 23.5.

One of the smaller (excuse the pun) enhancements is that the binary packages we put on PyPI for Linux are now stripped.

With the previous python-oracledb 2.2 release the install footprint was:

$ du -sh /home/cjones/.local/lib/python3.11/site-packages/oracledb/
49M /home/cjones/.local/lib/python3.11/site-packages/oracledb/

Now with python-oracledb 2.3 :

$ du -sh /home/cjones/.local/lib/python3.11/site-packages/oracledb/
7.2M /home/cjones/.local/lib/python3.11/site-packages/oracledb/

For you micro-optimizers, we know every byte counts!

This small change brings Linux into parity with macOS, where the install footprint remains around the 11M mark.

Installing or Upgrading python-oracledb

You can install or upgrade python-oracledb by running:

python -m pip install oracledb --upgrade

The pip options --proxy and --user may be useful in some environments. See python-oracledb Installation for details.

Python-oracledb References

Home page: oracle.github.io/python-oracledb/index.html

Installation instructions: python-oracledb.readthedocs.io/en/latest/installation.html

Documentation: python-oracledb.readthedocs.io/en/latest/index.html

Release Notes: python-oracledb.readthedocs.io/en/latest/release_notes.html

Discussions: github.com/oracle/python-oracledb/discussions

Issues: github.com/oracle/python-oracledb/issues

Source Code Repository: github.com/oracle/python-oracledb

Wondering how to install Oracle Instant Client on Oracle Linux from Oracle’s yum repository? This is what you need to know.

Oracle Instant Client enables development and deployment of applications that connect to Oracle Database, either on-premise or in the Cloud. The Instant Client libraries provide the necessary network connectivity and advanced data features to make full use of Oracle Database. The libraries are used by the Oracle APIs of popular languages and environments including Python, Node.js, Go, PHP and Ruby, as well as providing access for Oracle Call Interface (OCI), Oracle C++ Call Interface (OCCI), JDBC OCI, ODBC and Pro*C applications. Tools included in Instant Client, such as SQL*Plus, SQL*Loader and Oracle Data Pump, provide quick and convenient data access.

Oracle Instant Client is available on many platforms as simple zip downloads from Oracle. This post shows how the packages can also be installed using Oracle Linux yum repositories.

Packages

18c & 19c OL7:

• https://yum.oracle.com/repo/OracleLinux/OL7/oracle/instantclient/x86_64/
• https://yum.oracle.com/repo/OracleLinux/OL7/oracle/instantclient/aarch64/

sudo yum install oracle-release-el7
sudo yum install oracle-instantclient19.23-basic

19c OL8:

• https://yum.oracle.com/repo/OracleLinux/OL8/oracle/instantclient/x86_64/
• https://yum.oracle.com/repo/OracleLinux/OL8/oracle/instantclient/aarch64/

sudo dnf install -y oracle-release-el8
sudo dnf install -y oracle-instantclient19.23-basic

19c OL9:

• https://yum.oracle.com/repo/OracleLinux/OL9/oracle/instantclient/x86_64/

sudo dnf install oracle-instantclient-release-el9
sudo dnf install oracle-instantclient19.19-basic

21c OL7:

• https://yum.oracle.com/repo/OracleLinux/OL7/oracle/instantclient21/x86_64/

sudo yum install oracle-instantclient-release-el7
sudo yum install oracle-instantclient-basic

21c OL8:

• https://yum.oracle.com/repo/OracleLinux/OL8/oracle/instantclient21/x86_64/

sudo dnf install oracle-instantclient-release-el8
sudo dnf install oracle-instantclient-basic

23ai OL8:

• https://yum.oracle.com/repo/OracleLinux/OL8/oracle/instantclient23/x86_64/

sudo dnf install oracle-instantclient-release-23ai-el8
sudo dnf install oracle-instantclient-basic

23ai OL9:

• https://yum.oracle.com/repo/OracleLinux/OL9/oracle/instantclient23/x86_64/

sudo dnf install oracle-instantclient-release-23ai-el9
sudo dnf install oracle-instantclient-basic

Notes

For 18c and 19c, you need to explicitly give the RU (the ‘Release Update’ patch number) in the filename when installing. To upgrade, you need to remove the old package and install the new one. Please check the repos and keep up to date!

From 21c, any yum or dnf update will automatically pull in the latest RU. If
you need to prevent an update, then use versionlock, as discussed in the
Installing Oracle Instant Client Using RPMs documentation, e.g :

dnf versionlock oracle-instantclient-release-23ai-el8

Containers

Sample Dockerfiles:

• https://github.com/oracle/docker-images/tree/main/OracleInstantClient

Prebuilt containers — with Basic, SQL*Plus, and SDK packages:

• https://github.com/oracle/docker-images/pkgs/container/oraclelinux7-instantclient
• https://github.com/oracle/docker-images/pkgs/container/oraclelinux8-instantclient
• https://github.com/oracle/docker-images/pkgs/container/oraclelinux9-instantclient

Use like:

docker pull ghcr.io/oracle/oraclelinux9-instantclient:23

Instant Client Installation Documentation

Oracle Linux installation instructions: https://yum.oracle.com/oracle-instant-client.html

Oracle Client 23ai installation documentation: Installing Oracle Instant Client Packages

Oracle Client 21c installation documentation Installing Oracle Instant Client Packages

Oracle Database 23ai JSON-Relational Duality Views allow data to be stored as rows in tables to provide the benefits of SQL access in a relational model, while also allowing read and write access as JSON documents to the exact same data. This blog shows how to use the new views in Python.

The relational model is great: you can avoid data duplication; data consistency is guaranteed; and you have access via a very powerful, very efficient language — SQL. But, for developers, the requirement to define a relational schema — to decide on tables, columns and data types — before beginning to code is a chore. It’s not easy to predict future uses for the system, some of which may be difficult with the chosen schema.

This is why you love JSON. A JSON object can contain all the information for one use case without the need to use SQL to join tables. Access is via a simple query, or a single call to a database API. JSON is schema-flexible so, as your use cases evolve over the lifetime of a system, you can easily modify applications. But there are drawbacks: a single hierarchy may only allow a few use cases. Data can end up being duplicated, which affects not just space but makes it very hard to keep consistent. Optimization is harder. So what appears to be a simple model can end up causing long term complexity.

This is where Oracle Database 23ai JSON-Relational Duality Views are an outright game changer — not a phrase I use lightly. These duality views build on Oracle Database’s long history of JSON support and longer history with the relational model.

Let’s start by creating two relational tables:

drop table if exists AuthorTab;
drop table if exists BookTab;create table AuthorTab (
    AuthorId number generated by default on null as identity primary key,
    AuthorName varchar2(100)
);create table BookTab (
    BookId number generated by default on null as identity primary key,
    BookTitle varchar2(100),
    AuthorId number references AuthorTab (AuthorId)
);insert into AuthorTab values (1, 'Isabel M. Rich');
insert into AuthorTab values (2, 'Bobbie Cool');
insert into AuthorTab values (3, 'Charlie Shore');
insert into BookTab values (1, 'The Mysterious Dog', 1);
insert into BookTab values (2, 'The Mysterious Pony', 1);
insert into BookTab values (3, 'The Mysterious Tiger', 1);
insert into BookTab values (4, 'Self Help for Programmers', 2);
insert into BookTab values (5, 'More Self Help for Programmers', 2);
insert into BookTab values (6, 'Travel Guide Volume I', 3);
insert into BookTab values (7, 'Travel Guide Volume II', 3);commit;

These can obviously be accessed via relational SELECT, INSERT etc statements which are trivial to execute in Python using python-oracledb, for example:

import oracledbconnection = oracledb.connect(user="cj", password="MySecret", 
                              dsn="localhost/orclpdb1")with connection.cursor() as cursor:
    sql = """select AuthorName, BookTitle
             from AuthorTab, BookTab
             where AuthorTab.AuthorId = BookTab.AuthorId"""
    for r in cursor.execute(sql):
        print(r)

The output lists the authors and their books:

('Isabel M. Rich', 'The Mysterious Dog')
('Isabel M. Rich', 'The Mysterious Pony')
('Isabel M. Rich', 'The Mysterious Tiger')
('Bobbie Cool', 'Self Help for Programmers')
('Bobbie Cool', 'More Self Help for Programmers')
('Charlie Shore', 'Travel Guide Volume I')
('Charlie Shore', 'Travel Guide Volume II')

The fun is to create a duality view. In your SQL editor run:

create or replace json relational duality view BookDV as
BookTab @insert @update @delete
{
   _id: BookId,
   book_title: BookTitle,
   author: AuthorTab @insert @update
   {
       author_id: AuthorId,
       author_name: AuthorName
   }
};

This creates a view that is accessible using JSON. The syntax shown here is GraphQL, letting the database work out relations. Alternatively you could use a simplified SQL syntax — see the documentation.

With the BookDV duality view created, you can access it as a document or by using the power of python-oracledb and Oracle Database’s JSON features. For example, to find the book with ID of 1 using simple dot-notation access you can access the special column DATA:

sql = """select b.data.book_title, b.data.author.author_name
	 from BookDV b
         where b.data."_id" = :1"""
for r in cursor.execute(sql, [1]):
    print(r)

This shows:

('The Mysterious Dog', 'Isabel M. Rich')

To access the duality view with simple API calls, use Oracle Database’s SODA functionality (this requires python-oracledb be run in Thick mode). For example, to run a query-by-example on the view, open it as a collection:

soda = connection.getSodaDatabase()
collection = soda.openCollection('BOOKDV')qbe = {'book_title': {'$like': 'The%'}}
for doc in collection.find().filter(qbe).getDocuments():
    content = doc.getContent()
    print(content["book_title"])

This prints the matching book titles:

The Mysterious Dog
The Mysterious Pony
The Mysterious Tiger

The real magic of duality views comes when you need to update or insert new data. You can use the view for that too, for example with an INSERT statement:

data = dict(
    _id=101,
    book_title="Cooking at Home",
    author=dict(author_id=201, author_name="Dave Smith"),
)
inssql = "insert into BookDV values (:1)"
cursor.setinputsizes(oracledb.DB_TYPE_JSON)
cursor.execute(inssql, [data])
connection.commit()

The duality view understands the relationship between the tables and both tables get updated, as can be seen with a canonical query in your SQL editor:

SQL> select BookId, BookTitle, AuthorName, AuthorTab.Authorid
  2  from AuthorTab, BookTab
  3  where AuthorTab.AuthorId = BookTab.AuthorId;BOOKID BOOKTITLE                      AUTHORNAME             AUTHORID
---------- ------------------------------ -------------------- ----------
         1 The Mysterious Dog             Isabel M. Rich                1
         2 The Mysterious Pony            Isabel M. Rich                1
         3 The Mysterious Tiger           Isabel M. Rich                1
         4 Self Help for Programmers      Bobbie Cool                   2
         5 More Self Help for Programmers Bobbie Cool                   2
         6 Travel Guide Volume I          Charlie Shore                 3
         7 Travel Guide Volume II         Charlie Shore                 3
       101 Cooking at Home                Dave Smith                  201

Simple! You can also update the view via SODA, if that is your preferred style.

A runnable example is in json_duality.py

Conclusion

Oracle Database 23ai JSON-Relational Duality Views marry the best of the relational models with the best of JSON. Both JSON and SQL can be used in many Oracle Database language drivers, giving you a wide choice of client technologies and data models.

See the JSON-Relational Duality Developer’s Guide for more information.

Also, I recommend you join my colleague Beda Hammerschmidt for his free JSON in Oracle Database Office Hours series. You can watch recordings of previous presentations, or sign-up for upcoming live ones.

And finally never forget that Oracle Database is a database with many models, including: JSON, NoSQL, Graph, Object, Spatial, XML, Vectors, and Relational. This gives you enormous power, and lets you keep all your data safe in the one system.

References

• Documentation: JSON-Relational Duality Developer’s Guide
• Oracle Forum: ORDS, SODA & JSON in the Database
• Software: python-oracledb driver
• Software: Oracle Database 23ai

When using Python to interact with Oracle Database PL/SQL stored procedures, you may encounter PL/SQL code that uses the %ROWTYPE attribute. This is used to declare a record that represents either a full or partial row of a database table or view. This blog post shows how to use %ROWTYPE in python-oracledb.

Let’s start with a PL/SQL function that returns a row of the sample schema LOCATIONS table using LOCATIONS%ROWTYPE as the return type. You might create this in Python like:

import getpass
import oracledbun = 'cj'
cs = 'localhost/orclpdb1'
pw = getpass.getpass(f'Enter password for {un}@{cs}: ')
connection = oracledb.connect(user=un, password=pw, dsn=cs)with connection.cursor() as cursor:
    cursor.execute("""
        create or replace function TestFuncOUT return locations%rowtype as
          p locations%rowtype;
        begin
           select * into p from locations where rownum < 2;
           return p;
        end;""")
    if cursor.warning:
        print(cursor.warning)

The %ROWTYPE attribute defines a record, so you can use python-oracledb’s gettype() object functionality to get the record "shape", and pass this as the expected return type to callfunc() when you invoke TestFuncOUT:

    rt = connection.gettype("LOCATIONS%ROWTYPE")
    r = cursor.callfunc("TESTFUNCOUT", rt)

The python-oracledb type of r is oracledb.DbObject. You can manipulate and view it using that class's methods. A handy convenience function that prints an instance is dump_object() defined in samples/object_dump.py. Calling it on the return value from TestFuncOUT:

dump_object(r)

gives:

{
  LOCATION_ID: 1000
  STREET_ADDRESS: '1297 Via Cola di Rie'
  POSTAL_CODE: '00989'
  CITY: 'Roma'
  STATE_PROVINCE: None
  COUNTRY_ID: 'IT'
}

which is the row returned from the query.

If you need to construct a similar object directly in python-oracledb to pass into the database, you can use newobject() and set any desired fields. For example:

    rt = connection.gettype("LOCATIONS%ROWTYPE")
    r = rt.newobject()
    r.CITY = 'Roma'

Passing a value from Python into a PL/SQL block that expects a %ROWTYPE parameter or bind variable is straightforward. Let’s create a PL/SQL procedure that has an IN parameter of type %ROWTYPE:

cursor.execute("""
        create or replace procedure TestProcIN(p in locations%rowtype, city out varchar2) as
        begin
            city := p.city;
        end;""")
    if cursor.warning:
        print(cursor.warning)

Now you can call callproc() passing the variable r from the previous callfunc() or newobject() examples in the appropriate parameter position, for example:

    c = cursor.var(oracledb.DB_TYPE_VARCHAR)
    cursor.callproc("TESTPROCIN", [r, c])
    print(c.getvalue())

This prints:

Roma

Conclusion

PL/SQL %ROWTYPE attribute values map to python-oracledb’s DbObject instances, allowing easy manipulation and interaction between Python and Oracle Database.

Installing or Upgrading python-oracledb

You can install or upgrade python-oracledb by running:

python -m pip install oracledb --upgrade

The pip options --proxy and --user may be useful in some environments. See python-oracledb Installation for details.

Python-oracledb References

Home page: oracle.github.io/python-oracledb/index.html

Installation instructions: python-oracledb.readthedocs.io/en/latest/installation.html

Documentation: python-oracledb.readthedocs.io/en/latest/index.html

Release Notes: python-oracledb.readthedocs.io/en/latest/release_notes.html

Discussions: github.com/oracle/python-oracledb/discussions

Issues: github.com/oracle/python-oracledb/issues

Source Code Repository: github.com/oracle/python-oracledb

This is how that looks like:

My open source packages were downloaded roughly 138945563 times.

Like most live counters, this number isn’t tracked in real-time. Instead it just uses a start number and updates the number based on an average number of downloads.

One day it might be nice to make it live, but this is a static blog and this would need proper hosting and a database.

Why is this number so high?

This data comes from NPM and Packagist, and they both count any download. So this number doesn’t represent necessarily 138 million users, but simply this many downloads by any means including bots, CI environments and so on. I think for both package managers the goal was probably not to have a realistic representation of users, but rather a number that makes developers feel good. And I like that. It’s nice to see a number go up and it’s still a nice proxy for relative popularity.

The Web Component

This seemed like a good use-case for a web component. Always wanted to build one! I was surprised how easy it was.

This is how this looks in the HTML:

<p>
  My open source packages were downloaded roughly
  <strong>
    <download-counter inc-per-day="157444" date="2024-06-29T15:34:00Z" >
      138945563
    </download-counter>
  </strong> times.
</p>

What’s nice is that if Javascript is not enabled, or Web Components are not supported, this will just fall back on showing the static number.

I included 3 parameters:

• The last recorded download count (in the element value)
• When that number was recorded (date)
• Average number of downloads per day.

I wanted to include the date because I only intend to update these numbers rarely, so we need to know how many downloads have elapsed since the last time.

Writing the web component was surprisingly straightforward too. Here is it in its fully glory:

class DownloadCounter extends HTMLElement {
  connectedCallback() {
    this.count = +this.textContent;
    this.date = new Date(this.getAttribute('date'));
    this.inc = (+this.getAttribute('inc-per-day')) / (3600 * 24 * 1000)
    this.calculateCurrentDownloads();
  }
  calculateCurrentDownloads() {
    const currentDownloads =
      Math.floor(
      this.count +
      (Date.now()-this.date) * this.inc
      );
    // Intl.NumberFormat adds thousands separators
    this.textContent = Intl.NumberFormat().format(currentDownloads);
    setTimeout(
      () => this.calculateCurrentDownloads(),
      // Add some randomnes
      Math.floor(Math.random() * 150)+50,
    );
  }
}
customElements.define(
  'download-counter',
  DownloadCounter,
);

Now just include the file in the HTML and you’re good to go:

<script type="module" src="/assets/js/downloadcounter.mjs"></script>

Obtaining the data

I’ve published 20 PHP libraries on packagist and 30 Javascript libraries on NPM. This is too much to count, so instead I wrote some scripts that pull in the data for their respective APIs.

You cannot easily ask these APIs what the download count or download rate at a given date was, and for both the numbers might be a bit delayed. So I’ve opted to simply:

• Get a grand total of all downloads up until this date.
• Wait at least 24 hours or more.
• Get another grand total and use the difference to caclulate average download rate.

Below is my approach for getting the numbers from NPM and Packagist. It’s a bit messy and imperative.

NPM

The NPM api gave me weird, incomplete results for getting the whole list of packages I’ve authored, so I worked around this by hardcoding some package names, and then augmenting this with the result of 3 searches.

This gives us the full list of packages I’m interested in:

// Searches
const npmSearches = [
  // by author
  'author:evrt',
  // by org name
  'scope:badgateway',
  'scope:curveball',
];
// Hardcoded extra packages that for some reason didn't get returned with
// the searches
const npmPackages = {
  'davclient.js': 0,
  'structured-headers': 0,
  'bigint-money': 0,
  'fetch-mw-oauth2': 0,
  'hal-types': 0,
  'react-ketting': 0,
  'ketting': 0,
  'html-form-enhancer': 0,
  'changelog-tool': 0,
};
async function fetchNpmPackageList() {
  for(const search of npmSearches) {
    const res = await fetch('https://registry.npmjs.org/-/v1/search?text=' + search);
    const body = await res.json();
    for(const object of body.objects) {
      npmPackages[object.package.name] = 0;
    }
  }
}

NPM doesn’t return absolute total download counts but instead we can get a count for a specific time range with a maximum range of 18 months. I’ve opted to instead to get download counts per year, counting backwards for each package until get a result of 0.

async function fetchNpmDownloadCounts(packageName) {
  let year = new Date().getFullYear();
  let count = 0;
  let yearCount;
  do {
    yearCount = await fetchNpmDownloadCountsByYear(packageName, year);
    count += yearCount;
    year--;
  } while (yearCount > 0);
  console.log('%s: %i', packageName, count);
  return count;
}
async function fetchNpmDownloadCountsByYear(packageName, year) {
  const res = await fetch(`https://api.npmjs.org/downloads/point/${year}-01-01:${year}-12-31/${packageName}`);
  const body = await res.json();
  return body.downloads;
}

Packagist

Getting totals from packagist was considerably easier:

const packagistOrgs = [
  'sabre',
  'evert',
];
const packagistPackages = {};
async function fetchPackagistPackages() {
  for(const vendor of packagistOrgs) {
    const res = await fetch(`https://packagist.org/packages/list.json?vendor=${vendor}`);
    const body = await res.json();
    for(const pkg of body.packageNames) {
      packagistPackages[pkg] = 0;
    }
  }
}
async function fetchPackagistDownloadCounts(packageName) {
  const res = await fetch(`https://packagist.org/packages/${packageName}/stats.json`);
  const json = await res.json();
  const count = json.downloads.total;
  console.log('%s: %i', packageName, count);
  return count;
}

Putting it together

async function main() {
  await fetchNpmPackageList();
  await fetchPackagistPackages();
  for(const pkg of Object.keys(npmPackages)) {
    npmPackages[pkg] = await fetchNpmDownloadCounts(pkg);
  }
  for(const pkg of Object.keys(packagistPackages)) {
    packagistPackages[pkg] = await fetchPackagistDownloadCounts(pkg);
  }
  const packagesCombined = [];
  for(const [packageName, downloads] of Object.entries(npmPackages)) {
    packagesCombined.push({
      ecosystem: 'npm',
      packageName,
      downloads
    });
  }
  for(const [packageName, downloads] of Object.entries(packagistPackages)) {
    packagesCombined.push({
      ecosystem: 'packagist',
      packageName,
      downloads
    });
  }
  console.log(packagesCombined);
  console.log('Total: %i', packagesCombined.reduce((acc, cur) => acc+cur.downloads, 0));
}
await main();

Conclusion

Whenever I need small bits of Javascript to enhance a web application (and not a full-blown framework) I tend to write code that looks for an element with a particular class, and then start my logic and hook up events.

Web Components seems like a really great replacement for that.

Got comments? Liked this aticle? You can reply to this Mastodon post to make the comments show up here.

Applications need to be resilient to infrastructure glitches. Oracle Database has powerful features that help your users get their jobs done, without them being aware of planned or unplanned database downtime.

My colleagues Veronica Dumitriu and Nancy Ikeda previously posted a blog How to Make Application Continuity Most Effective in Oracle Database 23c covering AC (“Application Continuity”) and the related TAC (“Transparent Application Continuity”) features. These High Availability technologies are part of Oracle Database and Oracle Client. They allow applications to continue uninterrupted if there is an outage accessing one database node. They function by the driver client libraries recording the “work” that an application does. If the database instance becomes unavailable before all that “work” is committed to the database, the driver transparently connects to another database instance and replays all the recorded work, before letting the app continue as if there had been no outage. AC and TAC need a database that is configured with multiple instances, e.g RAC.

Veronica and Nancy’s post uses examples from the C language Oracle Call Interface API, which is one of the primary APIs for data access. Here I want to share some working notes about AC and TAC with python-oracledb, the Oracle Database driver for Python.

Before my sabbatical at the start of the year, I had been looking at python-oracledb with AC and TAC, in particular how to run the ‘acchk’ tool to check application coverage. AC and TAC are implemented in Oracle Client libraries, and so are transparently available to python-oracledb apps when using Thick mode against Oracle Datase (on-premise or in the Cloud). AC and TAC are similarly available to other drivers like node-oracledb in Thick mode.

AC and TAC are primarily used for unplanned database outages. Applications that use an Oracle connection pool will handle planned downtime gracefully.

My working notes show how to enable AC and TAC on Oracle Autonomous Database and see what protection they give a sample application. As good as AC and TAC are, there are some things they can’t (and shouldn’t) recover.

You can use acchk to review which code paths in your app will be protected and what database work will be replayed by AC or TAC in the event the database instance currently in use has a failure.

A few general app suggestions:

• Use python-oracledb Thick mode. This mode is a must.
• Use a connection pool. Python-oracledb doesn’t (yet) expose the OCIRequestBegin() and OCIRequestEnd() functionality.
• Use ORDER BY or GROUP BY in SQL statements
• Set Connection.module and Connection.action
• Use the Oracle Database 19.11+ acchck coverage report tool
  — This is the successor to Ora*CHK
  — Use it by enabling AC or TAC and then:
  — Use PL/SQL DBMS_APP_CONT_ADMIN package to enable recording
  — Run your app
  — Use DBMS_APP_CONT_ADMIN to get a coverage report

Testing acchk

I was testing using Oracle Autonomous Database. To enable acchk in an on-premise database, review your Oracle documentation.

Enable AC:

sqlplus -l admin@'tcps://adb.xxxx.oraclecloud.com:1522/xxxxx_cjdb_tpurgent.adb.oraclecloud.com?wallet_location=/opt/oracle/cjdb'execute dbms_app_cont_admin.enable_ac('xxxxx_CJDB_tpurgent.adb.oraclecloud.com', 'LEVEL1', 600);select name, failover_type from dba_services;

Or enable TAC:

execute DBMS_APP_CONT_ADMIN.ENABLE_TAC('xxxxx_cjdb_tpurgent.adb.oraclecloud.com', 'AUTO', 600);

Give myself access:

grant acchk_read to cj;grant execute on dbms_app_cont_admin to cj;

Enable recording:

sqlplus -l cj@'tcps://adb.xxxx.oraclecloud.com:1522/xxxxx_cjdb_tpurgent.adb.oraclecloud.com?wallet_location=/opt/oracle/cjdb'execute dbms_app_cont_admin.acchk_set(true);

The run your Python application eg.:

python acchk-test.py

My app is a web service so put some load on it:

ab -n 100 -c 4 http://127.0.0.1:7000/

View the acchk report:

set serveroutput on format wrapped linesize 160
execute dbms_app_cont_report.acchk_report(dbms_app_cont_report.FULL)

Or a summary report:

set serveroutput on format wrapped linesize 160
execute dbms_app_cont_report.acchk_report(dbms_app_cont_report.SUMMARY)

And disable the analysis:

execute dbms_app_cont_admin.acchk_set(false);

And clear records:

execute dbms_app_cont_admin.acchk_purge(purge_all=>true);

References

• Application Continuity Protection Check (acchk)

For those that like ‘official’ things, technical briefs Using Oracle Database 23ai Vectors in python-oracledb and Using Oracle Database vectors in node-oracledb discuss using the Oracle Database 23ai VECTOR data type with the respective database drivers.

It’s reasonably common that classes that write to a log take a PSR\Log\LoggerInterface in their constructor and then use that for logging. This usually logs to the error handler for pushing to Sentry or whatnot. This makes testing easy!

For testing we can use the TestHandler which stores the logs it receives and then they can then be inspected.

We set up like this:

<?php
namespace Test\Integration;
use Monolog\Handler\TestHandler;
use Monolog\Logger;
use PHPUnit\Framework\TestCase;
class FooTest extends TestCase
{
    protected TestHandler $testHandler;
    protected Logger $logger;
    public function setUp(): void
    {
        $this->testHandler = new TestHandler();
        $this->logger = new Logger('test', [$handler]);
    }
    // tests here
}

Then we can use it in a test like this:

public function testFoo(): void
{
    $foo = new Foo($this->logger);
    $result = $foo->bar();
    $this->assertTrue($result);
    $this->assertTrue($this->testHandler->hasInfoThatContains('Created Baz'));
}

TestHandler has a number of methods that allow you to determine if a specific log has been written, along with getRecords(), clear(), etc to allow full inspection and control of the test log. Peruse the source for the class to see all the options, noting particularly the set of convenience methods in the DocBlock.

If your system under test uses a PSR-3 logger, then Monolog’s TestHandler is a lovely way to test that your logs are as expected.

When a long-running application keeps a database connection open for the app lifetime, it is likely that the connection is often idle and not being used to execute SQL statements. However this holds the connection’s server process and session memory open on the database host, ultimately limiting the number of other users who can connect. When re-architecting to use an Oracle driver connection pool isn’t possible, a simple connection string change can make use of Oracle Database 23ai “Implicit Connection Pooling” to share the database host resources. This reduces the memory required, and allows the database to be more scalable.

My previous blog post DRCP helps apps that don’t use a Connection Pool showed an example which connected and disconnected frequently but didn’t use connection pooling. It explained how the app could benefit from Database Resident Connection Pooling (DRCP). This is because those frequent application disconnection calls let the database know when it is safe to reuse database server processes for other application users.

The following example differs because a disconnection call happens only when a long-running application terminates. Pure DRCP won’t help because there is no connection-usage “boundary” to indicate when to reuse the database server processes. Luckily, Oracle Database 23ai “Implicit Connection Pooling” can figure it out.

Let’s look at a demo. The following simple Python app long_run.py opens a connection and then loops. In each iteration a query is executed and there is a sleep simulating user "think time" between queries. The connection is closed when the app finishes (handled by the end of the with context block). This simulates a long-running app that holds onto a connection even when not executing a SQL statement:

# long_run.pyimport os
import timeimport oracledb
import sample_env  # Contains credentials, see python-oracledb samples# Number of queries to execute in the life of this script
NUMSQLS = globals().get("NUMSQLS", 0)# Number of seconds to wait between executing SQL statements
SLEEPTIME = globals().get("SLEEPTIME", 0)# The name of the end "user", unique per execution of this script
APPUSERNAME = globals().get("APPUSERNAME", None)sql = """select unique sid||'-'||serial# as sidser,
                       current_timestamp as ct
         from v$session_connect_info
         where sid = sys_context('USERENV', 'SID')"""with oracledb.connect(
    user=sample_env.get_main_user(),
    password=sample_env.get_main_password(),
    dsn=sample_env.get_connect_string(),
) as connection:
    for i in range(NUMSQLS):
        with connection.cursor() as cursor:
            for s, d in cursor.execute(sql):
                print(f"{APPUSERNAME} sid-ser {s} at time {d}")
        time.sleep(SLEEPTIME)

The application query gets the current time and the Session Identifier/Serial Number — this pair shows the database server process session in use. Each iteration’s query will output a line like:

User01 sid-ser 407-62786 at time 2024-06-11 20:29:34.632666

The connection string is a simple one like localhost/orclpdb1 . There is no DRCP involved.

A helper script, runner.py, is used to simulate a number of users concurrently running long_run.py. The helper script invokes long_run.py 10 times (set in "NUMUSERS") via the use of threads. Each invocation of long_run.py is passed a unique "user name" e.g. User01 (via "APPUSERNAME"), the number of SQL statements to execute ("NUMSQLS"), and the sleep time ("SLEEPTIME") between executions. The sleep time emulates the application user being idle and not executing SQL statements. The helper script is:

# runner.pyimport threading
import timeimport sample_env  # Contains credentials, see python-oracledb samplesNUMUSERS = 10      # number of times to concurrently invoke long_run.pydef start_app(tn):
    app_globals = {
        "APPUSERNAME": "User{:02d}".format(tn + 1),  # Name of the "user" running long_run.py
        "NUMSQLS": 5,     # Number of queries to execute in the life of long_run.py
        "SLEEPTIME": 5,   # Number of seconds to sleep between SQL statements
    }
    exec(open("long_run.py").read(), app_globals)def start_workload():
    thread = []
    for i in range(NUMUSERS):
        t = threading.Thread(target=start_app, args=(i,))
        t.start()
        thread.append(t)for i in range(NUMUSERS):
        thread[i].join()if __name__ == "__main__":print(f"Using connection string: {sample_env.get_connect_string()}")start = time.time()
    start_workload()
    elapsed = time.time() - start
    print("All done!")
    print("Time {:04.2f} seconds".format(elapsed))

As coded, each long_run.py process will run 5 queries with a sleep time of 5 seconds between each query. (In practice, your applications may stay open much longer). Add in a small amount of time for overheads, connecting, and executing statements gives a total expected end-to-end time of bit more than 25 seconds:

$ python3 runner.py
Using connection string: localhost/orclpdb1
User01 sid-ser 407-62786 at time 2024-06-11 20:29:34.632666
User03 sid-ser 26-48119  at time 2024-06-11 20:29:34.802162
User02 sid-ser 172-37948 at time 2024-06-11 20:29:34.973973
User04 sid-ser 272-26116 at time 2024-06-11 20:29:35.146129
User05 sid-ser 398-8361  at time 2024-06-11 20:29:35.315068
User07 sid-ser 33-58745  at time 2024-06-11 20:29:35.485322
User08 sid-ser 154-64891 at time 2024-06-11 20:29:35.667774
User09 sid-ser 268-40439 at time 2024-06-11 20:29:35.837789
User10 sid-ser 408-62081 at time 2024-06-11 20:29:36.013276
User06 sid-ser 10-45069  at time 2024-06-11 20:29:36.189829
User01 sid-ser 407-62786 at time 2024-06-11 20:29:39.653528
User03 sid-ser 26-48119  at time 2024-06-11 20:29:39.821690
User02 sid-ser 172-37948 at time 2024-06-11 20:29:39.990345
User04 sid-ser 272-26116 at time 2024-06-11 20:29:40.162383
User05 sid-ser 398-8361  at time 2024-06-11 20:29:40.330363
User07 sid-ser 33-58745  at time 2024-06-11 20:29:40.503150
User08 sid-ser 154-64891 at time 2024-06-11 20:29:40.686834
User09 sid-ser 268-40439 at time 2024-06-11 20:29:40.854444
User10 sid-ser 408-62081 at time 2024-06-11 20:29:41.029533
User06 sid-ser 10-45069  at time 2024-06-11 20:29:41.206820
User01 sid-ser 407-62786 at time 2024-06-11 20:29:44.666896
User03 sid-ser 26-48119  at time 2024-06-11 20:29:44.835387
User02 sid-ser 172-37948 at time 2024-06-11 20:29:45.005330
User04 sid-ser 272-26116 at time 2024-06-11 20:29:45.177302
User05 sid-ser 398-8361  at time 2024-06-11 20:29:45.343248
User07 sid-ser 33-58745  at time 2024-06-11 20:29:45.520129
User08 sid-ser 154-64891 at time 2024-06-11 20:29:45.704843
User09 sid-ser 268-40439 at time 2024-06-11 20:29:45.868719
User10 sid-ser 408-62081 at time 2024-06-11 20:29:46.046426
User06 sid-ser 10-45069  at time 2024-06-11 20:29:46.225250
User01 sid-ser 407-62786 at time 2024-06-11 20:29:49.679691
User03 sid-ser 26-48119  at time 2024-06-11 20:29:49.848716
User02 sid-ser 172-37948 at time 2024-06-11 20:29:50.021498
User04 sid-ser 272-26116 at time 2024-06-11 20:29:50.191881
User05 sid-ser 398-8361  at time 2024-06-11 20:29:50.359139
User07 sid-ser 33-58745  at time 2024-06-11 20:29:50.542895
User08 sid-ser 154-64891 at time 2024-06-11 20:29:50.715070
User09 sid-ser 268-40439 at time 2024-06-11 20:29:50.880684
User10 sid-ser 408-62081 at time 2024-06-11 20:29:51.062274
User06 sid-ser 10-45069  at time 2024-06-11 20:29:51.245149
User01 sid-ser 407-62786 at time 2024-06-11 20:29:54.694733
User03 sid-ser 26-48119  at time 2024-06-11 20:29:54.866731
User02 sid-ser 172-37948 at time 2024-06-11 20:29:55.038168
User04 sid-ser 272-26116 at time 2024-06-11 20:29:55.206117
User05 sid-ser 398-8361  at time 2024-06-11 20:29:55.372266
User07 sid-ser 33-58745  at time 2024-06-11 20:29:55.560076
User08 sid-ser 154-64891 at time 2024-06-11 20:29:55.730851
User09 sid-ser 268-40439 at time 2024-06-11 20:29:55.899642
User10 sid-ser 408-62081 at time 2024-06-11 20:29:56.078866
User06 sid-ser 10-45069  at time 2024-06-11 20:29:56.260751
All done!
Time 26.85 seconds

The important point is that each of the 10 processes will open a connection and keep it open for the duration the long_run.py app run. This is shown by each user script always executing its queries with the same session identifier/serial number combination. For example User01 always uses 407-62786. This is 10 connections open all the time, each of which has a corresponding server process and session memory consuming database host resources.

Oracle Database 23ai Implicit Connection Pooling

Implicit Connection Pooling is an Oracle Database 23ai feature that makes use of Database Resident Connection Pooling to share database server processes and session memory. It is supported by the popular Oracle Database drivers, including python-oracledb, node-oracledb and JDBC.

Implicit Connection Pooling is suitable for applications that hold onto connections when not doing database work. This includes applications which implement their own connection pool solution holding the underlying connections to the database open for the life of the pool.

Implicit Connection Pooling works by transparently recognizing the boundary when an application isn’t using an open connection. It then lets another application connection use the server process and session memory of the first (currently idle) connection. When the first application subsequently initiates another database request, a free database server process is allocated back to it and the application continues without knowing about the temporary ‘theft’ of the server process. No application changes are needed. The database’s resources are shared, allowing greater scalability.

Implicit Connection Pooling differs from pure DRCP because the mapping and unmapping of database server processes is done implicitly by Oracle. With pure DRCP, the mapping and unmapping only happens when the application initiates get-connection and close-connection calls.

Because long_run.py only has one open/close pair (the close is internally done at the end of the with context), it is an ideal candidate for Implicit Connection Pooling. All that is needed is to start DRCP in the database, change the connection string to use a DRCP pooled server, and additionally set a POOL_BOUNDARY parameter to the connection string. The application code is not modified. Re-running the demo gives the output:

$ python3 runner.py
Using connection string: localhost/orclpdb1:pooled?pool_boundary=statement
User01 sid-ser 399-25678 at time 2024-06-11 20:28:46.803668
User02 sid-ser 399-25678 at time 2024-06-11 20:28:46.830118
User05 sid-ser 399-25678 at time 2024-06-11 20:28:46.840759
User04 sid-ser 399-25678 at time 2024-06-11 20:28:46.867337
User03 sid-ser 152-55977 at time 2024-06-11 20:28:46.908656
User08 sid-ser 152-55977 at time 2024-06-11 20:28:46.978751
User09 sid-ser 152-55977 at time 2024-06-11 20:28:47.045773
User06 sid-ser 152-55977 at time 2024-06-11 20:28:47.106757
User10 sid-ser 152-55977 at time 2024-06-11 20:28:47.169244
User07 sid-ser 152-55977 at time 2024-06-11 20:28:47.229357
User01 sid-ser 152-55977 at time 2024-06-11 20:28:51.844053
User02 sid-ser 399-25678 at time 2024-06-11 20:28:51.844449
User05 sid-ser 152-55977 at time 2024-06-11 20:28:51.855879
User04 sid-ser 152-55977 at time 2024-06-11 20:28:51.883601
User03 sid-ser 152-55977 at time 2024-06-11 20:28:51.940661
User08 sid-ser 152-55977 at time 2024-06-11 20:28:51.990124
User09 sid-ser 152-55977 at time 2024-06-11 20:28:52.058879
User06 sid-ser 152-55977 at time 2024-06-11 20:28:52.120180
User10 sid-ser 152-55977 at time 2024-06-11 20:28:52.182709
User07 sid-ser 152-55977 at time 2024-06-11 20:28:52.243295
User01 sid-ser 152-55977 at time 2024-06-11 20:28:56.874489
User02 sid-ser 399-25678 at time 2024-06-11 20:28:56.874487
User04 sid-ser 399-25678 at time 2024-06-11 20:28:56.895112
User03 sid-ser 399-25678 at time 2024-06-11 20:28:56.954245
User05 sid-ser 277-27334 at time 2024-06-11 20:28:56.969050
User08 sid-ser 277-27334 at time 2024-06-11 20:28:56.997923
User09 sid-ser 277-27334 at time 2024-06-11 20:28:57.072845
User06 sid-ser 277-27334 at time 2024-06-11 20:28:57.131504
User10 sid-ser 277-27334 at time 2024-06-11 20:28:57.195656
User07 sid-ser 277-27334 at time 2024-06-11 20:28:57.257700
User01 sid-ser 399-25678 at time 2024-06-11 20:29:01.903889
User02 sid-ser 277-27334 at time 2024-06-11 20:29:01.903889
User04 sid-ser 152-55977 at time 2024-06-11 20:29:01.910282
User03 sid-ser 152-55977 at time 2024-06-11 20:29:01.968523
User05 sid-ser 152-55977 at time 2024-06-11 20:29:01.997715
User08 sid-ser 152-55977 at time 2024-06-11 20:29:02.011130
User09 sid-ser 152-55977 at time 2024-06-11 20:29:02.086866
User06 sid-ser 152-55977 at time 2024-06-11 20:29:02.139241
User10 sid-ser 152-55977 at time 2024-06-11 20:29:02.209716
User07 sid-ser 152-55977 at time 2024-06-11 20:29:02.271144
User01 sid-ser 152-55977 at time 2024-06-11 20:29:06.940760
User02 sid-ser 277-27334 at time 2024-06-11 20:29:06.940760
User04 sid-ser 399-25678 at time 2024-06-11 20:29:06.940760
User03 sid-ser 399-25678 at time 2024-06-11 20:29:06.982961
User05 sid-ser 399-25678 at time 2024-06-11 20:29:07.010853
User08 sid-ser 399-25678 at time 2024-06-11 20:29:07.023677
User09 sid-ser 399-25678 at time 2024-06-11 20:29:07.102189
User06 sid-ser 399-25678 at time 2024-06-11 20:29:07.152508
User10 sid-ser 399-25678 at time 2024-06-11 20:29:07.223001
User07 sid-ser 399-25678 at time 2024-06-11 20:29:07.284733
All done!
Time 25.96 seconds

This shows server process and session memory being reused across different users’ queries, for example the first two lines have the same session identifier/serial number:

User01 sid-ser 399-25678 at time 2024-06-11 20:28:46.803668
User02 sid-ser 399-25678 at time 2024-06-11 20:28:46.830118

If you crunch the output, you can see that in this run only three servers are used for all queries being executed. The session identifier/serial number pairs are:

152-55977
277-27334
399-25678

This is many fewer than the ten servers needed without Implicit Connection Pooling. Your absolute results will be different, depending on factors like timing.

The total time difference between the two runs is close but in this small test isn’t significant for comparison. In general, because Implicit Connection Pooling shares resources and the Oracle stack has to do some extra work, it’s possible the total time will be slower when moving to Implicit Connection Pooling — if the difference is even accurately measurable. This would depend on how frequently applications execute SQL. However, the benefit is that the database tier uses less memory due to the lower number of server processes, so it may be more efficient — and it certainly can handle more user connections from other applications.

Configuring Implicit Connection Pooling

DRCP provides the configurable pool of database server processes used by Implicit Connection Pooling. DRCP needs to be enabled, monitored, and tuned in the database, see the technical brief Extreme Oracle Database Connection Scalability with Database Resident Connection Pooling.

On the application side, your connection string simply needs to request a DRCP pooled server, e.g. with “:pooled" or "(SERVER=POOLED)". It additionally needs to contain a new POOL_BOUNDARY parameter. The connection string used in this blog post was an Easy Connect string localhost/orclpdb1:pooled?pool_boundary=statement. In a tnsnames.ora file, an equivalent connect descriptor would contain "...(POOL_BOUNDARY=STATEMENT)..."

The POOL_BOUNDARY parameter can have the value STATEMENT or TRANSACTION:

• STATEMENT: a connection is released back to the DRCP connection pool when the connection is implicitly stateless, i.e. when there are no active cursors in the connection (all the rows of the cursors have been internally fetched), no active transactions, no temporary tables, and no temporary LOBs.
• TRANSACTION: a connection is released back to the DRCP connection pool when a commit or rollback is initiated by the app. It is recommended to not enable any driver “autocommit” setting when using Implicit Connection Pooling. If you do, then you will be unable to fetch any data that requires multiple round-trips to the database, such as streaming LOB data.

In line with standard DRCP recommendations for security, you should add a POOL_CONNECTION_CLASS parameter to the connection string, using the same value for all applications that are alike, for example:

localhost/orclpdb1:pooled?pool_boundary=statement&pool_connection_class=myappname

The DRCP “purity” used by Implicit Connection Pooling defaults to SELF, which allows reuse of the server process session memory. Adding the connection string parameter POOL_PURITY=NEW will change this and cause each use of a connection to recreate the session memory.

You can optionally create a PL/SQL package ORA_CPOOL_STATE with procedures to get and reset connection session state, see the Oracle Call Interface documentation.

When not to use Implicit Connection Pooling

If you can restructure your application to use an Oracle-provided driver connection pool, this can be the most efficient outcome. A driver connection pool provides resource sharing, and provides Oracle Database high availability features. Using a driver connection pool (such as the python-oracledb connection pool or node-oracledb connection pool) is often best suited for multi-user applications, but even single user applications that infrequently use connections can benefit from them. The blog post Desktop applications with node-oracledb and electron shows one such scenario. If an application is efficiently using an Oracle application connection pool, and getting and releasing connections frequently, then Implicit Connection Pooling provides no benefit.

If your application can’t be changed to use an Oracle driver connection pool, but the app opens and closes connections frequently, then you may be able to benefit from DRCP directly without enabling Implicit Connection Pooling. See my blog post DRCP helps apps that don’t use a Connection Pool.

At an application level, there are some cases where Implicit Connection Pooling isn’t a good match. As you can see from the example in this post, the session identifier and serial number that a “user” saw for each of their SQL statements varied during the life of that user’s script. So if your app relies on the values not changing, then Implicit Connection Pooling isn’t a good fit.

Another example of a fit that may have issues, is using a statement boundary of TRANSACTION when multiple cursors are in use by the application, or when you are streaming LOBs. In this scenario any application commit can invalidate any open cursors, and also prevent LOB streaming.

You should thoroughly test your application when using Implicit Connection Pooling to ensure that the internal reuse of database servers and sessions does not cause any problems.

Conclusion

Implicit Connection Pooling allows applications that hold connections open for a long time to share their database server processes and session memory. This reduces memory pressure on the database host, and makes the overall system more scalable. Implicit Connection Pooling can be helpful for apps that open single connections, or that implement their own connection pooling solution instead of using an Oracle-provided driver connection pool. It is supported by the popular Oracle Database drivers.

This post discussed how Implicit Connection Pooling makes use of DRCP. It can additionally make use of PRCP, which is available when your system uses Oracle Connection Manager in Traffic Director Mode.

For a video on Implicit Connection Pooling see Effortless Connection Management with Implicit Pooling in Oracle Database 23c by my colleague Sharad Chandran R.

References

• Video: Effortless Connection Management with Implicit Pooling in Oracle Database 23c (video).
• Python-oracledb Documentation: Implicit Connection Pooling.
• Node-oracledb documentation: Implicit Connection Pooling.
• Blog post: Always Use Connection Pools — and How.
• Blog post: DRCP helps apps that don’t use a Connection Pool
• Technical Brief: Extreme Oracle Database Connection Scalability with Database Resident Connection Pooling (DRCP).
• Technical Brief: CMAN-TDM — An Oracle Database connection proxy for scalable and highly available applications.

18 June 2024 - Post source at 🐙

Updating (PHP) packages to ReactPHP Promise v3, and test your types with PHPStan

With version 3, react/promise adds template types to communicate what kind of data the promise will hold once resolved. Add react/async 4.2 with https://github.com/reactphp/async/pull/40 to that and return type hints are a thing and both PHPStan and Psalm will understand them.

Photo by Kindel Media

These past few weeks I’ve been working on upgrading all my PHP packages to 8.2+ and the update for ReactPHP packages also will include Promise v3. This means that if you want to use any of my packages with Promise v3, you’ll need to be on PHP 8.2. (With sponsorships that number can be lowered, but only to a certain point.)

During the work on https://github.com/reactphp/promise/pull/247 it became apparent that we needed a way to validate our expected return types. PHPStan provides a way to do that with the PHPStan\Testing\assertType.

Let’s assume we have a function named futurePromise that returns a promise which resolves when the event loop has done another tick. The implementation is pretty simple:

/**
 * Promise that resolves once future tick is called.
 *
 * @param mixed $value Value to return on resolve.
 */
function futurePromise($value = null): PromiseInterface
{
    $deferred = new Deferred();
    Loop::futureTick(static function () use ($deferred, $value): void {
        $deferred->resolve($value);
    });
    return $deferred->promise();
}

This is a pretty straightforward implementation: * Created deferred * Schedule tick * Return promise * One tick happens resolve the promise with the value passed into the function

To make it communicate the type the promise will hold, it had to be changed to:

/**
 * Promise that resolves once future tick is called.
 * 
 * @param T $value Value to return on resolve.
 *
 * @return PromiseInterface<T>
 *
 * @template T
 * @phpstan-ignore-next-line
 */
function futurePromise(mixed $value = null): PromiseInterface
{
    /** @var Deferred<T> $deferred */
    $deferred = new Deferred();
    Loop::futureTick(static function () use ($deferred, $value): void {
        $deferred->resolve($value);
    });
    return $deferred->promise();
}

And we feed PHPStan these two tests for type safety.

<?php
declare(strict_types=1);
use function PHPStan\Testing\assertType;
use function WyriHaximus\React\futurePromise;
assertType('React\Promise\PromiseInterface<bool>', futurePromise(true));
assertType('React\Promise\PromiseInterface<null>', futurePromise());

In total that is 4 changed lines and one new file in this package (ignoring the PHPStan ignore there for the argument with default value). To ensure PHPStan (and Psalm) understand whatever you put into the first argument, comes out through the promise. It works by telling PHPStan that $value is of type T, and that we return it by wrapping it into a promise<T>. There is a lot more happening behind the scenes, which is why we had to point out that $deferred is Deferred<T>.

Ultimately, we want to get to this, but for now promise typing is a major first step:

<?php
declare(strict_types=1);
use function PHPStan\Testing\assertType;
use function WyriHaximus\React\futurePromise;
assertType('bool', futurePromise(true));
assertType('null', futurePromise());

This will be done by using await() inside futurePromise in this case. Or as close as possible to the promise returned. But for this post we’re focusing on adding it to a package.

While the removal of done() and erroring on handled rejections might be the most impactful changes in how react/promise affects users directly. The added type safety's value isn’t to underestimated. PHPStan’s assertType function is incredible valuable to test and proof your functions/methods return types. I strongly encourage all package owners out there to add these tests if you use @template types in your packages.

With version 3, react/promise adds template types to communicate what kind of data the promise will hold once resolved. Add react/async 4.2 with https://github.com/reactphp/async/pull/40 to that and return type hints are a thing and both PHPStan and Psalm will understand them. Photo by Kindel Media

Bildgröße auf xmpp-files.cweiske.de prüfen

which translates to

Checking image size on xmpp-files.cweiske.de

The other XMPP client Dino showed the images, though.

In Conversations bug report #240 it was observed that the Content-Length header was missing, and my server exhibited the same problem:

$ curl -I 'https://xmpp-files.cweiske.de/share_v2.php/23/42.jpg
HTTP/1.1 200 OK
Date: Sat, 08 Jun 2024 12:38:47 GMT
Server: Apache/2.4.59 (Debian)
Access-Control-Allow-Methods: GET, PUT, OPTIONS
Access-Control-Allow-Headers: Content-Type
Access-Control-Max-Age: 7200
Access-Control-Allow-Origin: *
Content-Security-Policy: "default-src 'none'"
X-Content-Security-Policy: "default-src 'none'"
X-WebKit-CSP: "default-src 'none'"
Content-Type: image/jpeg

No Content-Length. I'm using the mod_http_upload_external Prosody module for file uploads together with the share_v2.php provided by it. That PHP script does set a Content-Length header, but nobody receives it!

Even a PHP script that only sends out a Content-Length header does not work:

<?php
header('X-Test: 23');
header('Content-Length: 42');

$ curl -I https://xmpp-files.cweiske.de/test.php
HTTP/1.1 200 OK
Date: Sat, 08 Jun 2024 13:18:34 GMT
Server: Apache/2.4.59 (Debian)
X-Test: 23
Content-Type: text/html; charset=UTF-8

The header is missing.

The cause​

Then I found Apache bug report #68973: Content-Length header missing in 2.4.59 is a breaking change which explained the symptom I experienced:

Apache version 2.4.59 fixed security issue CVE-2024-24795 by preventing CGI-like scripts (such as PHP) from sending out Content-Length headers.

A new environment variable ap_trust_cgilike_cl was introduced that restores to the old behavior.

Solution​

I re-enabled the Content-Length header in my PHP applications by creating an apache configuration file

SetEnv ap_trust_cgilike_cl 1

enabling it and restarting apache2:

$ a2enconf cweiske-content-length
$ systemctl reload apache2

Besides packaging long-term support versions of PHP, we also are publishing a product called ZendHQ. This is a combination of a PHP extension, and an independent service that PHP instances communicate with to do things like monitoring and queue management.

It's this latter I want to talk about a bit here, as (a) I think it's a really excellent tool, and (b) in using it, I've found some interesting patterns for prepping it during deployment.

What does it do?

ZendHQ's JobQueue feature provides the ability to defer work, schedule it to process at a future date and time, and to schedule recurring work. Jobs themselves can be either command-line processes, or webhooks that JobQueue will call when the job runs.

Why would you use this over, say, a custom queue runner managed by supervisord, or a tool like Beanstalk, or cronjobs?

There's a few reasons:

• Queue management and insight. Most of these tools do not provide any way to inspect what jobs are queued, running, or complete, or even if they failed. You can add those features, but they're not built in.
• If you are using monitoring tools with PHP... queue workers used with these tools generally cannot be monitored. If I run my jobs as web jobs, these can run within the same cluster and communicate to the same ZendHQ instance, giving me monitoring and code traces for free.
• Speaking of using web workers, this means I can also re-use technologies that are stable and provide worker management that I already know: php-fpm and mod_php. This is less to learn, and something I already have running.
• Retries. JobQueue allows you to configure the ability to retry a job, and how long to wait between retries. A lot of jobs, particularly if they rely on other web services, will have transient failures, and being able to retry can make them far more reliable.

So, what about queue warmup?

When using recurring jobs, you'll (a) want to ensure your queue is defined, and (b) define any recurring jobs at application deployment. You don't want to be checking on each and every request to see if the queues are present, or if the recurring jobs are present. Ideally, this should only happen on application deployment.

When deploying my applications, I generally have some startup scripts I fire off. Assuming that the PHP CLI is configured with the ZendHQ extension and can reach the ZendHQ instance, these scripts can (a) check for and create queues, and (b) check for and create recurring jobs.

As a quick example:

use ZendHQ\JobQueue\HTTPJob;
use ZendHQ\JobQueue\JobOptions;
use ZendHQ\JobQueue\JobQueue;
use ZendHQ\JobQueue\Queue;
use ZendHQ\JobQueue\QueueDefinition;
use ZendHQ\JobQueue\RecurringSchedule;
$jq = new ZendHQ\JobQueue();
// Lazily create the queue "mastodon"
$queue = $jq->hasQueue('mastodon')
    ? $jq->getQueue('mastodon')
    ? $jq->addQueue('mastodon', new QueueDefinition(
        QueueDefinition::PRIORITY_NORMAL,
        new JobOptions(
            JobOptions::PRIORITY_NORMAL,
            60, // timeout
            3, // allowed retries
            30, // retry wait time
            JobOptions::PERSIST_OUTPUT_ERROR,
            false, // validate SSL
    ));
// Look for jobs named "timeline"
$jobs = $queue->getJobsByName('timeline');
if (count($jobs) === 0) {
    // Job does not exist; create it
    $job = new HTTPJob('http://worker/mastodon/timeline', HTTPJob::HTTP_METHOD_POST);
    $job->setName('timeline');
    $job->addHeader('Content-Type', 'application/my-site-jq+json');
    $job->setRawBody(json_encode([
        'type' => MyApp\Mastodon\Timeline::class,
        'data' => [ /* ... */ ],
    ]);
    // Schedule to run every 15 minutes
    $queue->scheduleJob($job, new RecurringSchedule('* */15 * * * *'));
}

That's literally it.

The takeaway points:

• You can check for an existing queue and use it, and only define it if it's not there. You could also decide to suspend the queue and delete it before creating it, if you know that the existing jobs will not run with the current deployment.
• If you give a job a name (you don't actually have to, but it helps you identify related jobs far easier if you do), you can search for it. In the example above, if I find any jobs with that name, I know it's already setup, and I can skip the step of scheduling the job.

Running one-off jobs on deployment

Something else I also like to do is run one-off tasks at deployment. Often these are related to recurring tasks, and I might want to fetch the content at initialization rather than waiting for the schedule. In other cases, I might want to do things like reset caches.

Because these scripts run before deployment, which might mean restarting the web server, or, more often, waiting for the php-fpm container and/or web server to be healthy, I cannot run the jobs immediately, because there's nothing to answer them.

The answer to this is to queue a job in the future:

use DateTimeImmutable;
use ZendHQ\JobQueue\ScheduledTime;
$queue->scheduleJob($job, new ScheduledTime(new DateTimeImmutable('+1 minute')));

(I find it usually takes less than a minute for my FPM pool and/or web server to be online after running these scripts.)

The beauty of this approach is that my bootstrapping scripts now tend to be very fast, as I'm not trying to do all of this stuff before launching the site updates. The jobs then execute very soon after the site is up, and there's no noticeable differences in content or behavior.

Closing notes

I know I'm biased around ZendHQ. I'm also generally one of my own biggest critics. I had my team re-implement a lot of features that were present in Zend Server that I was never terribly keen on, and was hugely worried that we were going to make some of the same mistakes I felt we'd made with that product. However, the end result has been something that I am delighted to use, and which has opened up a ton of possibilities for how I build sites. The ability to warm my queues and manage them just like the rest of my PHP application is hugely powerful. I'm looking forward to seeing what others build with it!

I used to make these subtitles with an online service called Otter.io, but they stopped allowing uploading of video files.

And then I found Whisper, which allows me to upload audio files to create subtitles. Whisper is an API from OpenAI, mainly known for ChatGPT.

I didn't like having to upload everything to them either, as that means that they could train their model with my original video audio.

Whisper never really worked that well, because it broke up the sentences in weird places, and I had to make lots of edits. It look a long time to make subtitles.

I recently found out that it's actually possible to run Whisper locally, with an open source project on GitHub. I started looking into this to see whether I could use this to create subtitles instead.

The first thing that their documentation tells you to do is to run: pip install openai-whisper.

But I am on a Debian machine, and here Python is installed through distribution packages, and I don't really want to mess that up. apt-get actually suggests to create a virtual environment for Python.

In a virtual environment, you can install packages without affecting your system setup. Once you've made this virtual environment, there's actually Python binaries symlinked in there, that you can then use for installing things.

You create the virtual environment with:

python3 -m venv `pwd`/whisper-local
cd whisper-local

In the bin directory you then have python and pip. That's the one you then use for installing packages.

Now let me run pip again, with the same options as before to install Whisper:

bin/pip install -U openai-whisper

It takes quite some time to download. Once it is done, there is a new whisper binary in our bin directory.

You also need to install fmpeg:

sudo apt-get install ffmpeg

Now we can run Whisper on a video I had made earlier:

./bin/whisper ~/media/movie/xdebug33-from-exception.webm

The first time I ran this, I had some errors.

My video card does not have enough memory (2GB only). I don't actually have a very good video card at all, and was better off disabling it, by instructing "Torch" that I do not have one:

export CUDA_VISIBLE_DEVICES=""

And then run Whisper again:

./bin/whisper ~/media/movie/xdebug33-from-exception.webm

It first detects the language, which you can pre-empt by using --language English.

While it runs, it starts showing information in the console. I quickly noticed it was misspelling lots of things, such as my name Derick as Derek, and Xdebug as XDbook.

I also noticed that it starts breaking up sentences in a odd way after a while. Just like what the online version was doing.

I did not get a good result this first time.

It did create a JSON file, xdebug33-from-exception.json, but it is all in one line.

I reformatted it by installing the yajl-tools package with apt-get, and flowing the data through json_reformat:

sudo apt-get install yajl-tools
cat xdebug33-from-exception.json | json_reformat >xdebug33-from-exception_reformat.json

The reformatted file still has our full text in a line, but then a segments section follows, which looks like:

"segments": [
    {
        "id": 0,
        "seek": 0,
        "start": 3.6400000000000006,
        "end": 11.8,
        "text": " Hi, I'm Derick. For most of the videos that I make, I also like to have subtitles, because",
        "tokens": [
                            50363, 15902, 11, 314, 1101, 9626, 624, 13, 1114, 749, 286,
                            262, 5861, 326, 314, 787, 11, 314, 635, 588, 284, 423, 44344,
                            11, 780, 50960
        ],
        "temperature": 0.0,
        "avg_logprob": -0.20771383965152435,
        "compression_ratio": 1.5128205128205128,
        "no_speech_prob": 0.31353551149368286,
            },

Each segment has an id, a start and end time (in seconds), the text for that segment, and a bunch of auxiliary information.

But it is still sort of a sentence at a time, which isn't really what we want. Additionally, I really don't want it to say XDbook and misspell my name.

For both of those there are actually options that we can use.

To make things easier for us later, we're turning on word timestamps. That allows us to by word see where the time index actually was. You do that with the --word_timestamps True option.

To provide a hint on what the video is about, and to get better words, we can use the initial prompt. The option that I used for this video was: --initial_prompt="This video by Derick introduces a new Xdebug feature, regarding exceptions".

To make things more accurate or less accurate, there's also an option that you can specify which is which model to use. Normally the standard one medium is fine, but in order to speed up generation, you can use --model tiny. This will give less accurate results.

If the model has not been downloaded the before, Whisper will automatically do this.

It is also possible to specify the language, which makes things go faster if it's English only as well: --language English. It supports a bunch of languages.

The full command that I used is:

CUDA_VISIBLE_DEVICES="" \
        ./bin/whisper ~/media/movie/xdebug33-from-exception.webm \
        --word_timestamps True \
        --initial_prompt="This video by Derick introduces a new Xdebug feature, regarding exceptions" \
        --language English

In the output file, we now see another elements in each sentence section:

"words": [
    {
        "word": " Hi,",
        "start": 3.6400000000000006,
        "end": 4.16,
        "probability": 0.6476245522499084
    },
    {
        "word": " I'm",
        "start": 4.28,
        "end": 4.44,
        "probability": 0.9475358724594116
    },
    {
        "word": " Derick.",
        "start": 4.44,
        "end": 4.8,
        "probability": 0.12672505341470242

For each word, it has the start and end, as well as the probability of it being correct. You see that for Derick it was only 13% certain.

With this information you can do some analysis to create an actual subtitle script out of this. For that I have written a PHP script.

It loops over all the segments, and for each of the segments over all the words. If the difference in time between the end of a word and the start of a new word is more than a quarter of a second, it emits a new section of subtitles.

Similarly if a sentence is longer than 60 characters it also breaks it up into an extra line. This keeps all the subtitles reasonably well sorted.

The emit function formats it like how the SRT files are supposed to be.

With this script, I now create a new SRT file, overwriting the one that Whisper had created:

php whisper-to-srt.php xdebug33-from-exception.json > xdebug33-from-exception.rst

The output looks like:

0
00:00:03,640 --> 00:00:04,799
Hi, I'm Derick.
1
00:00:05,919 --> 00:00:10,980
For most of the videos that I make, I also like to have subtitles,
2
00:00:11,640 --> 00:00:15,779
because sometimes it's easier to just read along for various
different reasons.

These subtitles I can now add when uploading a video to YouTube. I might also create a similar script to generate output that can be used as the base for an textual article. Not everybody likes learning from watching videos.

I would also prefer not to use a model that has been trained with questionable sources. I will be investigating if I can use Mozilla's Common Voice project's data instead.

In the last months I have not spent a lot of time on Xdebug due to a set of other commitments.

Since my last update in November a few things have happened though.

Truncated by Planet PHP, read more at the original (another 1644 bytes)

python-oracledb 2.2, the extremely popular Oracle Database interface for Python, is now on PyPI.

Today we’re pleased to release python-oracledb 2.2 to coincide with the just-announced, general availability of Oracle Database 23ai.

Python-oracledb is an open source package for the Python Database API specification with many additions to support advanced Oracle Database features. It is the new name for the cx_Oracle driver.

Main Changes for Oracle Database 23ai

Support for the Oracle Database 23ai VECTOR data type

A major feature of Oracle Database 23ai is AI Vector Search (which is part of the database and is available at no additional charge in Enterprise Edition, Standard Edition 2, Database Free, and all Oracle Database cloud services).

Python-oracle supports this feature with its native capabilities for binding and fetching the new VECTOR data type. For example, given the table:

SQL> drop table if exists mytab_v; 
SQL> create table mytab_v (v64 vector(3, float64));

Python code to insert a vector could look like:

vector_data_64 = array.array("d", [11.25, 11.75, 11.5])cursor.execute(
    "insert into mytab_v (v64) values (:1)",
    [vector_data_64]
)

A query:

cursor.execute("select * from mytab_v")
for row in cursor:
    print(row)

would then show:

(array('d', [11.25, 11.75, 11.5]),)

Other examples are on GitHub.

(And did you note the new DROP TABLE IF EXISTS syntax I used?!

Support for the Oracle Database 23ai BOOLEAN data type

Oracle Database 23ai’s new SQL BOOLEAN data type lets you store true and false values in database tables. (Previously this type was only available in PL/SQL).

Python-oracledb 2.2 supports binding and fetching SQL BOOLEAN values. For example:

bv = cursor.var(oracledb.DB_TYPE_BOOLEAN)
bv.setvalue(0, True)
for r, in cursor.execute("select :bv", [bv]):
    print(r)

displays:

True

Note there is no FROM DUAL in the query! One Oracle Database 23ai enhancement was to make the clause optional in this common query idiom.

Support for the Oracle Database INTERVAL YEAR TO MONTH data type

The not-so-common INTERVAL YEAR TO MONTH data type is not new to the database, but python-oracledb 2.2 now has support via anoracledb.IntervalYM class. The class is a named tuple with two integer attributes, years and months. The code:

bv = cursor.var(oracledb.DB_TYPE_INTERVAL_YM)
bv.setvalue(0, oracledb.IntervalYM(years=3, months=10))
cursor.execute("insert into mytab_b (iym_col) values (:1)", [bv])for r, in cursor.execute("select * from mytab_b"):
    print(r)

displays:

IntervalYM(years=3, months=10)

Oracle Database 23ai JSON and SODA improvements

One big Oracle Database 23ai announcement is JSON Relational Duality, letting you leverage the power of relational and the simplicity of JSON development within a single app. Also check out this blog post.

Various enhancements to python-oracledb improve its support for JSON and SODA features in Oracle Database 23ai.

Oracle Database 23ai Implicit Connection Pooling

Implicit connection pooling is a great feature that leverages Oracle Database’s popular DRCP pooling. The intended use case is for applications that are not using python-oracledb connection pooling, but have become popular and are now causing excess connection load on the database server. Typically legacy applications can’t be modified to explicitly use application connection pooling calls. However they can make use of implicit connection pooling simply by starting DRCP in the database and then using a new connection string option POOL_BOUNDARY. The driver then automatically associates and disassociates DRCP pooled servers from application connections as they are actually used by the application to do database work. This can occur transparently between the application's explicit oracledb.connect() call and the connection.close() (or equivalent application's connection release at end-of-scope). By allowing each database pooled server to be used by multiple application connections, the load on the database host is reduced.

Implicit connection pooling is available in both python-oracledb Thin and Thick modes. It works with DRCP and also with CMAN-TDM’s Proxy Connection Pooling (PRCP). CMAN-TDM is Oracle Database’s mid-tier proxy solution.

As an example, to use implicit connection pooling with DRCP, your python-oracledb connection string might look like:

dsn = "localhost:1521/orclpdb:pooled?pool_boundary=statement"

Oracle Database 23ai Improved Connections

During the Oracle Database 23ai development cycle there were improvements to the database stack, and also in the Oracle Cloud networking infrastructure, that help connection performance in various scenarios. Some changes reduce the time to connect to the database, while others improve the transport across established connections. Oracle Database’s language drivers benefit in different ways. I can’t point to a specific “tell all” benchmark because there are lots of different connection configurations (such as the speed of your network, whether dedicated or pre-spawned server process are being used, whether encryption is being used, what language stack is used, are you in the cloud etc., etc.) but overall we’re very happy with the improvements.

Some of the specific changes in the python-oracledb driver include:

• Internal optimization in Python-oracledb Thin mode (and Thick mode when using Oracle Client 23ai libraries) to reduce overheads and connection establishment times when connecting to Oracle Database 23ai.
• Internal support for an Oracle Database 23ai network protocol improvement that directly benefits python-oracledb’s asyncio (and node-oracledb’s) performance when fetching large columns. This lets python-oracledb 2.2 more efficiently handle internal packet management.

Oracle Database 23ai SQL Annotations

The python-oracledb query metadata attribute FetchInfo.annotation returns a dictionary containing the database annotations associated with a fetched column. Annotations provide a mechanism to store application metadata centrally in the database so that they can be shared across applications, modules and microservices.

Oracle Database 23ai SQL Domains

The python-oracledb query metadata attributes FetchInfo.domain_schema and FetchInfo.domain_name support data use case domain information associated with a fetched column.

A use case domain is high-level dictionary object that belongs to a schema and encapsulates a set of optional properties and constraints. You can define table columns to be associated with a domain, thereby explicitly applying the domain’s optional properties and constraints to the columns.

Oracle Database Error URLs

One recent database group project was a review and rewording of common database error messages. An error portal now contains database messages with their Cause and Action help information. Python-oracledb will show a link to the relevant portal entry when printing database error messages. By following the link, further troubleshooting information is available to you. For example:

$ python t.py ORA-01017: invalid credential or not authorized; logon denied 
Help: https://docs.oracle.com/error-help/db/ora-01017/

These help links are displayed in Thin and Thick modes. With Thick mode, Oracle Client 23ai libraries are needed.

Summary

See the python-oracledb Release Notes for more improvements and bug fixes. Some recent versions had limited support for pre-release Oracle Database 23ai and the Oracle Database 23.3 Free edition features, but we strongly encourage you to upgrade to python-oracledb 2.2 if you are using any edition of Oracle Database 23.4 because of recent internal changes.

Don’t forget there are many great Oracle Database 23ai features that are part of the database itself, or are usable through SQL and PL/SQL — or are in other tools. See the Oracle Database New Features guide.

For examples, did you know SQL*Plus 23ai has a new PING command and a matching command line -P option to ping the database? This removes the need to install tnsping to check database availability. Did you know that you can use JavaScript for Oracle Database stored procedures? Did you know that our C language Oracle Call Interface API now supports database request pipelining? Database pipelining is something we'd like to expose in a future python-oracle version, so stay tuned to our release announcements!

Finally, if you are taking advantage of Oracle Database 23ai Free in Oracle Cloud, then our documentation Connecting to Oracle Cloud Autonomous Databases will be useful.

Installing or Upgrading python-oracledb

You can install or upgrade python-oracledb by running:

python -m pip install oracledb --upgrade

The pip options --proxy and --user may be useful in some environments. See python-oracledb Installation for details.

Python-oracledb References

Home page: oracle.github.io/python-oracledb/index.html

Installation instructions: python-oracledb.readthedocs.io/en/latest/installation.html

Documentation: python-oracledb.readthedocs.io/en/latest/index.html

Release Notes: python-oracledb.readthedocs.io/en/latest/release_notes.html

Discussions: github.com/oracle/python-oracledb/discussions

Issues: github.com/oracle/python-oracledb/issues

Source Code Repository: github.com/oracle/python-oracledb