How we use hyperfine to measure PHP Engine performance

One of our recurring jobs at Tideways is to ensure that all of our instrumentation works with the new and upcoming PHP versions. For us, “working” doesn’t just mean that the results are correct, but that your PHP extension is fast, gathering insights for our customers with a minimal performance overhead.

While we have a comprehensive automated test suite, especially for new PHP features or library updates, sometimes we manually investigate — and one of our main tools for this job is hyperfine.

Hyperfine is a command line benchmarking tool that takes care of handling the challenging parts of performance measurement correctly. By presenting command runtime output in a consistent, reliable and human-readable way to ensure clear communication of the results. Beneath the surface, it also takes care of all the minute details that pose a challenge during benchmarking, thereby making it easier to get good and correct results.

And all you need to do is pass it different command invocations, and it will compare them:

Now, how does that work for PHP?

We test two main things: The same PHP code with different PHP binaries and different PHP code with the same binary.

Let’s look at PHP 8.4s sprintf optimization as an example!

We run the following test script with two PHP versions 8.3 and the latest 8.4 beta:

We could now simply compare the two versions by running the following command:

hyperfine 'php-8.3/sapi/cli/php bench-sprintf.php' 'php-8.4/sapi/cli/php bench-sprintf.php'

Let’s first make use of hyperfine’s placeholder parameter to make this a bit nicer:

hyperfine -L version 8.3,8.4 'php-{version}/sapi/cli/php bench-sprintf.php'

Using placeholders makes it easy to see what we’re comparing and keep the command length short.

OPcache and CLI

By default, OPcache isn’t enabled for CLI scripts. This is done to avoid the cost of the optimization for one-time scripts. For our case, we want to use OPcache as this will provide the relevant insights for web-requests or long-running scripts:

Here we see PHP 8.4 executing 1.34 times faster, already an impressive difference, but we’re also still paying the cost of the function calls which also adds some overhead.

So let’s isolate the change and see how the numbers change:

Over six times faster! A good reminder that measuring things and making definitive statements is always tricky, and it’s easy to get things wrong. If we, for instance, forget to enable OPcache, the example above only shows a 1.8x difference between PHP 8.3 and 8.4 instead of the over 6x we’re seeing here.

This happens because of our very simple test case. Now that PHP 8.4 can turn \sprintf into an interpolated string containing only literals, OPcache will recognize that this string isn’t used and optimize that out. Using a couple of variables as parameters or assigning the result to something will change this behavior. Try it out yourself! A lot of the value in benchmarking comes from being surprised and then learning something while trying to explain the results.

Same PHP version

Now let’s make some comparisons within one PHP script:

Here, we can nicely see that the optimization only takes effect when PHP can see that sprintf is the global sprintf. Without the \ it might be defined locally in the bench namespace and thus PHP can’t optimize the call for you.

Note that this specific optimization is not performed by OPcache, but rather happens in the compiler itself. Other types of optimization are specific to OPcache, though.

Outro

We hope this serves as a nice overview of benchmarking with hyperfine and showcases some key considerations to keep in mind when testing PHP.

Related Posts

• New in PHP 8.4: engine optimization of sprintf() to string interpolation
• PHP 8.4 Property Hooks: Can we get rid of getters/setters now? A benchmark.