How we use hyperfine to measure PHP Engine performance

One of our recurring jobs at Tideways is to ensure that all of our instrumentation works with new and upcoming PHP versions. For us, “working” doesn’t just mean that the results are correct, but that your PHP extension is fast, gathering insights for our customers with a minimal performance overhead.

While we have a comprehensive automated test suite, especially for new PHP features or library updates, sometimes we manually investigate – and one of our main tools for this job is hyperfine.

What is hyperfine

Hyperfine is a command-line benchmarking tool that takes care of handling the challenging parts of performance measurement correctly. By presenting execution times in a reliable, consistent and human-readable way, it ensures clear communication of the results. Beneath the surface, hyperfine also takes care of all the minute details that pose a challenge during benchmarking, thereby making it easier to get good and correct results.

All you need to do is pass it different command invocations, and it will compare them using multiple executions to get reliable data..

Let’s start with a simple example. We run the sleep command for one, two and three seconds.

The output shows us that “sleeping for 1 second” was 1.98 ± 0.01 times faster than “sleeping for 2 seconds”. One of the things to note is that hyperfine always shows you the measurement uncertainty.

With sleep this uncertainty is very low, showing 1.020 s ± 0.006 s, meaning the execution is confidently measured to take 1.014 to 1.0026 seconds. Other programs can have a lot more variance and therefore uncertainty.

For example, when comparing 603.9 ms ± 170.8 ms and 502.1 ms ± 400.8 ms we can’t make any statement regarding what is actually faster, as the errors overlap too much. Being mindful of this can be very useful to focus on meaningful improvements.

Measuring PHP performance

Now, how does that work for PHP?

We test two main things: The same PHP code with different PHP binaries, for example different versions or build flags, and different PHP code with the same binary.

Let’s look at PHP 8.4’s sprintf optimization as an example.

In the following test, we run a million sprintf calls using PHP 8.3 and PHP 8.4:

hyperfine '/opt/php-8.3/sapi/cli/php bench-sprintf.php' \
     '/opt/php-8.4/sapi/cli/php bench-sprintf.php'

In cases where the command gets quite long, for example, when adding ini settings with -d we can also use Hyperfine’s placeholder parameter.

hyperfine -L version 8.3,8.4 \
  '/opt/php-{version}/sapi/cli/php bench-sprintf.php -d opcache.enable_cli=1'

Testing on the CLI: Be mindful of OPcache

By default, OPcache isn’t enabled for CLI scripts. This is done to avoid the cost of the optimization for one-time scripts. For our case, we want to use OPcache as this will provide the relevant insights for web-requests or long-running scripts:

hyperfine '/opt/php-8.3/sapi/cli/php -d opcache.enable_cli=1 bench-sprintf.php'  \
   '/opt/php-8.4/sapi/cli/php -d opcache.enable_cli=1 bench-sprintf.php'

Here we see PHP 8.4 executing 1.47 times faster, already an impressive difference, but we’re also still paying the cost of the function calls, which also adds some overhead.

So let’s isolate the change to just the sprintf call.

And see how the numbers change:

hyperfine -L version 8.3,8.4  \
   '/opt/php-{version}/sapi/cli/php -d opcache.enable_cli=1 \
   bench-minimal.php'

Over six times faster! Measuring things and making definitive statements is always tricky, and it’s easy to get things wrong. If we, for instance, forget to enable OPcache, the example above only shows a 1.8x difference between PHP 8.3 and 8.4 instead of the over 5x we’re seeing here.

When testing with PHP versions before PHP 8.5, it’s also possible that the OPcache extension isn’t loaded and PHP will silently ignore the OPcache ini settings, invalidating the results. Double-checking with php -m | grep -i opcache or with some code in the test script can prevent that issue.

This happens because of our simple test case. Now that PHP 8.4 can turn \sprintf into an interpolated string containing only literals, OPcache will recognize that this string isn’t used and optimize that out. Using a couple of variables as parameters or assigning the result to something will change this behavior. Try it out yourself! A lot of the value in benchmarking comes from being surprised and then learning something while trying to explain the results.

Benchmarking code chances

Now let’s make some comparisons within one PHP script:

hyperfine -L mode with_backslash,without_backslash  \
   '/opt/php-8.4/sapi/cli/php -d opcache.enable_cli=1 bench.php {mode}'

Over 1.3 times faster by adding a single character, or using the equivalent use function sprintf import.

Here, we can nicely see that the optimization only takes effect when PHP can see that sprintf is the global sprintf. Without the \ PHP needs to look up if a locally defined \bench\sprintf namespaced function exists, and thus PHP can’t optimize the call for you.

Note that this specific optimization is not performed by OPcache, but rather happens in the compiler itself. Other types of optimization are specific to OPcache though. Hence, we suggest you always turn it on to make sure your tests match how you will later run your code.

Outro

We hope this serves as a nice overview of benchmarking with hyperfine and showcases some key considerations to keep in mind when testing PHP.