Debugging a running PHP process by attaching GDB

We are noticing that some of our requests are starting to get slow and server load increases. Checking the process list of our server, for example with htop reveals that our FPM workers are taking up all of our CPU time.

Checking the health with our basic toolset of lsof to show open network connections and strace to show syscalls does not reveal any activity. This means that the workers are spending time processing data without any externally visible activity. This could be an infinite loop or just an inefficient algorithm.

Unfortunately, we were never able to reproduce the issue ourselves, so our regular PHP debugging toolkit in our development setup doesn’t work. How can we get additional information from these processes when we catch them red-handed?

Using a system-level debugger, specifically gdb, comes to the rescue. On common Linux distributions, it is available within the aptly named package gdb. We’re using Debian Trixie for this post but have made available instructions for other distributions at the bottom.

Let’s start by installing gdb:

Besides gdb, we also need debug symbols for PHP. These debug symbols allow gdb to map the processor instructions back to the statements and variables in the C source code of PHP. Since we are using Debian Trixie’s native PHP 8.4 packages, these debug symbols are available with debuginfod, which we enable by setting an environment variable:

Setup is now finished, and we can launch gdb using the gdb command. We need the PID of one of our stuck workers, 7154 in this case, and then use attach $pid (thus attach 7154) to attach to the worker. Note that the worker will be completely frozen while we look at it in gdb; thus, the request that is currently processed might run into timeouts depending on how long the worker is stopped with gdb. Use detach after extracting all the relevant information to let it continue running from where it stopped.

The bt command allows you to retrieve a C-level backtrace, which can already provide some insight.

We are stuck in ZEND_ADD_SPEC_TMPVARCV_TMPVARCV_HANDLER from zend_vm_execute.h, meaning we are adding two values in userland code, but additions can happen everywhere, which is not particularly useful to find out what is going wrong. It would be nice if we could find out which line of PHP code is currently being executed. Ideally with a full PHP-level stacktrace.

Turns out this is possible! PHP provides an official gdb script that provides helpful commands to retrieve information from PHP. It’s available as .gdbinit in the root of the php/php-src repository. We’ll need to make sure to select the right branch, PHP-8.4 in our case: https://github.com/php/php-src/blob/PHP-8.4/.gdbinit

After placing it somewhere convenient on the server, we load it into gdb using the source command:

Now the zbacktrace function will be available, which will output a stacktrace containing the names of the PHP functions rather than the C code executed internally.

Oh, that is nice. Apparently we are stuck evaluating a deep tree of Fibonacci numbers. Let’s check what data they are working on. We can print all local variables (internally called “CV”) using the print_cvs command, which also was provided by the .gdbinit script. We’ll need to pass the ID of the stack frame we’re interested in, FibonacciService->__invoke(33) in this case:

$n is 33 (which makes sense). It already calculated $n1 which is 2178309, the 32nd Fibonacci number, $n2 (the 31st) is not yet available, which also makes sense, since the function would be finished already otherwise – except for the addition, which should be so fast that it is challenging to catch reliably.

Now we know why the worker is using so much CPU, but we still don’t know what data was included in the request (e.g., $_GET parameters) to reach this situation. This information is available from the superglobals. To access them, we first need to obtain a list of all global variables using the print_global_vars command:

Afterwards, we can print out an individual variable using the printzv command with the variable pointer as the parameter ($_GET in this case):

Okay, our script was accessed with ?n=45 as the query string. Testing this in our development setup confirms it hangs, which means we can now start debugging and optimizing it!

Appendix

Debian Native Packages

The Debian Wiki explains how to get a backtrace. The easiest solution is by enabling debuginfod, as we have done above:

export DEBUGINFOD_URLS="<https://debuginfod.debian.net>"

Alternatively the main/debug APT archive needs to be enabled and the corresponding -dbgsym package for each PHP package needs to be installed. Most importantly the phpX.Y-cli-dbgsym or phpX.Y-fpm-dbgsym package is required, since it provides the debug symbols for the Zend Engine.

Debian with Sury’s PPA

The -dbgsym packages need to be installed as explained in the section for Debian Native Packages. No changes to the sources list are required, since the -dbgsym packages are part of the main archive.

Ubuntu Native Packages

On recent Ubuntu versions debuginfod is enabled by default and everything should work out of the box. In other cases, the Ubuntu Server documentation about debuginfod explains the setup.

Ubuntu with Sury’s PPA

The main/debug APT archive needs to be enabled by adding the main/debug component to your sources list of the PPA and -dbgsym packages need to be installed, as explained in the section for Debian Native Packages.

The Ubuntu Wiki on Debugging Program Crashes provides more detail.

RockyLinux Native Packages

debuginfod is enabled by default, but it appears to be unavailable in our test. Instead debuginfo packages can manually be installed via:

dnf debuginfo-install php-cli […]

AlmaLinux Native Packages

debuginfod does not appear to be set up. Debuginfo packages can be installed as explained in the section for RockyLinux Native packages.

Docker

No prebuilt debug symbols are available. Instead the Dockerfiles available in the docker-library/php GitHub repository need to be modified to remove the logic that strips the debug symbols and the Docker image needs to be rebuilt from the modified Dockerfile.

More details are available in docker-library/php#1538.