The difficulty of Memory Profiling in PHP
Did you ever have a memory leak in your PHP program and couldn’t locate the exact source in your code? From my experience with memory profiling in PHP, this is caused by the PHP engine and how it manages memory.
PHP uses a custom memory manager on top of the native memory management in C for multiple reasons:
- Better performance by allocating larger blocks of consecutive memory
- Ability to easily implement shared nothing, freeing all memory at the end of the request
So if you create a new string of size 200 KB (or any other kind of variable) in a function of your program, three things could happen in the PHP engine:
- The current and peak memory of the script are (nearly) the same, then the PHP Memory Manager needs to go to the kernel to allocate more memory and in turn the peak memory usage increases by (roughly) 200 KB.
- The current memory usage of the script is much lower than the peak memory that was allocated due to previouvly executed functions. PHP can re-use the memory and doesn’t need to talk to the kernel.
- The garbage collector could trigger and make the function look like it actually decreased memory usage.
This makes it difficult to find out about memory problems by looking at the changes of memory_get_usage() and memory_get_peak_usage() between function calls, because it highly depends on how your program is structured to understand where memory is allocated and freed. Sometimes it is obvious to see in a memory profiler but often it is not.
New Allocation Memory Hooks in PHP 7
This is why I was excited to see that PHP 7 easily allows to hook into the Memory Manager and count the number of allocations and memory. We added this feature to our XHProf fork last week.
Counting allocations is how most Ruby Memory Profilers work, so I hoped that their approach would yield better results.
Turns out, it does not yield good results.
Looking at two Composer runs in comparison between v1.3.3 and v1.4, which included a huge memory optimization , I realized that while peak memory is only 260 MB (1.3.3) and 120 MB (1.4), measuring the total amount of allocations both versions need 733 MB (1.3.3) and 859 MB (1.4) respectively.
So while Composer 1.4 allocates more memory in total, it does so in a way that only needs 40% of peak memory at the same time, which leads to a performance improvement. Allocating memory that the kernel already provided to the PHP script is not nearly as big a problem.
That means looking at total allocated memory can be quite misleading, because it is possible that its freed immediately and doesn’t affect the peak memory level.
The same is true for looking only at memory_get_usage(), because again for the most memory hungry function it is much larger on Composer 1.4 (322 MB) than on 1.3.3 (149 MB) even though peak memory usage is halfed.
The problem with the memory_get_usage() profiling approach is that we have no way of correlating if a function creates memory and its kept around permanently until the script ends, or if its (almost) immediately freed.
You just have to hope that looking at increases in memory_get_peak_usage() for each function points you to the memory leak.
Correlating allocation and free with php-memprof
If memory_get_peak_usage() is of no help there is another approach.
The alloc/free correlation problems can be worked around by using php-memprof extension, which allows to find out how much memory a function has allocated that is still in use at the end of the script.
It does this by keeping a large internal structure of every single memory allocation that happend during the execution of a function. I haven’t tested the CPU and memory overhead, but I assume this intensive approach prohibits its permanent use in production.
