I've spent quite a bit of time chasing an apparent problem where VSZ (and RSS) grow without boundary, even though there is no leak from the application code itself. The code is a fastcgi application and thus the "correct" behavior is it never exits.
What appears to occur is that when using functions in system libraries and elsewhere that allocate something, and then the program frees it (e.g. an EVP Context to use digest functions in the openssl library) a subsequent allocation request will fail to find a consolidated part of the memory space to reuse and instead goes back to the kernel with another request for more. This can become pathological some of the time and if it does ultimately will result in an out-of-RAM condition.
After various efforts I found that if I allocate system library things (e.g. openssl) contexts at program start and instead of freeing and getting another buffer as processing occurs I keep the original handle and reset it in the case of openssl functions, along with making sure that I release everything I can reasonably release for each process through (rather than attempting to reallocate only when I need to as a buffer is not large enough) I can greatly minimize this -- and with a sentinel that exits if it happens anyway the code is stable This is arguably not a great answer; ideally if you can reuse allocated memory taking the overhead to free and then reallocate a buffer you require is less-efficient.
Recently I brought jemalloc-5.3 (14.x uses the previous release) down, built it (defaults other than installing in /usr/local) and if I link the app with that version instead RAM consumption remains stable -- with no changes to the code at all. While it will be weeks (or more) before I'm absolutely certain that the pathology is completely gone it certainly appears to be in the somewhat-limited runtime I've put on it thus far.
If I'm reading the github release dates correctly 5.3.0 is from May of 2022.
Is there a reason FreeBSD has not rolled this forward?
What appears to occur is that when using functions in system libraries and elsewhere that allocate something, and then the program frees it (e.g. an EVP Context to use digest functions in the openssl library) a subsequent allocation request will fail to find a consolidated part of the memory space to reuse and instead goes back to the kernel with another request for more. This can become pathological some of the time and if it does ultimately will result in an out-of-RAM condition.
After various efforts I found that if I allocate system library things (e.g. openssl) contexts at program start and instead of freeing and getting another buffer as processing occurs I keep the original handle and reset it in the case of openssl functions, along with making sure that I release everything I can reasonably release for each process through (rather than attempting to reallocate only when I need to as a buffer is not large enough) I can greatly minimize this -- and with a sentinel that exits if it happens anyway the code is stable This is arguably not a great answer; ideally if you can reuse allocated memory taking the overhead to free and then reallocate a buffer you require is less-efficient.
Recently I brought jemalloc-5.3 (14.x uses the previous release) down, built it (defaults other than installing in /usr/local) and if I link the app with that version instead RAM consumption remains stable -- with no changes to the code at all. While it will be weeks (or more) before I'm absolutely certain that the pathology is completely gone it certainly appears to be in the somewhat-limited runtime I've put on it thus far.
If I'm reading the github release dates correctly 5.3.0 is from May of 2022.
Is there a reason FreeBSD has not rolled this forward?
