C Memory leak, or maybe not?

S

symlink

Hello,

I'm writing a software in C. As you know, in C we don't have standard library provided hash tables, so I'm relying on a header-only library that I found on github.

While designing this software I was very careful with memory: I pre-allocated the maximum amount of session objects, and I made sure that the hash table which is bookmarking sessions is large enough to minimize the number of reallocations. The program will evict sessions in LRU fashion if we have too many, hence freeing the handler in the hash table. A little testing highlighted that, despite my efforts, the hash table implementation needs to do some re-shuffling from time to time. This happens fairly infrequently, so I think it is acceptable.

Once my software seemed baked and ready to be tested, I made it run on my FreeBSD 15 box. It is a daemon, it needs to work for long time, so memory footprint is a thing to check. I found that the process keeps piling memory, despite address sanitizers and valgrind not detecting any defect.

I could eventually isolate the issue, and the culprit seems to be to the hash table I mentioned above. The implementation will sporadically issue `free(3)` for a certain buffer, followed by a new `malloc(3)`. ...which is perfectly OK, on my book, since allocated blocks are freed on the next re-shuffle, and everything is cleaned when the process is terminated. I concluded that the memory handling is reasonable. Using a memory profiler seems to confirm it. Moreover, under Linux the very same daemon does not show any memory leaking behavior.

I start wondering if this problem could be a non-problem, and maybe just being related to how FreeBSD handles memory. E.g. I'd like to test what would happen if I capped memory to a certain maximum: will I end up out of memory?

Do you have any recommendation?
 
Do you have any unit tests? Oh wait, I misread. There's not enough info here. Are all the allocations the same size? Can you re-use a prior allocation? Are you expecting memory to drop following a re-shuffle?
 
What memory counters are increasing? Is it possible that you are touching new pages, increasing the address mapped space, rather that just using more memory?
 
You may not have a memory leak but you may have a program logic bug. Put in some counters and checking code. Also watch /proc/<daemon pid/map and see if over time it is showing some unusual behavior. Some time even just printing stuff can help (print a counter every N malloc and every N free etc.). You can also add a conditional breakpoint in gdb/lldb and manually invoke the checking code. Finally, write a separate test program just for the hashtable and simulate the kind of usage your appn makes.
 
symlink

POSIX does have standard hash tables hcreate(3). You can create multiple tables with hcreate_r(3).

Regarding your implementation, if you are looking at the memory use from the system's perspective, you may see it as leaking memory. FreeBSD uses jemalloc(3). Its behaviour may be different from you might see in other systems. It does not immediately give memory back to the OS if it expects that it may have to allocate from the OS again.
 
cracauer@ said:
If Valgrind is happy it is more likely a memory fragmentation issue.
Click to expand...
Ahh, what is a memory leak?

If allocs and frees are matched that means the program is "freeing 1 for each alloc 1" but that does not mean the OS reclaims at the same rate. This is often a problem on Java or other garbage collecting systems.

That comes back to what cracauer@ points out: program is balanced on allocs and frees, but the OS needs to reclaim which may lead to fragmentation.
 
What's the program main loop control? If it's 1 thread I/O event-driven, there must be an action to associate with the increasing resident menory that doesn't happen in all iterations. A function call related to a hashtable?
I never used any hashtables that C books describe. It's mostly my own construction... Are you sure nothing is written beyond a defined data field boundary?
 
As a quick hack, grep for every use of *alloc and free in the library and instrument it with a simple integer increment/decrement.

Obviously realloc is more difficult (and many uses of it are simply wrong, i.e realloc with size 0 is *not* the same as free. I see this so often....)
 
If it was a plain leak, as in not freeing something that was malloced or realloced, a simple bookkeeping would detect it. If Valgrind is happy it's not that.

How do you measure the size of the program, anyway?
 
cracauer@ said:
If it was a plain leak, as in not freeing something that was malloced or realloced, a simple bookkeeping would detect it. If Valgrind is happy it's not that.
Click to expand...
Yeah, to clarify, I also mean things like putting a manual inc/dec with all allocs, i.e: glGenTextures, glDeleteTextures. If this is outside of Valgrind's VM (which graphical elements sometimes are due to display servers), sometimes it can't track it.
 
You must log in or register to reply here.
Back
Top