C Memory allocated with malloc/calloc

[ogogon]

Colleagues, please tell me what happens to memory arrays allocated using the malloc()/calloc() functions when the program terminates.

Will it remain allocated, will it be automatically and unconditionally released, or is it implementation dependent and may be unstable?

Let's assume that my program is not fully debugged and does not call free() on all such areas. Will this make the system more unstable or should I not worry about it at all? (I remember the times when something like this was very important and could almost lead to the downfall of the system. How is it now?)

Thanks for the answers,
Ogogon.

 

[cracauer@]

When the process ends all its memory is going away. No further problems.

 

[6502]

You can do simple test. Call malloc() with block like 2GB, see the free memory with top/htop, then close the program without free() and check again.

 

[bgavin]

Memory leaks are chronic, hence the eventual reboot required to pick up fragmented memory.

If your app goes fatal and the memory manager is locked out or disabled you have a memory leak.

I have no need to write C anymore so don’t have need for Malloy.

YMMV

 

[cracauer@]

You can do simple test. Call malloc() with block like 2GB, see the free memory with top/htop, then close the program without free() and check again.

Just malloc()ing a 2 GB block without touching the memory in it will not allocate physical memory and will not decrease the "free memory" count. You would actually have to fill the memory with data.

 

[Crivens]

Memory leaks are chronic, hence the eventual reboot required to pick up fragmented memory.

This should not be a problem for a looong time now. Since IOMMUs we can deal with fragmented main memory pretty good, no reboot required.

If your app goes fatal and the memory manager is locked out or disabled you have a memory leak.

You have a leak if you allocate some memory and don't free it. The system has no way to tell which of your memory is still reachable or not. Calling free() is YOUR job. You can see this in any modern browser which will, over time, fill up all the memory when you have pages with f.e. javascript running. Many a browser is leaking memory left and right, and needs almost something like a cron job to restart it from time to time.

I have no need to write C anymore so don’t have need for Malloy.

YMMV

Classic case of auto carrot?

 

[6502]

Just malloc()ing a 2 GB block without touching the memory in it will not allocate physical memory and will not decrease the "free memory" count. You would actually have to fill the memory with data.

OK. +1 line for memset().

 

[_martin]

OK. +1 line for memset().

I'm just nitpicking here but even memset is not needed. It's faster to write one byte to each page.

 

[6502]

Faster for CPU, for operator is faster to write memset() + params than for-loop.

 

[jbo@]

Let's assume that my program is not fully debugged and does not call free() on all such areas. Will this make the system more unstable or should I not worry about it at all? (I remember the times when something like this was very important and could almost lead to the downfall of the system. How is it now?)

When the process ends all its memory is going away. No further problems.

Well "it depends". If your program is a service/daemon or something else that runs for a long time (think web server), then it will become a problem even now days. What cracauer@ is hinting at is that a typical modern general computing operating system will free any memory that a process/program allocated once the process/program terminates. But if your program does not terminate (for example because it's designed to run constantly like a daemon or a driver), the operating system will not free up the memory. Hence a memory leak can still bring down an otherwise healthy system.
There are other mechanisms such as OOM killers but again - "it depends...".

 

[6502]

It is not very hard to catch memory leaks. malloc() and free() can be hooked or replaced with macros. Malloc-ed addresses are stored in array, then free() hook removes address from array. At program end or after 1 hour / 1 day of work the array is checked for forgotten malloc-s.

Storage Allocation Hooks in C

Hi all, this is my first time posting on these forums. I am hoping that I picked the right forum for this topic. Please let me know if there is a better place for this. I am currently porting a user space threads package called "capriccio" to FreeBSD for a research project. The...

forums.freebsd.org

 

[Paul Floyd]

Colleagues, please tell me what happens to memory arrays allocated using the malloc()/calloc() functions when the program terminates.

Will it remain allocated, will it be automatically and unconditionally released, or is it implementation dependent and may be unstable?

Let's assume that my program is not fully debugged and does not call free() on all such areas. Will this make the system more unstable or should I not worry about it at all? (I remember the times when something like this was very important and could almost lead to the downfall of the system. How is it now?)

Don't worry about the OS.

Do worry at least a bit about your executable.

It's generally best to deallocate resources when you no longer need them. If your application is small and short lived it probably doesn't matter too much. If your application is big and/or long running then failing to deallocate will probably result in

1. A major slowdown of your application and system as it starts swapping.
2. Your application failing either due to the inability to obtain resources that it needs or due to being killed by the kernel.

Consider checking for leaks with Valgrind or Address/Leak Sanitizer.

If your memory use is really important, consider using a heap profiler to analyse your memory use.

 

[ralphbsz]

It is not very hard to catch memory leaks. ...

Dijkstra: There are two kinds of programs. Those that are so small they obviously have no leaks. Those that are so large that they have no obvious leaks.

In a very small program (dozens or hundreds lines of source, runs for seconds, doesn't use all memory on the machine), don't even bother calling free(), ever. It just makes things more complicated and introduces bugs. Allocate memory like a maniac, use it, and then the program shuts down, leaving no trace (*)

In a very large program (millions of lines, 1000s of calls to malloc, many of them hidden inside other functions, some memory is used for the duration of the program and doesn't need to be freed, other memory is bound up in interesting data structures that have lots of cross-references, like a tree whose leaf nodes are also threaded into a list), figuring out whether memory is useful and reachable is a very tedious, error-prone, and inexact science, which requires lots of human interaction.

Real-world anecdote: "Fred, why do you allocate memory in line 1234 of module foo/bar/blatz.c, and never free it?" Answer: "Because Charlie said that he'll need that data structure in a few months, for his bangblitz project." Reply: "Fred, has anyone told you that Charlie has left our team, moved with his family to Timbuktu, and now works on an Artificial Dumbness project?" Outcome: "In that case, I guess you can free() that data structure and everything hanging from it. But wait, next week our new manager starts, and I heard the executives want to revive the bangblitz project." So now, do you free the memory or not? How much do you trust your new manager? Do you understand what executives really want? Who is going to do the work, with Charlie far away?

(*) Footnote: Motto of the park service: Take only memories, leave only footprints. Or something like that.

 

[Paul Floyd]

Of course the real problem is in the title, the need for explicit memory allocation and reallocation in C.

If you use Rust or follow the C++ core guidelines then you will have no problems.

C++ Core Guidelines

The C++ Core Guidelines are a set of tried-and-true guidelines, rules, and best practices about coding in C++

isocpp.github.io

(well, as long as your structures form a DAG, otherwise things are a bit tricky in all languages).

 

[Crivens]

(well, as long as your structures form a DAG, otherwise things are a bit tricky in all languages).

Not really. There is garbage collection in a lot of languages. But that needs to be part of the complete language and runtime libs, or it will not work. So no calling C libraries. But if you have it down to the lowest level, you have no memory leaks. Now for different resources we start all this again.

 

[cracauer@]

Dangling pointer create memory buildup in GCed languages very nicely. It might technically not be "leaked" memory, but the program grows nontheless. Always wipe out your pointers.

 

[drhowarddrfine]

In that case, I guess you can free() that data structure and everything hanging from it.

And then you find out someone else used it for something else.

 

[mer]

Dijkstra: There are two kinds of programs. Those that are so small they obviously have no leaks. Those that are so large that they have no obvious leaks.

Classic. It's been a while since I've seen this referenced.

Dangling pointer create memory buildup in GCed languages very nicely. It might technically not be "leaked" memory, but the program grows nontheless. Always wipe out your pointers.

And then you run into GC tuning: GC takes a finite amount of time, run it more often to keep ahead of memory build up you take away runtime from code doing real work. Increase the interval between runs of GC and then when it runs you have huge stall every where else in the code. High/low watermarks to influence how often it runs, picking them becomes non trivial.
Everytime I do something in java/javascript/other language and I see a "new" I want to find the matching "free".

And then you find out someone else used it for something else.

"What you copied that pointer to another variable and then use after free?" In theory, "smart pointers" in C++ are supposed to help with this.

I think modern browsers are good examples of memory leaks; lots of folk complain about using all the memory when they have 100 tabs open and playing videos.

 

[Crivens]

GC takes a finite amount of time, run it more often to keep ahead of memory build up you take away runtime from code doing real work.

Many algorithms still are single threaded. Some can not be parallel, no matter what. But we have a growing number of cores, so having some of them run the GC can actually speed up things. It will parallelize allocations and GC/free in different cores, something you can't do with manual free().

 

[Paul Floyd]

Everytime I do something in java/javascript/other language and I see a "new" I want to find the matching "free".

Ugh. "delete" I hope! Do java and javscript even have a "free"?

"What you copied that pointer to another variable and then use after free?" In theory, "smart pointers" in C++ are supposed to help with this.

Not just in theory. Using smart pointers should still be your second to last resort. If you can, start with standard containers and algorithms. In my experience that usually results in shorter, more readable and safer code with performance that is good enough.

 

[ccammack]

Dangling pointer create memory buildup in GCed languages very nicely. It might technically not be "leaked" memory, but the program grows nontheless. Always wipe out your pointers.

I once worked on a project that ran on a simplified Flash interpreter for the Nintendo DS. Those two things don't really go together, and we eventually had to add a function to dump the entire object tree to the console at the end of each level to find things to nil out.

 

[mer]

If you can, start with standard containers and algorithms. In my experience that usually results in shorter, more readable and safer code with performance that is good enough.

I don't disagree with this; C++ STL has a lot of good stuff in it designed by people a lot smarter than me. But a lot boils down to what is available, what do I know, what are the requirements.
Some of the more current C++ specifications are good/reasonable but take a little time to wrap head around.
But one thing I do like is that almost all containers can be iterated with relative ease (at least in code).

 

[cracauer@]

Many algorithms still are single threaded. Some can not be parallel, no matter what. But we have a growing number of cores, so having some of them run the GC can actually speed up things. It will parallelize allocations and GC/free in different cores, something you can't do with manual free().

Parallel GC (as in program pausing, multiple threads doing GC) isn't much faster than single-threaded - because it is memory bandwidth/latency bound in either case. SBCL had to realize that just recently.

Concurrent GC (no program pauses) has huge overhead in real time and CPU time of programs due to the read and write barriers required for all mainline code operations.

But malloc()/free() also has skeletons in the basement.

 

[cracauer@]

I once worked on a project that ran on a simplified Flash interpreter for the Nintendo DS. Those two things don't really go together, and we eventually had to add a function to dump the entire object tree to the console at the end of each level to find things to nil out.

You can do gc-by-fork in some situations, such as query-based systems. After receiving a request you fork and after you complete the query you throw away that child and make a new one for the next request.

Creates problems if you want to keep permanent data structures such as for statistics, but as such it works really well.

You can also do gc-between-queries where you run through queries without GC and then do a full GC between two queries. That costs CPU time and physical memory, but query latency will be excellent - better than malloc/free.

 

[Crivens]

Parallel GC (as in program pausing, multiple threads doing GC)

No. Parallel as in fork()ing and checking what is dead storage. What is dead in the copy is dead in the parent. You only need some memory mapped so that you can transfer that information back to the parent. Parent forks, and goes on. You could even do that in a thread and collect unreachable objects in a list to add to the free list. Then the main code only needs to wait in case that free list is empty. I'm currently toying around with something like that.