Something uses up all my memory

tarkhil

tarkhil

Hello

On one of my servers, after 12.1->12.2 upgrade, strange things happen.

Processes number is stable, memory uses by processes is stable, but memory usage grows and grows and in about 8-10 hours all memory and all swap is used out.

I've tried to set vm.pageout_oom_seq=120 because laundry memory usage becomes high, but without any visible effect.

Overall, system is slow and again I cannot find the source of problem - no CPU usage, no heavy disk usage, no visible database load, but some sites responds slow to timeout.

Here are my graphs


In 20+ years with FreeBSD, it's the first time when I can't even guess what is leaking.
 
  • Since you haven't got any reply here for 2 days on a topic that IMHO should get some of the wizzards to chime in, I'd like to recommend you ask on one of the mailing lists, too. Maybe freebsd-questions or freebsd-hackers? Also ask if there's another list that is better suited.
  • \top -atSzo res -s 3 should show you what's eating RAM. The backslash tells the csh(1) not to use your alias for top(1), but the plain utility. This way I'm shure it has the options applied right, e.g. if you already have -a, a 2nd -a would revert that.
PS You'll have to subscribe to post.
 
what is top saying about zfs ARC size ... maybe limit the arc size to something like 4 or 6 GB. also I do not grasp the failed and resets in the netstat graph ... is this figure millions? if so you might be under attack and the system uses lots of memory for temporary storing/handling the many half open tcp states. do you run mongodb - quite a well known memory hog, also java programs like tomcat
 
This is most likely to be a red herring - but in my attempts to work out why MySQL eats swap sometimes (to the extent that OOM kicks in) it seems that when FreeBSD is under memory pressure from long-running processes it prefers to give them swap to use rather than sorting out Inactive memory. It seems to decide that what is in Inactive memory is worth keeping when the memory pressure is from a long-running process, and hands out swap.

But my research has not been scientific, and it's certainly not related to moving from 12.1 to 12.2.

On one machine (11.4) I know mysqld will eat swap during a mysqldump operation - FreeBSD will give mysqld swap space (that will not be released) rather than anything from Inactive memory.

So on that machine, just before I run mysqldump - I have a short-lived process that asks for 8Gb of RAM, populates it, then exits. It gets that 8GB from Inactive memory, then when it exits the RAM becomes Free. Then mysqldump runs and starts to chew through the now-Free memory (instead of hitting swap.)

Most memory issues reported around here seem to be ZFS or MySQL. Not sure if you are using ZFS but looks like you aren't using MySQL.

You could try restarting services and seeing if swap drops drastically for any given service. But that might not be so feasible on a production machine.

EDIT - no, nothing to do with what you are seeing, so please ignore.

Looking closer at your memory chart: http://munin.sub.ru/sub.ru/gab2.sub.ru/memory.html

Looks like at Tuesday 1 p.m. inactive drops, wired jumps up massively, laundry jumps up massively, and you roll straight onto hitting swap. Same thing at about 8 p.m.
 
I also have not always the best fun with memory going inactive and increasing swap over time.
Sorry, I have no solution. But maybe this helps a little bit...

richardtoohey2 said:
So on that machine, just before I run mysqldump - I have a short-lived process that asks for 8Gb of RAM, populates it, then exits. It gets that 8GB from Inactive memory, then when it exits the RAM becomes Free. Then mysqldump runs and starts to chew through the now-Free memory (instead of hitting swap.)
Click to expand...
I also have observed the "inactive" memory increasing permanently over time, on my desktop which doesn't really do much work.
I suspected this behaviour for the desktop becoming sluggish sometimes, but I have no real clue.
So I wrote a swapstresser some years ago, which I used for some experiments.

Your comment made me dig it out again and try a bit.
The inactive memory already had piled up to 9GB.
So I chose to stress with 15 chunks each 1GB, watching top -atSzo res -s 3 (Thanks, Mjölnir 👍) on another window.
./swapstresser -r 100 -b 1024 -m 15 -f 3 -a 2 -s 1 -d 100
And bingo 👍 I watched it eating away the inactive memory, increasing laundry and free memory a lot.
So the trick richardtoohey2 described indeed works, at least for me. Thanks man 👍

However, when I stress so much memory that it dives into swap, the swap usage increases, and doesn't "recover" fully after that. Don't know why...

(If you like to play with the swapstresser, just get swapstresser.c and the Makefile, and make it. Running the program without options gives an overview of options.)
 
richardtoohey2 said:
This is most likely to be a red herring - but in my attempts to work out why MySQL eats swap sometimes (to the extent that OOM kicks in) it seems that when FreeBSD is under memory pressure from long-running processes it prefers to give them swap to use rather than sorting out Inactive memory. It seems to decide that what is in Inactive memory is worth keeping when the memory pressure is from a long-running process, and hands out swap.
Click to expand...
That's not a red herring. The wizzards are investigating ways to improve the swap handling (espc. the out-of-swap killer) for about a week or so. Subscribe to the freebsd-hackers if you're interested in such topics.
 
Thank you, that worked.
For some reason gmail search for "freebsd-hackers" didn't show me all of the mail. Need to find out how to search correctly or what went wrong otherwise.

Imho the discussion goes into a wrong direction.
Main part of the discussion was about increasing swap space to avoid the OOM kick in.
There is no discussion at all about the problem of ever-growing inactive memory, causing the memory manager stuff to swap out used memory instead of just releasing that unused "inactive" memory to the free memory pool to be used.
So my personal laymans' impression is the path "just increase swap so you don't hit OOM" might be like doctoring the symptoms instead of the cause.

I need to set up a test machine to explore further to find out more about the phenomenon, will do that soon when I am finished with the video hardware autodetection and autoconfiguration in the postinstaller.
 
You can set up a test environment in bhyve(8) except for stuff that needs bare metal, like BIOS & hardware drivers. Even with these, using bhyve(8) works in some cases (with BIOS image/PCI pass-thru). The VM system does not move inactive -> free, but laundry -> free on memory pressure. See the excellent article in the wiki. FMLU the point is that some applications (falsely?) set flags that prevent the OS from reusing their memory. Maybe that was discussed in another mailing list, freebsd-current instead? OT: would you chime in a collective order for FreeBSD T-shirts? See Off-topic -> FreeBSD Mall orders. If yes, answer there.
 
Mjölnir said:
The VM system does not move inactive -> free, but laundry -> free on memory pressure.
Click to expand...
That's what I've read but not what (I think!) I see.

There are some knobs to try turning that I haven't done so. But the OP's problem seems to be more Inactive->Wired, maybe via Laundry - he's not seeing lots still in the Inactive state.
 
Mjölnir said:
See the excellent article in the wiki.
Click to expand...
Link please. I looked at the wiki front page and still do not know which article you mean.
Aside of that, the Wiki articles are often outdated and sometimes just plainly wrong.
I have given up trying to mail the wiki maintainers because in my personal experience, the reaction of the large majority of the wiki maintainers seems to equal zero.
(This is exactly why I am thinking about mirroring the wiki as soon as the postinstaller can set up a webserver jail for etherpad, so we, the user community can work on improving it and encourage the wiki maintainers to adapt the corrections and other improvements.)

richardtoohey2 said:
That's what I've read but not what (I think!) I see.

There are some knobs to try turning that I haven't done so. But the OP's problem seems to be more Inactive->Wired, maybe via Laundry - he's not seeing lots still in the Inactive state.
Click to expand...
Sadly that already scrolled out of the first week display and I cannot see enough detail there anymore.

Anyway, what I observed is yesterday after the first swapstresser run, the laundry grew from 6GB to 9GB.
Today the inactive memory was back from zero to 6GB. Now after another swapstresser run, the laundry grew to 12GB.

Regarding the currently available knobs, I am afraid there is no way except sifting through the source code and make a list of these, as many wiki pages contain instructions that applied to historic FreeBSD versions and don't have any effect on later ones.
I guess it could be interesting to set up a test system with a clearly-defined memory usage profile that collects all the interesting data (possibly in higher time resolution), to maybe get an idea of the system behavior when analyzing the effect of these tunable settings.

Regarding the current state of the OP's system, to me it looks like as if he did some changes, as the system behavior does not follow the pattern he observed after upgrading anymore.

Hope the OP will give us an update...
 
The FreeBSD implementation book is good as well, but it's only up to FreeBSD version 11.x.

Design and Implementation of the FreeBSD Operating System, The: McKusick, Marshall, Neville-Neil, George, Watson, Robert: 9780321968975: Amazon.com: Books

Design and Implementation of the FreeBSD Operating System, The [McKusick, Marshall, Neville-Neil, George, Watson, Robert] on Amazon.com. *FREE* shipping on qualifying offers. Design and Implementation of the FreeBSD Operating System, The
www.amazon.com

But I wasn't sure (my limitation!) at times if it was talking about swapping in/out of processes versus use of swap space for VM.

I don't think VM is an easy problem so I think there are always going to be edge cases where the defaults will be sub-optimal. But in my case (working with MySQL) it also seems to be an interaction with the memory allocator - jemalloc versus tcmalloc - so not clear cut. And just when I thought it was definitely jemalloc on FreeBSD 12, I started seeing the same behaviour on FreeBSD 11 with the older version of jemalloc. From my brief & basic checking on FreeBSD 13.0 with a newer jemalloc - it's a lot better. But don't know if just down to jemalloc or other changes in FreeBSD.
 
Yes, the OP's issue smells to be an issue that pages do not get recycled, I mean returned from laundry to free pool. If it's an applications fault, then the OP would have seen that in top and known where to investigate (20+ years FreeBSD veteran/wizzard).
 
Mjölnir said:
Please supply some keywords and/or links about what has significantly changed since.
Click to expand...
Because I'm not kernel developer, it's most likely my idea of difference between VM in 3.X/4.X and 13.X is going to be some superficial differential comparisons.
We have to wait till some kernel developers post a proper answer to this question. In the mean time, I'll ask around, and hope to find someone to answer it here.
 
Thanks again Mjölnir and vigole!
When reading while eating, I recognized that the article was written by the FreeBSD God Wizard per se himself.

Like vigole I guess what the God Wizard and his friends once built is still the base, and I am afraid the detail information about the detail changes since then is spread in years of mailing list archives.
I guess searching there with keywords like "inactive, wired, dirty, page, coloring, laundry, ..." might help finding information.
Another approach might be finding all current knobs by code review and then search for these, collecting the information necessary for an up-to-date wiki article. A code review/audit might also reveal hints to understand this strange behavior observed by some of us...
 
  • Thanks
Reactions: a6h
Snurg said:
I guess what the God Wizard and his friends once built is still the base, and I am afraid the detail information about the detail changes since then is spread in years of mailing list archives.
Click to expand...

Here is the main problem with comparing code between different versions, merely based upon static analysis of program code,
specially when it comes to something like Virtual Memory (VM). The system has to go under pressure, using different machines,
state, etc. Then bugs will show up, fix this one, create another one, so forth and so on. As you have mentioned correctly, specific
details are buried under mailing list and bug reports. Also, there are discussions on IRC, which by its nature, hard to find on web.

[EDIT]: To avoid thread hijacking, I'll continue the Off-Topic portion of discussion (IRC) in my profile.
 
True!
If there cannot be found any somewhat-up-to-date information document, code sifting would be inevitable to merely know which tuning variables actually exist.

What I suspect is that there could be an issue with systems that insufficiently stress the memory management.
And to me, the graphics the OT's particular server shows don't look like heavy system pressure.

For many desktop/laptop users suspend/resume does not work (at least in the default configuration).
Many of these users boot their PCs daily, thus don't accumulate sufficiently long uptimes to have problems show up.
Another possibilty might be that the default tuning configuration doesn't work out well when the system parameters are somehow unbalanced, for example less-than-typical load combined with more-than-usual memory.

For example, personally I do not recall having observed this accumulation of inactive and laundry memory (which could be considered as sort of wasted resources due to possibly unbalanced autotuning) on systems with 8, 16 GB or less, no matter how big or small their load.

For this reason I believe an up-to-date tuning guide could possibly help solve such problems with a little bit of tuning...

vigole said:
Also, there are discussions on IRC, which by its nature, hard to find on web.
Click to expand...
True too, sadly...
On the FreeBSD IRC Wiki page I didn't find any hint of chat archives.
Googling didn't show me any indication there could maybe some chat logs archived either. Possibly I just didn't search well enough?

If there aren't such archives, I guess the only chance to salvage this possibly crucial information is to ask whether somebody has logged chats and can upload these for public use.
 
Mjölnir said:
Great! Have you been advised on the mailing list to try that, or did you just try because there was no better idea?
Click to expand...
No better idea. Upgrading to beta was the only thing I did not try. 13.0RC1 caused some strang killings ("Out of swap" with plenty of swap), RC5 ... well... for 19:30 uptime still did not.
 
You must log in or register to reply here.
Back
Top