Memory usage on FreeBSD

V

VulcanRidr

Have a question regarding memory usage on a couple of servers. I bought two Lenovo M710q mini-servers, with i5-7500 CPUs. They came with 8GB of RAM (2 4GB sticks), but I have since upgraded to a pair of 16GB sticks of RAM. Both machines are running FreeBSD 14.0-RELEASE. On the primary box, running 8 or 9 jails. I have noticed swap space going to anywhere between 45% and 85% usage, however top shows:

Code: 
267 processes: 1 running, 266 sleeping
CPU:  4.5% user,  0.0% nice,  3.0% system,  0.0% interrupt, 92.5% idle
Mem: 840M Active, 344M Inact, 224K Laundry, 3474M Wired, 172M Buf, 26G Free
ARC: 711M Total, 275M MFU, 244M MRU, 7240K Anon, 7592K Header, 177M Other
     375M Compressed, 1013M Uncompressed, 2.70:1 Ratio
Swap: 2048M Total, 881M Used, 1167M Free, 43% Inuse, 188K In

The other server, with no jails running on it, has 30G free.

So why is the server with the jails on it using so much swap when there is 26G of main memory free?

Thanks,
--vr
 
richardtoohey2 said:
What’s in the jails? Are you using MySQL?

You need to find out what is using the swap.
Click to expand...
Jail1: bind9
Jail2: zabbix server (zabbix server, web frontend, postgresql backend database)
jail3: mailserver (postfix + dovecot)
jail4: git server
jail5: mediawiki server (mysql backend)
jail6: ansible server
jail7: backuppc server
jail8: plex server

So yes, I have postgres and mysql running in two separate jails. So how can I tell what is using the swap? My understanding was that swap is not used if memory (especially that much memory) is available...Apparently this is not the case.
 
When you run out of free memory during a big SQL select for example the server start using the swap. You can check top -Sw then sort by swap using "o" and type "swap" to see if currently any process is using the swap.
 
VladiBG said:
When you run out of free memory during a big SQL select for example the server start using the swap.
Click to expand...
I see this happen during mysqldump so I have a small PHP script that requests 8GB of RAM before mysqldump runs. I also drastically cut down the amount of RAM MySQL was allowed to request.

It seems that when MySQL puts too much pressure on memory it gets allocated swap and that swap never gets freed (unless you restart the MySQL process). So running something that asks for RAM before mysqldump makes FreeBSD free up "real" RAM for mysqldump to then use.

Another thing that seemed to work was making MySQL use tcmalloc instead of jemalloc.

It seemed to be worse under FreeBSD 12 and then improved under FreeBSD 13 (might have been the newer version of jemalloc.)

I suspect that when you did that top other processes had requested RAM that got released, so when you looked at top at that point it seemed like you had a lot of free RAM.

I made a script that dumped top's output every minute to see when things were happening.

If you have a window when it's safe to do so you could try a service mysql-server restart to see if that makes any change to swap.

This MIGHT all be a red herring, but something to eliminate, anyway.
 
VulcanRidr said:
So yes, I have postgres and mysql running in two separate jails. So how can I tell what is using the swap? My understanding was that swap is not used if memory (especially that much memory) is available...Apparently this is not the case.
Click to expand...

There is a difference between starting to use swap and continuing to use swap.

You start using swap when memory is moderately low-ish.

But you don't put those pages back into RAM just because RAM becomes available.

To find out what is currently in swap kill -9 random things. If it was residing partially in swap swap usage will go down.
 
Either something at some point put pressure on RAM or if I recall inactive pages become a candidate to be preemptively though slowly added to swap so that swapping happens quicker when actually needed but their memory is still in RAM so you don't actually swap them back in on use. If ARC has not had its limits adjusted, the filesystem has a lot of program and datafiles, and the system has not been up long (uptime was clipped out of the top output) then something had to put pressure on the system to get ARC so small but it looks like it is below.
 
cracauer@ said:
To find out what is currently in swap kill -9 random things. If it was residing partially in swap swap usage will go down.
Click to expand...
I still use that a lot on firefox processes as it has trash memory management in my experience. You can open top, press 'w' to display the swap column, and 'o' then accepts swap as a sortable category. Though you will not need to kill processes to use this, it also doesn't show all swap use properly in my experience (ex: firefox is listed as 0B on my machine with multiple gigs of firefox swap currently in use). If you have enough free RAM then you can use swapoff to restore swapped contents to RAM. If you have enough free swap on multiple pieces, you can swapoff some of them which may shuffle things around.
 
You see, the memory that is pushed to swap space can even be loaded back into memory without the swap being reclaimed. What happens is that the swap will be freed when the page is released (by the kill -9 cracauer@ hinted at, For maximum effect, kill "init" or "login" ;) ) OR when the page is then changed in memory. The reason for this is that the page, while in sync with the swap, can simply be taken from memory without writing it out again. So even if the swap in use stays high, no worries. That actually is a good thing.
 
You also need to run a tool like databases/mysqltuner and check the various buffers and caches. More often then not people completely misconfigure those settings and MySQL ends up using way more memory than the machine actually has. That is a bad thing, never let MySQL use more memory than the machine has. Tweak your buffers, caches and max_connections until you get a satisfactory result. Take into account that all your other processes need memory too. Also limit arc_max. While ARC usually does a good job managing memory you can get in a situation where both MySQL and ARC start fighting over memory.

Also increase your swap size, 2GB isn't much. With 43% used of the 2GB, increase swap to 4GB. Then it'll be around the 25% mark. Some swap usage is fine, as long as it doesn't keep swapping things in and out.
 
top(1), sorting by RES and again sorting by SIZE will tell you which processes are your memory hogs.
 
Any take on swap use for SSD wear when it goes under higher load like firefox swapping things in+out actively and nearly nonstop after it put too much pressure on the system?
 
cy@ said:
top(1), sorting by RES and again sorting by SIZE will tell you which processes are your memory hogs.
Click to expand...
Top only shows currently running processes, RES doesn't include swap, and SIZE includes memory that has been requested but not actually used/allocated to the process (silly programs like Baloo run that up to 256GB last I looked). I wish memory use was easier to read. I thought some inaccuracies in #s were present because it was hard to get right or done inaccurately for performance reasons but don't know if that is still the case; an option for lower performance but better #s would be handy to have if we don't already.
 
Of course RES doesn't include swap. SWAP does. SIZE includes all memory that has been allocated (malloc()ed). malloc()ed memory doesn't necessarily mean it's used.

Processes that are swapped out are always the "victims" of memory shortage. Those swapped processes may or may not be the cause. In a DBMS environment or large running java app, other applications may be swapped out to make room for the culprits. This is why you look at RES first and secondly SIZE.

The simplistic thing is to only look at SWAP.

Personal observation: The vast majority of people (including sysadmins) do not understand how virtual memory works. This is why people base their memory decisions on output from programs such as linux free(1). These same sysadmins explain this falsehood to customers who in turn approve unneeded expenditures.

Before all this, the first simplistic thing people do is only look at free RAM.
 
BTW, we are talking about paging most of the time. Swapping is when your whole process gets thrown out. Paging is when only parts of your memory gets written to swap and reclaimed for other things.
 
Correct. But, top(1) and other tools report swap used. Paging uses swap space. All operating systems except z/OS do this. On z/OS (formerly MVS) paging and swap went to different datasets (mainframe term for files).

Swapping involves throwing many data structures defining the process to disk too. Paging pages out process pages only.

Like z/OS, the BSDs page and swap. Linux no longer swaps. It only pages.
 
You must log in or register to reply here.
Back
Top