11GB swap used - but what data is actually swapped out?

R

rowan194

Something I'm a little confused about. Here is the output of top -osize:

Code: 
32 processes:  1 running, 31 sleeping
CPU:  3.5% user,  0.0% nice,  1.6% system,  0.0% interrupt, 94.8% idle
Mem: 2701M Active, 1405M Inact, 8454M Laundry, 109G Wired, 24K Buf, 3452M Free
ARC: 99G Total, 80G MFU, 15G MRU, 4123M Anon, 148M Header, 558M Other
     92G Compressed, 243G Uncompressed, 2.66:1 Ratio
Swap: 32G Total, 11G Used, 21G Free, 35% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 3181 mysql       108  20    0    13G    12G select   7 3398.1 101.62% mysqld
25255 rowan         1  20    0   130M  1572K ttyin   10   0:06   0.00% mysql

Note that 11GB (35%) of swap is used.

One thing that confuses me: if 11GB of data is swapped out..... where did that data come from? The most hungry process, by far, is mysqld, but that appears to have almost all of its allocation resident.

The other 30 processes (not shown) have a total of about 409MB allocated. The server pretty much just does one thing, so the large majority of resources are used by mysqld.

With all user processes apparently using a total allocation of around 13.5GB, with 12GB+ of that being resident, how can 11GB be swapped out? Am I misunderstanding the meaning of resident?
 
gpw928 said:
There's a few ideas here.
Click to expand...
I actually posted in that thread a few years back. :)

I ended up ending the mysqld process. Used swap went back to a few megs. mysqld was using 11GB of swap.

Still don't understand:

13GB is allocated.
Of that, 12GB is resident.
But 11GB is somehow swapped out??

I presume SIZE is the total allocation, regardless of whether the allocated memory is physical or virtual. In other words, a process that begins swapping out data will still have the same SIZE, but RES will reduce?
 
Is this FreeBSD 12 or 13 and what version of MySQL?

I found 12.x seemed to make MySQL 5.7 use more and more swap; things were a lot better under 13.x.

I can’t explain the memory usage in terms of what is showing in top.
 
richardtoohey2 said:
Is this FreeBSD 12 or 13 and what version of MySQL?

I found 12.x seemed to make MySQL 5.7 use more and more swap; things were a lot better under 13.x.

I can’t explain the memory usage in terms of what is showing in top.
Click to expand...
FreeBSD 12.2-p6 (need to upgrade), mysql 5.7.33 (need to upgrade)

Use of swap is not necessarily a bad thing. I'm quite happy for the OS to swap out idle data and instead reuse that precious physical RAM for ARC. But now I'm starting to question whether "11GB swapped out means an extra ~30GB of ARC cache" is actually an accurate assumption.
 
VladiBG said:
forums.freebsd.org

Solved - FreeBSD 12.x and MySQL 5.7 and importing file with lots of small lines exhaust RAM and swap

Thought I'd try out the new 12.1 beta and RC releases but have got side-tracked down a MySQL 5.7 rabbit hole. When I try and import a 17Gb database dump file from mysqldump my machines run out of RAM then they swap, then they run out of that, then the OOM killer kicks in with sad results...
forums.freebsd.org forums.freebsd.org
Read my post #22 from the thread above
Click to expand...
Thanks, I had that issue too - back in 2019 - but it's unrelated to this particular issue/quirk. The machine is not running out of memory.
 
rowan194 said:
With all user processes apparently using a total allocation of around 13.5GB, with 12GB+ of that being resident, how can 11GB be swapped out? Am I misunderstanding the meaning of resident?
Click to expand...

No, but you are misunderstanding swap, it stopped being an either/or long before FreeBSD existed.

Writing pages to swap creates clean pages of ram which can be reassigned instantaneously when they are needed. You can also create clean pages by reading data back in from swap, which is likely to be what happened here (unless it's just cruft on tmpfs).
 
How much RAM are you giving MySQL? On a 32G machine (not dedicated to MySQL, and using UFS rather than ZFS) I started with 12G here, but ended up with 5G to reduce the swapping:
Code: 
innodb_buffer_pool_size = 5G
SirDice will be along shortly telling you to use mysqltuner (on my TODO list!)
 
cracauer@ said:
Do you have a memory filesystem for /tmp?
Click to expand...

No memory file systems or anything backed by swap.

Note also that when I killed mysqld the used swap went back down to (almost) zero, so it seems reasonable to assume the majority of the 11GB was associated with that process.

bob2112 said:
No, but you are misunderstanding swap, it stopped being an either/or long before FreeBSD existed.

Writing pages to swap creates clean pages of ram which can be reassigned instantaneously when they are needed. You can also create clean pages by reading data back in from swap, which is likely to be what happened here (unless it's just cruft on tmpfs).
Click to expand...

Sorry, I'm not understanding this. ELI5?

richardtoohey2 said:
How much RAM are you giving MySQL? On a 32G machine (not dedicated to MySQL, and using UFS rather than ZFS) I started with 12G here, but ended up with 5G to reduce the swapping:
Code: 
innodb_buffer_pool_size = 5G
SirDice will be along shortly telling you to use mysqltuner (on my TODO list!)
Click to expand...

11GB for the innodb_buffer_pool_size. Again, lack of memory is not an issue. The process hasn't been killed by OOM (jemalloc leaks) since I changed to tcmalloc back in 2019. The server runs fine, I'm just trying to understand the apparent overlap (?) between swap and resident.

Also, the increase in swap used is gradual, over several days. It's not thrashing in and out due to lack of memory.
 
rowan194 said:
Again, lack of memory is not an issue. The process hasn't been killed by OOM (jemalloc leaks) since I changed to tcmalloc back in 2019.
Click to expand...
I think it is, but don't understand it well enough myself to explain - but think it is what bob2112 is (maybe) getting at.

tcmalloc stopped the rush-to-OOM, but it didn't fix everything.

I think swap is used when there is memory pressure - you want to use more physical RAM than you've got. When you look at top and the amount of RAM you've got it doesn't seem like there should be memory pressure - but there is. But I'll stop here - I can't explain something I don't understand entirely myself. For my situation, going to FreeBSD 13.x and drastically cutting back what I thought I could give MySQL reduced the memory pressure to where I only have a very gradual swap increase.

On the machine with the buffer pool set to 5G, uptime of 30 days:
Code: 
Mem: 5559M Active, 21G Inact, 365M Laundry, 2741M Wired, 1543M Buf, 1749M Free
Swap: 3840M Total, 60M Used, 3780M Free, 1% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
 3859 mysql        47  20    0  6963M  5987M select   1 117.8H  15.92% mysqld

MySQL staying in its box, swap slightly used. My situation a bit different in that MySQL seems to get given swap instead of Inactive being cleared up fast enough (unless I run a RAM grabber script just before running a big import).

I think you are seeing that the swap IS being used by MySQL - so FreeBSD decided there wasn't enough space in physical RAM for the memory requested - so it's swap time.
 
rowan194 said:
I'm just trying to understand the apparent overlap (?) between swap and resident.
Click to expand...
I've done a bit more reading and it looks like RES does mean in physical memory/RAM, so that may mean the contents of swap are the result of more memory requests from MySQL- they can't be satisfied by physical RAM, so they get allocated to swap.

Over time that mounts up - between them MySQL and FreeBSD don't seem to realise that the swapped "stuff" isn't required.

That's what I was seeing with the imports - ran through all available RAM then all swap and then OOM.

Moving to FreeBSD 13.x fixed it for me, using tcmalloc fixed it for you - the OOM part. But the creeping memory usage issue is still there, it just doesn't happen as quickly. For me I also had to more-than-halve what I thought MySQL could use to keep the swap usage down.

Like you, I'd love the answer to "what data is actually swapped out?" - I didn't find anything to answer that question.

Got to move to MySQL 8.x this year (5.7 goes EOL in 2023), so will be "interesting" to see if that behaves any differently.
 
If you want to not to keep data in SWAP if it's not absolutely necessary then:

sysctl vm.pageout_update_period=0

But if the RAM frequently is not enough maybe it will make your system slower as it needs to pageout harder.
 
This thread is based on a false assumption that a page of data is stored either in a page of ram or in swap; it can be in both at the same time and so can contribute to both the RES values and the Swap value in top.

If the cpu tries to access a region of virtual address space that's fully swapped-out, the data is copied from swap into a page of ram that's then mapped into that address range. For a read, the data in swap is still valid, so the VM system leaves it there to avoid having to repeat the write at a later time.

A page of ram that's consistent with a backing store is called a "clean page", as opposed to a "dirty page" that has to be written-out before it can be recycled. If a system has too little memory, a good supply of clean pages helps the VM system to run more smoothly. Older versions of FreeBSD did seem more proactive at creating free pages well before they were needed.
 
You must log in or register to reply here.
Back
Top