Swap use growing even though plenty of free RAM

B

bschwand

top gives me:
Code: 
last pid: 61466;  load averages:  1.27,  1.00,  0.95                                                                                  up 159+04:03:05 15:22:39[/ICODE]
158 processes: 1 running, 157 sleeping
CPU:  4.3% user,  0.0% nice,  0.3% system,  0.0% interrupt, 95.4% idle
Mem: 938M Active, 66G Inact, 4732M Laundry, 281G Wired, 21G Free
ARC: 239G Total, 96G MFU, 110G MRU, 13M Anon, 2567M Header, 31G Other
     164G Compressed, 426G Uncompressed, 2.59:1 Ratio
Swap: 4096M Total, 2586M Used, 1510M Free, 63% Inuse

There is 384GB physical RAM, 21G free and plenty used by the ARC which I assume would be freed if processes need it.
Swap use is growing slowly. Why is it even used at all ?
 
a bunch of stuff, some storage nodes. Neither mysql or mariadb.

But why would it matter ?

my point is that intuitively I'd think if there is free RAM then no swap would be used. And before any swap is even used, I assume some ARC would be flushed and RAM freed. Is that not how it's supposed to work ?
 
Possibly a memory leak somewhere? Memory allocated but unused that the kernel pages out. I've seen it many times on Linux and even prepared a script to identify the culprit but unfortunately it does not work on FreeBSD.
 
You have 93% of your RAM in use.

While the percentage of swapspace used is high you only have 1.5 GB in absolute terms. That is a very small amount compared to RAM in use.
 
cracauer@ said:
You have 93% of your RAM in use.

While the percentage of swapspace used is high you only have 1.5 GB in absolute terms. That is a very small amount compared to RAM in use.
Click to expand...
Sure 93% in use, which is good otherwise it's wasted. But notice it's used by ARC. It could be flushed if processes need memory.
So still my question why anything in swap at all ? it makes no sense to me.
 
bschwand said:
Sure 93% in use, which is good otherwise it's wasted. But notice it's used by ARC. It could be flushed if processes need memory.
So still my question why anything in swap at all ? it makes no sense to me.
Click to expand...

Evicting ARC is actually a fairly involved activity. It might slack off and the pager gets impatient.

Which version of ZFS are you running? A threaded/parallel eviction was introduced... I forgot but recently.
 
bakul said:
My guess is ARC doesn't free up memory fast enough. Try setting vfs.zfs.arc.max to something like half of physmem and experiment. sysutils/zfs-stats can be used to see how well ARC is used.
Click to expand...
I would say it's well used

Code: 
$ zfs-stats -A -E

------------------------------------------------------------------------
ZFS Subsystem Report                            Fri Mar 21 14:23:55 2025
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                97.48   m
        Mutex Misses:                           36.30   k
        Evict Skips:                            11.81   m

ARC Size:                               63.68%  243.86  GiB
        Target Size: (Adaptive)         63.70%  243.93  GiB
        Min Size (Hard Limit):          33.43%  128.00  GiB
        Max Size (High Water):          2:1     382.93  GiB
        Compressed Data Size:                   166.97  GiB
        Decompressed Data Size:                 433.15  GiB
        Compression Factor:                     2.59

ARC Size Breakdown:
        Recently Used Cache Size:       46.96%  114.55  GiB
        Frequently Used Cache Size:     53.04%  129.38  GiB

ARC Hash Breakdown:
        Elements Max:                           16.79   m
        Elements Current:               67.32%  11.30   m
        Collisions:                             348.59  m
        Chain Max:                              7
        Chains:                                 851.13  k

------------------------------------------------------------------------

ARC Efficiency:                                 63.36   b
        Cache Hit Ratio:                99.60%  63.11   b
        Cache Miss Ratio:               0.40%   250.85  m
        Actual Hit Ratio:               99.21%  62.86   b

        Data Demand Efficiency:         95.29%  3.84    b
        Data Prefetch Efficiency:       2.11%   31.79   m

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             0.37%   231.75  m
          Most Recently Used:           2.62%   1.65    b
          Most Frequently Used:         96.99%  61.21   b
          Most Recently Used Ghost:     0.00%   2.12    m
          Most Frequently Used Ghost:   0.02%   14.72   m

        CACHE HITS BY DATA TYPE:
          Demand Data:                  5.80%   3.66    b
          Prefetch Data:                0.00%   671.22  k
          Demand Metadata:              91.26%  57.60   b
          Prefetch Metadata:            2.94%   1.85    b

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  72.10%  180.87  m
          Prefetch Data:                12.41%  31.12   m
          Demand Metadata:              3.63%   9.10    m
          Prefetch Metadata:            11.86%  29.76   m

------------------------------------------------------------------------

Max size seems a bit excessive given the physical RAM installed is 384GB....
I set vfs.zfs.arc.max to 320GB now. We shall see.

Code: 
$ zfs version
zfs-2.2.0-FreeBSD_g95785196f
zfs-kmod-2.2.0-FreeBSD_g95785196f
 
I'm not going to suggest that I understand all the nuances of modern Unix memory management systems, because they are very complex. But I do think that generalisations can help.

You have 384 GB physical RAM, with 93% in use. Having "plenty of ram" is a concept subject to dynamic change.

Consider a long running process which allocates and modifies vast amounts of memory which is "used" once, during initialisation, and never referenced again. If the memory management system comes under any sort of pressure, then these "unused" memory resident pages may present the "best option" for the memory manager to page out.

So using swap can be a good thing. And having only 1% of total RAM allocated to swap is restricting the full set of options available to the memory manager.
 
bschwand said:
if you run out, bad things happen
Click to expand...
Only if you don't have any main memory pages that can be reclaimed.

and I don't like the idea of having some memory paged out to disk when I have plenty of RAM
Click to expand...
But that RAM might be able to do a job that is more profitable than the pages that are currently in swap.

You seem to be under the misconception that swap should only be used then the system has completely run out of RAM.
 
Which FreeBSD version do you use? I'm wondering that no one have asked it so far in this thread.
For me I have had significant improvements with ARC after upgrading from 13.x to 14.
In 13.x I always needed to limit Arc's memory use, In 14 not anymore at all. Arc is just doing it's job without me.
With 32GB Ram I'm far away from your 384GB but have the same amount of swap. The 4GB swap are usually not touched at all.
 
meaw229a said:
Which FreeBSD version do you use? I'm wondering that no one have asked it so far in this thread.
For me I have had significant improvements with ARC after upgrading from 13.x to 14.
In 13.x I always needed to limit Arc's memory use, In 14 not anymore at all. Arc is just doing it's job without me.
With 32GB Ram I'm far away from your 384GB but have the same amount of swap. The 4GB swap are usually not touched at all.
Click to expand...
I run FreeBSD 14
 
ralphbsz said:
Only if you don't have any main memory pages that can be reclaimed.


But that RAM might be able to do a job that is more profitable than the pages that are currently in swap.

You seem to be under the misconception that swap should only be used then the system has completely run out of RAM.
Click to expand...
I think it should be a last resort to page RAM out to disk, since it is so much slower. Isn't that how it generally works ?
I suppose what ends up in swap is not necessarily moved back into RAM when the RAM frees up, so it might be there lingering from a previous high RAM pressure situation ?
 
dark.initr0 said:
maybe this article can shed some light on your question
https://klarasystems.com/articles/exploring-swap-on-freebsd/

But if you don't want to use swap - turn it off
Click to expand...
excellent article, thanks
I must be seeing this case

It is thus common to see moderate swap space usage even when plenty of free memory is available[3]: at some point in the past, demand for free memory triggered page-outs to swap, and the swapped-out data remained valid.
 
bschwand said:
I think it should be a last resort to page RAM out to disk, since it is so much slower. Isn't that how it generally works ?
Click to expand...

Well, that depends. Linux does what you imply and is very hesitant to move read-write pages out to swapspace. But of course that means that it drops readonly pages very frequently. I found that behavior unfortunate and prefer FreeBSD's.

I suppose what ends up in swap is not necessarily moved back into RAM when the RAM frees up, so it might be there lingering from a previous high RAM pressure situation ?
Click to expand...

Correct. Plus, your RAM didn't free up yet.
 
bschwand said:
I think it should be a last resort to page RAM out to disk, since it is so much slower. Isn't that how it generally works ?
Click to expand...
That is clearly not optimal: A very rarely accessed page should not be in RAM; because RAM is fast but at a premium, it should be used for pages that are frequently accessed. Even if the page that is kept in RAM is clean.

Other operating systems implement this differently. Often less than optimally for mixed workloads. This is a huge area of R&D; one joke I heard at a CS conference several decades ago was "2000 papers about memory management in the year 2000".
 
You must log in or register to reply here.
Back
Top