swap always full / swap_pager_getswapspace

S

Sebastian

Hey,

I upgraded my system to 11.2 a month ago. Since than my, swap partition is always full.

Dmesg shows a lot of this events.

Code: 
swap_pager_getswapspace(13): failed
swap_pager_getswapspace(8): failed
swap_pager_getswapspace(8): failed
swap_pager_getswapspace(6): failed
swap_pager_getswapspace(8): failed
swap_pager_getswapspace(3): failed
swap_pager_getswapspace(16): failed

Code: 
Device          1K-blocks     Used    Avail Capacity
/dev/ada0p4       2097152  2093892     3260   100%


Code: 
last pid:  2638;  load averages:  0.41,  0.29,  0.26                                                                                                                       up 18+20:00:33  18:49:50
81 processes:  1 running, 78 sleeping, 2 zombie
CPU:  0.4% user,  0.0% nice,  0.1% system,  0.0% interrupt, 99.5% idle
Mem: 424M Active, 1337M Inact, 7032M Wired, 6991M Free
ARC: 3175M Total, 807M MFU, 1657M MRU, 18M Anon, 59M Header, 634M Other
     1692M Compressed, 4669M Uncompressed, 2.76:1 Ratio
Swap: 2048M Total, 2045M Used, 3132K Free, 99% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
26581    975       25  52    0  3922M   105M uwait   2  94:01   0.82% mongod
26455    975      858  52    0  2852M   430M uwait   0  18:09   0.09% java
2532    933       94  52    0  2158M   302M uwait   2 119:39   0.29% java
6597 unbound       1  20    0  1245M   104M kqread  0  15:11   0.00% unbound
26463    922       14  52    0  1167M 11948K uwait   7  23:30   0.84% mongod
12209 root         28  20    0  1083M  4996K kqread  1   8:08   0.03% bhyve
4903    907       22  20    0   338M   169M uwait   6  90:43   0.67% influxd
9861 root          1  20    0   170M  2188K select  2   0:25   0.00% smbd
10114 root          1  20    0   170M  3164K select  5   0:03   0.00% smbd
4933 ssmtp        44  20    0   146M 66500K kqread  3  36:14   0.09% python3.6
2584 sharky        7  20    0   132M 30112K select  3 342:26   0.24% python2.7
9986 root          1  20    0   128M  2068K select  1   0:02   0.00% smbd
9981 root          1  20    0   127M   892K select  1   0:02   0.00% smbd

Code: 
sh freebsd-memory.sh
SYSTEM MEMORY INFORMATION:
mem_wire:        7373950976 (   7032MB) [ 44%] Wired: disabled for paging out
mem_active:  +    445153280 (    424MB) [  2%] Active: recently referenced
mem_inactive:+   1403105280 (   1338MB) [  8%] Inactive: recently not referenced
mem_cache:   +            0 (      0MB) [  0%] Cached: almost avail. for allocation
mem_free:    +   7327490048 (   6988MB) [ 44%] Free: fully available for allocation
mem_gap_vm:  +      -221184 (      0MB) [  0%] Memory gap: UNKNOWN
______________ ____________ ___________ ______
mem_all:     =  16549478400 (  15782MB) [100%] Total real memory managed
mem_gap_sys: +    480661504 (    458MB)        Memory gap: Kernel?!
______________ ____________ ___________
mem_phys:    =  17030139904 (  16241MB)        Total real memory available
mem_gap_hw:  +    149729280 (    142MB)        Memory gap: Segment Mappings?!
______________ ____________ ___________
mem_hw:      =  17179869184 (  16384MB)        Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used:        8449273856 (   8057MB) [ 49%] Logically used memory
mem_avail:   +   8730595328 (   8326MB) [ 50%] Logically available memory
______________ ____________ __________ _______
mem_total:   =  17179869184 (  16384MB) [100%] Logically total memory

I can't image that my 16gb ram aren't sufficent for my system. An as you can see from top . My system has 6991M free memory , so it should't swap , right ?

Any hints ?

best regards, sebastian
 
Sebastian said:
I can't image that my 16gb ram aren't sufficient for my system.
Click to expand...
It can be. But it depends on how your services are configured. If you run a database and tell it to use 32GB your 16GB isn't going to be enough.

Sebastian said:
My system has 6991M free memory , so it should't swap , right ?
Click to expand...
Common mistake. It has 6991 MB free because the system swapped out a bunch of unused stuff that was only keeping memory occupied. Things can get swapped out preemptively in order to provide actual memory immediately should an active process request it. Swap usage in and of itself is not a problem and is typically done to increase performance for the active processes. Why keep tons of stuff locked in memory if it's rarely used?

Keep in mind that you have various things running all competing for a chunk of that 16GB. I see Java, MongoDB, bhyve and ARC. All fairly memory hungry processes.
 
Having thought this through a bit, do you have only one VM running or more? I did notice on my bhyve server that there's probably a memory leak somewhere on 11.x. After booting the server no swap is used, after about a day or so half of the swap was in use. That's fine, things kept working as it should without any adverse affect. But leaving this running for 2 or 3 weeks I could see the memory graphs slowly showing less and less memory and using more and more swap. It's not obvious if you look at a daily graph but it's obvious if you look over a monthly period. After updating the server from 11-STABLE to 12-STABLE I can see a definite improvement in memory usage. I still have around half of my swap used but this seems to be quite stable now over several weeks.

Looking back at the commit logs of 12 I see there have been many improvements for bhyve. So if you want to use bhyve more I can definitely recommend updating to 12.0.
 
Thank you SirDice. I learned a lot from your posts. I don't have any problems at the moment , maybe the machine is a bit slow , but it is only a home server project.

And yes I'm only running one byhve instance .

But I'm running it for exactly this kind of situations where I can troubleshoot such a problem. I want to investigate it further without updating to freebsd 12 . If the server is stable again , I will update soon.

I will update the thread if I find something useful or how I solved the problem.
 
Instead of upgrading to 12 you could start with 11-STABLE. 11-STABLE is basically a beta of the next 11.x release.

For the time being you could limit ARC, the default setting would allow it to use all memory minus 1GB. As you have a bunch of other processes running they're all going to be competing for the free memory.
 
You must log in or register to reply here.
Back
Top