out of swap

[YuryG]

I've seen some similar problem but could not find it here now. So, may be it's a different question.

During uptime of my computer I've noticed the error in dmesg:

swap_pager_getswapspace(20): failed
swap_pager_getswapspace(18): failed
swap_pager_getswapspace(9): failed
swap_pager_getswapspace(3): failed

But the top shows no big processes, only Wired and Inact parts of memory are huge. I have 16GiB of RAM installed. 11.2-STABLE FreeBSD r346801 amd64
So, top some hours after the error in dmesg sorted by size:

Mem: 976M Active, 5220M Inact, 380M Laundry, 8605M Wired, 1538M Buf, 490M Free
Swap: 6144M Total, 5941M Used, 203M Free, 96% Inuse

  PID USERNAME       THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
3683 yury             2  20    0   183M 32648K uwait   3 138:49   1.41% Xorg
60116 yury             4  20    0   164M   134M kqread  5   1:33  14.43% transmission-d
3760 yury             4  20    0 94748K 20060K select  7  80:05   1.81% mate-terminal
53244 yury             1  20    0 78404K  5676K nanslp  2   0:58   0.05% mplayer
70585 yury             1  20    0 78404K  5656K select  1  18:35   0.08% mplayer
3702 yury             4  20    0 61660K 18844K select  1  27:41   0.25% marco
3770 root             1  20    0 42720K  6068K select  3   0:30   0.00% httpd
  936 root             1  20    0 37312K 32432K physrd  3   0:25   4.07% ntfs-3g
3884 yury             1  20    0 25292K  9588K fu_ans  6   0:48   2.33% mc

You see, nothing's so big!

Why this huge Wired and Inact? May be there's a way to flush them? Or how to work it out?

 

[YuryG]

It happen to have something with Xorg. For when I logged out from Xs (MATE environment, namely), it freed 14GiB!!! almost all Wired and Inact memory was freed.
In Xorg.0.log I see constant probes of my monitor (VGA, I did not try to unplug it!) every 4.5 hours on average since start, nothing more.

 

[tingo]

My pet peeve with the current way (as in FreeBSD 11.x, I haven't tested 12.x long enough to know if things have changed) swap handling (for lack of a better phrase) works in FreeBSD is this:
1) notice that the machine is using swap, but has a lot of free memory
2) run # swapoff -a && swapon -a but swapoff complains that it is unable to do its job because of too little free memory
3) start a large program (example Falkon), wait some seconds after it has started, and stop it again
4) run # swapoff -a && swapon -a this time it succeeds

Why was swapoff(8) unable to do its job the first time?

 

[Crivens]

The swapoff will need to have real memory to read in the data from the swap space to drop. So when you have a lot of memory in use and/or wired (ZFS!) then this will not work.

 

[rsronin]

A few months ago I had similar issues. The culprit was a theme setting in KDE5.
Thread

 

[YuryG]

The swapoff will need to have real memory to read in the data from the swap space to drop. So when you have a lot of memory in use and/or wired (ZFS!) then this will not work.

For me, I do not use ZFS, and no Free memory (as seen from top output), only plenty of Inact and Wired.
So, the problem is these fractions, not swap as it is _needed_ later.

 

[tingo]

The swapoff will need to have real memory to read in the data from the swap space to drop. So when you have a lot of memory in use and/or wired (ZFS!) then this will not work.

Yes - it will. However, how come "program X" (Falkon in my example) is able to not only allocate enough memory to function, but also cause some sort of housekeeping that make enough real memory available for swapoff to do its job after "program X" has exited? And why isn't such housekeeping programmed into swapoff itself?
(ZFS is not relevant for my example: none of these machines have zfs)

 

[PMc]

The wired memory is an interesting thing, and I would love to have a means to figure out which kind of subsystem is actually claiming that. But it is most certainly not maintained by the pagedaemon - due to their nature of being wired (i.e. locked in memory) those pages should be excluded from such maintenance.
As I have observed, there are means to reduce the wired mem when the system is in need of ram: the count may suddenly go down for no obvious reason. But, since these pages are supposed to be excluded from regular page maintenance, it would be entirely in the discretion of those subsystem that do claim the wired memory, to free it again when that seems fit, i.e. when they notice that the system is low on ram.
The pagedaemon (or in this case, swapon/off) cannot do much about that.

I my case, I have solved problems by limiting ZFS to a sensible amount of RAM, and compensating for the lost performance with an SSD. It is not the way I like it, because that SSD has already 10 times more writes than reads, but it seems to help for now

Why this huge Wired and Inact? May be there's a way to flush them? Or how to work it out?

Inact is not an issue, these are maintained by pagedaemon and get free as soon as they are needed (not earlier, as that would be a waste).
But if You indeed are NOT using ZFS, then this confirms my assumption that there must be some other candidate(s) beyond ZFS, which grossly abuse Wired.

 

[YuryG]

…Inact is not an issue, these are maintained by pagedaemon and get free as soon as they are needed (not earlier, as that would be a waste).
But if You indeed are NOT using ZFS, then this confirms my assumption that there must be some other candidate(s) beyond ZFS, which grossly abuse Wired.

How's “Inact not an issue” if I get “out of swap” error while 5GiB in Inact? Or does that error mean that it wanted not to disturb Inact, but have to do it since swap is out of space?
Yes, I am definitely not using ZFS.
By the way, after last reboot I'm now seeing (the problem was after some days uptime in Xs)

Mem: 1482M Active, 10G Inact, 947M Laundry, 2428M Wired, 1538M Buf, 207M Free
Swap: 6144M Total, 346M Used, 5798M Free, 5% Inuse
 Displaying threads as a count
  PID USERNAME       THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 4839 yury            61  21    0  2024M   559M select  1  42:47  11.08% firefox
 4842 yury            49  20    0  1863M   416M select  4  35:45   1.56% firefox
28031 yury            49  20    0  1731M   335M select  1  35:00   0.24% firefox
42077 yury            46  20    0  1633M   302M select  1   0:46   0.24% firefox
91478 yury            37  44    0  1542M   261M select  7   2:18  34.67% firefox
 4843 yury            30  20    0  1624M   236M select  7  13:37   0.24% firefox
91290 yury            39  20    0  1563M   234M select  3  12:36   1.27% firefox
43642 yury            37  20    0  1475M   210M select  1   0:30   0.54% firefox
28215 yury            38  20    0  1521M   195M select  0   5:25   0.68% firefox

 

[Crivens]

I would need to check the docs, or the source code, but I assume the swapoff() system code is running at so low a level that it needs free memory and is not allowed to lock the other memory lists. Swapoff may read A LOT of data in one operation, and blocking all memory lists for so long is out of the question.

But I would check this, it's from the top of my head.

 

[PMc]

How's “Inact not an issue” if I get “out of swap” error while 5GiB in Inact? Or does that error mean that it wanted not to disturb Inact, but have to do it since swap is out of space?

Because Inact is maintained. When space is needed, the pagedaemon will see where to get that page, based on the recent usage of the pages. And I suppose it knows what to do, so I don't worry about that.

But then, Inact just means that the page wasn't used for some time. I might suppose it does NOT mean that the page doesn't contain data that is in use - and if it contains such data (aka, is "dirty"), then that has to go to swap first. The pagedaemon doesn't care about that, it just decides what is the supposedly best action in terms of performance.

 

[zirias@]

The wired memory is an interesting thing, and I would love to have a means to figure out which kind of subsystem is actually claiming that.

vmstat -m gives you some insight, it shows all the memory allocated by kernel malloc(9), sorted by type argument.

 

[PMc]

vmstat -m gives you some insight, it shows all the memory allocated by kernel malloc(9), sorted by type argument.

That one did work, but then they came up with UMA ( vmstat -z), and I didn't yet get a real clue about how that is to be interpreted - but some discussion on the mailinglists seem to state that UMA can allocate memory (and thereby push other things out or into swap) which isn't even used.