ZFS Constantly out of swap space

Charlie Brown

Charlie Brown

Hey y'all hope everything is going well on your corner of life, I'm doing ok, today I open this thread because I have an issue with my memory, I don't understand because I have plenty : 16 GB but I constantly run out of swap space, so two things happen: my session freeze or reboot itself.

Here is an excerpt from /var/log/message

sh: 
Nov 15 23:43:40 nomonoru kernel: swap_pager: out of swap space
Nov 15 23:43:40 nomonoru kernel: swp_pager_getswapspace(13): failed
Nov 16 00:07:19 nomonoru kernel: swp_pager_getswapspace(3): failed
Nov 16 00:34:15 nomonoru kernel: swp_pager_getswapspace(9): failed
Nov 16 01:42:05 nomonoru kernel: swap_pager: out of swap space
Nov 16 01:42:05 nomonoru kernel: swp_pager_getswapspace(30): failed
Nov 16 03:09:22 nomonoru kscreenlocker_greet[46074]: in _pam_exec(): pam_sm_setcred: pam_get_authtok(): authentication token not available
Nov 16 13:10:42 nomonoru kscreenlocker_greet[87297]: in _pam_exec(): pam_sm_setcred: pam_get_authtok(): authentication token not available
Nov 16 13:11:03 nomonoru kernel: swp_pager_getswapspace(29): failed
Nov 16 13:16:36 nomonoru kernel: swp_pager_getswapspace(5): failed
Nov 16 15:08:18 nomonoru kscreenlocker_greet[20650]: in _pam_exec(): pam_sm_setcred: pam_get_authtok(): authentication token not available
Nov 16 15:33:36 nomonoru kernel: drmn0: [drm] *ERROR* Atomic update failure on pipe A (start=14147120 end=14147121) time 1000 us, min
Nov 16 15:33:42 nomonoru kernel: swp_pager_getswapspace(15): failed
Nov 16 15:34:43 nomonoru kernel: swap_pager: out of swap space
Nov 16 15:34:43 nomonoru kernel: swp_pager_getswapspace(1): failed
Nov 16 15:35:54 nomonoru kernel: swp_pager_getswapspace(11): failed
Nov 16 15:38:36 nomonoru kernel: swp_pager_getswapspace(10): failed
Nov 16 15:39:36 nomonoru kernel: drmn0: [drm] *ERROR* Atomic update failure on pipe A (start=14169488 end=14169489) time 2001 us, min
Nov 16 15:48:00 nomonoru kernel: swap_pager: out of swap space
Nov 16 15:48:00 nomonoru kernel: swp_pager_getswapspace(22): failed
Nov 16 15:57:43 nomonoru kernel: swap_pager: out of swap space
Nov 16 15:57:43 nomonoru kernel: swp_pager_getswapspace(1): failed
Nov 16 16:17:17 nomonoru kernel: swap_pager: out of swap space
Nov 16 16:17:17 nomonoru kernel: swp_pager_getswapspace(6): failed
Nov 16 16:29:36 nomonoru kernel: drmn0: [drm] *ERROR* Atomic update failure on pipe A (start=14349342 end=14349343) time 0 us, min 1073,
Nov 16 16:30:45 nomonoru kernel: swap_pager: out of swap space
Nov 16 16:30:45 nomonoru kernel: swp_pager_getswapspace(28): failed
Nov 16 17:05:37 nomonoru kernel: drmn0: [drm] *ERROR* Atomic update failure on pipe A (start=14478479 end=14478480) time 0 us, min 1073,
Nov 16 17:23:37 nomonoru kernel: drmn0: [drm] *ERROR* Atomic update failure on pipe A (start=14544383 end=14544384) time 3999 us, min

sh: 
Mem: 4375M Active, 6695M Inact, 311M Laundry, 3872M Wired, 1414M Buf, 429M Free
ARC: 1123M Total, 186M MFU, 535M MRU, 132K Anon, 8066K Header, 393M Other
426M Compressed, 898M Uncompressed, 2.11:1 Ratio
Swap: 2048M Total, 2048M Free

Code: 
 freecolor -m -o 
            total       used       free     shared    buffers     cached 
Mem:         15628       8278       7349          0          0          0 
Swap:         2048          0       2048
 
we would take a look at vmstat -m and vmstat -z — redirect the output of each to a file immediately after a reboot, and then when it starts doing this, run the commands again to a different file, and then compare the two.
 
For me the problem seems not related to ZFS.
Charlie Brown said:
swp_pager_getswapspace(30): failed
Click to expand...
The fact that none of your 2G swap is used, makes me wonder if swap is not just configured, but also accessable on your system.
The correct command to check is the following:
Code: 
# swapinfo  
Device          1K-blocks     Used    Avail Capacity
/dev/gpt/swap0    8388608        0  8388608     0%
/dev/gpt/swap1    8388608        0  8388608     0%
/dev/gpt/swap3    8388608        0  8388608     0%
/dev/#C:0x223     1048576        0  1048576     0%
Total            26214400        0 26214400     0%
Where the 4th line is a temporary swap I created and than removed the disk for demonstration purposes.
What does swapinfo show on your system?
 
You have 16 GB RAM, and 2 GB swap. According to the size of you ZFS pool(s), the amount of data to be handled, and what your machine else is doing at the same time, this can be a bit meagre.
 
Maybe the swap config on disk is damaged? Just enough there to mount it, not enough to allocate from it? Or is it a swap file?
 
SirDice said:
Besides ZFS, what else is running on that system?
Click to expand...
I attached a .txt file to this message called psaux.txt.
tingo said:
Is this your desktop (or laptop) FreeBSD machine? Do you run a web browser (or two) with many windows open?
Click to expand...
Yes my Firefox browser have a lots of tabs I did not know that could be an issue ?
k.jacker said:
What does swapinfo show on your system?
Click to expand...

here is the output :
Code: 
free@nomonoru:~ $ swapinfo
Device          1K-blocks     Used    Avail Capacity
/dev/da1p3        2097152  1047712  1049440    50%
free@nomonoru:~ $ gpart show
=>        34  1000215149  nda0  GPT  (477G)
          34        2014        - free -  (1.0M)
        2048      204800     1  efi  (100M)
      206848       32768     2  ms-reserved  (16M)
      239616   510976000     3  ms-basic-data  (244G)
   511215616   487424000     4  ms-basic-data  (232G)
   998639616        2048        - free -  (1.0M)
   998641664     1570816     5  ms-recovery  (767M)
  1000212480        2703        - free -  (1.3M)

=>        6  976746229  da0  GPT  (3.6T)
          6  976746229    1  freebsd-ufs  (3.6T)

=>        40  1953525088  da1  GPT  (932G)
          40      532480    1  efi  (260M)
      532520        1024    2  freebsd-boot  (512K)
      533544         984       - free -  (492K)
      534528     4194304    3  freebsd-swap  (2.0G)
     4728832  1948794880    4  freebsd-zfs  (929G)
  1953523712        1416       - free -  (708K)
Click to expand...

atax1a said:
we would take a look at vmstat -m and vmstat -z
Click to expand...
I attached accordingly to each vmstat1 vmstat2 to this messsage.
 

Attachments

freebsd doesn't have any specific format to the swap partition (no mkswap as on linux), so long as it shows up in swapctl -l it should be used.

also those files are only the baseline readings, keep an eye on your kernel messages and run those commands again when you're seeing the swap-space errors show up.
 
Charlie Brown said:
Is there a way to check if the swap partition's healthy ?
Click to expand...
As k.jacker suggested, please show us the output of the swapinfo command.

We are interested in seeing the "Device", and knowing if it's a swap file or a disk partition.

Swap files, especially if resident in a ZFS file system, will take us in a completely different direction to swap resident on a disk partition.
 
gpw928 said:
As k.jacker suggested, please show us the output of the swapinfo command.

We are interested in seeing the "Device", and knowing if it's a swap file or a disk partition.

Swap files, especially if resident in a ZFS file system, will take us in a completely different direction to swap resident on a disk partition.
Click to expand...
Hi thanks for your answer, I already did provide the output asked previously, you have to extend the quotations the see that output.
 
k.jacker said:
The fact that none of your 2G swap is used
Click to expand...
that was right after the computer reboot itself, the same command now that it is live since at least 24h will have a different output:
Code: 
free@nomonoru:~ $ freecolor -m -o
             total       used       free     shared    buffers     cached
Mem:         15628       7116       8511          0          0          0
Swap:         2048       1011       1036
 
Charlie Brown said:
maybe i should allocate more swap space ?
Click to expand...
Besides you better also check what the others suggest, (you better get to the problem's root), this was what I would try first, yes: increase swap (8 GB) and see if this helps - quick'n'dirty. But this was only temporary, and no real solution. As long as don't figured out what's going on on your machine, increasing swap only may help shorttermed, but being delusive: You may think "more swap - violá, problem solved." But later you will find: "Damned. Something else is going on." (For example, your machine became slow.) Because the problem was not solved.

You need to figure out if you're really running out of memory.
If so, more swap helps only temporary, but not satisfactory. Then you need to think of how to reduce memory usage, or increase memory size. If not, and your problem lies somewhere else, more swap wouldn't bring any useful result at all.
 
atax1a said:
we would take a look at vmstat -m and vmstat -z — redirect the output of each to a file immediately after a reboot, and then when it starts doing this, run the commands again to a different file, and then compare the two.
Click to expand...
VladiBG said:
ccording your ps aux RSS the top two memory processes are:
librewolf 2 814.82 MB (web browser)
baloo 1 334.48 MB (search index)
Click to expand...
But this command show that ZFS take much of the memory no ? :
Code: 
vmstat -m | sort -k3 -nr
         solaris 1184855 267790672 274058487 16,32,64,128,256,384,512,1024,2048,4096,8192,16384,32768,65536
 
It take the rest of the memory up to vfs.zfs.arc.max size. The problem is when there's not enough memory and some process try to allocate more memory the ARC cannot free so fast the required memory and some allocated memory going into swap.

systat -zarc

docs.freebsd.org

Chapter 22. The Z File System (ZFS)

ZFS is an advanced file system designed to solve major problems found in previous storage subsystem software
docs.freebsd.org docs.freebsd.org
 
You must log in or register to reply here.
Back
Top