Solved FreeBSD 10.2 without swap

jarek6

jarek6

Hi
I recently upgraded my machines from old 7.4 to latest FreeBSD 10.2. Most of machines was fully reinstalled + reinstalled/rebuild all ports/packages.

At earlier versions - eg. 7.x I never used swap as my servers have plenty of RAM (yes, RAM is cheap and much faster than HDD, even SSD). After upgrade to 10.2 I see dying processes:
Code: 
Dec 24 21:13:56 Amnesiac kernel: pid 780 (named), uid 0, was killed: out of swap space
Dec 24 21:14:03 Amnesiac kernel: pid 41059 (screen), uid 0, was killed: out of swap space
Dec 24 21:15:07 Amnesiac kernel: pid 834 (ntpd), uid 0, was killed: out of swap space
Dec 24 21:15:07 Amnesiac kernel: pid 46968 (bash), uid 666, was killed: out of swap space
Dec 24 21:17:02 Amnesiac kernel: pid 47106 (named), uid 0, was killed: out of swap space
'named' and 'ntpd' are most affected
Also simple test cat * | egrep -i mail at directory where I have 10 files, each is around 2GB killing processes due to 'out of swap space'

MALLOC_PRODUCTION=YES in /etc/make.conf + recompile world and kernel didn't resolved my issue. Temporary I enabled swap ... on MD based RAM disk which resolved my problems. But I'm looking less dirty solution.

Example host running sshd, named, dhcpd, ntpd for LAN purposes:
Code: 
FreeBSD Amnesiac 10.2-STABLE FreeBSD 10.2-STABLE #0 r291457M: Sun Nov 29 23:49:51 CET 2015     root@Amnesiac:/usr/obj/usr/src/sys/AMNESIAC  amd64

$ free
             total       used       free     shared    buffers     cached
Mem:          1981        536       1445          0          0         22
Swap:         4096        232       3863

$ top
last pid: 17683;  load averages:  0.24,  0.14,  0.09                                                                                                                 up 36+08:09:11  08:37:19
53 processes:  2 running, 51 sleeping
CPU:     % user,     % nice,     % system,     % interrupt,     % idle
Mem: 105M Active, 1396M Inact, 430M Wired, 22M Cache, 213M Buf, 28M Free
Swap: 4096M Total, 233M Used, 3863M Free, 5% Inuse
 
Even if you have plenty of memory don't disable swap. Every modern OS uses some swap. Mostly for house keeping. Swap usage in and of itself isn't an issue. It's excessive swapping that causes performance problems.

MALLOC_PRODUCTION=YES in /etc/make.conf + recompile world and kernel didn't resolved my issue.
Click to expand...
Because it's already enabled by default. As far as I know only -CURRENT uses it.
 
We don't use swap in Windows 2012
We don't use swap in Linux
We don't use swap in Solaris.
We don't use swap in FreeBSD 7.4.

Above OS'es just works without swap as expected.
You can write that all above are NOT modern OS'es... But this post is not about writing which OS is better.

More - Swap is NOT recommended for Virtual Machines as it's sacrifice Storage I/O.
99% of our storage is NFS based (shared storage), so I really don't want have SWAP over network.
I have a lot of RAM in my production machines - eg. 384GB RAM.

I don't catch your understanding why Swap is better than RAM and why FreeBSD kernel is moving some memory pages to swap as it has 99% free memory. RAM is better, as it's faster. Modern computers has plenty of memory - even my 12" laptop have 32GB RAM which was opposite in the past - 4-8GB per server.

Today I did two simple tests. On HP DL360p Gen8 machine with 256GB RAM (yeah, really plenty of RAM) running FreeBSD 10.2 (GENERIC kernel) running just OpenSSHd I have 1.9GB file called 'p1' (exact size=1987253400 bytes). I'm executing command cat p1 p1 p1 p1 p1 p1 p1 > p.out from Bash logged via SSH. Test was done two times - one time with swap and second one without swap. Swap is created via RAM disk (md0).

1) With swap: Success. Swap usage was 2MB during executing command:

Code: 
Tue Jan 5 13:00:06 CET 2016 /dev/md0            970336        0   970336     0%
Tue Jan 5 13:00:07 CET 2016 /dev/md0            970336        0   970336     0%
Tue Jan 5 13:00:08 CET 2016 /dev/md0            970336       40   970296     0%
Tue Jan 5 13:00:09 CET 2016 /dev/md0            970336       40   970296     0%
Tue Jan 5 13:00:10 CET 2016 /dev/md0            970336       40   970296     0%
Tue Jan 5 13:00:11 CET 2016 /dev/md0            970336     1600   968736     0%
Tue Jan 5 13:00:12 CET 2016 /dev/md0            970336     2184   968152     0%
Tue Jan 5 13:00:13 CET 2016 /dev/md0            970336     2184   968152     0%
Tue Jan 5 13:00:14 CET 2016 /dev/md0            970336     2184   968152     0%
Tue Jan 5 13:00:15 CET 2016 /dev/md0            970336     2184   968152     0%
Tue Jan 5 13:00:16 CET 2016 /dev/md0            970336     2192   968144     0%
Tue Jan 5 13:00:17 CET 2016 /dev/md0            970336     2192   968144     0%

RAM usage was maximum around 1GB. So kernel was using 2MB of SWAP having 255GB free RAM...

2) Without swap: Didn't complete the test.

Code: 
ssh process was killed by kernel due to out of swap memory afer 15 seconds:
Jan  5 13:20:27 test kernel: pid 1037 (sshd), uid 666, was killed: out of swap space
p.out file size was 3GB of expected 12GB (25% complete).
RAM usage was maximum 1GB, with 255GB free RAM.


Conclusion : Something is broken in memory management as kernel was looking for 2MB swap having 99,5% (255GB) free RAM. Imagine 2MB vs 255GB (!). Is it true "Modern OS Memory Management" ? I believe it's just a bug.
 
Last edited by a moderator:
jarek6 said:
I believe it's just a bug.
Click to expand...

A bug, perhaps, in the cat command implementation, assuming that the system will do whatever is necessary ("swap", even) to allow buffering a whole lot of data... you are asking it to do that with 7 x 2Gb files...

I'd bet telling cat not to buffer things with cat -u would make it run with a minimal amount of memory, and no swap space.
 
Let's check if cat -u changes something

Code: 
[20](root@test /mnt)# cat -u p1 p1 p1 p1 p1 p1 p1 > p.out
Connection to 10.254.254.149 closed by remote host.
Connection to 10.254.254.149 closed.

No, it didn't.

Code: 
Jan  5 22:32:54 test kernel: pid 1300 (sshd), uid 0, was killed: out of swap space
Jan  5 22:32:56 test kernel: pid 1302 (sshd), uid 666, was killed: out of swap space

But AFAIK it's not bug in cat. See my first post where we have dying processes like ntpd, named in situation where we still have enough free RAM. My workaround was creating 512MB RAM Disk, and use it as SWAP. Yes, I'm using RAM as SWAP. It's sounds funny, but helps. I'm looking for more clean solution.
 
Last edited by a moderator:
I also noticed excessive swap usage while having plenty of available RAM. FreeBSD 10-STABLE, running poudriere with 4 builders on 4-core CPU, 16 GB of RAM, 8 GB swap partition. At one point 60% of swap partition was used.
 
The kernel has a compile-time option for disabling swap entirely (from /usr/src/sys/conf/NOTES):
Code: 
# Disable swapping of stack pages.  This option removes all
# code which actually performs swapping, so it's not possible to turn
# it back on at run-time.
# 
# This is sometimes usable for systems which don't have any swap space
# (see also sysctls "vm.defer_swapspace_pageouts" and
# "vm.disable_swapspace_pageouts")
# 
options        NO_SWAPPING
 
I use md(4) to combine a RAM file system and swap space for compiling ports, or other file systems. It would seem redundant to set complete virtual swap space over RAM (when it is likely supposed to do that without swap), and I haven't tried this. I would think the program would give an error to try that.
 
Didn't helped. Added
Code: 
options         NO_SWAPPING
to kernel config, recompiled & installed & rebooted.

After that 'cat' test and again same result :(

Code: 
pid 594 (vmtoolsd), uid 0, was killed: out of swap space
pid 786 (sshd), uid 666, was killed: out of swap space
pid 744 (sshd), uid 0, was killed: out of swap space
pid 746 (sshd), uid 666, was killed: out of swap space

BTW. I didn't found it earlier As I was grepping /usr/src/sys/amd64/conf/NOTES instead mentioned by You /usr/src/sys/conf/NOTES. Next question which NOTES file is more valid for amd64 kernel?
 
Last edited by a moderator:
Did the test on bigger machine (256GB RAM) with new kernel - option NO_SWAPPING - and it worked :)
It fails on machine with 8GB RAM (VM) (my earlier post) - killed a lot of processes (ssh etc. ), but here really something eaten all memory according to 'top'

Something was eating RAM, finally up to all memory:

top during the 'cat' operation:
Mem: 24M Active, 2729M Inact, 1777M Wired, 1656M Buf, 3413M Free
Swap:
Code: 
  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
  933 root          1  52    0 12364K  2456K wdrain  3   0:04  27.10% cat

And another top but few seconds later
Mem: 24M Active, 5415M Inact, 1777M Wired, 1656M Buf, 727M Free
Swap:
Code: 
  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
  933 root          1  49    0 12364K  2456K CPU0    0   0:08  32.86% cat

How to investigate what eaten all memory ? Top doesn't seem to show any process was so memory consuming. RES of all processes running are less than 10000K.
 
Last edited by a moderator:
I did one more test on VM (8GB RAM).
Enabled 'swap' in RAM:

mdconfig -s 1g -u md0
swapon /dev/md0

Did 'cat' test. Worked. top shows very similar results to NO_SWAPPING - something was eating memory.
'Free' in top was counting to zero, after zero top starts to show some different outputs, eg:

Code: 
Mem: 29M Active, 6046M Inact, 1762M Wired, 43M Cache, 1654M Buf, 54M Free
Swap: 1024M Total, 1224K Used, 1023M Free

comparing to old one:

Mem: 24M Active, 5953M Inact, 1777M Wired, 1656M Buf, 189M Free

Conclusion#1: 'option NO_SWAPPING' is OK if you have a really lot of RAM, but it looks FreeBSD 10 is not prepared/tested well for that option
Conclusion#2: instead of 'option NO_SWAPPING' it's better to create SWAP in RAM using mdconfig

Still looking for less dirty solution...
 
Last edited by a moderator:
I just tried a cat f1.iso f1.iso f1.iso f1.iso f1.iso f1.iso > t1.out on my machine here:
Code: 
jimmy@tower1:~ % uname -a
FreeBSD tower1.jimmy.localnet 10.2-STABLE FreeBSD 10.2-STABLE #19 r292833: Mon Dec 28 17:44:41 CST 2015  root@tower1.jimmy.localnet:/usr/obj/usr/src/sys/GENERIC  amd64

I keep this machine updated with the latest from the -STABLE SVN branch .
The box has 2Gb RAM.
Swap turned off:
Code: 
jimmy@tower1:~ % swapinfo
Device  1K-blocks  Used  Avail Capacity
jimmy@tower1:~ %

f1.iso is a 4GB ISO image.

I have Xorg/i3 running, a couple of xterms, and FireFox going at the same time.

Oh, and no ZFS.

Things get really slow response-wise, but nothing dies with "out of swap space" message.

Monitoring things with systat -vm 2, and free memory dropped, but I don't think I saw it go lower than about 13000K.
 
Last edited by a moderator:
What shell you're using ?
It's not shell issue. I'm using bash. But tried tcsh and same bad effect.

I assume your /usr/src was synced at 28/Dec ?

Can you show me /etc/make.conf ?
Any changes to kernel config or you just using GENERIC ?
Any specials in /boot/loader.conf ?
 
I use tcsh, and just the GENERIC kernel config.

Last SVN update of /usr/src was on 28 Dec.

make.conf just set up for using ccache:
Code: 
jimmy@tower1:~ % cat /etc/make.conf
#PORTS_MODULES=emulators/virtualbox-ose-kmod

.if (!empty(.CURDIR:M/usr/src*) || !empty(.CURDIR:M/usr/obj*)) 
.if !defined(NOCCACHE) && exists(/usr/local/libexec/ccache/world/cc) 
CC:=${CC:C,^cc,/usr/local/libexec/ccache/world/cc,1} 
CXX:=${CXX:C,^c\+\+,/usr/local/libexec/ccache/world/c++,1} 
.endif 
.endif
Here's loader.conf:
Code: 
jimmy@tower1:~ % cat /boot/loader.conf
geom_stripe_load="YES"
kern.vty=vt
#i915kms_load="YES"
firewire_load="YES"
ichsmb_load="YES"
 
Last edited by a moderator:
Problem solved. It was a bug. Thanks all for help.

The problem was related to my /usr/src tree version - r291457M. Which was simply outdated. I changed /usr/src SVN repo to "head/stable" (r293232), rebuild everything and now all tests runs smooth. No problems with running 'cat' test. I'm upgrading now one production machine, where services such as squid, bind and some more are running - this was first place where I occurred a lot of issues where system was running without swap.
 
You must log in or register to reply here.
Back
Top