swap - memory

Nicola Mingotti

Well-Known Member

Reaction score: 129
Messages: 344

Eric A. Borisch, that discussion is outdated, I have red a few lines and they are talking about "vm.defer_swapspace_pageouts" which does not exist anymore !

I wrote the summary [the post before yours] exactly to prevent people to loose time on already discovered and discussed stuff. It is not a man page, but is a short report of our accumulated knowledge of the last weeks of discussion in this thread.
 

Eric A. Borisch

Well-Known Member

Reaction score: 219
Messages: 365

Yes, that particular sysctl is gone, but vim.disable_swapspace_pageouts still exists.

I think some of the distinctions drawn in that post (between paging and swapping) are worth noting in this discussion, too.

I personally tune these:

vm.v_free_target
vm.pageout_wakeup_thresh
vm.v_free_min


to adjust what amount of free memory the system tries to keep available. Depending on the system, workload, and goal, those may need to go up or down -- test for yourself to see if it impacts the behavior you are concerned about.

Refer to https://wiki.freebsd.org/AndriyGapon/AvgPageoutAlgorithm for what they're doing, or the explore the code itself if you're up for it: https://github.com/freebsd/freebsd/blob/master/sys/sys/vmmeter.h https://github.com/freebsd/freebsd/tree/master/sys/vm

Monitoring vm.stats while the system is “behaving poorly” or top -SHIz -mio to see where your IO is coming from can be informative, too.
 

Nicola Mingotti

Well-Known Member

Reaction score: 129
Messages: 344

Thank you for the links Eric A. Borisch I will give them a shot in the evening, this stuff become very complex very fast. About code, I can read user-space code (I did it for "top" for the first time a few weeks ago), kernel stuff nope, I never tried, I guess I am not prepared for it.
 

Nicola Mingotti

Well-Known Member

Reaction score: 129
Messages: 344

New experimental note

Today I used the system in swap-off, untill at some moment I was
near to eat all the memory so I turned the swap on.

But surprise, after 3 hours of use the system has not used a single byte of it !
I attach a picture for non "belivers";)

Since I did not see swap getting used I was a worried it would not have been used at all. So I closed many applications, and kept the minimum open. That is why you see a lot of Inactive memory in the screenshot.
 

Attachments

Snurg

Aspiring Daemon

Reaction score: 329
Messages: 794

Nicola Mingotti, your write-up is really very good :)
While looking into whether there is a non-too-complicated way of running X as normal user, I found that I apparently cannot do this.

Because, there is no Nouveau driver available for FreeBSD, and the Nvidia blob seems not to support KMS.

Ubuntu supports non-root X already since 2010, Debian even made non-root X the system default 2015.

So there on FreeBSD there seems no way to use the rctl stuff to limit memory usage if one uses Nvidia, like one can do with cgroups in conjunction with Nouveau on messy systemDOS :(

(BTW I had attempted to use a AMD/ATI card listed in the supported graphics cards FreeBSD wiki with the Radeon driver.
But the Radeon driver seems to have quite a lot of nasty issues.
For example it would hang when tearing was turned off in the xorg.conf, fail to resume, fail to switch between xorg and the consoles etc etc.
So switching to Radeon turned out to be no solution. :( )
 

Eric A. Borisch

Well-Known Member

Reaction score: 219
Messages: 365

But surprise, after 3 hours of use the system has not used a single byte of it !
I attach a picture for non "belivers";)
The pagedaemon won't wake up (writting pages to disk) until vm.stats.vm.v_free_count < vm.pageout_wakeup_thresh. If this condition hasn't been met since you've turned on swap, nothing will be paged out into the swap devices. (Unless you have vm.swap_idle_enabled=1, which will swap out (via vmdaemon) entire idle processes.)
 

Snurg

Aspiring Daemon

Reaction score: 329
Messages: 794

Eric A. Borisch, I am not sure whether this is going the right direction.
The problem is to avoid user programs to use more than a specified threshold of RAM.

Optimally this would include their X memory consumption. But one could just make a generous assumption how much memory X could consume, so that user-available memory can be set accordingly lower.

But X will not start for some reason if there is any setting for vmemoryuse for the user, no matter how big.
achix wrote that it could be related to X being setuid, but this doesn't seem plausible to me.

Why should a setting of 10 gigabytes prevent even a small program like xkbcomp to run, making X fail to start?

Does there exist a reasonable explanation for this behavior, or should it be considered as buggy?
 

Eric A. Borisch

Well-Known Member

Reaction score: 219
Messages: 365

Sorry, it seemed like there was a more general discussion also going on of how swap/paging is used in FreeBSD, and some ways to influence it.
 

Nicola Mingotti

Well-Known Member

Reaction score: 129
Messages: 344

Snurg , thank you !
From time to time I will try to keep that document updated adding new things emergin here.

About, having X running non-root, I guess you are going face a difficult challenge. When it was achieved in OpenBSD it was commented as "incredible milestone", in 2014, see here:
http://undeadly.org/cgi?action=article&sid=20140223112426

Personally, I am sick of dealing with unsupported consumer hardware. When I use Linux/FreeBSD/OpenBSD for desktop use -- that is all days, since many years, I always run them inside a VMWare virtual machine. And happliy leave those problems to younger programmers;)

For people, like me, who do not know well how X works and do not know why it needs root privilege here is the explanation:
https://www.reddit.com/r/linux/comments/1viqpt
 

Nicola Mingotti

Well-Known Member

Reaction score: 129
Messages: 344

Eric A. Borisch , thank you for the links, understanding how virtual memory is managed and make a nice expalanatory document for users (in the style of the HandBook) would be super !

Unfortunately, the links you have sent are quite criptic to me, I will need a good amount of study to understand what is written there. I will try in the near future.
 
OP
OP
fernandel

fernandel

Aspiring Daemon

Reaction score: 160
Messages: 716

Snurg , thank you !
From time to time I will try to keep that document updated adding new things emergin here.

About, having X running non-root, I guess you are going face a difficult challenge. When it was achieved in OpenBSD it was commented as "incredible milestone", in 2014, see here:
http://undeadly.org/cgi?action=article&sid=20140223112426

Personally, I am sick of dealing with unsupported consumer hardware. When I use Linux/FreeBSD/OpenBSD for desktop use -- that is all days, since many years, I always run them inside a VMWare virtual machine. And happliy leave those problems to younger programmers;)

For people, like me, who do not know well how X works and do not know why it needs root privilege here is the explanation:
https://www.reddit.com/r/linux/comments/1viqpt
Did you ever try HardenedBSD https://hardenedbsd.org/ ?
 

Nicola Mingotti

Well-Known Member

Reaction score: 129
Messages: 344

Did you ever try HardenedBSD https://hardenedbsd.org/ ?
No, fernandel, I see its name for the first time now.
Thank you for letting me know.

Usually when my maximum concern is security I use OpenBSD.
Indeed, in the base, OpenBSD has important things for me as:
mg (editor, Emacs like), dhcpd, perl [whatever scripting language
for me is fine, I don't like programming in the shell languages]

But I am getting used to FreeBSD ipfw and dummynet so I will
consider also HardenedBSD.

Moreover, OpenBSD updates very fast and changes (expecially pf)
quite frequently. This is a bit of a nuisance for me. I prefer FreeBSD
approach, from what I undestood till now.
 

ronaldlees

Aspiring Daemon

Reaction score: 304
Messages: 723

I made a best effort summary of this thread, it is here:
https://docs.google.com/document/d/1cubX4cRAyk3dyOR5ed4t0dngQrhrUjCec6VXpMl8zC0/edit?usp=sharing

If somebody wants to partecipate in the editing let me know.
I looked at the Google doc.

10 FACT. On ARM, especially BeagleBone Black, FreeBSD comes swapless by default.
Thanks for the great outline. I was looking at #10 on your list. I think there's another reason, in addition to the one about swap "wearing out" the disks more quickly, for there being no default swap setup on some of the ARM images:

Older USB and even some newer uSD storage devices are so slow that it disturbs the swap system. If the data can't be swapped fast enough due to slow IO, there is a chance that processes will be "killed off" even though there is technically "unused" swap space left on the slow drive. So, swap depends on pretty fast drives (hard drive speed).
 

Nicola Mingotti

Well-Known Member

Reaction score: 129
Messages: 344

I looked at the Google doc.
Thanks for the great outline. I was looking at #10 on your list. I think there's another reason, in addition to the one about swap "wearing out" the disks more quickly, for there being no default swap setup on some of the ARM images: ...
Thank you ronaldlees, I was actually asking myself why it was so.

I just updated the write up with the latest contributions.
 

zbsd

New Member

Reaction score: 1
Messages: 4

Yes, that particular sysctl is gone, but vim.disable_swapspace_pageouts still exists.

I think some of the distinctions drawn in that post (between paging and swapping) are worth noting in this discussion, too.

I personally tune these:

vm.v_free_target
vm.pageout_wakeup_thresh
vm.v_free_min


to adjust what amount of free memory the system tries to keep available. Depending on the system, workload, and goal, those may need to go up or down -- test for yourself to see if it impacts the behavior you are concerned about.

Refer to https://wiki.freebsd.org/AndriyGapon/AvgPageoutAlgorithm for what they're doing, or the explore the code itself if you're up for it: https://github.com/freebsd/freebsd/blob/master/sys/sys/vmmeter.h https://github.com/freebsd/freebsd/tree/master/sys/vm

Monitoring vm.stats while the system is “behaving poorly” or top -SHIz -mio to see where your IO is coming from can be informative, too.
Could you give an example of how this works please? Like the method used to configure the values you listed for those parameters. Does it have any effect on processes running as root? This does not involve the zfs file system.
 

Eric A. Borisch

Well-Known Member

Reaction score: 219
Messages: 365

Could you give an example of how this works please? Like the method used to configure the values you listed for those parameters. Does it have any effect on processes running as root? This does not involve the zfs file system.
You use these with sysctl:
sysctl vm.pageout_wakeup_thresh=16384 or in /etc/sysctl.conf to set on boot.

These values are in pages (4K), so 16384 = 64MB. Adjust as needed for your system.

My understanding is that the wakeup threshold is when the pagedaemon will be woken up to try to page out (and mark as cache (ready to be freed)) inactive memory. The target is what it shoots for before sleeping again. The _min setting determines when some allocations are delayed, if I recall. Check the wiki page I linked to.

It impacts your entire system. Root, ZFS, etc. I bump them all up from the defaults to try to start recovering memory sooner rather than later (closer to 0.) As with any VM tuning, YMMV; test under your usage case & hardware.
 

Nicola Mingotti

Well-Known Member

Reaction score: 129
Messages: 344

New data available.

After a couple of weeks of daily use I can state with near to certanity level of
confidence that : almost all my problems with swap were caused by Firefox.

Another factor I recognized is, when OSX is up since days the overall performance
of the VMWare virtual machine (in this case FreeBSD) deteriorates badly. Switching the VM on/off makes no effect. An OSX reboot is necessary.

Finally, the only other thing I toggled is, I increased swap space from 1G to 4G.
I don't know if this may had an influence, I had before and now 5G Ram for FreeBSD VM.

It seems all my swap problems are gone !

P.S. my OSX is 10.11.6, Apple does not let me upgrade.
 

Mage

Member

Reaction score: 16
Messages: 98

I’ve been using FreeBSD for many years. From day one, I used only ZFS and swap on ZVOL. It used to be okay.

Lately, as my projects were growing on the servers, I started running into the problem when the ARC is like 28G, but FreeBSD starts swapping (not only a percentage, also "in" and "out" all the time). The system slows down. I lowered the vfs.zfs.arc_max. The result is disturbing. I either have 4-10G free (not used) RAM on the servers or swapping occurs once in a while.

I decided to test turning off the swap before I do it on production. I’m still running the rest while writing the post. What I have seen so far, it surprised me. My super scientific test method:

Bash:
dd if=/dev/random of=1.file bs=1m count=2k
dd if=/dev/random of=2.file bs=1m count=2k

screen:
  for i in $(seq 0 100); do echo $i; xz -9e -T 4 -k -f 1.file; done
  for i in $(seq 0 100); do echo $i; xz -9e -T 4 -k -f 2.file; done

irb:
  size = 500; a = [0] * (1024 ** 2 * size); nil

top -o res
tail -f /var/log/messages
cat 1.file 2.file > /dev/null # to grow up ARC

The irb is a Ruby console. It allocates a huge object. I can control how much memory it takes by the "size" variable. In the VM I used for the testing when the two xz processes maxed out their memory usage, Ruby would use up the rest of the memory somewhere size = between 400 and 500.

Setup: VM, FreeBSD 11.2, 16G RAM, 8 cores. Most settings are FreeBSD default.

Result:
When there is a swap partition (8G ZVOL), FreeBSD 11.2 behaves like a mass murderer. It kills processes on sight. For example:

* Without the xz processes running, if the irb process’ memory usage would get near (but not that close) to the size of the ram, the kernel kills the process without using a considerable amount of swap, and without lowering the ARC to the minimum. Yeah, why not? He is the boss.

* When the two xz processes were running and used up about 4-5G RAM each, the kernel killed either the irb or one of the xz processes at about 6-7% swap usage. Maybe once I could go up to 60%, but most of the time a process got killed below 10%. (It might have made a difference whether used irb or xz first.)

* At the last test run, the kernel didn’t kill anything for a while. Neither it did anything. It was not frozen as I could type in the ssh terminal. But it couldn’t execute even an ls. When this happened, the top stopped refreshing. In this test run, I added the swap on the ZVOL only after logging in. I turned off the "org.freebsd:swap" on the ZVOL and rebooted. It has been stuck already for half an hour. There is a lot of free swap.

When I turned off the swap:

The kernel became peaceful. It went to the end to not kill anything. It emptied the Laundry (when there was one), the Inact, and most of the Wired. When it reached the wall, it slowed the execution of the processes. Instead of 400% CPU usage per xz process, they went down to 100% or even 0%.

When the above happened, the system still was responsive. The terminal with the top was refresing. It wasn’t usable though. I couldn’t even quit irb to free up the memory.

Conclusion:

When there was no swap in the system, it was terrible to run out of memory.

When there was a swap in the system, processes got killed way before the swap filled up. Other times (not in the test but on production servers) the system was swapping when there was a lot of ARC used. Without swap, ARC would have been lowered.

Once, when I ran out of physical RAM but I used almost zero swap, the system became and stayed unresponsive. Only a hard reset helped. It was worse than without swap and running out of memory.

I would say, the swap was used when it shouldn't have been used. It slowed down the system for nothing. Other times, the swap wasn’t used when it could have been used to save the system from a freeze. With swap, the kernel also much more likely killed processes even when, I thought, there was no reason to do so.

After I post this comment, I will turn off swap on all my servers, and remove the limit of the ARC.
 

Crivens

Moderator
Staff member
Moderator

Reaction score: 753
Messages: 1,704

Could you try to enable compression on the zvol, or swap on a physical device?

And you may need to check the block size of the ZVOL, swapping is done using 4k blocks. I can't look up the settings on my machine as I am not at home.
 

abishai

Aspiring Daemon

Reaction score: 141
Messages: 630

I have even worse experience with swap and VMs. After a day or two all VM memory is moved to swap. My server has 64GB and 32 GB swap and usually, the layout looks like 28GB of used swap and everything is ARC. Not only VMs are slow, but I have OOM events as well. Not sure what to do.
 
Top