Excessive swapping while using synth

S

scottro

As marino@ has pointed out, the question and information will probably be lost in the 35 plus page synth thread, so I've taken the liberty of copying my original post, and the responses.
pquote]
Synth using memory till swap is full



I see that someone had this issue back with FreeBSD-10.x, where synth started using swap at an unexpected rate. https://forums.freebsd.org/threads/56171/ Lately, this has started happening to me on 2 FreeBSD-11 systems, both with reasonably good processors, zfs, and 8 GB of RAM. Adding a second swap partition didn't help. This seems to happen when building large packages, such as libreoffice, however, using portmaster or just doing make install from a port doesn't do this--it will use a lot of CPU, but swap stays at unused or just a little bit used. This happens when I run synth prepare-system. I haven't seen any mention of it save for that one thread I linked, so it may be one of those Just Me(TM) things, but I do see it on two different machines. I see swap gradually climbing till it reaches 100 percent and the machine becomes unresponsive. Logs show things like
Code: 
 09:28:47 s kernel: swap_pager: out of swap space
Dec 26 09:28:47 s kernel: swap_pager_getswapspace(5): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s last message repeated 6 times
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(12): failed
Dec 26 09:32:50 s kernel: swap_pager_getswapspace(16): failed
[/quote]
marino@ responded

There are 2 possibilities:
  1. you've set too many builders + jobs for the resources you have for every case
  2. you've got a leak somewhere. Something is using but not releasing swap space. grep log directory for watchdog failures to see if those are happening (they really should never happen).
I answered


For what it's worth, I used defaults, and looking through logs I don't see any watchdog errors. This may mean more work in tracking down a memory leak than I have the ability (or time) to do right now.

@fernadel posted

I had the same problem still, now on FreeBSD 11-RELEASE. Not many times but it happened.
Not so long I use Synth and at the same time it built LibreOffice and Blender. SWAP raise to 36% and after LibreOffice was done dropped to 13.6%. And when was everything done swap stayed on 13%??

abishai posted
8Gb of ram is not enough for synth on my system. When 2 builders start to build Firefox/Thunderbird it starts swapping heavily and eventually system crashes. I disabled tempfs for workarea in Synth config and everything runs smoothly after that.

marino@ replied
how much swap do you have? maybe you find a cheap 64gb or higher SSD and use that (assuming we're not talking about a space constrained machine like a laptop here).

The original problem was likely your swap partition was way too small, but if the swap is a spinning disk, it's not much better than just turning off tmpfs for the workspace.

alternatively, you can shift the synth build root to an SSD partition so that building without tmpfs won't be such a big performance hit.

And I posted
Today I started a synth update, but first tried limiting vfs.zfs.arc_max from its default of RAM minus 1 GB to about 4 GB. I think something is leaking memory, but don't have the expertise to track it down. In my case (on a ZFS mirror in one case, on a single ZFS disk pool in another--haven't tried on that machine yet), it may have fixed the issue, with swap, so far, staying below 30%, though there are still many packages left to go. I'll update when the build is complete.


So, putting this in this new thread (some posts above have been somewhat edited) in case others run into the problem as there are two possible fixes listed, abishai and mine.

TLDR; From the synth thread, I posted about swap getting completely used, and I've posted the responses here, which include two solutions that have worked for two individuals.
 
Last edited:
Both machines with the issue have 2 GB of swap. On the second machine, I did try adding a 4 GB swap partition as well, but had the same problem. Swap usage during the build I'm doing now (with limitation on ZFS ARC) has risen to 65 percent of the 2 GB, and I'm not sure if there will be a libreoffice build on this run, which seems to frequently be the culprit. I should also mention (going back to marino@'s last post, that the other machine where this happened is on an SSD, though a fairly low end one.


EDIT:
Well, limiting ARC didn't help. So, as I saw swap climbing up over 90% I added a swap partition, 8 GB this time, and it seems to be working. I still think there's a memory leak somewhere, but as a workaround, this will do for now.
 
I have 8 GB of RAM and my swap partition is 3.6 GB. Should I resize swap partition.
As I wrote yesterday swap was used 36% when Synth build LibreOffice and Blender at the same time but I didn't run anything. Usually I am running GNOME 3 or Fluxbox and using GIMP or surfing with Netsurf or Firefox...
 
those swap partititions are too small for building. I would say you need a 4:1 ratio between ram and swap (so 4G ram requires 16G swap), etc. With cheap SSDs, you really could just throw a 64Gb fast swap at the problem as a permanent solution.

Less than 1:1 between ram and swap is not going to work. Don't be stingy here.

Added: This was directed as scottro@ but applies to fernandel@ even better.
 
I have observed that when building some of the mentioned packages (libreoffice, firefox, thunderbird, ...) RAM+SWAP usage reach approx. 4 or 5 GB for each builder (using tmpfs, of course), sometimes more, very approximately half for local base and half for work area.

That's a raw estimate but enough do some basic calculation about how many builder your system can support, with or without swapping.

Also, I noticed that disabling use of tmpfs result in a much increased disk I/O (understandably) and usually a performance penalty, not much because of disk throughput, but more because of disk concurrent access (read & write from each builder and additional I/O if you use ccache, like I do.), that's why an SSD help a lot.

Where possible I would favor swapping over turning off tmpfs, because that is usually faster than ZFS I/O or journaled UFS I/O, also use of tmpfs allow for much faster "deinstall/cleanup" of the work areas.

If you are forced to turn off tmpfs a NON journaled UFS partition will work faster than a journaled one, local base and work area are temporary, you may safely avoid journaling.
 
I decided to enlarge swap file because mine FreeBSD is installed on iMac and add SSD disk is not cheap and iMac is 11,1.
gpart show
Code: 
34 1953525101 ada0 GPT (932G)
34 6 - free - (3.0K) 
40 409600 1 efi (200M) 
409640 1216587112 2 apple-hfs (580G) 
1216996752 1269536 3 apple-boot (620M) 
1218266288 1024 4 freebsd-boot (512K) 
1218267312 727710720 5 freebsd-ufs (347G) 
1945978032 7547102 6 freebsd-swap (3.6G) 
1953525134 1 - free - (512B)

I never resize partition and my idea is to use OS X boot camp and take sam space from apple-hfs, than use gpart and delete swap file and make the new larger one or I am doing wrong, please?
 
I would be nervous about resizing partitions too.
If you can't add a second drive easily (or cheaply) then maybe you just have to keep tmpfs turned off.

alternatively, you can create a second synth profile that doesn't use tmpfs and just build libreoffice and other monsters with that new profile and use the old profile for everything else. That would be a good compromise.
 
fernandel said:
I decided to enlarge swap file because mine FreeBSD is installed on iMac and add SSD disk is not cheap and iMac is 11,1.
gpart show
Click to expand...

Or you could just stay with the current disk layout.

fernandel said:
have 8 GB of RAM and my swap partition is 3.6 GB. Should I resize swap partition.
As I wrote yesterday swap was used 36% when Synth build LibreOffice and Blender at the same time but I didn't run anything. Usually I am running GNOME 3 or Fluxbox and using GIMP or surfing with Netsurf or Firefox...
Click to expand...

It appear your system used approximately 1.2 GB of swap, turning off tmpfs for only local base (and leaving it on for work area) will give you enough free RAM to run the other apps safely. Or, follow the advice from marino@ of course.
 
ASX said:
It appear your system used approximately 1.2 GB of swap, turning off tmpfs for only local base (and leaving it on for work area) will give you enough free RAM to run the other apps safely.
Click to expand...

Right, but then TMPFS is never used leading to a very noticeable performance penalty. He has a swap issue < 1% of the time, so removing tmpfs 100% of the time is a big penalty.
 
marino@ said:
He has a swap issue < 1% of the time, so removing tmpfs 100% of the time is a big penalty.
Click to expand...

Of course, I agree that where possible tmpfs should be used, but I'm not sure it is only 1% of the time, quite a few packages require similar resources (on a average desktop system)., libreoffice, thunderbird, firefox, virtualbox, the webkit series, texlive-texmf, ... to mention some.

I can also see some difficult in identifying which one are those packages ... I can remember some of them now, after running Synth several times, but that is only after you already built them and monitored the resources usage.
 
it's not if swap is used. It is if the activity of building the package uses 100% of the swap. Out of 27,000 packages, the number of packages that require that are probably less than 0.1%. I'd bet it's less than 20 packages.

Added: obviously systems with absurdly low levels of swap are excluded here. 3.6G seems reasonable at first glance but it's simply insufficient for multi-builder/multi-job profiles in all cases.
 
I will leave as is. I have a problem just when I am building LibreOffice otherwise works pretty fast and it doesn't bother me whatever I am doing. I have concurrent builders and jobs per builders 2, 2, tmpfs work area and localbase true and I am using ccache too.
Thank you.
 
marino@ said:
it's not if swap is used. It is if the activity of building the package uses 100% of the swap
Click to expand...
Please do not take me wrongly, my previous answer was referred to fernandel's context that was "building while using some other apps". (to me that imply minimize swap usage).
 
Well, even adding gigs of swap (on a ZFS mirror) didn't stop it from eventually, while apparently building firefox, thunderbird, and libreoffice simultaneously, locking up the system. I may play with the config file to limit the number of builds (I think it's 6 on that workstation.) I did create enough swap to give me RAM*4 as marino@ suggested, but it wasn't enough. :-(

I may try again tomorrow with a 64 GB swap partition, or, as I said, limit the number of builds.
 
On my laptop I have 1.5GB swap and 8GB ram. So, for me, the situation looks normal as now I have 2 builders instead of portmaster's one.
As for tests, it is not nesessary to change partition layout at all. One can create swap file for synth and delete it after it finishes.
I compile FreeBSD kernel on 1 buck cheap 128MB Ram VDS with
Code: 
truncate -s 10G /usr/swap0
mdconfig -a -t vnode -f /usr/swap0 -u
swapon /dev/md0
The effectiveness of such swap is lower, but it's swap after all.
 
I had 672 port to rebuild. I run GNOME3. I start Synth before I went to bed and results are:
Time: 5:56:42
Swap: 22.7%
swapinfo shows:
Code: 
Device          1K-blocks     Used    Avail Capacity
/dev/ada0p6       3773548   383532  3390016    10%
Everything was okay, no errors. They were also LibreOffice, Firefox...webkits.
 
While I don't have a complete answer at this point, it seems that if I first build libreoffice with synth install editors libreoffice everything else goes fairly smoothly. I would have to try a few times to be sure, but for now, that's the solution I'll use with synth, as well as temporarily adding a lot of swap.
 
scottro said:
I may play with the config file to limit the number of builds (I think it's 6 on that workstation.) I did create enough swap to give me RAM*4 as marino@ suggested, but it wasn't enough. :-(
Click to expand...
I think 6 builders are beyond the capability of an 8 GB RAM machine, (that's the RAM size you mentioned in your first post).

From a performance point of view you should not make your system swap most of the time. (swap access even on SSD is thousands times slower than RAM, as you surely know).

Consider also that large packages will take longer to build, therefore they have a natural tendency to line up and build simultaneously, thus maximizing memory requirements all at the same time.

2 builders is the maximum I would suggest on 8 GB RAM (3 or 4 builders might work too but in my opinion will degrade the overall performance), if your CPU has many cores increase the number of jobs but keep down the number of builders.

To recap, configure the number of builders depending on available RAM, and configure the number of jobs depending on available CPU cores.
 
Thanks, I'll keep that in mind. The only times I've run into trouble though, were with libreoffice being one of the packages. It's 9 cores, so I am guessing that that's why synth.ini had 6 as the number of jobs (the man page says default is 75% of the cores).
 
scottro said:
Thanks, I'll keep that in mind. The only times I've run into trouble though, were with libreoffice being one of the packages. It's 9 cores, so I am guessing that that's why synth.ini had 6 as the number of jobs (the man page says default is 75% of the cores).
Click to expand...

I had to re-read the man page and do think that percentage refer to systems where RAM is very large, because it also state:
Generally memory is the limiting resource when tmpfs is used
Click to expand...
... and tmpfs should be used nearly always ...

The description of "Max Jobs per builder" seems to agree with my view:
If memory is constrained, it's often a better performance tradeoff to reduce the number of builders and increase the number of jobs per builder.
Click to expand...
 
As I understand that is okay that I live my system as is and don't thinking about resizing swap partition.
BTW LibreOffice build with ccache 28 minutes (last time).
 
Bah, I said 9 cores, I meant 8 of course. (See what I did there--cores, course, nudge nudge). So, at present, the only time I've had a problem is when libreoffice was building with something else. I think I'm going to try resetting build number from 6 to 2 or 3, but of course, may not remember about this whole issue until it bites me again.

Many thanks to all who have replied, and to marino@ who no only replied, but gave us synth and quickly replies to every query about it.
I was thinking of marking this solved, with the solution being to limit builds in /usr/local/etc/synth/synth.ini, but I'd like to wait till I try it again. It seems the most likely solution, perhaps combined with adding temporary swap for an upgrade.
 
ASX said:
I think 6 builders are beyond the capability of an 8 GB RAM machine......
....
Consider also that large packages will take longer to build, therefore they have a natural tendency to line up and build simultaneously, thus maximizing memory requirements all at the same time.

2 builders is the maximum I would suggest on 8 GB RAM (3 or 4 builders might work too but in my opinion will degrade the overall performance), if your CPU has many cores increase the number of jobs but keep down the number of builders.

To recap, configure the number of builders depending on available RAM, and configure the number of jobs depending on available CPU cores.
Click to expand...

Bullseye!

My builder server is 8GB ram, 4GB swap (gonna enlarge it I'm pretty sure based on this discussion). I too have decreased the defaults from 6 builders x 4 jobs, to 4x3. I've also tried 3x4, but I am still undecided. One thing I am doing, is when I see a hog port show up on the display I do a ctrl-q. This causes Synth to not start any more new builds. After the hog port is built, I then restart synth, using the exact same command that was previously launched. But that requires watching the machine. I'm still tinkering, but 4x3 seems to be a sweet spot.
 
You must log in or register to reply here.
Back
Top