Excessive swapping while using synth

Today, getting a bit of lagging when it built firefox, thunderbird, and chromium at once. I now have builds limited to 3, but when it was those 3 it lagged (though didn't freeze) the machine. I left jobs per builder at its default of 4. (I did add a large swap partition before starting, and it hasn't seemed to get close to overwhelming swap--I chose the RAM*4 that marino@ has suggested, 32 GB, so this combination seems to be working for me, though not perfectly.
 
PacketMan said:
I too have decreased the defaults from 6 builders x 4 jobs, to 4x3. I've also tried 3x4, but I am still undecided.
Click to expand...
Assuming your system has 4 cores, try with 3x4 or even 3x3, with 8 GB RAM + 4 GB swap you may still need a bit of luck to not run out of that resources, where two buiders will be probably the "safe" setting.

Synth, by using tmpfs, make large use of the RAM, and that's good, the drawback is that when the RAM is limited there is no room for 'disk cache', and that is bad, and even worse when forced to make extensive use of swap, the extreme case being running out of swap completely.

Fact is that many (most) packages will require little RAM (that is on average 1.25 GB x builder, as desumed from Synth man page), but some will require more RAM, in those case the system will fallback to use disk I/O. (tmpfs filesystems are backed from swap).

Synth build up a clean environment for each packages to be build, (local base), and to do so must read the base files from disk and write them to tmpfs, I would expect that very often the base files are the same for each port and fetching them from the disk cache would be much faster than reading from disk, similar reasoning for build dependencies, that's why in my opinion could be better to keep down the number of builders and possibly allow for more disk cache.

At the end, the whole job might be faster by using less builders.
 
ASX said:
At the end, the whole job might be faster by using less builders.
Click to expand...

Maybe. I find you get through the few big ones, but then you got say 50 or 100+ small ones and you are only using a fraction of the resources available. For a machine that is in production that might be perfectly desirable. For me however on a machine that is dedicated for only building ports/packages it seems to be of a balancing act. I could go buy lots more ram for an older machine, or just live it with. Marino had a good suggestion. Build your known big ones using less builders/jobs and then do the rest of the smaller ones using more builders/jobs. At the end of the day, something like 3x3, 4x3, 3x4 seems to be a sweet spot, and even if the machine has some idle resources, the outcome is still way better than what we had before with ports-mgmt/portmaster.

Just for testing and learning purposes only, I created a 30GB swap file on my builder. And yes its on the same spinny disk as the OS. Gonna try and pound the machine hard with a few big build jobs just to see what happens. I seem to learn best when I get experimental. I'll buy a newer machine before I put any money into the older ones.
 
PacketMan said:
Just for testing and learning purposes only, I created a 30GB swap file on my builder. And yes its on the same spinny disk as the OS. Gonna try and pound the machine hard with a few big build jobs just to see what happens. I seem to learn best when I get experimental. I'll buy a newer machine before I put any money into the older ones.
Click to expand...
Replacing the spinning disks by SSDs is as day and night. The invest is little and the effect is huge. It should increase the speed also if you just use a very small SSD and use if for temporary data as ccache or swap.
 
Hello!

Based on what was discussed here before, ports-mgmt/synth configured to run 4 builds and 8 jobs would be appropriated to (try to) avoid the use of swap on a system with 8 cores and 16GB ram?

How much swap would be recommended for that system/configuration?

Thanks! :)
 
you can't really avoid using swap when you hit the giant ports like chromium and libreoffice.

8 jobs per builder is lot with 4 builders. You might be better served with 4-6 jobs per builder.

I've been throwing out guidelines of swap to be 4 times RAM, so you'd probably want around 40 to 64 G swap, especially with the 32 jobs you're proposing.
 
marino@ Sorry, for some reason the jobs per builder option was in my mind as total jobs. What I was indeed proposing was 4 builds and 2 jobs per build.

Thank you!
 
I have played with these numbers a lot of a machine that I use as a dedicated builder. It does nothing else other than synth. My sweet spot seems to be 3 builders by 4 jobs. I tired 4x3 too. In the end there doesn't seem to be too much difference; all runs well until you hit the big ports like Marino mentioned. Only thing gonna handle that is lots of virtual memory (ram + swap).

My machine is 8 core with 8GB ram. Might try and add 8GB more, but its an older machine so just might keep my money for a new server.
 
So, the number of builders are what will determine the amount of memory needed and not of the jobs, am I correct?

Currently, using normal/single build system with 8 jobs, I do need about these amount of memory:

Code: 
Chromium - 10GB
Firefox - 4GB; 8GB if using debug
Libreoffice - 6GB

If I do 4 builds and 2 jobs, and hypothetically have 48GB of memory, I would not need the swap in any case. Correct?

Thanks!
 
both do. each builder has to provide a localbase and working area (both could be in the gigabytes). Each job needs memory to compile. The builders are the big ones though.

with 4 builders and 2 jobs and 48G RAM, you'd probably never touch swap. You could probably add several more jobs per builder.
You're really only sizing for the monster ports. 95+% of ports probably never get close to consuming all the memory (assuming synth is only thing running)
 
marino@ said:
You're really only sizing for the monster ports. 95+% of ports probably never get close to consuming all the memory (assuming synth is only thing running)
Click to expand...

When I build x11/gnome3 the swap on my machine hardly get used, until you hit the big ports. Then the gears grind and smoke blows out the back. My machine absolutly could use a lot more ram, but with the 40G of swap it clunkers through it. I just synth just-build x11/gnome3, walk away and then later do the pkg upgrade on the GNOME3 machine. Their home use machines; not enterprise production.
 
You must log in or register to reply here.
Back
Top