Limit CPU usage when building ports?

[dostoievsky]

Hey guys, since this isnt a closed post, I've decided to post here because I got some questions of the topics discussed here.
I am having the same issue as the OP, but I build one package at time, and this by just getting into port and doing a make install clean.
When the port is big, sometimes when it is building the dependencies the CPU gets at 100% and I started to search on internet things like "how limit threads usage when building ports" my ideia was like "maybe I can make it just uses 80% ou 90% so it leaves me a little to keep other tasks runnings at ease.
But you guys are telling this is happening because the quantity of outputs being printed by the build task. I dont use tmux (neither I want to), and like OP i am using urxvt.
Reading this got me some ideas but I dont think it's trully, so what you guys think:
Since I am doing on Xorg TTY, it would be better if I just open other TTY to manage builds, and then, get back to Xorg Session? Or in the end it would be the same thing, since it will not stop the number of outputs in that building session?

Unresponsive? My guess, it's either write bursts or memory thrashing during linking. So, try using IO scheduler (zio or gsched) to force system write often but in small amounts or tune VM to not swap out interactive apps.

I'll also try reading about thoose tools too, if my first question isnt true, and try it first to see if I get a better performace.

 

[SirDice]

since this isnt a closed post

Threads aren't closed. Even if they're resolved.

I've decided to post here because I got some questions of the topics discussed here.

That was a 12(!) year old thread.

 

[dostoievsky]

You created a new one thread?
Well, while I was wainting the post to be accepted I was also doing some digging.
I fould this post:

poudriere and jobs number

I'm trying to build packages for i386 machine on amd64 machine. As advised in 59131, I've installed mgmt/poudriere (-devel specifically). One question. For some reason poudriere builds in one thread each of PARALLEL_JOBS (I set PARALLEL_JOBS=3 in poudriere.conf because without setting it...

forums.freebsd.org

And a quote from YuriG:

building the only package of www/chromium
without ALLOW_MAKE_JOBS took 11:07:52 with only 1 core used,
with ALLOW_MAKE_JOBS=yes and MAKE_JOBS_NUMBER=4 took 03:25:13 with 4 cores used averaging (3.25x),
with ALLOW_MAKE_JOBS=yes and without MAKE_JOBS_NUMBER (that is on all 8 cores) took 02:13:06 with all 8 cores occupied (+usual background tasks) (5.01x).

Probably, my choice will be ALLOW_MAKE_JOBS=yes, PARALLEL_JOBS=3 (2?) and MAKE_JOBS_NUMBER=4.

Since he had a similar number of cpus threads of mine, I decided to try it myself, and runned:
make install clean ALLOW_MAKE_JOBS=yes PARALLEL_JOBS=3 MAKE_JOBS_NUMBERS=2
And it is really leaving for me some 5% to keep chilling on web.
Also, during my research I found on web if thoose variables are not used during make, it only runs with 1 job. It is trully?
make execution will use all cpus/threads to run a single job?
What is the most effective way to build things with make?
Also, there is a way to make it use more of my ram too, and it can somehow accelerate the process?

 

[VladiBG]

Yes the easy way is to limit the number of CPU cores which are used by limiting the number of jobs which make(1) is using. The other method is to build inside a jail and limit the entire jail cpu usage via rctl.

rctl(8)

man.freebsd.org

Controlling Resource Limits with rctl in FreeBSD

FreeBSD provides several methods to limit the system resources an individual may use. The rctl command can be effectively used for controlling resource limits.

klarasystems.com

 

[im]

What about simple solution like nice()?
I used it many times in this way nice make install clean
Of course, you can specify other non-default value for nice().

 

[bakul]

May be you can use cpuset(1) for this? That is limit your ports building to only use specific cores. I haven't used this so don't know how well (or if) this works well.

I tend to think there is a lot of similarity between network and cpu scheduling. That is, time a process is running on a cpu core should be somewhat comparable to % bandwidth of a specific network channel used for a given flow. Though I am not aware of any system exploiting this similarity.

 

[Eric A. Borisch]

Yes, as bakul suggests, cpuset is another very effective way to limit the total CPU usage of any multi-threaded task. (As it works on per-cpu granularity, the lowest it can restrict is to 100% of a single CPU.)

 

[VladiBG]

limiting a single process is not enough as the build often spawn many sub processes.
Both synth(1) and poudriere(8) support limit of the maxjobs

 

[bakul]

I believe subprocesses started by a process limited to a cpu set will also be similarly limited. Experiment to see if this is the case.

 

[Eric A. Borisch]

I believe subprocesses started by a process limited to a cpu set will also be similarly limited. Experiment to see if this is the case.

Yes, cpuset is effective at limiting the entire spawned process tree, regardless of what unexpected parallelism the tree attempts to generate.

 

[CraigW]

This discussion seems relevant to leaving some CPU time for the user.
I still do it this way to increase responsiveness while poudriere churns away.

pkg-static eats alot of cpu

poudriere, make make-depends, pkg-static consumes alot of cpu. Playing youtube videos is interrupted i.e. an ULE scheduler issue ? As even "nice -n 20" does not fix it.

forums.freebsd.org