pkg-static eats alot of cpu

Alain De Vos

Alain De Vos

poudriere, make make-depends, pkg-static consumes alot of cpu. Playing youtube videos is interrupted i.e. an ULE scheduler issue ?
As even "nice -n 20" does not fix it.
 
Building which port?

For multimedia/webcamd, here, after a build time of around eight-and-a-half minutes, packaging takes three seconds without excessive use of the CPU:

Code: 
…
[00:00:56] [01] [00:00:30] Status   multimedia/webcamd | webcamd-5.17.1.0: build
[00:09:34] [01] [00:09:08] Status   multimedia/webcamd | webcamd-5.17.1.0: run-depends
[00:09:34] [01] [00:09:08] Status   multimedia/webcamd | webcamd-5.17.1.0: stage
[00:09:36] [01] [00:09:10] Status   multimedia/webcamd | webcamd-5.17.1.0: package
load: 1.43  cmd: sh 50275 [nanslp] 573.38r 0.60u 2.17s 0% 3672k
mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_timedwait_sig+0x12 _sleep+0x1c2 kern_clock_nanosleep+0x1c1 sys_nanosleep+0x3b amd64_syscall+0x10c fast_syscall_common+0xf8
[main-default] [2022-04-14_01h20m22s] [parallel_build:] Queued: 3  Built: 0  Failed: 0  Skipped: 0  Ignored: 0  Fetched: 2  Tobuild: 1   Time: 00:09:35
 ID  TOTAL                ORIGIN   PKGNAME            PHASE PHASE    TMPFS     CPU% MEM%
[01] 00:09:13 multimedia/webcamd | webcamd-5.17.1.0 package 00:00:03 2.26 GiB 26.1% 0.5%
[00:09:39] Logs: /usr/local/poudriere/data/logs/bulk/main-default/2022-04-14_01h20m22s
[00:09:39] [01] [00:09:13] Status   multimedia/webcamd | webcamd-5.17.1.0: install
[00:09:40] [01] [00:09:14] Status   multimedia/webcamd | webcamd-5.17.1.0: deinstall
[00:09:40] [01] [00:09:14] Finished multimedia/webcamd | webcamd-5.17.1.0: Success
…

1649896474462.png
 
Yes, I've seen the same when using ports (rather than poudriere) directly, IIRC it might be python - but there's definitely some times it really (pkg-static) "sticks" at 100% CPU for a while doing "something".

I've not investigated further, just saying it does happen sometimes.
 
Alain De Vos said:
poudriere, make make-depends, pkg-static consumes alot of cpu. Playing youtube videos is interrupted i.e. an ULE scheduler issue ?
As even "nice -n 20" does not fix it.
Click to expand...
My suspicion is that nice does not play nice "hard enough" for all the things (jails & its processes) that poudriere does behind the scenes.
Perhaps when you confine the poudriere (bulk) build as a whole to a separate jail, you'll have more direct influence and control to limit its resources.
 
Just rarely, I do notice the computer unusually busy (hogged) at the package stage, however it's never hogged long enough for me to think of it as a problem. IIRC when this happens, the (preceding) stage stage also hogs, and takes longer. Maybe only when I allow poudriere to use all CPUs, although I doubt this.

Like, I don't think of staging and packaging as a problem if the preceding build stage has taken hours, or days. When a build takes so long, I really want the final stages wrapped up without delay, at any cost.
 
Erichans said:
My suspicion is that nice does not play nice "hard enough" for all the things (jails & its processes) that poudriere does behind the scenes.
Perhaps when you confine the poudriere (bulk) build as a whole to a separate jail, you'll have more direct influence and control to limit its resources.
Click to expand...
Nice is not nice enough. I think freebsd needs a nicer nice
 
From: https://hackmd.io/@dch/HkwIhv6x7
[edit]
The above disappeared so here:
https://web.archive.org/web/20220924012217/https://hackmd.io/@dch/HkwIhv6x7
[/edit]

A few weeks ago, I discovered a gem in the above. I've been running poudriere from a script like this:

Code: 
nice -n 18 idprio 29 poudriere -N bulk -f /usr/local/etc/poudriere.d/myports.txt -J 3 -j 13amd64 -p default

This syntax only seems to work only in a bourne shell or script.
It runs poudriere at a nice and idle priority.
I have not found any other examples on the internet of the two being combined.

I can say that youtube in firefox and other interactive stuff seems to run a lot more smoothly.

Thanks to dch@ - I believe he/Dave is a FreeBSD developer.
 
Last edited:
CraigW said:
From: https://hackmd.io/@dch/HkwIhv6x7

A few weeks ago, I discovered a gem in the above. I've been running poudriere from a script like this:

Code: 
nice -n 18 idprio 29 poudriere -N bulk -f /usr/local/etc/poudriere.d/myports.txt -J 3 -j 13amd64 -p default

This syntax only seems to work only in a bourne shell or script.
It runs poudriere at a nice and idle priority.
I have not found any other examples on the internet of the two being combined.

I can say that youtube in firefox and other interactive stuff seems to run a lot more smoothly.

Thanks to dch@ - I believe he/Dave is a FreeBSD developer.
Click to expand...
Thanks, I was unaware of idprio.
But it's weird how the choice of shell could have an influence.
Currently trying on an 8-core with,
Code: 
nice -n 15 idprio 23 /usr/local/bin/poudriere bulk -J 5 -j $JAIL -p $PORTS -f $PACKAGES
With this firefox-youtube seems to play acceptable.
 
Not quite; it has to do with how a shell interprets and executes commands.

When looking at the tcsh, various actions (such as alias and variable expansion) are performed before the complete line with its commands is executed. Because of the nature of builtin commands (they execute faster because they don't have to be loaded from disk) a possible command (like nice) is first looked for as a tcsh builtin command and ... that happens to be the case. As covacat already noted, the Bourne shell (sh(1)) does not have a builtin nice.

From tcsh(1) at subsection "Builtin and non-builtin command execution":
When a command to be executed is found not to be a builtin command the
shell attempts to execute the command via execve(2). Each word in the
variable path names a directory in which the shell will look for the command.
Click to expand...
This means that only after the shell decides a command is not a builtin (=internal) command (and also not an alias) that command is searched for externally, that is: in the filesystem. When searching in the filesystem, that is where the search path comes into play when the command is not preceded with an absolute or relative path.

Code: 
nice -n 18 idprio 29 poudriere -N bulk -f /usr/local/etc/poudriere.d/myports.txt -J 3 -j 13amd64 -p default
When the above command is issued from a tcsh shell, the builtin nice is selected and executed. The tcsh(1) builtin commands which and where* are very useful to check in advance what is going to be executed**:
Code: 
% which nice
nice: shell builtin command.
% where nice
nice is a shell builtin
/usr/bin/nice
To bypass the builtin nice, one can of course specify the path and the command explicitly with its absolute path (in that case it doesn't even have to be in the search path). However, because the matching for an internal command is based on a character by character basis you can also bypass the builtin nice command and let the search process go via the search path by "tweaking its spelling" a bit by preceding it with a backslash: \nice. The backslash is a way to specify a character literally; thereby losing its special meaning for the shell. In this particular case however, at first the string "\nice" will be taken by the tcsh as one word and it tries to match it with a builtin command: that doesn't succeed. Further on in the processing, the "\n" is converted to an ordinary character "n"; in the end the string "nice" is matched to the file /usr/bin/nice
Code: 
% which \nice
/usr/bin/nice
% where \nice
/usr/bin/nice
In this case I have chosen to specify the character "n" literally. Preceding the command with a backslash seems the easiest when you type it. I could have chosen any and all characters from the word "nice". In the most extreme case that would be \n\i\c\e.

___
* from tcsh(1):
where command
Reports all known instances of command, including aliases, builtins and executables in path.
Click to expand...
If the output is more than one line then the ordered lines represent the instances where command can be found; the first one is executed.

** other shells have comparable commands.
 
So pkg-static just done this from a portmaster -r curl, after rebuilding mysql57-server:

Code: 
===>>> Creating a backup package for old version mysql57-server-5.7.37_1
Creating package for mysql57-server-5.7.37_1
load: 2.95  cmd: pkg-static 74876 [running] 49.17r 46.63u 0.20s 99% 105424k

Just matches what the OP said about using a lot of CPU. This is normally when I see it - after the "Creating a backup packge" message.
 
You must log in or register to reply here.
Back
Top