poudriere and jobs number

Y

YuryG

I'm trying to build packages for i386 machine on amd64 machine.
As advised in Thread 59131, I've installed mgmt/poudriere (-devel specifically).

One question. For some reason poudriere builds in one thread each of PARALLEL_JOBS (I set PARALLEL_JOBS=3 in poudriere.conf because without setting it pourdriere ate all my 16GiB RAM making 8-core AMD build system unresponsive). As far as I can see, MAKE_JOBS_NUMBER in make.conf doesn't work (setting it or commenting out). Building in one thread, say, www/chromium make all the idea on cross-building almost useless. In one thread it is almost as long as on my i386 target machine itself.
 
Poudriere works mostly well with default configuration options, making it easy to configure.
Poudriere uses hw.ncpu to determine the number of builders. This makes sense, as the bulk build should be done with all resources to build fast. While Poudriere is building, the system used should be considered "dedicated" to build fast, so user interaction may be delayed. Shrinking JOBS to 3 makes little sense. If you need to make the system more responsive for user interaction allow 7 for Poudriere (but I'd give all 8).

As having been said here before unused RAM is useless RAM. But you can limit to MAX_MEMORY=10 for Poudriere depending on how you allocated RAM i.e for ZFS or other processes.

You may post your poudriere.conf for review.

/etc/make.conf is never seen by Poudriere as it has it's own per jail in /usr/local/etc/poudriere.d for setting global ports-options.
 
Dear YuryG,
I did not find anything about jobs in make.conf and I am fine by configuring this in poudriere.conf. Regarding the number of jobs I found the explanation of ASX in Thread 59107 post #19 very useful for my small machine.
 
YuryG said:
I set PARALLEL_JOBS=3 in poudriere.conf because without setting it pourdriere ate all my 16GiB RAM making 8-core AMD build system unresponsive
Click to expand...
I wonder if there's other reasons for that. Slow disks or controllers perhaps? My home server is an older Core i5 (2 cores, 4 threads) and has 8GB. Almost half of that is used for ARC and I also have a bunch of services running (Samba, Transmission and some Java stuff). It builds a full Gnome desktop with poudriere without a hitch. My swap is barely touched. Even building something as massive as GCC or OpenJDK doesn't make the machine any less responsive than without the builds.
 
My computer has a CPU Intel(R) Pentium(R) D CPU 3.40GHz (3391.61-MHz K8-class CPU) with 2GB RAM. With two builder it uses a lot of swap when graphics/dri and devel/llvm37 are build in parallel. The system is still very responsive. The file system is UFS on a gmirror(8) on two SSDs. Now I have limited poudriere to one builder and the swap is not touched. I am not sure if the complete build time has changed. The load by the builder is not noticable anymore. I might could think about to increase the number of jobs. On the other hand I do not care if the build is a few percent faster or not unless it does not exceed some practical limit.
 
Well, after just installing podriere I started it without modifying jobs and the system stale, even mouse pointer moved with too much lagging, not to tell about anything else. So, I had to limit something: PARALLEL_JOBS=3. May be even 6 would be O.K., but suppose not 8. MAX_MEMORY to 10??? I do not have 30GiB of RAM, even memory with swap for 3 simultaneous jobs.

I do not need to build packages as-fast-as-possible. I need it done in a background and considerably faster than it would be on i386 target system.

Concerning the question I stated initially. MAKE_JOBS_NUMBER seem to disappear from make.conf (thank you, I know it's not /etc/make.conf), but that's the only thing that I could find concerning number of simultaneous processes for each build (analogue of -j flag for make). And even on target i386 I can start a couple of building processes simply with make from within port's directory and not loosing that much of system (with single core!). If I use poudriere I use almost as much time for building devel/llvm37 or www/chromium or editors/libreoffice. 10–17 hours??? for each of these. "Small" ports are not problem, obviously. They are all done when such "huge" one still builds. On 3GHz amd64 8-core system? That's just because every building job is done on a single CPU thread. It's obvious from top output.

May be someone can instruct me where this behaviour changes?
 
Aside from NO_ZFS=yes FREEBSD_HOST USE_TMPFS="wrkdir data" (tried with deault " yes"), WRKDIR_ARCHIVE_FORMAT=txz and now set PARALLEL_JOBS=3 everything else is default in poudriere.conf.
 
I have
Code: 
NO_ZFS=yes
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
USE_PORTLINT=no
USE_TMPFS=yes
DISTFILES_CACHE=/usr/ports/distfiles
SVN_HOST=https://svn.FreeBSD.org/ports/head
CHECK_CHANGED_DEPS=yes
CCACHE_DIR=/usr/local/poudriere/ccache
NOLINUX=yes
ALLOW_MAKE_JOBS=yes
ATOMIC_PACKAGE_REPOSITORY=yes
COMMIT_PACKAGES_ON_FAILURE=yes
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=2
BUILDER_HOSTNAME=pkg.esprimo.local
PRESERVE_TIMESTAMP=yes
but currently limit the number of builder via command line. devel/ccache is extreme helpful to speed up re-occuring builds. The build time for devel/llvm of 17 hours is by far too much. The latest build of of the cache took less than 10 minutes. The latest log I have showed a build time of 2:30 hours, likely with empty cache. May be it is better to limit the number of builders which can be achieved by the configuration file or as command line parameter by -J <number>. About the howtodo may be you have already found Thread 38859.
 
SirDice said:
I wonder if there's other reasons for that. Slow disks or controllers perhaps?
Click to expand...

I'm with SirDice on this, in my experience unresponsiveness is often bound to slow or excessive disk I/O, (or excessive swapping)

YuryG said:
As far as I can see, MAKE_JOBS_NUMBER in make.conf doesn't work (setting it or commenting out).
Click to expand...

I'm using Synth and not Poudriere, and installed poudriere just to look into the man page and docs, but was unable to find any reference to MAKE_JOBS_NUMBER, nor in poudriere manpage nor in /usr/local/etc/poudriere.conf.sample.

YuryG said:
If I use poudriere I use almost as much time for building devel/llvm37 or www/chromium or editors/libreoffice. 10–17 hours??? for each of these. "Small" ports are not problem, obviously. They are all done when such "huge" one still builds. On 3GHz amd64 8-core system? That's just because every building job is done on a single CPU thread. It's obvious from top output.
Click to expand...

Those times are way too long, it seems to me you are missing this setting:
Code: 
ALLOW_MAKE_JOBS=yes
It would be a good idea to post your poudriere.conf, that will provide the whole picture.
Also, as noted from chrbr, if possible make use of ccache.

getopt said:
Poudriere uses hw.ncpu to determine the number of builders. This makes sense, as the bulk build should be done with all resources to build fast. While Poudriere is building, the system used should be considered "dedicated" to build fast, so user interaction may be delayed. Shrinking JOBS to 3 makes little sense. If you need to make the system more responsive for user interaction allow 7 for Poudriere (but I'd give all 8).
Click to expand...

I agree about maximizing the resources usage, but there are cases where some bottleneck come in to play, and disk I/O, or quantity of RAM are often limiting factors. I would think one can still maximize the CPU cores usage, by means of ALLOW_MAKE_JOBS, while containing other resources like RAM by limiting the number of parallel builds.
 
I made a little benchmark with the configuration as above and a parallel build of devel/cmake and graphics/ImageMagick. I have chosen this ports because the build time is comparable and building them in parallel touches swap. The first set of tests have been with ccache disabled in the poudriere configuration. The second set of tests use the existing cache. For both tests the number of builders is either two which is chosen by devel/poudriere since the CPU has two cores. The result is compared with a single builder only. The time in the table of the ports are taken out of the poudriere log files. The last row shows the time devel/poudriere reports after the job has been completed.
Code: 
                    | without CCACHE |with CCACHE
-------------------------------------------------
Number of builder   |    2    1      |   2    1
swap                |  72M  26M      |  82M  no change
devel/cmake         | 14:22 9:48     |  3:44 2:31
graphics/ImageMagick| 13:23 6:30     |  4:38 3:25
-------------------------------------------------
poudriere total     | 15:44 17:46    |  5:59 7:23
For the run with one builder the swap might not have been released completely. With one builder each port is build faster. There seems to be some overhead due to start and stop of the builders and start and stop of devel/poudriere. For my old machine using two builders give only a little improvement in speed. For larger ports the relation might be even worse when more swap is allocated. The effect of devel/ccache is clear. I am sure that this differs from machine to machine. May be I will repeat the test with 1GB RAM only. I think for my machine I can already confirm the statement of ASX with respect to the number of builders and RAM.
 
O.K. Here's my poudriere.conf
Code: 
NO_ZFS=yes

FREEBSD_HOST=ftp://ftp2.ru.FreeBSD.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
USE_PORTLINT=no
USE_TMPFS="wrkdir data"
DISTFILES_CACHE=/usr/ports/distfiles
PARALLEL_JOBS=3
WRKDIR_ARCHIVE_FORMAT=txz
Due to add ALLOW_MAKE_JOBS=yes right now.
MAKE_JOBS_NUMBER was sometime ago in make.conf.
 
The NO_ZFS=yes is the culprit. This causes a huge amount of I/O when setting up (and tearing down) the jails. This results in significant slowdowns even on the dual 6 core Xeons I have at work. Setting PREPARE_PARALLEL_JOBS helps a bit but not much.

All I can say is, switch to ZFS.
 
or switch to synth. :)



(synth uses null mounts to create equivalent of jails, no build-ups or tear-downs are occurring)
 
Well, I suppose, ALLOW_MAKE_JOBS=yes is what I was asking about.
But now not only memory, but also CPU is a bottleneck, now all (at least half) of CPU is in building www/chromium alone.
 
SirDice said:
The NO_ZFS=yes is the culprit. This causes a huge amount of I/O when setting up (and tearing down) the jails…
Click to expand...
Am I correct, this happens only at the start and at the end of a whole bulk process? Because I see in terminal only this stages marked as "Creating jail" or "Starting jail". So, if I build 5 hundred packages, twice a two-minute lag is not relevant?
 
No, it happens for every single package. For every new build a new jail is set up and torn down when it's finished. With ZFS it's simply cloned from a base snapshot, this only takes a fraction of a second and barely uses I/O. But without ZFS it uses cpio(1) to extract a new jail, which is subsequently rm(1)'ed when finished. And this causes a very high I/O load.
 
YuryG said:
But now not only memory, but also CPU is a bottleneck, now all (at least half) of CPU is in building www/chromium alone.
Click to expand...

This is not a bottleneck, this is maximizing CPU cores usage, assuming building chromium took 10 hours when using a single core, by using 4 cores the build time will be approximately 2.5 hours, I though it was what you asked for.
 
ASX said:
This is not a bottleneck, this is maximizing CPU cores usage, assuming building chromium took 10 hours when using a single core, by using 4 cores the build time will be approximately 2.5 hours, I though it was what you asked for.
Click to expand...
I am saying that if I add PARALLEL_JOBS=8 and add enough of other packages to build simultaneously, I'm out of all resources: RAM, CPU, swap, disc I/O? Now it's about finding reasonable values for parameters (PARALLEL_JOBS, namely, may be others).
And thank you, yes, that's what I was asking for: to build heavy packages with all (almost) available cores, because I see that all other packages are built or awaiting dependent heavy ones.
 
YuryG said:
If I use poudriere I use almost as much time for building devel/llvm37 or www/chromium or editors/libreoffice. 10–17 hours??? for each of these. "Small" ports are not problem, obviously. They are all done when such "huge" one still builds. On 3GHz amd64 8-core system? That's just because every building job is done on a single CPU thread. It's obvious from top output.
Click to expand...
It should be worth to take one of the bigger ports and measure the effect of ALLOW_MAKE_JOBS. If you start with a smaller one as devel/cmake you should already notice the effect. You can use poudriere testport... or poudriere bulk -C ....
 
YuryG said:
Now it's about finding reasonable values for parameters (PARALLEL_JOBS, namely, may be others).
Click to expand...
Yes, it is. PARALLEL_JOBS=3 is a starting point, which is mostly OK in my opinion, but I don't have direct experience with poudriere, therefore my opinion is very approximate.
 
marino@ said:
or switch to synth. :)
Click to expand...
Yes, that's what I'm using and also what I would suggest. I did my initial choice based nearly exclusively on reading the documentation, which I have found much cleaner than poudriere.
It was also reported that synth is faster than poudriere, and the fact it doesn't use jails let me think it is true.
 
Well, now I see that ALLOW_MAKE_JOBS=yes is right for me, that is when you have several very big packages to build (like devel/llvm37, devel/llvm39 (which all unfortunately needed as they have dependencies like graphics/dri or www/chromium and somehow www/firefox pulls devel/llvm36 and actually do not build to work with other compilers saying something about wrong Infinity), lang/gcc5 (well, it needed by me and some ports pulls lang/gcc), www/firefox, editors/libreoffice or the most monstrous of these www/chromium, without which I can live but it's monstrousness is too obviuos not to take in account).
MAKE_JOBS_NUMBER actually works in make.conf, that's nice. Because having allowed ALLOW_MAKE_JOBS and set PARALLEL_JOBS to more than 1, I am definitely in a great resources pressure.

Building the only package of www/chromium
without ALLOW_MAKE_JOBS took 11:07:52 with only 1 core used,
with ALLOW_MAKE_JOBS=yes and MAKE_JOBS_NUMBER=4 took 03:25:13 with 4 cores used averaging (3.25x),
with ALLOW_MAKE_JOBS=yes and without MAKE_JOBS_NUMBER (that is on all 8 cores) took 02:13:06 with all 8 cores occupied (+usual background tasks) (5.01x).

Probably, my choice will be ALLOW_MAKE_JOBS=yes, PARALLEL_JOBS=3 (2?) and MAKE_JOBS_NUMBER=4.
 
  • Thanks
Reactions: 0mp
Dear YuryG,
it is good to know that you found a solution. A different approach would be to create jails for the huge ports and install them in the jails as packages. Thread 53362 is a perfect guide for that. The method is not limited to a certain jail management. This is what I use with sysutils/ezjail.
 
You must log in or register to reply here.
Back
Top