Hardware Requirements to build packages using poudriere

M

mizhka

Hi,

Does anybody have list of hardware requirements to build packages using ports-mgmt/poudriere? On my laptop (2 cores, 4 threads, 16GB RAM, ZFS+SSD) I'm using it for half of year and discovered that sometimes 16GB RAM is not enough for parallel build, but enough for single build.

Now I want to buy powerful desktop, so I want to size it to suite frequent Poudriere builds.

Thanks you in advance,
Michael.
 
The requirements are not significantly more than what the requirements are for building packages in the first place using make -j n install where n is the number of parallel jobs. Poudriere is really just a wrapper around ports(7) and jail(8) for the most part. The only part where it consumes some extra resources compared to ports(7) is the construction of dependency graphs to figure the optimal build order. If you can build a large port with many parallel jobs using make -j n install the same system should be able to build the same ports using Poudriere.

You can in the extreme case limit the number of parallel jobs used by ports-mgmt/poudriere by using this setting in /usr/local/etc/poudriere.conf:

Code: 
# parallel build support.
#
# By default poudriere uses hw.ncpu to determine the number of builders.
# You can override this default by changing PARALLEL_JOBS here, or
# by specifying the -J flag to bulk/testport.
#
# Example to define PARALLEL_JOBS to one single job
# PARALLEL_JOBS=1
 
Thank you! I totally argee with you, but my problem is memory consumption of LLVM compiler & linker. For instance, if Chromium & Libreoffice is being built in parallel, there is high probability that chromium build will be failed due to lack of free memory. I found that "O2" compilation requires several GB of free RAM in specific cases. Another issue is linker, obviously it requires many memory to link huge project (like Chromium or webkitX-gtkX).

But memory is quick cheap now, and it's not a problem to meet memory requirements to build huge packages. What I'm looking for is number, number of RAM required for current state of ports / packages.

Another possible option is to find tool which gather memory footprint / timing statistics of builds. Something like poudriere-perf-stats?...

Thanks!
 
Build the problematic ports using -J 1 option for poudriere bulk, slower but should allow the ports to be built at least.
 
Cka3o4Huk said:
...sometimes 16GB RAM is not enough for parallel build, but enough for single build. Now I want to buy powerful desktop, so I want to size it to suite frequent poudriere builds...
Click to expand...

Here's my data/experience:
Intel Core i5-2520M (2.5 GHz, low thermal cpu) on Gigabyte Sandybridge micro-atx board with 16 GB DDR memory.
FreeBSD on zfs 128 GB SanDisk SSD
home directory on ZFS mirror pool with two WD Red 1TB hd.

That CPU has four real cores and I let Poudriere use four build jails simultaneously.

I use Poudriere to build my local repository of 2000 packages, including Libreoffice, Firefox, Thunderbird, Java, PostgreSQL, KDE, Haskell, etc. I haven't had any problem building everything and even use the build machine for browsing the web or watching lectures and videos under Fluxbox while poudriere is busy. I haven't noticed any problem with running out of memory but sometimess when all four build jails are working on big packages the CPU load gets above 5 and video playback gets choppy for a while. Of course I generally try to run Poudriere while I'm sleeping. It's always possible to interrupt a Poudriere bulk build when you need the full power of your computer and then restart it later, so you could just let Poudriere run every night and reclaim your computer for other uses every morning.

The nice thing about that particular cpu is that it never gets hot even at maximum load (highest temp I see is 120 degree F with 3 variable-speed case fans still running almost silently).

If I were building the machine over I would probably still install only 16 GB of memory because (a) it's enough, and (b) I'm poor. 32 GB would be very sweet :)
 
It should work fine with 4 GB even. My system at home only has 4 GB and it builds all my packages just fine. Granted, it's not the fastest machine but it gets the job done.
 
Cka3o4Huk said:
Thanks you, garry and SirDice!

It sounds reasonable, but could you please tell me what is swap partition size on your machines? Thx!
Click to expand...
I set the swap partition for 8 GB. My boot drive is bigger than necessary for the os even with some zfs snapshots, so I want to allot an excess of swap to ensure that the system never runs to the end of swap space even under extremely rare conditions.

While running poudriere bulk builds with four jails I see that occasionally a tiny bit of swap is used (tens of megabytes). I've never (but should have) checked for the highwater mark. I took a peek once when video temporarily got too choppy to watch and there was about 200 Mb of swap in use. Of course that implies that my 16 Gb of ram was "full" so you see that four jails building four big packages simultaneously can eat up a lot of memory. If I limit poudriere to 2 jails I never even notice that it's running.
 
You must log in or register to reply here.
Back
Top