Building images with Poudriere has taught me a lesson...

Phishfry

Phishfry

I have spewed some bad information about compiling FreeBSD source.

Many times over I can be heard saying that for fast compiling you need fast storage and many cores.

It turns out I was wrong. Super fast storage is not needed. Max throughputy I see with speedy storage system is around 435MB/sec.

A single SATA3 SSD drive would almost do.

Its all about the core count.

Whats your experience?
 
I'd say it really depends on your particular setup.

My perspective as one that is using a different OS that needs all packages compiled locally (Gentoo) is that having the build directory in a tmpfs helps. Not only is it slightly faster but also it limits the wear on the drives. Also distcc is a blessing when the packages are not compiled on my 32core machine.

And yes, more cores help. Also not having any overhead from things like virtualization is a plus.

later edit
some numbers, for context. I'm compiling the exact same package (portage/net-analyzer/wireshark-4.4.1) in three slightly different environments:

build directory on a fast NVME drive: 609.75 seconds
build dir in tmpfs: 597.21 seconds
build directory in tmpfs and distcc: 263.86 seconds
 
Has anybody used the -P option with poudriere jail ?

-P patch Apply the specified patch to the source tree before
building the jail.

So I can provide a diff file but I need to include path to patch?
Where should /usr/src/ patches reside?

poudriere-jail

man.freebsd.org man.freebsd.org
 
I regularly compile on very slow storage (NFS over 1 Gb Ethernet). Empty make recursion and `make clean` hurt a lot, but complete recompilations do not.

Having lots of RAM helps, too.

/tmp should be a memory FS or -pipe should be in compiler flags.
 
I think for poudriere jail patching I have looked at handbook for patch and diff and there is no specific path needed.
So I hope to stash my patch under /poudriere the root of my builds. That way I can apply changes only to select board images.
And I don't contaminate my /usr/src/ with hand patches. That has bit me already.
If poudriere jail can patch; then only its source build will get contaminated. That is ideal.

Unified diff I am guessing is the default.
 
Back on point I am sure the Write Speed would increase with core count. Each cpu process writing to disk.
Still for me it seems bursts of 435MB/sec were less than 99.9% of the time. Most writes are sporadic and low like 35MB/sec.

So I have to test that difference with my dual machine with 44 cores x 2.
Over double the core count and see how much disk writes increase. Beefy storage there too.
 
cracauer@ said:
Having lots of RAM helps, too.
Click to expand...
Thats very interesting. On amd64 I see zero memory problems. I use UFS so maybe some difference there but it seems to me compiling doesn't scratch above a couple of gigs of usage. Regardless of 64GB or 128GB machines seems like extreme overkill.
Slap ZFS and NFS on top I could see more memory needed. I am doing bare metal builds.

For speeds I am doing 35min for cross compiling with QEMM bin emulator for arm.armv7 and 45 min on arm64.aarch64 builds.
I have not done native builds yet. Only cross compiling.

I am also staring at ccache and trying to decide if I want to deal with that option.
I don't think I need it. But shared file builds are not terrible idea.
 
My disk subsytem is not that fast on this machine. A mini cube MATX server LGA2011 AsrockRack E5-2698v4. 64GB Mem.
WOL works so that makes it nice....Calling this my roadtrip server. Has translucent side which I find stupid. But nice that Asrock board has LED Post Code right there. It ended up nice for a Non-Lighted case guy. Truly functional window. Silverstone Sugo01.
Once POST codes cycle the LED turns off. Perfectly. For a headless server.

I gmirror'ed two samsung M.2 on a card with bifurcation.
Code: 
Transfer rates:
    outside:       102400 kbytes in   0.080953 sec =  1264932 kbytes/sec
    middle:        102400 kbytes in   0.043478 sec =  2355214 kbytes/sec
    inside:        102400 kbytes in   0.048834 sec =  2096900 kbytes/sec

# nvmecontrol devlist
 nvme0: SAMSUNG MZ1LB1T9HALS-000MV
    nvme0ns1 (1831420MB)
 nvme1: SAMSUNG MZ1LB1T9HALS-00007
    nvme1ns1 (1831420MB)
 
It did not think I needed TMPFS on quick machine but I did notice I was allowing poudriere to use it. I will give it a try.

Code: 
# Use tmpfs(5)
# This can be a space-separated list of options:
# wrkdir    - Use tmpfs(5) for port building WRKDIRPREFIX
# data      - Use tmpfs(5) for poudriere cache/temp build data
# localbase - Use tmpfs(5) for LOCALBASE (installing ports for packaging/testing)
# all       - Run the entire build in memory, including builder jails.
# yes       - Enables tmpfs(5) for wrkdir and data
# no        - Disable use of tmpfs(5)
# EXAMPLE: USE_TMPFS="wrkdir data"
USE_TMPFS=yes

# How much memory to limit tmpfs size to for *each builder* in GiB
# (default: none)
#TMPFS_LIMIT=8

So I should be setting this to:
USE_TEMPFS=all

Then setup tmpfs mount in fstab ????

i wonder if 64GB is enough to build on 40 cores. I will see....
 
poudriere jail is really failing me for patching.

poudriere jail -a arm64.aarch64 -c -j rock3c -v 15.0-CURRENT -b -J40 -m src=/usr/src15.0 -S /usr/src15.0 -P /usr/src15.0/pci_dw.patch -x

If I give a wrong patch location I get error but when it finds patch file and builds it seems to skips the patch stage.
The source inside jail never gets patched.

Code: 
[00:00:01] Copying /usr/src15.0 to /poudriere/jails/rock3a/usr/src... done
[00:00:22] Starting make buildworld with 40 jobs
 
I broke down and ran the patch manaully and discovered I had to be in source tree.
So I tried to place patch there and trigger it like I did manually.
/usr/src15.0/sys/dev/pci/ is where it needed to run from.

poudriere jail -a arm64.aarch64 -c -j rock3a -v 15.0-CURRENT -b -J40 -m src=/usr/src15.0 -P /usr/src15.0/sys/dev/pci/pci_dw.patch -x
So it copies patch over into jail correctly in the correct location and still won't fire off patch.

I have to assume this is only a Poudriere Jails Ports Feature unless somebody proves me wrong.

I even tried jail.sh variable SRCPATCHFILE
poudriere jail -a arm64.aarch64 -c -j rock3a -v 15.0-CURRENT -b -J40 -m src=/usr/src15.0 -P SRCPATCHFILE=/usr/src15.0/pci_dw.patch -x
I now know this was wrong path but it was not proper syntax.

Can somebody please help me turn on logging in Poudriere. I know it has html stuff but I need logs.
NanoBSD builds provided the most useful logs ever. It would have shown this Patch Phase in detail.
I don't want html logs what are the options?
I liked the webserver when I first ran poudriere as a cross compiling package builder. My cut and paste days.
I don't need that right now I want basic logs.

I am sorry to be talking about -CURRENT so much. I really never used it until testing on Arm boards that don't work.
 
You must log in or register to reply here.
Back
Top