Poudriere failures

garyh

New Member

Reaction score: 7
Messages: 14

Hi,

Since reinstalling the OS to 13 I have not been able to build llvm90 llvm10 or llvm11
I continually get messages about a runaway process. I have also noticed it taking a very long time to build any packages.

Has anyone else seen this or have a resolution for it?

Regards,

Gary
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,601
Messages: 37,927

Those have been building fine for me.

You might want to add llvm* to ALLOW_MAKE_JOBS_PACKAGES in /usr/local/etc/poudriere.conf to speed up the builds (depends on your system of course).
 

Lamia

Aspiring Daemon

Reaction score: 203
Messages: 764

Those are two of the best practices (ccache, ALLOW_MAKE_JOBS_PACKAGES) for using Poudrière.
There are several other recommendations. Are llvm* in your priority builds in the conf file? Contents of your make.conf - src.conf may also affect it regardless of them working flawlessly before until now. Perhaps, there is a blocker in them.
Runaway process may also need you extend the allocated time to build a PKG before failure. You may also need increase resources.
 
OP
garyh

garyh

New Member

Reaction score: 7
Messages: 14

To be clear, the OS has been installed from scratch, I have not used ccache and have a pretty much empty make.conf and src.conf, there is no priority set at all. When on 12 llvm10 took just over 6 hours to build, now on 13 and it is not complete after 36 hours.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,601
Messages: 37,927

Are you using ZFS with poudriere? Although it doesn't matter much for a single port it does matter when building a lot of ports.

When poudriere is building one of the LLVMs have a look at top(1), is there only one core building? Or is the build spread over multiple cores? Did you set ALLOW_MAKE_JOBS_PACKAGES already?
 
OP
garyh

garyh

New Member

Reaction score: 7
Messages: 14

The whole system is using zfs but poudriere is not, as in no zfs options in the conf file, did not seem needed to me. I have set the ALLOW_MAKE_JOBS_PACKAGES to allow for all llvm ports.
 

Alain De Vos

Daemon

Reaction score: 546
Messages: 1,890

Playing Youtube music is currently better , non-interrupted, during poudriere builds.
My poudriere.conf file for a cpu with 8cores and 8G memory,
Code:
TIMESTAMP_LOGS=yes
HTML_TRACK_REMAINING=yes
USE_TMPFS=""
PARALLEL_JOBS=12
PREPARE_PARALLEL_JOBS=24
ALLOW_MAKE_JOBS=YES
ALLOW_MAKE_JOBS_PACKAGES="llvm10 rust qt5-webengine"
The CPU usage remains low ...
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,601
Messages: 37,927

poudriere is not, as in no zfs options in the conf file, did not seem needed to me.
It is. It uses ZFS to clone the build jails. So you really need to configure ZFS in the configuration file. Otherwise it's going to use cpio(1) to clone the base jails and this is quite an I/O intensive and slow process.
 
OP
garyh

garyh

New Member

Reaction score: 7
Messages: 14

Ok understood, but why would this behaviour only display in 13 and not 12? That's what I don't understand, the base jail has been installed from src cleanly after a src upgrade of the os, all config files are as they were on 12.
 
OP
garyh

garyh

New Member

Reaction score: 7
Messages: 14

I'll let you know after it's completed, have recreated the jail to use ZFS and set it going again.
 
OP
garyh

garyh

New Member

Reaction score: 7
Messages: 14

I'm going to go out on a limb here and say this is not just limited to Poudriere, make buildworld is taking a long time now too, has been going for near on 36 hours, when on FBSD 12 it would take maybe 4 hours, starting to think there is something wrong with this install. Admittedly this is building on an Intel Core i3 2 core, 2 thread processor but this is very slow even for that processor.
 

Alain De Vos

Daemon

Reaction score: 546
Messages: 1,890

On a 8-core processor compiling chromium-browser on FreeBSD-12 takes a full day. So it depends...
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 11,601
Messages: 37,927

make buildworld is taking a long time now too, has been going for near on 36 hours, when on FreeBSD 12 it would take maybe 4 hours, starting to think there is something wrong with this install.
Yes, something seems off. No hard figures, but building world; 13-STABLE takes around the same time as building 12-STABLE on my ancient Core i5. Certainly not that much longer, it may even be a little quicker.
Admittedly this is building on an Intel Core i3 2 core, 2 thread processor but this is very slow even for that processor.
Even my Raspberry Pi 3 is able to build world in 36 hours and that's a really, really slow system.

How's your I/O doing? Compiling takes some CPU horsepower of course but it has to read and write a lot of intermediate files too. So slow I/O can have a really big impact.
 

Lamia

Aspiring Daemon

Reaction score: 203
Messages: 764

It is expected that the first build - be it kernel/world or ports with Poudrière - would take a LONG time.

Notwithstanding, something can be wrong. It is suggested to start on a clean slate for a new build particularly in a major release - new/update Poudrière, delete cache/*, delete /usr/obj/* etc. I would think that you have done all that.
 
OP
garyh

garyh

New Member

Reaction score: 7
Messages: 14

Not sure why it is expected for the first buildworld to take longer, even if that were true I have never seen a buildworld take longer than 8 hours on this system. It is indeed a fresh install of FreeBSD 13. I/O looks good, all disks are solid state, the layout is standard from install with a separate pool for /usr/home and /var/db/mysql which are on mirrored zpools each, poudriere is set up in /usr/home/poudriere as it's own dataset. Network performance looks good. IMAP access over the internet seems very responsive. I am at a loss.

Thank you all for your responses and help.
 
OP
garyh

garyh

New Member

Reaction score: 7
Messages: 14

Right so I have rebuilt world and kernel and installed both, recreated the build jail and things look a lot better, currently building llvm11 and at 40% after 3 hours 4minutes. So whatever the issue was it seems better now.

Thanks all.
 

Lamia

Aspiring Daemon

Reaction score: 203
Messages: 764

Right so I have rebuilt world and kernel and installed both, recreated the build jail and things look a lot better, currently building llvm11 and at 40% after 3 hours 4minutes. So whatever the issue was it seems better now.

Thanks all.
It is always irritating at first build.
 

Lamia

Aspiring Daemon

Reaction score: 203
Messages: 764

You can achieve a better performance with those recommendations above - ccache, priority builds, etc.
 
OP
garyh

garyh

New Member

Reaction score: 7
Messages: 14

Yeah I have set ccache and priority builds but had already done that on the run before I blew away the poudriere data and jail directories and re set them up.
 
OP
garyh

garyh

New Member

Reaction score: 7
Messages: 14

All of my poudriere builds have finished in a time I would have expected now. I have a feeling there was some odd debugging issue in my original OS install. Seems to be cured after that src upgrade I performed, either that or the jails world and kernel were out of sync.

Thanks to all that helped me resolve this.
 
Top