First time porting - setting up Poudriere-devel and it's slow

G

grandpa

Hello,

So, I have attempted to create a port - see this thread. Next step was to setup Poudriere to manage the port locally.

Basically I have used Vermadens guide on this link with a few small changes (thanks vermaden) and inspiration from threads in this Forum.

Poudriere is fun. It does what a build server should do and is simple enough for a first time porter to understand it. When I add a file in /usr/local/etc/pkg/repos pointing to http://localhost/packages/myportsrepo I can see my port with pkg search and my repo with pkg stats.

For me it is horrendously slow though. I have probably not configured it correctly.

I have set "latest" in /etc/pkg/FreeBSD.conf and I have installed ccache and memcache from ports.

Here's my poudriere.conf

Code: 
ZPOOL=zroot14
FREEBSD_HOST=https://download.FreeBSD.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
USE_PORTLINT=no
USE_TMPFS=yes
DISTFILES_CACHE=/usr/ports/distfiles
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
CCACHE_DIR=/var/ccache/
PARALLEL_JOBS=4
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=3
BUILD_AS_NON_ROOT=no
ALLOW_MAKE_JOBS=yes
ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py*"
PACKAGE_FETCH_BRANCH=latest
PACKAGE_FETCH_URL=pkg+http://pkg.FreeBSD.org/\${ABI}
PACKAGE_FETCH_BLACKLIST=""
PKG_NO_VERSION_FOR_DEPS=yes

I can feel that there is a complexity that comes with building a port that is dependant on moving targets in the ports tree.

But - is there any way to make Poudriere fetch more packages to speed up the build?

/grandpa
 
Before people are going to suggest various things, what are your systems specs?
CPU, amount of RAM (and amount you can spare if this box is used for something else that needs to stay online), HDD/SSD? , HDD/SSD space available and swap size?
 
grandpa said:
I have set "latest" in /etc/pkg/FreeBSD.conf
Click to expand...
Please don't edit /etc/pkg/FreeBSD.conf, if you want to switch to latest create a /usr/local/etc/pkg/repos/FreeBSD.conf. The /etc/pkg/FreeBSD.conf file might get 'reset' with an upgrade, removing your changes.

As for speeding up poudriere builds, yes, that heavily depends on CPU and memory. Faster CPU (more cores), and lots of memory will certainly help.
 
grandpa said:
But - is there any way to make Poudriere fetch more packages to speed up the build?
Click to expand...

Hi,

for a start - works with ports-mgmt/poudriere-devel - add -b latest to the poudriere bulk ... command.

If that does not work - try ports-mgmt/poudriere-dsh2dsh instead - its a modified Poudriere that uses packages more when its not needed to rebuild them with custom options.

Regards,
vermaden
 
diizzy said:
Before people are going to suggest various things, what are your systems specs?
Click to expand...

Hi diizzy,

Comparing the build time with the build time for manually creating a directory in /usr/ports/category/portname and doing make install clean from there. This might be an unfair comparison since the very first time I run Poudriere since there's nothing in the jail from start. Maybe the build time improves on next build?

I noticed that it was building qt6 and parts of rust and I was hoping that there would be a "magic" switch to say to Poudriere to install packages for those but I assume that the ports tree is ahead of the package tree in time causing the builds to be done(?).

I have this on a 10-year or so old laptop for now since this is where the port is. The software uses microphone and speakers and comes with a midi-interface for the usb ports. I guess I could move Poud to a virtual server to get some more oomph.

SirDice said:
Please don't edit /etc/pkg/FreeBSD.conf, if you want to switch to latest create a /usr/local/etc/pkg/repos/FreeBSD.conf. The /etc/pkg/FreeBSD.conf file might get 'reset' with an upgrade, removing your changes.
Click to expand...

Hello SirDice,

This is noted and corrected!

vermaden said:
Hi,

for a start - works with ports-mgmt/poudriere-devel - add -b latest to the poudriere bulk ... command.

If that does not work - try ports-mgmt/poudriere-dsh2dsh instead - its a modified Poudriere that uses packages more when its not needed to rebuild them with custom options.

Regards,
vermaden
Click to expand...

Hi vermaden,

Thanks for the guide!

Yes, I did -b latest. I also did try dsh2dsh as well as the bare poudriere but I wanted the option PKG_NO_VERSION_FOR_DEPS and I am not entirely sure, but I think this option is in -devel only?

If there are no more magic switches I am satisfied with my configuration and will set the thread to solved.

/grandpa
 
This is mine optimised,
cat poudriere.conf

Code: 
BUILDS=1

BUILDERS=1
PARALLEL_JOBS=1
PREPARE_PARALLEL_JOBS=1

MAKE_JOBS_NUMBER=6

#File Storage
#TMPFS_LIMIT=7
#CC usage
#MAX_MEMORY=7

#-----------------------------------------------------------------------
#ALLOW_MAKE_JOBS_PACKAGES="llvm* gcc* rust* waterfox firefox*"
#MAX_FILES=8192

ATIME=no
ZPOOL=SSD
ZROOTFS=/usr/local/poudriere
FREEBSD_HOST=https://download.FreeBSD.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
USE_PORTLINT=no
#USE_TMPFS="wrkdir data"
USE_TMPFS="all"
DISTFILES_CACHE=/usr/ports/distfiles

NOLINUX=yes
ALLOW_MAKE_JOBS=yes
TIMESTAMP_LOGS=yes
BUILDER_HOSTNAME=myfreebsd
USE_COLORS=yes
URL_BASE=http://127.0.0.1/poudriere/
PACKAGE_FETCH_BRANCH=quarterly
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes

Compiling qt6-webengine takes 20 hours.
[It's all about ram memory]
 
You must log in or register to reply here.
Back
Top