Poudriere options is the bottleneck

amiramix

amiramix

Maybe I am doing something wrong but the longer is the list of ports I want to compile with poudriere, the longer it takes to configure them versus compiling them.

I have a lot of CPUs and RAM and I compile everything in memory. Multiple ports at the same time, apart from when they depend on each other. But the longer is the list, the more can be compiled independently.

But "poudriere options" is always taking one package from the list at a time, and recursively calls all the dependent ports to configure them. If I add a new port to the list, I still have to go through all to configure. I know that I can configure just the individual ports that I added, but then it's a manual work when I am adding a few ports to configure them one by one, and it still takes time.

I had a list of 32 ports and just added npm and yarn. It took 10 minutes for "poudriere options" to complete and 28 seconds for "poudriere bulk" to complete.

What's the typical way people work with options? Is configuring them a common scenario or maybe most would leave the defaults and only configure specific ports they care about? Or maybe there is another better way?

Probably just going through all the options for all the ports in a linear fashion would be quicker than allowing poudriere to traverse depedencies for each port from the list, as often ports share dependencies, and so the complexity becomes exponential instead of linear. Is there any way for poudriere to write down options for all the dependencies that "poudriere bulk" uses?
 
% ls -1 /usr/local/etc/poudriere.d/143amd64-default-options/ | wc -l
743

I get used to it within time, also it is way easier to notice when you just start off from the beginning. I've already had configured most of the ports I use.

I think there's two ways of reducing time poudriere options takes:
1. Use non-recursive config (not make config-recursive but make config instead)
poudriere-options(8)
Code: 
OPTIONS
             -n           Do not be recursive
2. Maybe unset/set some options in make.conf, this may reduce the time poudriere takes if one port's options are automatically had configured by make.conf. Also you can just press enter after dialog just appeared, because you already selected/unselected some of them if exist.
 
I've not used poudriere but from a bunch of threads on the forums, I think one can prespecify the desired options for each port. It may take more effort up front to get your options as desired in make.conf (as nxjoseph suggests), but I think once you do that then "configure" should be almost a noop.
 
This is how I have this scripted

  • Process no configuration options for any ports at all -> do not call /usr/local/bin/poudriere options
  • Process conditional configuration options for these ports only -> /usr/local/bin/poudriere options -n -C -j ${jail} -f /list/of/ports
  • Process conditional configuration options recursively -> /usr/local/bin/poudriere options -C -j ${jail} -f /list/of/ports
  • Process configuration options for these ports only -> /usr/local/bin/poudriere options -n -c -j ${jail} -f /list/of/ports
  • Process configuration options for these ports recursively -> /usr/local/bin/poudriere options -c -j ${jail} -f /list/of/ports
 
BTW, configuring each and every port and all of its dependencies is sheer madness.
You'd only do that (and most of the time non-recursively) if you have a very small set of ports (say 10-20) with very specific non-standard settings.
Doing a fully recursive configuration on even a small number of ports will offer you ports and options you've never even heard of, so they make no sense and you'll probably just hit 'enter' a lot.
And even then: you do it once, not every single time, of course.
Configurations are saved under /usr/local/etc/poudriere.d/.
 
poudriere options touches *A LOT* of tiny files, so the type of storage plays a huge factor here.
Ideally poudriere should have everything (including its config directory, where it keeps the port options) on an nvme-backed, dedicated pool for best performance. Even the difference between nvme and SAS SSDs is already significant, and I wouldn't want to find out how abysmally slow a full poudriere options -f list-of-ports run with ${etcdir}/poudriere.d/<options> on spinning rust would be...

And as DutchDaemon already mentioned: running a full recursive, unconditional (-c) poudriere options on a regular basis is pure madness. Usually you specify port options once when adding a port to your list; then an occasional poudriere options -C (~every month or after a new quarterly branch) is sufficient to catch all new/changed options. Yes, this may take a while to wade through all ports and dependencies, but IMHO doing this ~once every quarter is bearable...
Of course this is only necessary if you specifically want to deviate from the default options (but that's usually the main reason for maintaining a poudriere build environment). For ports where you don't care you can just omit poudriere options and use the defaults. You can also use different lists of ports to build, e.g. 'portlist-default-options' and 'portlist-custom-options', and then only call poudriere options for the latter list.
 
I'd recommend not using poudriere-options(8) and instead put your options in the make.conf. Much easier to keep track off, and you don't have to redo the poudriere-options(8) for every new jail you create.

Here's an example from my make.conf, once you know how you can set/unset options it's actually pretty easy to figure out.
Code: 
DEFAULT_VERSIONS+= mysql=10.11m php=8.2 java=17 linux=rl9

OPTIONS_SET+= OPTIMIZED_CFLAGS

devel_cmake_UNSET= DOCS
devel_git_UNSET= GITWEB SEND_EMAIL
devel_log4cplus_UNSET= DOCS
devel_py-molecule_SET= DOCKER
editors_neovim_SET= PYNVIM
java_openjdk8_UNSET= CUPS X11
java_openjdk11_UNSET= CUPS
math_gmp_SET= CPU_OPTS
net_samba416_SET= MDNSRESPONDER
net_samba416_UNSET= AD_DC AVAHI CUPS FRUIT
print_ghostscript10_UNSET= CUPS
security_strongswan_SET= CURL GCM KDF SWANCTL VICI
security_sudo_SET= INSULTS
sysutils_cpupdate_SET= INTEL
sysutils_vm-bhyve_SET= BHYVE_FIRMWARE GRUB2_BHYVE
textproc_py-docutils_SET= PYGMENTS

You can even get creative, by setting a 'common' make.conf and additional settings specific to a set, jail or ports tree. They will end up getting merged together.
Besides the 'common' make.conf above I also have a server-make.conf specifically for my "server" set of packages, and a desktop-make.conf for a "desktop" set of packages.
server-make.conf:
Code: 
DEFAULT_VERSIONS+= imagemagick=7-nox11
OPTIONS_UNSET+= X11

www_nginx_SET= HTTP_FANCYINDEX
mail_exim_SET= CONTENT_SCAN DKIM DNSSEC MYSQL SA_EXIM
mail_dovecot_SET= MYSQL
mail_spamassassin_SET= MYSQL
net_miniupnpd_SET= UPNP_IGDV2
net-mgmt_zabbix64-server_SET= IPMI LDAP MYSQLDV NMAP
net-mgmt_zabbix64-server_UNSET= MYSQL
net-mgmt_zabbix7-server_SET= IPMI LDAP MYSQLDV NMAP
net-mgmt_zabbix7-server_UNSET= MYSQL
print_ghostscript10_UNSET= CUPS X11
sysutils_openipmi_SET= SSL
security_suricata_SET= NSS
sysutils_vm-bhyve_SET= BHYVE_FIRMWARE GRUB2_BHYVE
graphics_ImageMagick7-nox11_UNSET= OPENEXR
graphics_ImageMagick7_UNSET= OPENEXR SVG PANGO RAQM
graphics_graphviz_UNSET= DOCS
graphics_openexr_UNSET= DOCS
graphics_vips_UNSET= MATIO
databases_mongodb70_SET= NOAVX

# Kerberos
security_heimdal_SET= LDAP LMDB
net_openldap26-server_SET= GSSAPI
desktop-make.conf:
Code: 
audio_pulseaudio_UNSET= AVAHI

comms_sdr++_UNSET= PLUTOSDR
devel_geany_SET= THEMES VTE
devel_libudev-devd_SET= GPL
devel_py-isort_SET= COLORS
devel_sdl20_SET= UDEV
devel_qt6-base_UNSET= CUPS

graphics_ImageMagick7-nox11_UNSET= OPENEXR
graphics_ImageMagick7_UNSET= X11 OPENEXR

graphics_libmng_SET= MNG_OPTIMIZE

multimedia_libdvdread_SET= DVDCSS_DLOPEN
multimedia_mplayer-skins_SET= ALL

print_qt5-printsupport_UNSET= CUPS

www_firefox_UNSET= JACK

x11_sterm_SET= SCROLLBACK XRESOURCES

x11-drivers_xorg-drivers_SET= EVDEV
x11-drivers_xorgxrdp_SET= DRI3

x11-toolkits_gtk20_UNSET= CUPS
x11-toolkits_gtk30_UNSET= CUPS
x11-toolkits_gtk40_UNSET= CUPS

x11_nvidia-driver-390_SET= ACPI_PM LINUX
x11_nvidia-driver_SET= ACPI_PM LINUX
#x11_nvidia-driver_UNSET= LINUX

games_libretro-puae_SET= DEBUG
games_retroarch_SET= DEBUG

You do have to review those options from time to time, as options can be added or removed, dependencies might shift, etc. But because this is all conveniently done in a small set of files it's very easy to review and adjust.

Code: 
   Create optional make.conf
     You can also specify a global make.conf which will be used for all the
     jails.  Any of the following are allowed and will all be used in the
     order shown:

           /usr/local/etc/poudriere.d/make.conf
           /usr/local/etc/poudriere.d/<setname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-<setname>-make.conf
           /usr/local/etc/poudriere.d/hooks/plugins/<plugin>/make.conf
From poudriere(8)
 
SirDice said:
and you don't have to redo the poudriere-options(8) for every new jail you create.
Click to expand...
You can run poudriere options without the -j flag, so the options are not jail-dependent.
I usually use the -z and -p flags, so I end up with options for specific combinations of portstree (latest and quarterly) and sets (e.g. servers, desktop). The resulting directories are then named e.g. 'quarterly-servers-options' or 'latest-laptop-options' and I have matching makefiles for those tuples ('quarterly-servers-make.conf') where I put distinct options for that tuple, global options go to the global make.conf in the poudriere.d directory.
 
sko said:
You can run poudriere options without the -j flag, so the options are not jail-dependent.
I usually use the -z and -p flags, so I end up with options for specific combinations of portstree (latest and quarterly) and sets (e.g. servers, desktop). The resulting directories are then named e.g. 'quarterly-servers-options' or 'latest-laptop-options' and I have matching makefiles for those tuples ('quarterly-servers-make.conf') where I put distinct options for that tuple, global options go to the global make.conf in the poudriere.d directory.
Click to expand...
Still rather annoying to keep track off. For a long time used poudriere-options(8) myself, until I figured out make.conf made it so much easier.

Careful with mixing those poudriere-options(8) with make.conf though, poudriere-options(8) overrule whatever was set in make.conf, unless you use *_SET_FORCE/*_UNSET_FORCE (see Mk/bsd.options.mk).
 
  • Thanks
Reactions: sko
You must log in or register to reply here.
Back
Top