Best way to manage ports?

I have been running FreeBSD 14-Current for about a month now and have been enjoying it besides the odd crashes here and there. It is not as many crashes as OpenBSD 7.2-Current was giving me however. I am stuck with 14-Current for now because on my Intel 2.5GB Lan and RX 6600 GPU not being supported in 13.1. I am coming from Gentoo and have been messing with ports a little bit on and off and can't seem to figure out an easy way to manage them and was curious what others are doing.

I don't like that pkg can't handle ports on it's own and requires some outside program like Poudriere (horrible name btw) or portmaster. I have tried both and run into the same issues some packages will only install using 1 core on the system which is a huge drag on something like LLVM, and the other issue is updating and installing the packages I can't just run pkg upgrade and it automatically calls one of the 2 packages to start building the new updates.

It seems to me like ports was just a huge after thought in the system are we not supposed to build most things from source? My other question is what if I drop Poudriere/portmaster and just cd into the ports directory and make install them? Will pkg then know that when an update comes in to build them from source and not just install the binary?

The last thing I want to ask is not an issue yet but it has been on Gentoo for me every now and again. The options or use flags whatever you want to call them for the packages how do I know which ones I need for X package to work with Y? There is no error messages telling me if I remove say wasm in LLVM that firefox may break or something.
 

SirDice

Administrator
Staff member
Administrator
Moderator
FreeBSD 14-Current
You must be new here.

Poudriere (horrible name btw)
Interesting bit of trivia; Poudriere is French for 'tinderbox', which was the name of the original package building tool (ports-mgmt/tinderbox)

It seems to me like ports was just a huge after thought in the system are we not supposed to build most things from source?
It's the other way around actually. Ports have been a part of FreeBSD going all the way back to when we didn't even have packages. Heck, freebsd-update(8) didn't even exist until 9.0, and the only way to update/upgrade the system was by building from source. I've spent many hours since 3.0 watching compiler output scroll past. Some time around 9.0 we migrated from the old pkg_* tools to PKGNG, which is the current packaging system. https://wiki.freebsd.org/pkgng/CharterAndRoadMap https://wiki.freebsd.org/PkgPrimer

My other question is what if I drop Poudriere/portmaster and just cd into the ports directory and make install them?
It'll just work. What were you expecting to happen?

Will pkg then know that when an update comes in to build them from source and not just install the binary?
Ports build packages, and it's this package that eventually gets installed. So everything is a package in the end.

The options or use flags whatever you want to call them for the packages how do I know which ones I need for X package to work with Y?
Look at the port options.

There is no error messages telling me if I remove say wasm in LLVM that firefox may break or something.
Nope. So you may end up building a dependency without a certain option that's required for another port. That's why certain options are set by default (because some other port requires them). If you want to muck around with options than you will need to figure out which ports require what dependencies. And you're going to run into problems, and rebuild everything again, and again.
 
You must be new here.
Yes I understand what current is thanks as mentioned in the post I am stuck with it for the time being.
Interesting bit of trivia; Poudriere is French for 'tinderbox', which was the name of the original package building tool (ports-mgmt/tinderbox)
Gotcha! Cool history hard to type correctly.
It'll just work. What were you expecting to happen?
Ports build packages, and it's this package that eventually gets installed. So everything is a package in the end.
These 2 refer to the same thing. I understand they are a package in the end my question is will pkg understand that when it needs to update that package to build it from ports directly by running pkg as with Poudriere and portmaster I need to update the packages through separate commands and need to have separate package list from what's installed on the system it comes off as an after thought. If it was a priority pkg should handle it directly a good post about this you can read here - http://michael.orlitzky.com/articles/motherfuckers_need_package_management.xhtml
Look at the port options.
Again pretty manual just comes off as an after thought.
Nope. So you may end up building a dependency without a certain option that's required for another port. That's why certain options are set by default (because some other port requires them). If you want to muck around with options than you will need to figure out which ports require what dependencies. And you're going to run into problems, and rebuild everything again, and again.
Something to improve on portage for example detects this automatically and will inform you before wasting time building.

I appreciate the response I'm just looking for a more automatic way of maintaining ports without much manual intervention.
 

SirDice

Administrator
Staff member
Administrator
Moderator
I understand they are a package in the end my question is will pkg understand that when it needs to update that package to build it from ports directly by running pkg as with Poudriere and portmaster I need to update the packages through separate commands and need to have separate package list from what's installed on the system it comes off as an after thought.
It won't tell the difference and pkg(8) will just install everything from the repository you've set. So, be careful when mixing building from ports and using the official package repositories. You can and will run into dependency issues. Options disabled on a port you built will clash with the default options the official repositories use. That's why you set up Poudriere to build your own repository, and disable the FreeBSD repositories. Then you'll have everything from the same repository using the same default settings and options set. That gives you the benefits of ports (setting your own defaults and options) while keeping the ease of management of packages.

Again pretty manual just comes off as an after thought.
Ports and their options existed before packages. Once the package has been built you can't change any of its options or dependencies anymore. Everything starts with a port, no port no package.

I'm just looking for a more automatic way of maintaining ports without much manual intervention.
ports-mgmt/poudriere or ports-mgmt/synth and build your own repository.
 
It won't tell the difference and pkg(8) will just install everything from the repository you've set. So, be careful when mixing building from ports and using the official package repositories. You can and will run into dependency issues. Options disabled on a port you built will clash with the default options the official repositories use. That's why you set up Poudriere to build your own repository, and disable the FreeBSD repositories. Then you'll have everything from the same repository using the same default settings and options set. That gives you the benefits of ports (setting your own defaults and options) while keeping the ease of management of packages.
Is there anyway to solve the packages that only use 1 job to compile then?
Ports and their options existed before packages. Once the package has been built you can't change any of its options or dependencies anymore. Everything starts with a port, no port no package.
Of course this should be obvious.
 

SirDice

Administrator
Staff member
Administrator
Moderator
Is there anyway to solve the packages that only use 1 job to compile then?
See /usr/local/etc/poudriere.conf:
Code:
# By default MAKE_JOBS is disabled to allow only one process per cpu
# Use the following to allow it anyway
# ALLOW_MAKE_JOBS=yes

# List of packages that will always be allowed to use MAKE_JOBS
# regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports
# which holdup the rest of the queue to build more quickly.
#ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py*"

As for options, check the manual, you can set options via specific make.conf files. Those will be applied to your builds, thus you can have things build consistently with certain options set (or unset). For example, this is my /usr/local/etc/poudriere.d/make.conf:
Code:
DEFAULT_VERSIONS+= mysql=10.5m samba=4.13 java=17 php=8.1 lua=54

OPTIONS_SET+= OPTIMIZED_CFLAGS

devel_cmake_UNSET= DOCS
devel_py-molecule_SET= DOCKER
math_gmp_SET= CPU_OPTS
net_samba413_SET= MDNSRESPONDER
net_samba413_UNSET= AVAHI CUPS FRUIT
net_samba416_SET= MDNSRESPONDER
net_samba416_UNSET= AVAHI CUPS FRUIT
security_strongswan_SET= CURL GCM KDF SWANCTL VICI
security_sudo_SET= INSULTS
sysutils_cpupdate_SET= INTEL
sysutils_vm-bhyve_SET= BHYVE_FIRMWARE GRUB2_BHYVE
textproc_py-docutils_SET= PYGMENTS

And I have another that's specific to a 'server' repository, /usr/local/etc/poudriere.d/server-make.conf
Code:
OPTIONS_UNSET+= X11

www_nginx_SET= HTTP_FANCYINDEX
mail_exim_SET= CONTENT_SCAN DKIM DNSSEC MYSQL PGSQL SA_EXIM
security_suricata_SET= NSS

And a desktop-make.conf:
Code:
audio_pulseaudio_UNSET= AVAHI

devel_geany_SET= THEMES VTE
devel_libudev-devd_SET= GPL
devel_py-isort_SET= COLORS
devel_sdl20_SET= UDEV

graphics_libmng_SET= MNG_OPTIMIZE

multimedia_libdvdread_SET= DVDCSS_DLOPEN
multimedia_mplayer-skins_SET= ALL

print_qt5-printsupport_UNSET= CUPS

www_firefox_UNSET= JACK

x11_sterm_SET= SCROLLBACK SCROLLBACK

x11-drivers_xorg-drivers_SET= EVDEV
x11-drivers_xorgxrdp_SET= DRI3

x11-toolkits_gtk20_UNSET= CUPS
x11-toolkits_gtk30_UNSET= CUPS

x11_nvidia-driver-390_SET= ACPI_PM LINUX
x11_nvidia-driver_SET= ACPI_PM LINUX
#x11_nvidia-driver_UNSET= LINUX

I build 4 repositories, for 13.1-RELEASE and 13-STABLE, and a 'desktop' and 'server' set for each.

Code:
   Create optional make.conf
     You can also specify a global make.conf which will be used for all the
     jails.  Any of the following are allowed and will all be used in the
     order shown:

           /usr/local/etc/poudriere.d/make.conf
           /usr/local/etc/poudriere.d/<setname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-make.conf
           /usr/local/etc/poudriere.d/<tree>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<setname>-make.conf
           /usr/local/etc/poudriere.d/<jailname>-<tree>-<setname>-make.conf
           /usr/local/etc/poudriere.d/hooks/plugins/<plugin>/make.conf

So a 'server' build will combine make.conf and server-make.conf. The 'desktop' build combines make.conf and desktop-make.conf.
 
See /usr/local/etc/poudriere.conf:
Code:
# By default MAKE_JOBS is disabled to allow only one process per cpu
# Use the following to allow it anyway
# ALLOW_MAKE_JOBS=yes

# List of packages that will always be allowed to use MAKE_JOBS
# regardless of ALLOW_MAKE_JOBS. This is useful for allowing ports
# which holdup the rest of the queue to build more quickly.
#ALLOW_MAKE_JOBS_PACKAGES="pkg ccache py*"
Yes I have tried that it works great with 32 jobs it just gets stuck on like 4 or 5 packages where it then decides it wants to use 1 job to make that package and takes hours obviously.
 

SirDice

Administrator
Staff member
Administrator
Moderator
Some ports are just big builds and may hold up the queue for a long time. Chromium for example takes more than 24 hours to complete on the really beefy FreeBSD package builders. You can keep track of those here: https://pkg-status.freebsd.org/ (the logs on the actual build servers are only accessible on IPv6 though).

If you have a bunch of ports that require LLVM15 for example, those cannot be build until LLVM15 actually finishes. But 1 job running doesn't mean it's only using one core (it does do this by default). That's what ALLOW_MAKE_JOBS_PACKAGES does, it'll tell the job to use all available cores to build that particular port. So llvm* is a good candidate to add to ALLOW_MAKE_JOBS_PACKAGES.

I have ALLOW_MAKE_JOBS_PACKAGES="pkg llvm* gcc* rust firefox node* mame webkit*", but this sometimes means it tries to build LLVM15 and GCC12 on all cores at the same time and that will choke my build server (running out of memory), causing both package builds to fail.
 
Some ports are just big builds and may hold up the queue for a long time. Chromium for example takes more than 24 hours to complete on the really beefy FreeBSD package builders. You can keep track of those here: https://pkg-status.freebsd.org/ (the logs on the actual build servers are only accessible on IPv6 though).
So this is a FreeBSD issue cause on Gentoo I can compile say LLVM is under 20 mins but it's using 32 jobs where I can see on poudriere it's only running 1 job at a time and take over 5 hours. Running a 7950X for context.
 

DutchDaemon

Administrator
Staff member
Administrator
Moderator
Developer
So this is a FreeBSD issue cause on Gentoo I can compile say LLVM is under 20 mins but it's using 32 jobs where I can see on poudriere it's only running 1 job at a time and take over 5 hours. Running a 7950X for context.
Look into PRIORITY_BOOST in poudriere.conf. This is an awesome tool, and it needs configuration. A common setting is
Code:
PRIORITY_BOOST="gcc* llvm* rust"
.
 

SirDice

Administrator
Staff member
Administrator
Moderator
So this is a FreeBSD issue
No, it's a Poudriere configuration issue. You need to adjust poudriere.conf for your situation.

And also note that it may be more beneficial to have less jobs running. Building/compiling also uses a lot of I/O, so it's sometimes faster to build with less jobs running because more jobs would also mean a lot more I/O and you may be bottlenecking there. So you need to play around a bit with the various options to get it to build as efficiently as possible for your hardware.

Depending on the amount of memory this is also a great option to play with:
Code:
# Use tmpfs(5)
# This can be a space-separated list of options:
# wrkdir    - Use tmpfs(5) for port building WRKDIRPREFIX
# data      - Use tmpfs(5) for poudriere cache/temp build data
# localbase - Use tmpfs(5) for LOCALBASE (installing ports for packaging/testing)
# all       - Run the entire build in memory, including builder jails.
# yes       - Enables tmpfs(5) for wrkdir and data
# no        - Disable use of tmpfs(5)
# EXAMPLE: USE_TMPFS="wrkdir data"
USE_TMPFS=yes
Setting this to all for example would be really fast but it'll also consume an enormous amount of memory.
 
One question related to this, is there any relation between the pkg repo and the ports version? For example, if I want to compile a lot of ports, should I follow the quartely or the latest repo? Or there is no correlation?
 
Look into PRIORITY_BOOST in poudriere.conf. This is an awesome tool, and it needs configuration. A common setting is
Code:
PRIORITY_BOOST="gcc* llvm* rust"
.
I'll have to look more into this thanks.
No, it's a Poudriere configuration issue. You need to adjust poudriere.conf for your situation.

And also note that it may be more beneficial to have less jobs running. Building/compiling also uses a lot of I/O, so it's sometimes faster to build with less jobs running because more jobs would also mean a lot more I/O and you may be bottlenecking there. So you need to play around a bit with the various options to get it to build as efficiently as possible for your hardware.

Depending on the amount of memory this is also a great option to play with:
Code:
# Use tmpfs(5)
# This can be a space-separated list of options:
# wrkdir    - Use tmpfs(5) for port building WRKDIRPREFIX
# data      - Use tmpfs(5) for poudriere cache/temp build data
# localbase - Use tmpfs(5) for LOCALBASE (installing ports for packaging/testing)
# all       - Run the entire build in memory, including builder jails.
# yes       - Enables tmpfs(5) for wrkdir and data
# no        - Disable use of tmpfs(5)
# EXAMPLE: USE_TMPFS="wrkdir data"
USE_TMPFS=yes
Setting this to all for example would be really fast but it'll also consume an enormous amount of memory.
I have tmpfs enable to yes already, all is to much for me sadly. Poudriere is a great tool for someone like you building packages for multiple machines it overly complex for the person like myself building for the machine its compiling on.
 
One question related to this, is there any relation between the pkg repo and the ports version? For example, if I want to compile a lot of ports, should I follow the quartely or the latest repo? Or there is no correlation?
I would assume the ports are based on the same versions as in your release.
 
Look into PRIORITY_BOOST in poudriere.conf. This is an awesome tool, and it needs configuration. A common setting is
Code:
PRIORITY_BOOST="gcc* llvm* rust"
.
I think this has fixed my issue build times are much more inline with what I was expecting thanks! I still think it's annoying to use a separate build tool and then add it as a repo just to use the packages though. Would be nice to see something integrated directly into pkg just my 2 cents.
 
This is my line. All the big ones are in it,
Code:
PRIORITY_BOOST="ghc llvm* gcc* node* rust* electron* qt5-webkit qt5-webengine firefox-esr chromium iridium ungoogled-chromium"
 
Would be nice to see something integrated directly into pkg just my 2 cents.
Since you are new here, I just want to make sure you are clear about the following:
You do not _need_ to compile packages on your own. There are also precopiled packages you can install within seconds. Example: pkg install chromium will download and install Chromium.
 
Since you are new here, I just want to make sure you are clear about the following:
You do not _need_ to compile packages on your own. There are also precopiled packages you can install within seconds. Example: pkg install chromium will download and install Chromium.
Well aware of that I think this is a huge plus for a package manager but I also would have to say if all that package manager can do is install binaries it sucks... I shouldn't need extra tools to build a package. I like the optimizations and build options of fine tuning what packages I need and don't need on my system and the only way to do that is to build packages. The ideal solution I would say is to at the very least be able to do both so people can compile lighter things like libraries themselves and use binaries for things like firefox or llvm as they can be a real pain on some hardware. Ik Poudriere can do this by running 2 repos but again this should be something native IMO. Nothing is perfect...
 
I have been running FreeBSD 14-Current for about a month now [...]
Since you are new here, I just want to make sure you are clear about the following:
You do not _need_ to compile packages on your own.
When running -CURRENT, you do need to do that.

When running a supported -STABLE, using pkg(8) with either the quarterly or latest repository works almost 100% of the time. See the first page (mainly) of the informative thread: Proper way to use -STABLE
 
SirDice DutchDaemon ok so after building the system I am running into common place issues when removing certain options. On Gentoo when I change an option in my make.conf the package manager portage finds all the affected packages with that option and allows me to change it and of course there is a custom package.use file so set specific different options for different packages that may differ from the global. On Poudriere can I do something similar where I change the options in make.conf is there a command I can run where it will update the options for those affected packages or at least see the config again for only those packages without manually deleting the config?

Also when a package fails to build in Poudriere it just says failed build is there like a log somewhere to find out why?

Lastly Is there a way to setup certain variables to fix said issues with compiling with something similar to this or do I need an entire separate poudriere jail for just one or two packages that may act up? - https://wiki.gentoo.org/wiki//etc/portage/package.env

Thanks!

When running -CURRENT, you do need to do that.

When running a supported -STABLE, using pkg(8) with either the quarterly or latest repository works almost 100% of the time. See the first page (mainly) of the informative thread: Proper way to use -STABLE
For what its worth I can install binaries on -CURRENT just fine and they work fine.
 
And -RELEASE with either the quarterly or latest repository?
Yes exactly, but that sort of goes without saying :) Binary packages are built and tested for specific -RELEASE-s (major.minor versions and architecture). Using pkg(8) could nowadays be considered "the default" when the binary packages with the options with which they were built on FreeBSD servers are what you want. Building your own stuff from ports used to be the default.

One question related to this, is there any relation between the pkg repo and the ports version? For example, if I want to compile a lot of ports, should I follow the quartely or the latest repo? Or there is no correlation?
Both methods: either building/compiling from the source ports or adding binary packages via pkg(8) are at your disposal. However, note that "either ... or" should be a choice made on the basis of a FreeBSD installation as a whole, not on the basis of software X or Y. If you make the choice on the basis of "this" or "that" software, you are mixing ports and packages; be forewarned not to do that: Warning.

FreeBSD management of software outside of the FreeBSD base installation is not set up/designed to "just work" when you both build your software from ports and are using pkg(8) for direct binary installation (=mixing ports and packages).

When the choice has been made to build your own stuff and get the source from ports, I think one usually chooses to get the sources "from head" in the ports collection, that is "latest". However, you can just as well choose to get the source from quarterly; see Installing the Ports Collection. Again, make a choice for the system as a whole: get the source from either "latest" or "quarterly". You may of course decide to switch between latest and quarterly as a whole and rebuild al your ports software from source.

Also, I recommend: [Guide] About ports and (binary) packages
 
Top