portmaster best practices?

eldaemon

Member

Reaction score: 11
Messages: 98

I compile all of my own ports into packages and keep the distfiles so I can rebuild it all offline if need be.

I recently switched over to the 2020Q4 ports tree. Normally, my process is svn update (in /usr/ports), portmaster -a -B -D -G -y --no-confirm -g. This seems to work pretty well when doing updates before the quarterly jump, but after a quarterly jump it gets more dicey.

For instance, it failed this time. I think libffi was updated and then glib2 failed because Python 3 was built against the older libffi. I rebuilt Python 3 which fixed the issue and I'm able to run portmaster again to keep upgrading. Also, I don't think rebuilding a package bumps the package version number, so pkg upgrade across my systems wouldn't upgrade it which is another issue.

Is there an extra switch I should be using? Should I be using a different build tool to make this more bulletproof?

Thank you!
 

vigole

Aspiring Daemon

Reaction score: 681
Messages: 628

ports-mgmt/portmaster Best practice:

Step #1:
pkg delete ports-mgmt/portmaster

Step #2:
Ports:
cd /usr/ports/ports-mgmt/poudriere && make install clean
cd /usr/ports/devel/ccache && make install clean
OR
Binary:
pkg install ports-mgmt/poudriere
pkg install devel/ccache

Step #3:
Why?
 

olli@

Aspiring Daemon
Developer

Reaction score: 845
Messages: 798

I do not like the black&white attitude “poudriere is the only correct way to build ports, and everything else is wrong”. It’s not that simple.
For example, I’m used to build from ports with some helper scripts, and this works very well for me and it’s comparatively fast. There are no real disadvantages doing it that way.

I don’t use portmaster (anymore), though, so I can’t answer the OP’s particular question.
 

Minbari

Well-Known Member

Reaction score: 254
Messages: 493

I'm using -ad -w (to save old shared libraries before deinstall) and sometimes when I don't want to update big ports -i (llvm's, ruby, gcc, etc.). Also, I consider poudriere/synth are useless when I'm running FreeBSD on a single machine which is using quarterly ports.
 

chrcol

Well-Known Member

Reaction score: 41
Messages: 436

First comment, I dont like the spamming on here from some members trying to push Poudriere, its ok maybe in its own thread, but when someone asks how to do something with portmaster, the answer is not use Poudriere instead.

Now I dont usually use portmaster, not a fan of its decision making, but I have checked the man page to try and see what went wrong with the flags chosen. Your flags wouldnt manipulate the build order so I dont know sadly.
 

anonymous9

Member

Reaction score: 43
Messages: 70

Ports management in FreeBSD needs to coalesce around one method. Having 20 ways to do the same thing does not always make something better because there are so many choices, especially when dealing with ports. Nobody would ever say, "You know, we have a dozen different methods to upgrade FreeBSD to a new version or release, and that's what makes FreeBSD so great. So many choices."

While choices are good for databases, firewalls, web servers, load balancers, or many other applications, I don't think it should extend to some domains and ports is one of them. Invariably, mixing ports and packages screws over too many, too, which also needs to be addressed in some way.

ports-mgmt/synth
ports-mgmt/poudriere
ports-mgmt/portupgrade
ports-mgmt/portmaster
 

richardtoohey2

Active Member

Reaction score: 125
Messages: 246

Sure, but who gets to choose which one? portmaster has been fine for me for many years but it doesn't work for everyone. If I was less lazy I'd look at what I actually use it for and figure out how to just extract that part (I think that would be like what olli has described above - a helper script or two, which is probably what portmaster started out as). But it still wouldn't be poudriere.

I'm not helping the OP either - I follow the latest ports rather than quarterly - so 🤐 from me now.
 

Oclair

Member

Reaction score: 7
Messages: 32

I primarily use portmaster, and use pkg to do some trimming. I don't get into trouble any more generally. There are many ins and outs,

for example a basic updating run
first check UPDATING,
then run #rm -R /usr/ports/distfiles/*
then #portsnap fetch
and finally #portmaster -ay

Many times things break or as I like to think go bump in the night and stop portmaster, at such times generally going into the dependent port it's self
#cd /usr/port/libsomething/brokeportmaster
# make deinstall reinstall clean

and then things seem to work out

Other times UPDATING (should) mention what to do, which might mean modifying /etc/make.conf to actually set what your own personal preferences are
(here's one of mine) mileage varies
##/etc/make.conf override

DEFAULT_VERSIONS+= php=7.3
DEFAULT_VERSIONS+= mysql=10.2m
DEFAULT_VERSIONS+= python3=3.7
DEFAULT_VERSIONS+= ssl=openssl
#DEFAULT_VERSIONS+= ssl=openssl111

MAKE_JOBS_UNSAFE=yes
FORCE_MAKE_JOBS=yes
MAKE_JOBS_NUMBER=8

#LICENSES_ACCEPTED+=NONE

OPTIONS_UNSET+= X11 GUI CUPS NLS HAL GSSAPI_BASE KRB_BASE KERBEROS
OPTIONS_SET+= GSSAPI_NONE KRB_NONE
WITHOUT_X=YES
WITH_X=NO
ENABLE_GUI=NO
 

drhowarddrfine

Son of Beastie

Reaction score: 1,665
Messages: 3,711

Invariably, mixing ports and packages screws over too many, too, which also needs to be addressed in some way.
Packages are created from ports so it is addressed. The reasons for not mixing them is well known.

I always thought synth was the ed of managers and poudriere was for those installing on multiple servers and workstations. I thought portupgrade was going away and recently read that portmaster was too. Not that I've paid that much attention. I primarily use make and pkg for large programs that take forever to build.
 

olli@

Aspiring Daemon
Developer

Reaction score: 845
Messages: 798

Ports management in FreeBSD needs to coalesce around one method. Having 20 ways to do the same thing does not always make something better because there are so many choices, especially when dealing with ports. Nobody would ever say, "You know, we have a dozen different methods to upgrade FreeBSD to a new version or release, and that's what makes FreeBSD so great. So many choices."
Having choices is always a good thing, IMHO, and this is one of the main advantages of UNIX-derived systems. You can take the way that is most suitable for you, and that doesn’t necessarily has to be the way suitable for most others.

Maybe there are not a dozen methods to upgrade FreeBSD, but I can think of four or five from the top of my head, maybe more. And this is good. There is no “one size fits all”.
 

Oclair

Member

Reaction score: 7
Messages: 32

I really do not understand what the obsession some folks have to remove a tool which manages the ports system. Perhaps it's good they blame the messenger than the Ports system it's self. Don't mess with Portmaster, I don't care how you manage your ports pal.
 

vigole

Aspiring Daemon

Reaction score: 681
Messages: 628

olli@ You're right. My reply could be count as a hijack-post. I apologise for that.

Onwards.

There's only one minor issue. Somebody else has suggested, because I was mentioning poudriere, then I'm a SPAMMER! I don't think promoting a program, which has a section in the FreeBSD handbook, and has been repeatedly suggested by FreeBSD committers, developers and system administrators counts as SPAM. Because I'm not knowledgeable enough to compare the technical details of different solutions, I have to rely on expert in the field, in this case developers and administrators. That's why I was suggesting to use poudriere instead of portmaster. Also I'm not the developer of poudriere, so I don't have a dog in the fight. All that said, I admit that my reply could be a hijack-form of posting, thus I'm not going to argue further about this topic.
 

olli@

Aspiring Daemon
Developer

Reaction score: 845
Messages: 798

There's only one minor issue. Somebody else has suggested, because I was mentioning poudriere, then I'm a SPAMMER!
Calling you (or anybody else here) a “spammer” is completely inappropriate.

The discussions here should focus on technical contributions, not personal attacks. I guess I’m not innocent either, calling you a hijacker was not the right thing to do, and I have to apologize for that. When someone asks a question regarding portmaster, it’s not completely unreasonable to mention poudriere as a potential alternative, even though it’s not really an answer to his question.

However, statements along the lines of “poudriere ist the only way to go, and everything else is just wrong” are not helpful. It provokes opposition and disagreement, and it causes conflict and division. We can observe that very well in these threads.
 

anonymous9

Member

Reaction score: 43
Messages: 70

Probably should be a different thread, but no, multiple ways does not make it better just because you can do it. Take jails as another example:
sysutils/iocage
sysutils/ezjail
sysutils/cbsd
sysutils/bastille
sysutils/qjail
sysutils/quickjail

And I'm certain I'm missing a few. What I hear is, "This is great, so people have options." That's absurd for some things. Jails are in the base, so there should be one official way that is maintained to manage jails. Not at least a half dozen methods, some of which aren't fully implemented and some probably not maintained.

Someone new to FreeBSD comes along and asks, "How do I install packages?" And some say binary packages, while others say only use ports and use poudriere, while others say portmanager, then some say synth. Then the OP starts reading on poudriere and says "I only want to install a package." Gives up and leaves.

Someone new to FreeBSD comes along and asks, "How do I manage jails?" And some say ezjail while some say cbsd. Then others say bastile, and others say iocage. Then more say to just use the base utilities to manage jails. Gives up and leaves.

If you're using SmartOS and you want to install zones you use imgadm and vmadm. They come with the base install. That is sensible and doesn't make SmartOS worse because there isn't 10 ways to install a zone.
 

olli@

Aspiring Daemon
Developer

Reaction score: 845
Messages: 798

Someone new to FreeBSD comes along and asks, "How do I install packages?" And some say binary packages, while others say only use ports and use poudriere, while others say portmanager, then some say synth. Then the OP starts reading on poudriere and says "I only want to install a package." Gives up and leaves.
What’s wrong with having one recommended way that works for most users (in particular for people new to FreeBSD), but offering more choices for users who have special requirements?

In the case of installing packages, the recommended way is to use pkg, of course (see the FreeBSD Handbook). If you want to compile from the ports collection yourself, the recommended way seems to be poudriere (although I’m not sure if this is an “official” recommendation). But if portmaster works better than poudriere for a certain user, well, why would you forbid him to use it? Would you want to ban all tools except for poudriere? If you want a monoculture of tools, then you shouldn’t be using an open source operating system with public APIs.

By the way, there are even multiple different ways to install binary packages beside pkg. For example, you can use bsdconfig(8) which provides a full-screen UI for selecting packages. There are also several GUI tools for X11. Having choices is good. ;)
 

drhowarddrfine

Son of Beastie

Reaction score: 1,665
Messages: 3,711

Having choices is good.
Having too many choices is bad. There is even a UX term for it that I can't recall, having just woken up.

There should be one general purpose manager one can point a new user to so they can get started. After they get comfortable, let them discover other installers on their own. Perhaps this is a documentation issue more than anything else. When I would train people, I would say, "Here's how you do this. There are other ways you might like better but this is good for now. You'll find there are other ways later that you'll like better but, for now, do it this way."

I do think the Handbook is good about that--limiting choice to two methods of doing things.
 

mjollnir

Daemon

Reaction score: 830
Messages: 1,267

What’s wrong with having one recommended way that works for most users (in particular for people new to FreeBSD), but offering more choices for users who have special requirements?
Nothing, except that the defaults present a pitfall for newbies: the version mismatch of ports(7) tree (latest branch) vs. packages (built from the quarterly branch). The (bad) consequence is that dependencies will be pulled in from another branch when you start to build a single software package from ports on a system installed from binary packages.
In the case of installing packages, the recommended way is to use pkg, of course (see the FreeBSD Handbook). If you want to compile from the ports collection yourself, the recommended way seems to be poudriere (although I’m not sure if this is an “official” recommendation).
ports-mgmt/poudriere is semi-official in that the packages for the official repositories are built via Poudrière. The strength of Poudrière is to handle profiles, whereas ports-mgmt/synth might be much easier for the case when users have a small number of FreeBSD hosts.
But if portmaster works better than poudriere for a certain user, well, why would you forbid him to use it? Would you want to ban all tools except for poudriere? If you want a monoculture of tools, then you shouldn’t be using an open source operating system with public APIs.
It's not about forbidding, but to help newbies to prevent to shoot themselves in their foot. Other than that, portmaster manages to let experienced users to shoot into their feet, too... (see post #1 and other threads). Denying the obvious bugs of portmaster à la "portmaster has worked fine for me for decades" is not helpful either. Note that it is a shell script of >1 kLoC, beeing in a refactoring process & rewritten in LUA since more than a year now; that's a heavy task because some of it's behaviour is broken-by-design.
By the way, there are even multiple different ways to install binary packages beside pkg. For example, you can use bsdconfig(8) which provides a full-screen UI for selecting packages. There are also several GUI tools for X11. Having choices is good. ;)
bsdconfig(8) and other (G)UI tools use pkg(8) under the hood. Please note that some ports are self-contained in the ports(7) tree (scripts) and do not exist as binary package; i.e. these can only be installed from ports(7). On the genuine topic, my recommendations are
  • Use the same version of ports(7) tree & package repository, i.e. either use the latest package repository in /etc/pkg/FreeBSD.conf, or manage your quarterly ports(7) tree with svnlite(1), net/svnup (don't mix these two!) or git(1) instead of portsnap(8). Portsnap only manages the latest ports tree & will vanish soon, anyway.
    While this is not strictly neccessary, this will enable you to safely use portmaster(8)'s -P and -PP switches. Frontends & GUI tools for Subversion/svn & git(1) exist, e.g. psearch -c devel -s git -s GUI
  • Work around portmaster(8) bugs where possible, e.g. use make -C /usr/ports/category/someport fetch-recursive instead of the corresponding portmaster(8) command. In some cases, you can derive the neccessary tasks from RTFM, in some it means RTSL.
  • Use ports-mgmt/portconf in combination with portmaster.
  • Using devel/ccache and/or devel/distcc will speed up long builds regardless of the method you choose.
 

olli@

Aspiring Daemon
Developer

Reaction score: 845
Messages: 798

mjollnir, I basically agree with most of what you wrote (with minor disagreement in some details, but that’s really not that important).
Just some notes and additions …

Nothing, except that the defaults present a pitfall for newbies: the version mismatch of ports(7) tree (latest branch) vs. packages (built from the quarterly branch). The (bad) consequence is that dependencies will be pulled in from another branch when you start to build a single software package from ports on a system installed from binary packages.
Yes, I agree, those defaults can cause trouble. This is why newbies are told to not mix ports and packages (and this is pretty much independent from the tools you use, whether portmaster or something else).

Those defaults have historic reasons. The concept of quarterly branches is relatively new – Not too long ago there was only “current” ports (a.k.a. “head”). Shortly after the quarterly branches were introduced, the decision was made to use them by default for the package repository of new releases, because they were assumed to be more stable than “head” and in a more consistent state, and therefore newbies would have less problems with them. On the other hand, portsnap continued to only deliver “head” – partially because of technical reasons (without major changes, portsnap itself and the infrastructure behind it were not able to handle multiple branches), partially because people who decided to use portsnap and build from the ports collection themselves were assumed to be advanced users who prefer to have the most current software.

There are other tools beside portsnap, of course. For example, net/svnup is quite popular for fetching source and ports. Unlike portsnap, it supports branches, and there is no default. You have to specify the branch that you want. So you can use svnup to fetch a quarterly branch and build from it without problems. Of course, you can do the same with svn, svnlite or git if you want. The advantage of svnup is that it just fetches the files, without all of the overhead of local repository data, so the end result is exactly the same as if you used portsnap, except that you can use branches.

For that reason I’m not afraid of portsnap going away, because svnup is a good replacement. And even if svnup didn’t exist (or stopped working for some reason), svnlite can be used as well. Since there’s a git-to-svn gateway, it’ll continue to work even when the FreeBSD repositories switch over to git. (However, I assume that there will be specialized tools for git at some point in the future, maybe an svnup-workalike called “gitup” or something like that.)

It's not about forbidding, but to help newbies to prevent to shoot themselves in their foot.
Well, as the saying goes:
“UNIX does not prevent you from doing stupid things, because that would also prevent you from doing clever things.”

But seriously … Well, yes, portmaster has some bugs and shortcomings. I think nobody is denying that. It is probably fair to say that it should not be the first choice for newbies. However, the OP of this thread doesn’t appear to be a newbie.

Note that it is a shell script of >1 kLoC
Uh … Are you sure? I assumed it was much larger. I have to admit I’ve never looked at it myself, only read about it at various occasions.

My own script is currently 2.2 kLoC, plus a few small helper scripts that add up to another 0.5 kLoC, roughly. (One third of that is comments, though.) Of course, it started as a much smaller script. If I had known at the beginning how it would evolve, I would have written it in Python instead, probably. But I’m not going to rewrite it now. It’s still manageable as a shell script since I made some efforts to have it structured in a useful way.

bsdconfig(8) and other (G)UI tools use pkg(8) under the hood.
Yes, of course. Likewise, all of the various ports building tools use the ports collection’s Makefile framework under the hood.

Please note that some ports are self-contained in the ports(7) tree (scripts) and do not exist as binary package; i.e. these can only be installed from ports(7).
There are several reasons why no binary packages are provided for certain ports. For example, some software has license restrictions that prohibit binary redistribution, or you have to confirm a license agreement when you build the port. Also, some ports cripple their options when building a package for redistribution – for example, the binary package of audio/sox has no LAME support (for mp3 encoding), even though the LAME option is enabled by default when you build the port yourself. Furthermore, some ports must always be compiled from source because the package is specific to the local machine for some reason (e.g. it puts hardware-specific things in the binary, or a different key is generated each time you build the port, or similar). Currently there are about 250 ports affected by things like that.

On the genuine topic, my recommendations are […]
Agreed on all points. 👍
 
Top