Auto-lock/unlock packages from ports & running pkg upgrade

Dealing with pkg upgrade wanting to clobber your custom optioned ports seems to be a common issue...

This doesn't seem like it's going to get fixed in a nice way any time soon, so combining some of the suggestions from some my searches, I think I've come up with a pretty good one-line solution.

First, use the annotations system to add the tag "from-ports" to any package you want built from ports (apr in this example). The value can be anything, but noting which options to select may help your future self.

sudo pkg annotate -M apr from-ports 'With SQLite option'

Then run this huge one-liner to automatically lock those packages, run pkg upgrade, and then unlock the packages

pkg query '%n %At' | grep from-ports | awk 'BEGIN {FS=" "};{printf "%s\n",$1}' | sudo xargs -L 1 pkg lock -y ;sudo pkg upgrade;pkg query '%n %At' | grep from-ports | awk 'BEGIN {FS=" "};{printf "%s\n",$1}' | sudo xargs -L 1 pkg unlock -y

  • The pkg query / grep / awk combo is to overcome the fact that pkg query -e doesn't appear to be able to match on annotation tag names.
  • awk using \n along with the -L argument to xargs is to overcome pkg lock not supporting multiple package names in the command.
  • I know about custom repos. Like most people who have encountered this problem, they are not an appropriate solution compared to this one.
  • %R in pkg query isn't reliable to key on for packages built from ports. For instance, ports may have built a package as a dependency, but it doesn't have any special options, so the binary update would be just fine.

Please leave a comment if I've overlooked something!
Why not simply stop mixing ports and packages?

If you need to customize port options then make sure you build your whole setup from ports, otherwise simply use packages.

There really isn't a big problem which needs fixing here, the system works fully as intended. It all boils down to dependencies: if you mess with a package dependency then it obviously can't sustain itself. (edit): Trying to circumvent this with clever tricks will, in my opinion, only postpone the inevitable: ending up with a broken system (and most likely not knowing what the problem is anymore).
I expected this might instantly devolve into "mixing ports and pkg is doing it wrong"
  1. It takes too long to compile everything from ports
  2. Typically only a couple things need to be customized
  3. Maintaining custom repos is a lot of work
People want to mix them, it's set up so you can mix them, building everything from ports functionally works the same as this, and if pkg would just stop pretending ports didn't exist, things would be nicer.

FWIW, I've tried maintaining the entire system from ports most of my time using FreeBSD, which goes back to the 2.2 series.

Anyway, again, if there's something specific that will go wrong with this method you can point out, please let me know.
Maintaining custom repos is a lot of work
Not really. It's a lot less work than doing it your way. Yours is going to haunt you sooner or later.

Most of the maintenance of a custom repository is automated, so even if it takes several hours you don't have to babysit it. Just let it churn away during the night.

if pkg would just stop pretending ports didn't exist, things would be nicer.
And this is a big misconception. There is NO difference between a port and a package. A port builds a package and it's this package that gets installed. So from the system's point of view they're all packages, even if you installed everything from ports. You're trying to mash packages with different options and dependencies together. Once a package has been built its dependencies are set in stone and cannot be changed.

Just to give you an example how much "work" I have to do to maintain 20 or so FreeBSD servers using a custom repository. I go to that client every Wednesday. Every Tuesday evening poudriere updates itself (cronjob). It then starts building updated packages. So every Wednesday morning I come in and find a freshly updated repository. This takes me zero to no effort. I then proceed to run pkg-upgrade(8) everywhere and be done before lunch. No fuss, no issues, no dependency misery.
Personally, I'm the arbiter of how I spend my time and CPU power, and from where I see it, downloading binaries of packages that don't have their options changed takes less time and space than building them from ports. My understanding of why pkgng exists and why binaries are even available is a tacit acknowledgement of this. Your specific example assumes that every server I administer needs the same options for the packages on all the servers, which is not true for me.

You say "haunt," which is a nebulous way of saying there's a problem with what's going on here. Do you have any specific examples instead? The only problems I've had with mixing pkg and ports are when I forget which one I built from ports. It would be nice if pkg would do that for me.

I don't think what I said about pkg is wrong. If there's no difference between a port-built package and a binary installed package, then pkg is indeed "pretending that ports doesn't exist." All I'm trying to do by adding an annotation to my ports-built packages is making it so that I can know the source. If pkg knew ports existed in this way, it would be easier to exclude them from pkg-upgrade.

Indeed, once a package has been built and its dependencies can't be changed. Fortunately this information is encoded into the package and neatly managed. This is no different than binary packages. I don't see that as being a problem with mixing pkg and ports.
Your specific example assumes that every server I administer needs the same options for the packages on all the servers, which is not true for me.
Your assumption about my systems is wrong. I have several repositories. Typically at least two, although both are similar, one for AMD64 and one for i386. But during various transition periods I've also had more different repositories. When we had to migrate from Apache 2.2 to 2.4 for example (2 x 2 sets; ie. 4 repositories). And again when we migrated from MySQL 5.5 to 5.6 to 5.7 (1 set based on 5.5, 1 set based on 5.6, 1 set based on 5.7). I've also had to battle with Ruby 1.8 all the way through 2.4 (this was long before FLAVORs existed). But yes, I do end up with "everything" the same. Because maintaining more than a few servers while trying to maintain your own sanity requires standardization.

You say "haunt," which is a nebulous way of saying there's a problem with what's going on here. Do you have any specific examples instead?
It's going to mess up with renamed, moved or deprecated ports/packages. I also see problems if you lock a package another package depends on. So if packageA is locked but PackageB needs to be updated and requires an updated PackageA. That's going to fail or at the very least leave you with a broken system. I actually see a lot of potential problems.
So you have to build the packages twice? Again, that's CPU and disk space I don't believe is necessary for me to use.

Thanks for the specificity. Locking ports-compiled packages during upgrade doesn't absolve this method of running portsnap update. Solving those sorts of origin problems by paying attention to the UPDATING file and using -o options seems standard fare.
So you have to build the packages twice?
Yes, but who cares? It's a computer, its primary function is to do mundane tasks, repeatedly. And it happens at night when nobody's around. Power is already paid for and on that system I have lots of CPU and disk space to spare. If building things twice or more times results in package upgrades I can blindly trust then I'll happily to do so.
I expected this might instantly devolve into "mixing ports and pkg is doing it wrong"
There is a good reason for that too. Please realize that most of us aren't just spouting opinionated theory which somehow got accepted as factual over time. Most of us have seen plenty of examples during troubleshooting where it turned out that the cause of the problems was in fact mixing ports & packages.

Also be aware that I never said that doing so is wrong. Mixing ports and packages isn't wrong per definition, under specific circumstances it can easily work out just fine. But it will become a serious problem over time if you start trying to fight the symptoms you experience because of it. I've simply seen too many examples of that over the years I've been active within the FreeBSD community.

  1. It takes too long to compile everything from ports
I definitely agree that some ports can take ages to build, especially on older hardware. But there's software which can handle that for you. I'm a huge fan of ports-mgmt/portmaster which I use with a project of my own called portmaster-assist (basically a set of shell scripts to handle most common tasks). All I basically do to keep my server up to date is run ./refresh, then I check refresh-updating.log for any specific upgrade instructions (extracted from /usr/ports/UPDATING) and then I run ./update. Leave the system running while I do other stuff and I'm done. Usually the updates really don't take all that much time, a few hours at best.

The best part is that Portmaster can also create a collection of packages which can be directly used for a repository of your own. Very little customization is required, so another server I maintain is using binary packages but from my own repository.

  1. Maintaining custom repos is a lot of work
Wrong. It's pretty much an automated process. And keep well in mind that I don't even rely on automated tools such as Poudriere because I actually somewhat dislike those ;)

Just to give a bit of background info: I told Portmaster to not only install the package(s) build from the port(s) but to also copy those to /usr/ports/Packages/All (default location). Everything which gets build & installed on my server will end up as a package in there.

I symlinked the directory into /usr/local/www/repos, I made a public & private key using OpenSSL to sign my repository (pkg-repo(8) has a working example for this) and now all I have to do is run # pkg repo /usr/local/www/repo /path/to/my/private.key to create / update the repository itself. This command gets run through a cronjob every so many days.


Well, other than providing the repository through HTTP (setting up Apache) and pointing my other server(s) to this repository of course (creating /usr/local/etc/pkg/repos/stuff.conf and copying my public key over). But that's the kind of stuff you only do once anyway.

It's honestly hardly as much work as you think it is. And keep in mind that I'm actually taking the "hard" route because I don't rely on tools which automate the whole lot (sort off, I do rely on Portmaster of course).

People want to mix them, it's set up so you can mix them, building everything from ports functionally works the same as this, and if pkg would just stop pretending ports didn't exist, things would be nicer.
Who says you can't mix them? You can, and it's quite easy too.

I think you seriously misunderstand the nature of the problem here. It's not that you can't mix them, it's simply that binary packages have a predetermined set of dependencies. And if you somehow interfere with those dependencies then you'll create a situation which you need to deal with. That's all which is happening here.

Those packages which pkg wants to remove? There's a good reason for that yet that reason has absolutely nothing to do with mixing ports & packages, it all boils down to dependencies. Make sure those dependencies are kept satisfied and you're home free.
How much work it is to set up a private repo is "just like, your opinion," or my opinion, which obviously differs. It's not necessary to to debate opinions. Now, if there are specific problematic facts about this setup I described, I specifically asked about that. I know about portmaster and I use it. I'm not unfamiliar with how the private repo works. I really do not want to spend the time, space, and more specifically maintenance of a private repo. If it's not "wrong" to set it up this way, then why did this thread devolve into telling me to stop trying to mix pkg and ports?

Comparing make config against pkg info is a valid concern. The "good reason" pkg has to remove my custom built ports is usually because it either has to remove them to upgrade them (something I'm doing by myself with portmaster) or it complains that the options have changed from the binary repo and wants to normalize this. Both of these situations are managed. The entire reason this technique temporarily locks the packages is because I expect to update them presently via ports.

I can understand that people mixing ports and pkg and not understanding things like your GSSAPI selection in mail/dovecot22 is going to have to match other binary ports makes answering questions on this forum more difficult. I also understand that trying to make this situation more commonplace by making it easier to mix them might make more people ask annoying questions. That's actually why I thought that holistically, pkg knowing about ports would be better in the long run because you could identify this problem earlier in a troubleshooting process, such as "please run pkg query '%R' | uniq -c and if you see FreeBSD-Ports, it's up to you to manage dependencies by yourself."

Really, though, I didn't want to re-litigate this issue again! I simply wanted to know if there was a problem with the way I kept pkg from clobbering my custom built ports, such as how I add the annotations, how to select on them, or a specific problem with mixing ports and pkg.
The real problem is the FreeBSD's ports system that forces a rigid set of dependencies on everything you build from ports. If a port requires xyz-1.2.3 then you have to have xyz-1.2.3 installed, xyz-1.3.0 won't work. The dependencies could be relaxed if the ports system was aware of the set of compatible (ABI compatible to be accurate) versions of the same port and ports-mgmt/pkg was modified to allow these relaxed dependencies to be satisfied. So to answer your question, it's a problem of mixing ports and packages in a way where you're not limiting the mixing to leaf ports (ports that are on needed by anything else) only. As soon as you start mixing different versions, different options, different dependencies on ports that are required by other ports you have just entered a minefield.

You can dance around problems with various unsatisfactory methods like pkg lock but the only real solution is to build your own repo and only use packages from it.

There are two main tools for doing that and there are two HOWTOs covering those, first one by yours truly:
Since packages are built from ports, a port requiring a lower version would be updated to requiring the newer version. Updating with portsnap along with pkg as I noted earlier would seem to solve this problem.

I've been doing this to add SQLite support to apr, which is not a leaf port, and there aren't any problems. If I was taking an option out or switching GSSAPI providers, yes, you'd have to keep track of packages that would need to be taken into management under ports, such as ones that pull actionable options out of make.conf. It's not impossible.
I am doing dual mode maintenance of ports and packages on several FreeBSD servers for years now, and it works just fine. I didn’t experience any of these show stopper problems which the nay-sayers always paint on the wall.

I wrote a shell script which does the job, and when I started working on this, I was not able to get pkg lock/unlock to behave well for me, so my script does the job differently (note, I don’t say better), by examining via sqlite the pkg metadata on the respective machines. Your method doing locking/unlocking on the fly seems to be a very reasonable one.

The general bottom line:
Dual mode maintenance does work, and it does work well, and it does save a lot of time and energy which otherwise would be spent on building ports, specially those which for arbitrary reasons keep-on insisting to depend on the ever most recent llvmXY, which takes hours for building and only minutes for loading as a package.