!!!!WARNING ALL FREEBSD DESKTOP USERS!!!! : Insane pkg's new version upgrade of installed apps

[T-Aoki]

Ouch. That's a high volume list if you don't procmail it.

I'm checking here with the archive from time to time, not subscribing.

 

[astyle]

Ouch. That's a high volume list if you don't procmail it.

I'd think that 'pkg being problematic' is one unfortunate reason for that ML to be THAT high-volume...

 

[Charlie Brown]

For my part, there is no reason for PKG to have acted like this. I'm running 14.3 stable and I just did the update upgrade and reboot. No ports installed. So if it works for you, why not for me ? But yes, other than that, lesson learned wait and simulate before update.

The problem is that one package was marked to be removed the others were mark to be updated and reinstalled, but an all bunch of packages marked to be reinstalled have been deinstalled but not reinstalled.

 

[T-Aoki]

poudriere bulk builder detects OSVERSION upgrade and delete all pkgs before starting actual builder jails,

Forgot to mention.
As far as I've experienced, poudriere forces full rebuilds at least when whichever of below happenes.

• When OSVERSION (generated based on __FreeBSD_version defined in /usr/include/sys/param.h) is bumped (even in lowest significant digit!).
• When BRANCH (STABLE, ALPHA5, RELEASE,...) in /usr/src/sys/conf/newvers.sh is changed.

Note that when OSREL (13.5, 14.3, 15.0,... coming from REVISION in /usr/src/sys/conf/newvers.sh) is bumped, OSVERSION is bumped, too.

 

[astyle]

More and more gotchas... This is really firming up my resolve to stick to ports and fresh installs. The User Handbook does not really mention those pkg-upgrade(8) and freebsd-update(8) gotchas, either.

I know one is supposed to RTFM... but it does look like there's actually a long list of gotchas and details to line up so that pkg-upgrade(8) and freebsd-update(8) work as intended / advertised.

 

[cracauer@]

I'd think that 'pkg being problematic' is one unfortunate reason for that ML to be THAT high-volume...

Well, if you do use procmail you could filter out build failures for the packages you care about. That way you get an early warning.

 

[bakul]

If I understand you correctly, the workflow used by the system that creates packages for RELEASE is the following:

1. On October 1st, create a new empty package directory for the current quarter.
2. Begin populating it with newly built packages. This takes several days.
3. Use the created packages for the remainder of the quarter, unless they need to be regenerated.

And: if anyone runs a "pkg upgrade" command between steps 1 and 2 above (when the new quarterly repo is only partially populated), they will get some of their installed packages deleted. In other words, there is a window of several days every quarter (or roughly several percent probability) where an host is at high risk of having packages wrongly uninstalled.

If true, that would mean that the "pkg upgrade" mechanism has only a reliability of 1-2 nines (about 90% to 99%). That would be insanely bad.

*If* this is how packages are created, they should rename the directory (an atomic operation) *once* the pkg dir is fully populated. Partially populated pkg dir should never be visible to users. [A reason to not use a zfs mountpoint for pkg dir]

I usually do a pkg update && pkg upgrade and so far have not run into problems.

 

[astyle]

Partially populated pkg dir should never be visible to users.

This, I would upvote if I could!

I do see it as part of the problem (with the pkg infrastructure that the FreeBSD project provides to the users), and a very substantial one.

[A reason to not use a zfs mountpoint for pkg dir]

Now that, I disagree with, I see no technical reason for that. If there's reasons to rename a directory, it's usually pretty easy to add the necessary lines to do that just about anywhere.

 

[Useradd]

So if it works for you, why not for me ?

This is still not clear to me either. I have 14.3-RELEASE with Xfce installed on a modern, powerful workstation with a Radeon RX 6900 XT GPU. I saw that pkg was about to be upgraded which is a red flag to me since a previous bad experience. After upgrading it I ran pkg upgrade again. I do check the pkg upgrade output when it asks for confirmation, and when I saw the long list of packages about to be removed without corresponding reinstallations I aborted the upgrade.

I went to my old test workstation (which has an old but powerful Nvidia GPU) and ran the upgrade there. Sure enough, a number of important desktop applications were lost. I reinstalled them individually and then with help from vmisev on this thread resolved an additional problem with missing 64-bit linux support.

What is still a mystery to me is that when I returned to my main workstation to repeat the process there, the original problem was no longer there, and the package upgrade worked perfectly. I can only guess that something had changed in the meantime.

 

[bakul]

Now that, I disagree with, I see no technical reason for that. If there's reasons to rename a directory, it's usually pretty easy to add the necessary lines to do that just about anywhere.

Let us say (to illustrate) the publicly visible dir is pub_pkg, and you are building packages in priv_pkg. When you are done, you might want to do this: mv pub_pkg temp_pkg; mv priv_pkg pub_pkg; mv temp_pkg priv_pkg -- not strictly atomic but the downtime is very small. What I meant is if pub_pkg & priv_pkg are separate zfs filesystems, this would be far more painful (if at all possible) but if these dirs are *one* level down in the same zfs filesystem, a normal mv works. [But note that I am speculating based on what was described. I have never used poudriere]

 

[kevans]

There's a lot in here to unpack, but I'm just going to drop a few points from past visibility I've had into the process, in no particular order:

1. The builders run poudriere bulk -a in a loop
2. Package sets are not removed when a new one is started
3. When a build has finished, some heuristics are applied to avoid publishing a set that is effectively broken (% packages built in the run, maybe others)
4. if the run is to be published, it begins pushing to the mirrors. No change is visible to users yet.
5. When the set is synced, there's a symlink swapped over to point users at it instead -- quick, atomic swap over to the new set

No, pkg clients will not see a weird incomplete set of packages at the beginning of every new quarterly. Maybe things were broken when main branched off and they fail in the new build, but I don't think that's common enough to warrant making a sweeping generalization.

 

[bvdw78]

Stuff like Electron (vscode) occasionally dips out of the upstream repo. That thing is ginormous though. Nowadays I'm doing two things: run RELEASE on anything I care about, and build my own packages on my own Poudriere instance. Ever since I started doing that, things have been running like the proverbial wind. Why? Because I'm getting loud notifications about build failures/issues before any packages hit my systems. Back when I was responsible for a server farm for hundreds of web hosting customers this was my exact process. Build everything locally, and when the build runs cleanly push the result through DTAP in a staggered fashion. Do not depend on upstream binaries existing for anything truly critical.

 

[T-Aoki]

1. The builders run poudriere bulk -a in a loop
2. Package sets are not removed when a new one is started

poudriere bulk -a wants to rebuild everything depending upon anything actually bumped by default. And recursively delete already-built pkgs in its builder repo before starting builds.

And bump of OSVERSION causes everything to be rebuilt, even if ABI is *.

 

[kevans]

poudriere bulk -a wants to rebuild everything depending upon anything actually bumped by default. And recursively delete already-built pkgs in its builder repo before starting builds.

And bump of OSVERSION causes everything to be rebuilt, even if ABI is *.

Indeed, I'm not sure I see your point here. The project does not serve packages directly out of poudriere repositories (see: the rest of my post).

 

[cracauer@]

*If* this is how packages are created, they should rename the directory (an atomic operation) *once* the pkg dir is fully populated. Partially populated pkg dir should never be visible to users.

That is debatable. You don't want security fixes for packages that did built successfully to be held back for extended periods of time.

 

[T-Aoki]

Indeed, I'm not sure I see your point here.

I've just commented regarding my experiences related with the 2 items.
Nothing more, nothing less.

The project does not serve packages directly out of poudriere repositories (see: the rest of my post).

So if anything lacking on latest build && not marked as vulnerable are copied back to new (not yet swapped) repo, there would be much fewer claims / frustrations.

Yes, I understand default behaviour of poudriere is "the safest side", but overkill in a number of cases.

I dream if a kind of reliable "auto PORTREVISION bumper" is possible on ports repo side, poudriere can default to "-S" option safely.
But I cannot think of efficient and painless way to achieve it, at least for now.

 

[astyle]

That is debatable. You don't want security fixes for packages that did built successfully to be held back for extended periods of time.

That's what ports snapshots are for - if you build from ports, you get the stuff with security fixes... and not wait for pre-built stuff.

 

[RetroComputingCollector]

I can't wait until there is some sort of build failure or mistake in the pkgbase repo and it ends up uninstalling half your system. Hopefully that will end this crazy scheme.

 

[mer]

That's what ports snapshots are for - if you build from ports, you get the stuff with security fixes... and not wait for pre-built stuff.

I think is more generalized to "if you care that much about security fixes, you should probably be building from ports and not using pre built packages".

 

[bakul]

No, pkg clients will not see a weird incomplete set of packages at the beginning of every new quarterly. Maybe things were broken when main branched off and they fail in the new build, but I don't think that's common enough to warrant making a sweeping generalization.

Sorry about adding to the noise!

 

[cracauer@]

I think is more generalized to "if you care that much about security fixes, you should probably be building from ports and not using pre built packages".

I do not agree.