Proposal: a pkg upgrade mode that never silently drops installed packages

ZioMario

ZioMario

Hello to everyone.

I realized that when I do a "pkg upgrade",some installed packages silent drops : it means that they are removed without notice. For this reason,I asked Claude to elaborate a strategy for making a better upgrading / installing packages strategy.

Since I do "vibe coding",no one will be interested to the code I produce,so I will not be involved personally in this project. Anyway,I want to propose the plan to someone that can code it personally,using his brain :

So,this is what Claude (Opus 5.5) suggested :

1. It's not exactly "silent", but the policy is wrong.

pkg upgrade does show an Installed packages to be REMOVED section and asks for confirmation, unless -y or ASSUME_ALWAYS_YES is set. In a 200-line plan, though, that section is easy to miss. The real flaw is elsewhere: pkg's SAT solver (picosat) treats removal as an operation just as legitimate as an upgrade. There is no --never-remove-style option that turns a removal into an error.

2. The ecosystem limit.

FreeBSD repositories (quarterly/latest) carry only one version per origin. If py39-foo disappears from the repo or conflicts with the new py311 default, no solver can "find the best combination" using the repo alone, because that combination doesn't exist. The only way to give the solver a choice is to make additional versions available to it. So this is not just a better solver: it also needs a richer catalog.

Design​


A. Snapshot of the initial state.

For every installed package, record: origin, flavor, version, automatic flag, lock state, shlibs_provided and shlibs_required (via pkg query). Before touching anything, create a ZFS boot environment with bectl create.

B. Multi-source catalog with priorities.
  1. Remote repo (quarterly or latest).
  2. A local repo generated with pkg repo over /var/cache/pkg. It contains the currently installed packages, so "keep the old version" is always an available choice.
  3. Optional: a personal poudriere repo, for ports removed from the official tree.

C. Problem model (MaxSAT / pseudo-boolean, CUDF/Mancoosi style).

Hard constraints:
  • dependencies (origin/flavor);
  • declared conflicts and file conflicts;
  • at most one version per origin;
  • every shlib_required of the selected packages must be provided by a selected package or by the base system.
That last point is decisive. pkg metadata already includes shlibs, so the solver can verify whether an old package kept from the cache still works with the new libraries, instead of keeping it blindly and ending up with a broken binary.

D. Lexicographic objective, in strict order: a lower level can never compensate for a worse result at a higher one.
  1. Zero removals of explicitly installed (non-automatic) packages.
  2. Minimum number of removals of dependencies that are still required.
  3. Minimum number of packages kept at an old version or downgraded.
  4. Maximum number of packages brought to the newest version.
  5. Minimum number of newly installed packages.
E. If level 1 cannot reach zero, the system does not proceed.

It extracts the minimal unsatisfiable subset (MUS) and reports it in readable form, for example: "X cannot stay because it requires libfoo.so.3, provided only by Y, which conflicts with Z, required by W". It then lists the concrete alternatives: keep W at the old version, remove X, or build X in poudriere. The decision stays with the user and is never made implicitly.

F. Execution.

The plan is translated into explicit operations on specific .pkg files, not into a generic pkg upgrade that would recompute everything from scratch. Packages kept at an old version get pkg lock, so the next bare pkg upgrade doesn't throw them away.

G. Final verification.

Diff the initial snapshot against the final one and check shlibs with pkg check -d and pkg check -B. If something doesn't add up, roll back to the boot environment.

Implementation​

  • libsolv (BSD-3 license, the solver used by zypper and dnf). It already has the concepts of installed packages to preserve, "lock" and "favor" jobs, and problem explanations with alternative solutions. The main work is converting the pkg catalog (packagesite.yaml, one JSON object per line, with shlibs as "provides") into libsolv's format.

  • Alternative: direct MaxSAT with PySAT/RC2 (MIT). More control over the lexicographic objective, but MUS extraction and the explanations have to be written from scratch.
libsolv is the lower-risk choice: it is mature and has been in production for years on exactly this problem.

Immediate mitigation with current pkg​

A wrapper that runs pkg upgrade -n and aborts if the to be REMOVED section is not empty. It's crude and doesn't find alternatives, but it removes the surprise right away. There is also the BACKUP_LIBRARIES option in pkg.conf, which keeps removed shared libraries in a compatibility package and reduces ABI breakage for packages kept at old versions. It needs to be verified against the installed pkg version.
 
Why not get Claude to read the man pages

just use the -n option for a dry run
and create a bectl boot environment before you upgrade

pkg-upgrade - FreeBSD 15.1-RELEASE and Ports.quarterly Manual Pages

man.freebsd.org man.freebsd.org

Code: 
     -n, --dry-run
         Dry-run mode: show what packages have updates available, but do
         not perform any upgrades.  Repository catalogues  will  be  up-
         dated as usual unless the -U option is also given.

Code: 
doas pkg upgrade -n

Code: 
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
    glib-bootstrap: 2.84.4,2 -> 2.86.4,2 [FreeBSD-ports]

Number of packages to be upgraded: 1

there is already a pkg cache of installed packages on your machine
if memory serves me right

yes my memory was correct

Code: 
ls -l /var/cache/pkg
 
"Silently dropped"? I have no idea what that is supposed to mean... I've been using 'pkgng' since it first dropped (so when they replaced the Sun Solaris package manager?) and I never had a package "silently" removed, pkg always tells you what it's about to do.

(edit)

ZioMario said:
The real flaw is elsewhere: pkg's SAT solver (picosat) treats removal as an operation just as legitimate as an upgrade. There is no --never-remove-style option that turns a removal into an error.
Click to expand...
Actually, you can lock packages, look into pkg-lock(8) for that; this will ensure that some packages can't be removed "just like that". Otherwise you let the system do its thing.

I really fail to see the problem with any of this, because what you're describing can easily be avoided by checking for any potential upgrade messages instructions beforehand.
 
ShelLuser said:
"Silently dropped"? I have no idea what that is supposed to mean... I've been using 'pkgng' since it first dropped (so when they replaced the Sun Solaris package manager?) and I never had a package "silently" removed, pkg always tells you what it's about to do.

(edit)


Actually, you can lock packages, look into pkg-lock(8) for that; this will ensure that some packages can't be removed "just like that". Otherwise you let the system do its thing.

I really fail to see the problem with any of this, because what you're describing can easily be avoided by checking for any potential upgrade messages instructions beforehand.
Click to expand...

It means that if I installed, let's say, smplayer before upgrading the packages, sometimes, more than sometimes,after the upgrade process is complete, smplayer is no longer there. And I didn't remove it voluntarily. If I want it,I should reinstall it. But can happens that if I do that,another package will be removed. I don't know if this is the way how dependencies are resolved or...if this is a bug.
 
ZioMario said:
I realized that when I do a "pkg upgrade",some installed packages silent drops : it means that they are removed without notice. For this reason,I asked Claude to elaborate a strategy for making a better upgrading / installing packages strategy.
Click to expand...
My usage of the pkg command for a while has shown me that nothing gets silently deleted, but rather the user ignores the output or specifically does "pkg upgrade -y" to assume "yes".

Granted that is just my experience but when people have posted similar the output has shown user ignored messages or explicitly said "-y".

So "silently dropped" not that I've ever seen. "Dropped because user ignored messages" Yeah, probably.
 
ZioMario said:
But can happens that if I do that,another package will be removed.
Click to expand...
That is different than "silently dropped". I have run into Conflicts where installing package A has a conflict with package C so if you install/upgrade A it may delete package C. But the conflict will be (should be) flagged but the user needs to read the messages.
 

If you read this discussion on github,you will realize that they are talking about what I'm talking.

github.com

upgrade removes non-automatic package to upgrade its dependencies · Issue #2740 · freebsd/pkg

I had a virtual machine with 14.1 that I freebsd-upgrade'd to 14.4 then 15.1 and now when trying to upgrade packages it upgraded pkg from 1.21.3 to 2.8.4, it then mentions no removal in what will b...
github.com github.com


Ok. maybe the symptom of the problem manifests itself differently for you. I've explained how it manifests itself for me.

This is how Claude describe it :

1. It's not exactly "silent", but the policy is wrong.

pkg upgrade does show an Installed packages to be REMOVED section and asks for confirmation, unless -y or ASSUME_ALWAYS_YES is set. In a 200-line plan, though, that section is easy to miss. The real flaw is elsewhere: pkg's SAT solver (picosat) treats removal as an operation just as legitimate as an upgrade. There is no --never-remove-style option that turns a removal into an error.
Click to expand...
 
Ok, maybe I'm an outlier. But I have always run "pkg upgrade -n" and looked at the output; what gets upgraded, what gets installed, what gets reinstalled, what gets removed.
"gets removed" I pay attention to and investigate. "why removed" most times because it didn't build but that often will not remove the already installed package. I think there may be confusion as to what happens there.
The only conflict I can remember was one between gnucash and xsol (in my specific set of applications) but easily understood and worked around (install gnucash because needed vs xsol which was fun)

I think the biggest takeaway is
Don't run "pkg upgrade -fy"
 
You must log in or register to reply here.
Back
Top