pkg upgrade -n --why-would-you-want-to-do-that

I have a specific question, which I think leads into a more generic one about understanding pkg logic.

I'm in the midst of upgrading a system from 13.0 to 13.1, and pkg upgrade tells me

Code:
The following 17 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
    libpsl: 0.21.1_4
    openldap24-client: 2.4.59_4

Installed packages to be UPGRADED:
    bash: 5.1.12 -> 5.1.16
    ca_root_nss: 3.71 -> 3.78
    curl: 7.80.0 -> 7.84.0
    gettext-tools: 0.21 -> 0.21_1
    icu: 70.1_1,1 -> 71.1,1
    libltdl: 2.4.6 -> 2.4.7
    lmdb: 0.9.29,1 -> 0.9.29_1,1
    mercurial: 6.0 -> 6.1.4
    nss-pam-ldapd-sasl: 0.9.11_1 -> 0.9.12_1
    portmaster: 3.20 -> 3.21
    py38-setuptools: 57.0.0 -> 62.1.0_1
    python38: 3.8.12_1 -> 3.8.13_2
    sudo: 1.9.8p2 -> 1.9.11p3
    zsh: 5.8 -> 5.9

Installed packages to be REINSTALLED:
    dialog4ports-0.1.6_1 (option removed: ACCESSLOG)

I'm puzzled, because I don't understand why pkg wants to install openldap24-client. Checking with pkg info -d, none of the currently installed packages have a dependency on it. In the absence of a --why-on-earth-would-you-want-to-do-that option for pkg, I'm scratching my head.

This isn't out of the blue, because the machine does have openldap26-client and -server (ie, later versions) installed from ports:

Code:
% pkg lock -l
Currently locked packages:
cyrus-sasl-2.1.27_2
cyrus-sasl-saslauthd-2.1.27_2
openldap26-client-2.6.1
openldap26-server-2.6.1_1

But I don't see the logic that means that the presence of those portmaster-installed ports leads pkg update to try to downgrade them. Is pkg perhaps spotting the presence of files installed by openldap26-{client,server} and jumping to the conclusion that they were put there by openldap24, which it's therefore repairing? That seems... unlikely.

I would have thought that pkg info -d <packagename> would have represented all of the information that pkg uses for its upgrade decisions, but that's clearly not the case.

I am aware that the handbook does say ‘it is generally ill-advised’ to use pkg and ports together (quarterly vs HEAD, etc), but I'm also aware, from these forums and elsewhere, that it's OK in practice more often than not, and I do seem to have got away with this (convenient) configuration in the past.

So the questions are:

* Why is pkg trying to install openldap24-client?
* Is this in fact indicating that my pkg-vs-ports luck has run out? If so, what is pkg saying that makes that explicit, and how do I work out what the precise nature of that out-of-luck is? (that last question is really ‘am I really going to have to have another go with poudriere?’)
* How do I get pkg to be a little more forthcoming about the logic of its conclusions?

Thanks for any pointers,

Norman
 
My understanding:
pkg uses the list of currently installed packages (roughly pkg prime-list) to figure out "what do I need to upgrade" but I think it uses info from the repo on what the dependencies are.
I have a pkg alias I call "reverse-depends" to find out "what packages depend on this package"

reverse-depends 'query %rn-%rv'
basically pkg reverse-depends openldap24-client would tell me what installed packages have openldap24-client as a dependency.

I think if you change the alias a little you can do that on the repo. basically change the query to "rquery" in the alias. Take that list and compare to what you have installed.
 
Have you ever run pkg-autoremove(8)? Because to me it sounds like you still have some old (now orphaned) dependencies installed. A pkg upgrade will just upgrade anything and everything that's currently installed.

The "default" OpenLDAP client was recently changed from 2.4 to 2.6. You probably did some upgrades after that change (which pulled in openldap26) but openldap24 might still be there if it was never removed (pkg-autoremove(8)).


But I'm wondering how you managed to install net/openldap26-client while keeping net/openldap24-client, because those ports/packages would conflict and the system will refuse to install both.

Code:
.if defined(CLIENT_ONLY)
CONFLICTS_INSTALL=	${PORTNAME}2[0-57-9]-client ${PORTNAME}-client
.else
CONFLICTS_INSTALL=	${PORTNAME}2[0-57-9]-server ${PORTNAME}-server
.endif
 
To satisfy your curiosity: if you want to check afterwards, let the install proceed as proposed and then try
pkg required-depends openldap24-client
(required-depends is an alias, see pkg alias | grep depend for more predefined pkg aliases.)
Then you'll have your answer and can decide what packages you want to remove.

You can dry-run pkg-autoremove(8) with pkg autoremove -n

P.S.
[...] I would have thought that pkg info -d <packagename> would have represented all of the information that pkg uses for its upgrade decisions, but that's clearly not the case.

I am aware that the handbook does say ‘it is generally ill-advised’ to use pkg and ports together (quarterly vs HEAD, etc), but I'm also aware, from these forums and elsewhere, that it's OK in practice more often than not, and I do seem to have got away with this (convenient) configuration in the past.
Note that where pkg-query(8) searches the local pkg repository/database, both pkg-rquery(8) and pkg-search(8) apply to the remote repository. For example:
pkg search -d wireguard is the same as pkg search --depends-on wireguard or pkg search -Q depends-on wireguard and relate to the remote repository. Generally speaking, that means that the proposed installation of an additional package could also be triggered by the addition of an updated (new) version of a current (locally installed) package. In your case that is rather/very unlikely because it concerns another older version altogether (i.e. openldap24-client) that is being is proposed.
 
  • Like
Reactions: mer
Did you try to unlock all locked packages ? They can optionally depends on some version of openldap, depending on when they were installed.
My guess is that after unlock all locked package, they can be updated and there should no longer be an openldap24 dependency.
Then you can update your ports tree and rebuild your packages with openldap support (by the ways your version of cyrus-sasl is subject to CVE-2019-19906)
 
Greetings, all.

Thank you, all, for your thoughtful and insightful remarks.

I think I have some understanding (though incomplete) of what's
happened here. Skip to 'Resolution', below, if you're not interested
in the flailing around in the middle.

(Context: This machine is an LDAP server, which since 11.x has
required careful selection of things from the ports tree, in order to
get the up-to-date openldap2X-server with needed options, whilst
avoiding packages which depend on an old/pkg
openldap2X-{client,server} -- as SirDice notes, the
different versions install into conflicting places. I'm pretty sure
the same was true of 13.0, but I'm going to double-check whether 13.1
still forces me to do this.)

Investigations:

I had already run pkg autoremove -n, and it produced no
indication of a link to openldap24-client. I had done the local
reverse-depends thing the hard way (list all of the packages, then
list their dependencies one by one): pkg query %rn-%rv openldap24-client is a lot more straightforward -- thanks mer! It produces
nothing. But one of its rquery-discovered reverse dependencies is
nss-pam-ldapd-sasl 0.9.12_1, and that is the only one
of those reverse dependencies which I do have installed (and it is one
of those returned by pkg prime-list).

So that would appear to be the culprit. Except...

Code:
% pkg info -d nss-pam-ldapd-sasl
nss-pam-ldapd-sasl-0.9.11_1:
	openldap26-client-2.6.1

The locally installed nss-pam-ldapd-sasl seems to think
that it depends on openldap26-client. It's further surprising that
the local package database seems to think this depends on
openldap26-client, but a later version of the package
(0.9.12) apparently depends on openldap24-client.

But (aha....) this seems to link to Erichans' remark that
Generally speaking, that means that the proposed installation of an
additional package could also be triggered by the addition of an
updated (new) version of a current (locally installed) package. In
your case that is rather/very unlikely because it concerns another
older version altogether (i.e. openldap24-client) that is being is
proposed.
and the observation that pkg upgrade is proposing to
upgrade nss-pam-ldapd-sasl: 0.9.11_1 -> 0.9.12_1. But
Code:
% pkg rquery '%n-%v depends on %dn-%dv' nss-pam-ldapd-sasl
nss-pam-ldapd-sasl-0.9.12_1 depends on openldap24-client-2.4.59_4
Thus it appears that the newer version of nss-pam-ldapd-sasl is quoted
as depending on the older version of openldap.

But if I look at
<https://www.freebsd.org/cgi/ports.cgi?query=nss-pam-ldapd-sasl&stype=all>,
then I'm told that
Code:
nss-pam-ldapd-sasl-0.9.12_1
  Advanced fork of nss_ldap with sasl support
  Description : Changes
  Maintained by: zi@FreeBSD.org
  Requires: cyrus-sasl-2.1.28, openldap26-client-2.6.3

So what appears to be the case is that (a) my
system wants to update nss-pam-ldapd-sasl to 0.9.12 (which seems
right), but (b) also thinks, for some reason, that that updated
version depends on openldap24-client-2.4.59_4 which appears to be
incorrect. I wonder if I have some confusion in a local cache of a
remote dependency database (I think this is right -- see below).

Resolution:

What I've done is to (unlock and) remove the packages I installed from
ports, including nss-pam-ldapd-sasl. That allows me to pkg upgrade, and complete the upgrade to 13.1 (phew). At this point (ie,
after removing nss-pam-ldapd-sasl), if I ask to install
nss-pam-ldapd-sasl pkg still reports that it wants to
install openldap24-client: 2.4.59_4, and so does
portmaster --show-work. Hmm... portmaster(8)?... hmm...

And then I update the ports tree with portsnap(8).

And now portmaster --show-work reports that it
will install net/openldap26-client.

So I think I can blame an outdated ports tree for my problem. But
that also fails to completely explain things, because I
thought that the pkg system was independent of the local
ports tree. When eg portmaster does its work, it installs
packages where pkg can see them, but I didn't think that
pkg could be thrown off-track by the ports tree.

Whatever the explanation for that, I think my next step is to discover
whether the current pkg version of openldap26 now has the defaults I
need, or at least to take more explicit note of just why I need the
ports version.

And yes, thanks to the prompt from Monwarez, I have now found pkg audit.

Summary: So the things I've learned are:

* yes, it is indeed just the network of dependencies that pkg upgrade works with, but
* this comprises the local and remote dependencies,
* and my ideas about how to query that are now less muddled than
they were.

(I'm still a bit confused about the relationship between pkg and ports, though...; but I'm differently confused, so that's progress)

Thanks again,

Norman
 
  • Like
Reactions: mer
...but still

Code:
% pkg rquery %n-%v' depends on '%dn-%dv nss-pam-ldapd-sasl
nss-pam-ldapd-sasl-0.9.12_1 depends on openldap24-client-2.4.59_4
and that's the same on two different 13.1-RELEASE boxes.

I have nothing beyond the default aliases in pkg.conf, and pkg -d confirms that it's fetching from <http://pkgmir.geo.freebsd.org>.
 
(Context: This machine is an LDAP server, which since 11.x has
required careful selection of things from the ports tree, in order to
get the up-to-date openldap2X-server with needed options, [...]
...but still
Code:
% pkg rquery %n-%v' depends on '%dn-%dv nss-pam-ldapd-sasl
nss-pam-ldapd-sasl-0.9.12_1 depends on openldap24-client-2.4.59_4
and that's the same on two different 13.1-RELEASE boxes. [...]
My guess is that those "two different 13.1-RELEASE boxes" versus the "LDAP server" do not have the same package settings (quarterly versus latest) in FreeBSD.conf. Check:
  1. /etc/pkg/FreeBSD.conf
  2. /usr/local/etc/pkg/repos/FreeBSD.conf
The setting in /etc/pkg/FreeBSD.conf is the "general-default" setting (preferably not to be changed by you). When present, /usr/local/etc/pkg/repos/FreeBSD.conf should contain local settings that are to be retained when updating the FreeBSD OS; please refer to the FreeBSD Handbook: 4.4.2. Quarterly and Latest Ports Branches

I do not have net/nss-pam-ldapd-sasl installed; however, on 12.3-RELEASE, I get:
Rich (BB code):
% pkg rquery %n-%v' depends on '%dn-%dv nss-pam-ldapd-sasl
nss-pam-ldapd-sasl-0.9.12_1 depends on openldap24-client-2.4.59_4

<snap>

% pkg rquery %n-%v' depends on '%dn-%dv nss-pam-ldapd-sasl
nss-pam-ldapd-sasl-0.9.12_1 depends on openldap26-client-2.6.3
The second result is based on the latest pkg repository—as indicated by monwarez. The first result is based on quarterly.

My observation is that the change by this commit - Mk/bsd.ldap.mk: Use OpenLDAP 2.6 by default from 2022-07-08 08:06:07 +0000 according to this diff:
Rich (BB code):
-DEFAULT_OPENLDAP_VER?=    24
+DEFAULT_OPENLDAP_VER?=    26
has been committed after the quarterly branch of Q3 (=2022Q3) has been branched off the ports tree's main branch. This means that this commit is incorporated in the ports tree's main branch but, it is not incorporated in the 2022Q3 branch of the ports tree. Currently, the 2022Q3 branch of the ports tree is used for the quarterly setting. The main branch of the ports tree is used for the latest setting.

The quarterly setting will use the 2022Q4 branch when the next quarter has arrived. At Q4, first a (new) branch will be branched off from the ports tree's main branch, immediately after this it will be designated as the 2022Q4 branch and will be used by FreeBSD installs using the quarterly setting. It will, of course, incorporate the change of the "old" commit from 2022-07-08 08:06:07 +0000; because of that net/nss-pam-ldapd-sasl will then have openldap26-client as its dependency when using the quarterly setting.*

Concluding: this commit with this particular timestamp is the reason that, at this moment, net/nss-pam-ldapd-sasl has a different dependency (openldap24-client versus openldap26-client), depending on which remote package repository is being used.



Generally speaking, especially from an admin standpoint of view, it would be beneficial to have all your FreeBSD systems on the same remote pkg repository. If you don't do that or if that is not possible then, that is one reason to consider using a local package server. If you mix ports and packages and you do not have a (very) good knowledge of the consequences of mixing ports and packages and individual settings that govern how they influence each other then, that is another reason to consider using a local package server.

A local package (build) server (with its own local repository) creates an extra level of indirection that enables you to effectively control the update process of all your FreeBSD installs. That means that the update process of this local repository with source code from the external FreeBSD ports repository is fully under your control, independent from the remote FreeBSD ports tree's individual branches. The packages that have been built this way enable a uniform way by which the FreeBSD installs in a local network get their packages updated from this local repository.

Such a new configuration will require extra work to set up and an extra learning curve but the tools are there.

___
* edit: slightly reworded
 
Last edited:
Are you using quarterly or latest ? For me with latest I get
I think this is a key difference. I run that pkg rquery on 13.1-RELEASE that I know is pointing at quarterly and get

pkg rquery %n-%v' depends on '%dn-%dv nss-pam-ldapd-sasl nss-pam-ldapd-sasl-0.9.12_1 depends on openldap24-client-2.4.59_4

just like the OP. Another reminder that quarterly lags latest, by design.
And ports trees track whatever you last updated to, whether it's portsnap or a git clone.

I'd like to also point out that Erichans has a lot of good info in post #9
 
Thanks, Erichans and mer.

I think I'd managed to confuse myself in post #7. I'm aware of the distinction between the quarterlies (being what's in pkg) and latest (in ports), so I shouldn't have been surprised at what I saw there. Each of the machines in question has the default /etc/pkg/FreeBSD.conf and no local tweaks. My problem comes down to simultaneously wanting openldap26-server, which depends on openldap26-client (unsurprisingly) and net/nss-pam-ldapd-sasl which, in the current quarterly, depends on openldap24-client. And if I'd stepped back and walked around the block again, I'd have seen that pkg and portmaster were in fact telling me that.

My immediate problem is therefore solved by building net/openldap26-server, security/cyrus-sasl2-saslauthd and, crucially, net/nss-pam-ldapd-sasl from ports (and locking them and their immediate dependencies). The command portmaster --show-work shows ports dependencies, and I can install most of the ‘ordinary’ dependencies (but, from experiment, not net/nss-pam-ldapd-sasl) from pkgs.

From what Erichans says, I'm optimistic that, come the next quarterly, because openldap26-server now includes the options I want and because nss-pam-ldapd-sasl depends on the right OpenLDAP, I'll be able to install what I need entirely from quarterly pkgs. I'll therefore – despite Erichans's reassurance – be able to postpone learning about poudriere for a little longer (I think my bad experience with it before was because it was a hurried means to an end, which didn't finally work, as opposed to being a bit of strategic learning).

But more importantly, I think I now have a clearer idea of just what the manual is referring to when it says that ‘it is generally ill-advised’ to use pkg and ports together. It's not some arcane issue of subtly incompatible APIs (which, though not impossible, is a problem you can probably get away with ignoring quite a lot of the time), but instead exactly this problem of confusing and subtly incompatible dependencies, which can only really be worked out by iterative, trial-and-error, work with portmaster --show-work, and then taking careful notes, to minimise the total amount of magic.

So: the learning:

* one should repeatedly remind oneself of the relationship between pkg/quarterlies and ports/latest
* pkg query and rquery are your friend
* portmaster --show-work is what discloses the corresponding information for ports

(I'd still quite like a --why-on-earth-would-you-want-to-do-that option, though, a bit like make -dm).

Thanks again for very useful answers; I hope this was a useful question for the archive.
 
You are not the first one to get "bitten by" "confused by" ports and packages and quarterlies and latest. And probably not the last.
Since my systems are "home use" I tend to follow -RELEASE and quarterly packages simply for ease of use. I do pay attention to warnings and vulnerabilities, but look to see if I actually "care right now".

and then taking careful notes, to minimise the total amount of magic.
Words of wisdom to any system admin.
 
Back
Top