"pkg updating" is unreliable

rwv37

rwv37

This has bothered me in the past several times, but I was just reminded of it, so I figured I'd post about it before I forget again:

According to "man pkg-updating", it "scans the installed ports and shows all UPDATING entries that affect one of the installed ports." That sounds very useful, and in fact I use it frequently, and have for a long time. However, it's not actually true. The command is unreliable, and quite possibly dangerously so.

The problem is that it relies upon the messages in UPDATING being in a particular format (seemingly, the "AFFECTS" line has to have the package name), but there is nothing enforcing that they actually are in that format. So, for example, just a week or two ago, "pkg updating" did not show me that my installed package lang/python311 had an UPDATING entry that affected it. This seems to be because the entry was labeled with "AFFECTS: Python users" (which, to be clear, does not contain the string "lang/python311").

I haven't seen anything (e.g. in "man pkg-updating") about this, nor how to make it reliable. If there is such a way, I'd love to hear about it. If there's not, though, maybe something should be done about it. For example, perhaps instead of just a text file that maintainers can change arbitrarily, a little program to check the format of the text file before the change can actually be successfully submitted to source control. Or, maybe even have maintainers submit their entries to some DB-centric app instead of to a text file.
 
pkg updating seem to want ports name exactly appear in AFFECTS: line (with or without *) of UPDATING entries.
This would be because there are NO indexes for UPDATING available covering all combinations of category/port and PKGNAME.
 
It's not an easy task, see the thread Automating port search in UPDATING file. Without a specific format of AFFECTS entry, you always have to read all the content /usr/ports/UPDATING file, at least the ones at specific date after last update of each port (I hope I explained myself), that means for each installed port get the installed/last updated date with pkg query '%t' pkg/port name, then for each entry check AFFECTS if it match a pkg/port, if not and the date of the entry is after the earliest entry of the installed/update of all port, show it. This way the script filter all the entries for not installed ports except the one the script does not recognize which pkg/port it refer to and only if the date is after the last update of all installed pkg/ports. It's convoluted but it's the best I could do and for most ports.
 
Surely there could at the very least be something like a pre-commit check that looks for something of the form x/y?

Or even just a change to "pkg updating" itself that makes it (at least optionally) show things that don't conform to that?
 
rwv37 said:
Surely there could at the very least be something like a pre-commit check that looks for something of the form x/y?

Or even just a change to "pkg updating" itself that makes it (at least optionally) show things that don't conform to that?
Click to expand...
I believe pre-commit checks are done by the committer who commit the change. But basically it's "for natural human readers", not for pkg-updating(8).

UPDATING exists in this format from far before pkg-updating(8) appears, more, far before even first CFT (Call For Testing) of pkg(8) was done.

And unfortunately, AFFECTS: line is NOT always as simple as pkg-updating(8) wants to be.

In many cases, single changes to specific single port affects some (many) other ports to require some works on user's side, thus, AFFECTS: line sometimes become complexed and/or abstract.

These are good and informative for natural human readers, but not good for tools like pkg-updating(8).
 
Yes, so that sort of thing is why I suggested a change to "pkg updating" itself that makes it (at least optionally) show things that don't conform.

Temporal order of UPDATING versus "pkg updating" aside: As I said in the original post, its man page explicitly claims that it "scans the installed ports and shows all UPDATING entries that affect one of the installed ports." That claim is false, there seems to be nothing in there to indicate that it's an overgeneralization, and the (entirely reasonable) assumption of a reader that it must be true is potentially dangerous. I think an absolute low bar for what should be done about all this is would be changing the man page to reflect reality.

But I just don't get why something better also couldn't be done. I understand it's difficult or potentially impossible to make something perfect given the way UPDATING currently exists, but it doesn't have to be perfect in order to be a significant improvement; having a "pkg updating" option to explicitly show things that may be false positives would be a significant improvement.
 
T-Aoki and rwv37 your reasoning makes sense, pkg updating works halfway. For a user who uses ports this might not be a problem, he completely ignores the command and consults the file /usr/ports/UPDATING directly. For a package user who doesn't use /usr/ports and therefore doesn't have all the files it contains, the halfway functioning of that command becomes a trap, because it doesn't always receive truthful information. For many package users this is a problem.
 
The AFFECTS section should be more specific, or add another section to be specific to make pkg updating more specific. It's like making a PR for a port, it's specified to start the title of the PR with the category/port the PR refer to, this is not a strict rule is a convention that all user follows. This way it's easy for developers to recognize which port is affected by the PR.

EDIT: And it's easy to search all PRs for a specific port
 
Incidentally, I noticed (via a comment in the thread that freethread linked to earlier) that the Porter's Handbook actually does say what the format of the file (quote) "is" (that being with a machine-parseable AFFECTS line), and moreover gives an example saying that it (quote) "must" be in that format and (quote) "must not" be in a laxer one similar to the "Users of python" that I mentioned in the original post:

The format of an entry in this file is:

YYYYMMDD:
AFFECTS: users of portcategory/portname
AUTHOR: the name <the email address>

Special instructions
Click to expand...

(...)

It is recommended that the AFFECTS line contains a glob matching all the ports affected by the entry so that automated tools can parse it as easily as possible. If an update concerns all the existing BIND 9 versions the AFFECTS content must be users of dns/bind9*, it must not be users of BIND 9
Click to expand...

So it seems to me that there really should be something done about all this, beyond simply updating the man page, and even beyond something like a false-positive-friendly change to pkg updating. To be clear, I'd be happy with something like that, but it seems to me like there should be a "real" fix.

For example, as I mentioned in the original post, it could be changed so that it's driven by a DB-based app rather than by arbitrary text from the maintainers. That app could, in addition to forcing an explicit selection of applicable ports, generate /usr/ports/UPDATING (or something like it) as appropriate.
 
One way would be to allow multiple (and unlimited) number of AFFECTS lines per entry, terminated by following (first, once multiple lines are allowed) AUTHOR line, forbid wildcards and force one port per line.
And maybe something like AFFECTS_OPTIONAL and AFFECTS_INDIRECT would be wanted, too.
In this case, the order should be restricted like below.
  1. AFFECTS
  2. AFFECTS_OPTIONAL
  3. AFFECTS_INDIRECT
  4. AFFECTS_OPTIONAL_INDIRECT
  5. AUTHOR
And mechanism for automating AFFECTS{_INDIRECT} lines should be wanted, as maintainers can easily know direct and indirect dependencies, but too hard to know dependants (what depends on the specific port). And what AFFECTS line want is the latter, unfortunately.
For example, maintainers prepare UPDATING.template having blank AFFECTS line after already-filled AFFECTS line, then, the tool (or bsd.ports.mk target) scans the whole (local) ports tree and look for any ports depending on already-listed ones.

To be fair, AFFECTS_OPTIONAL, AFFECTS_INDIRECT and AFFECTS_OPTIONAL_INDIRECT can be simply made as AFFECTS to allow pkg updating to run properly, but OPTIONAL would be helpful for anyone installing via ports (as it indicates nothing is needed to do if default options are used) and INDIRECT would help maintainers of consumer ports to determine priorities (as it indicates works are not always needed).

But the above are infrastructural changes that needs intensive discussions.
 
T-Aoki said:
One way would be to allow multiple (and unlimited) number of AFFECTS lines per entry, terminated by following (first, once multiple lines are allowed) AUTHOR line, forbid wildcards and force one port per line.
And maybe something like AFFECTS_OPTIONAL and AFFECTS_INDIRECT would be wanted, too.
In this case, the order should be restricted like below.
  1. AFFECTS
  2. AFFECTS_OPTIONAL
  3. AFFECTS_INDIRECT
  4. AFFECTS_OPTIONAL_INDIRECT
  5. AUTHOR
And mechanism for automating AFFECTS{_INDIRECT} lines should be wanted, as maintainers can easily know direct and indirect dependencies, but too hard to know dependants (what depends on the specific port). And what AFFECTS line want is the latter, unfortunately.
For example, maintainers prepare UPDATING.template having blank AFFECTS line after already-filled AFFECTS line, then, the tool (or bsd.ports.mk target) scans the whole (local) ports tree and look for any ports depending on already-listed ones.

To be fair, AFFECTS_OPTIONAL, AFFECTS_INDIRECT and AFFECTS_OPTIONAL_INDIRECT can be simply made as AFFECTS to allow pkg updating to run properly, but OPTIONAL would be helpful for anyone installing via ports (as it indicates nothing is needed to do if default options are used) and INDIRECT would help maintainers of consumer ports to determine priorities (as it indicates works are not always needed).
Click to expand...
I do not think there is need to go deep in dependencies ('required' and 'depend on'), only the direct port reference of the entry is necessary to check if it is installed and display the entry to the user. For 99% of the ports port/pkg updating tools (like portmaster or pkg) make a good job scanning dependencies.

In my opinion the problem are the languages + related packages ports, like python, perhaps PHP but I'm not shure & Co, they use FLAVORS to set the version and for most of them you can have several version installed at the same time, so if you have (for example) python311 and some py311-xxx packages installed, if you upgrade python311 to python312, the py311-xxx still can be installed and will be not upgraded to py312-xxx. At the end of upgrading you end with python311 and python312 installed (I guess. I never tested this scenary). To really upgrade from python311 to python312 the upgrading tool (whatever it is) should guess if you want both versions or if you want to upgrade the stale version and all language packages.

For that reason exists the file /usr/ports/UPDATING, to advise you to pay attention in upgrading certain ports.

T-Aoki said:
But the above are infrastructural changes that needs intensive discussions.
Click to expand...
Yes. For this reason I prefer to manage on my own solution (if any) or something close to it.
 
freethread said:
I do not think there is need to go deep in dependencies ('required' and 'depend on'), only the direct port reference of the entry is necessary to check if it is installed and display the entry to the user.
Click to expand...
Unfortunately, it's not always correct.
There are hidden dependencies that even the maintainer don't notice.

This often happenes when the port is using some kind of toolkits and the port itself seemingly using the toolkit only, but some calls are implememted as macros, causing direct links to underlying library ports.

Another case is that dependencies are hidden behind USES framework and overlooked. In this case, dependencies can differ depending on parameters for USES.

The other case is intentional to simplify ports Makefile. If some dependencies are know always needed, omitting direct dependencies that some other dependencies are known to pull it in. An example can be seen at PR 286077.
 
T-Aoki Ok, you are right, there are a lot of things that plays this game and complicate the job, but returning to the initial problem: running pkg updating get me the list of installed ports that I must pay attention before upgrading them. It should at least show me all the items related to the installed ports that are listed in /usr/ports/UPDATING, without skipping the unrecognized items (that is: making unrecognizable entries recognizable). Then if some are not listed in that file because of unseen/hidden dependencies or something similar, this is another story.
 
freethread said:
T-Aoki Ok, you are right, there are a lot of things that plays this game and complicate the job, but returning to the initial problem: running pkg updating get me the list of installed ports that I must pay attention before upgrading them. It should at least show me all the items related to the installed ports that are listed in /usr/ports/UPDATING, without skipping the unrecognized items (that is: making unrecognizable entries recognizable). Then if some are not listed in that file because of unseen/hidden dependencies or something similar, this is another story.
Click to expand...
Note that pkg(8) is still under active development.
And I think this is why in-base pkg is just a bootstrap tool and actual one is provided via ports/pkgs.
 
T-Aoki said:
There are hidden dependencies that even the maintainer don't notice.
Click to expand...
I wonder if there is a way to get these.
"port X, what does it depend on and what depends on it" and recurse until you get to the leaf ports that depend on nothing but others have dependencies on it.
Long time ago if you were using ports there was a tool that would give you this information. Maybe "portsmaster"? Not sure how it derived the relationships but it seemed to work.

How about using ldd to get things and then pkg which to find out what port supplies the file.
 
mer said:
Long time ago if you were using ports there was a tool that would give you this information.
Click to expand...
Not sure about ports-mgmt/portmaster (I've never used it), but available from before it appears.

ports-mgmt/portupgrade{-devel}, which is (if I recall correctly) the very origin of tool to manage installing/upgrading ports, recurses to leaf / root.
portupgrade -R -f -n <ports> would give you on what ports <ports> depends to the root.
portupgrade -r -f -n <ports> would give you by what ports <ports> is depended upon to the leaves.

Note that, if I recall correctly, anything upgraded alone were shown without -f option.
And unfortunately, ports-mgmt/portupgrade{-devel} does NOT handle FLAVORs without patches proposed at bugzilla. So I'm using it only when I need to switch origin of installed ports that ports-mgmt/pkg_replace still doesn't support.

Ah, forgot to mention. portsclean that is bundled in ports-mgmt/portupgrade{-devel} is still quite useful! Need patch, though.
 
mer said:
How about using ldd to get things and then pkg which to find out what port supplies the file.
Click to expand...
As far as I know, it's the only way to detect hidden dependencies.
Note that indirect dependencies are NOT always hidden dependencies.

To clarify, what I mean is something like cases below.
  1. Port A depends on port B and use functions provided by port B only.
  2. Port B depends on port C.
  3. Even though port A doesn't call functions provided by port C directly, any of (shared) objects in port A are linked directly against port C.
In this case, if port A stop depending upon port B and switch to port D, and port D doesn't at all depends upon port C, port A would be broken.

This is what I call "hidden dependencies".
 
  • Like
Reactions: mer
To go down on the dependency chain is easy (get the list of port that a port depend on: Port A depends on Port B and Ports C, Port B depends on Port D ... and so on down on the chain for all branches). The other way is a bit more complicated (Port A is required by Port B and Port C. Port B is required by Port D ... and so on up on the chain for all branches), but it's not impossible, is not complicated to do in a program. In a sh script is a lot complicated, at least for me.
 
freethread said:
To go down on the dependency chain is easy (get the list of port that a port depend on: Port A depends on Port B and Ports C, Port B depends on Port D ... and so on down on the chain for all branches). The other way is a bit more complicated (Port A is required by Port B and Port C. Port B is required by Port D ... and so on up on the chain for all branches), but it's not impossible, is not complicated to do in a program. In a sh script is a lot complicated, at least for me.
Click to expand...
I think it also depends on how one gets the information.
Each Package has meta data on what it provides and what it depends on.
If the metadata for a package is not correct, but the package links against a library, it may miss a dependency or basically what T-Aoki says in #22. Don't depend on metadata which is generated by humans (or AI :) ) but on actual system linking (ldd).
 
You must log in or register to reply here.
Back
Top