updated to 13.1-RELEASE, portupgrade now broken

noodlefling

noodlefling

I just updated to 13.1-RELEASE from 12.3-RELEASE using freebsd-update -r 13.1-RELEASE upgrade

I did the first install run, then the second, then it told me to update the ports first.

So, I ran portupgrade -can and I got an error that told me it noticed a major version change, so please do a pkg bootstrap -f which I did.

Then I tried portupgrade again, it got a little ways through and the remote connection dropped.

I should mention that I also did a portsnap fetch ; portsnap upgrade which seemed to work fine.

I logged back in and tried portupgrade -can again and now I get the message:

Code: 
[missing key: virtual_categories] [Updating the portsdb <format:bdb_btree> in /usr/ports ... - 0 port entries found /usr/ports/INDEX-13:1:Port info line must consist of 13 fields.
 ..... done]
missing key: virtual_categories: Cannot read the portsdb!
database file error

If I do a pkg info everything is there, but I guess the database is hosed? What do you suggest?

Also, unless there is no other option, please don't suggest "just use poudriere it's super easy!". I have tried numerous times to switch and I can't make heads nor tails of the process. Maybe I am just thinking about things incorrectly, but I've found the startup procedure unfathomable, no matter how many "get started in 37 easy steps!" tutorials I've seen.
 
Try make fetchindex (or make index to generate it locally) in /usr/ports.

Note this index functionality is partially broken (IIRC since introduction of port flavors). A tool relying on it seems a bit fishy nowadays.

noodlefling said:
please don't suggest "just use poudriere it's super easy!". I have tried numerous times to switch and I can't make heads nor tails of the process.
Click to expand...
I actually have a hard time to imagine what could be the problem? poudriere comes with a config that works. All you need to do in preparation is create a jail with poudriere-jail(8), create a ports tree with poudriere-ports(8), and probably create a simple list of ports you want to build. Then just run poudriere-bulk(8) to do the build.
 
Sometimes I need a nonstandard config.

I don't understand how any of this poudriere stuff applies to what I know about installing ports. poudriere seems to be about setting up bundles of packages for use on multiple systems? And immediately the tutorials start talking about ZFS (which I'm not using) and jails (which I don't understand in this context), and nothing ever gets to the point where I can type in <somecommand> <portname> and have it pull down that thing plus anything it might need to run properly.

Are there any tutorials out there that explain the premise pragmatically? I have seen ones that talk in generalities, but those are no more helpful. A tutorial which explains how someone who understands things through the lens of ports would be extremely helpful.

Don't get me wrong, I understand that portupgrade is broken and a nightmare to use and frequently leads to dead ends and Catch-22 situations, but it also does eventually work if you bang your head against the wall long enough.

I'd love to use packages and have system updates take an hour instead of days. But I need something that will load all the necessary prerequisites and allow me to keep things updated with portsnap.

If you know of a system that will do this (while sometimes allowing me to install something with a modified config), I'm totally open to trying it. But not if there are words in the first sentence of the tutorial that don't make any sense to me.
 
So, if I want to switch to packages, the necessary functions would be:

1. upgrade any package
2. upgrade any package and its dependencies and/or things that depend on it
3. upgrade all packages that need it
4. upgrade all packages, whether or not the system thinks they need it

Plus some system that allows for set-and-forget manual configs of a few ports that differ from the default. If I have to compile those, that's fine, but they shouldn't screw up the system.

Also I don't want to manually keep track of what's installed. Something like pkg info should show what's there.

Is there anything like this? Port and package people seem to speak different languages, so maybe I'm asking the wrong questions, but I just want a simple system that doesn't get easily corrupted or out of whack.
 
Anyway, sorry for the port/package digression.
Zirias said:
Try make fetchindex (or make index to generate it locally) in /usr/ports.
Click to expand...
make index did not work:
Code: 
Generating INDEX-13 - please wait..--- describe.accessibility ---
--- describe.arabic ---
--- describe.accessibility ---
make[4]: "/usr/ports/Mk/Uses/python.mk" line 486: Cannot open /usr/ports/lang/python39/Makefile.version
make[4]: Fatal errors encountered -- cannot continue
===> accessibility/accerciser failed
*** [describe.accessibility] Error code 1

make[1]: stopped in /usr/ports
--- describe.arabic ---
make[4]: "/usr/ports/arabic/libreoffice/Makefile" line 5: Cannot open /usr/ports/arabic/libreoffice/../../editors/libreoffice/Makefile.i18n
make[4]: Fatal errors encountered -- cannot continue
===> arabic/libreoffice failed
*** [describe.arabic] Error code 1

make[1]: stopped in /usr/ports
2 errors

make[1]: stopped in /usr/ports

********************************************************************
Before reporting this error, verify that you are running a supported
version of FreeBSD (see https://www.FreeBSD.org/ports/) and that you
have a complete and up-to-date ports collection. (INDEX builds are
not supported with partial or out-of-date ports collections.
If that is the case, then
report the failure to ports@FreeBSD.org together with relevant
details of your ports configuration (including FreeBSD version,
your architecture, your environment, and your /etc/make.conf
settings, especially compiler flags and OPTIONS_SET/UNSET settings).

Note: the latest pre-generated version of INDEX may be fetched
automatically with "make fetchindex".
********************************************************************

*** Error code 1

Stop.
make: stopped in /usr/ports

make fetchindex seemed to work, but something is still screwed up. Trying to make ruby-bdb results in this:

Code: 
===>  Staging for ruby30-bdb-0.6.6_8
===>   ruby30-bdb-0.6.6_8 depends on file: /usr/local/bin/ruby30 - found
===>   Generating temporary packing list
/usr/bin/install -c -m 0755 bdb.so /usr/ports/databases/ruby-bdb/work/stage/usr/local/lib/ruby/site_ruby/3.0/amd64-freebsd12
/usr/bin/strip /usr/ports/databases/ruby-bdb/work/stage/usr/local/lib/ruby/site_ruby/3.0/amd64-freebsd13/bdb.so
strip: open /usr/ports/databases/ruby-bdb/work/stage/usr/local/lib/ruby/site_ruby/3.0/amd64-freebsd13/bdb.so failed: No such file or directory
*** Error code 1

Something is screwed up with the libraries, which I know, because I'm in the Twilight Zone of updating where the whole point is to refresh those libraries.

Also I see in the earlier error, it's complaining about python39, but this system has python38. I would use portupgrade to update, buuuut....

Should I just complete the third freebsd-update install and then try again, or will that only make everything worse? I don't understand the root cause of all this.
 
The error you get with make index looks like breakage in your ports tree itself. You could try to fetch a new one, but then, this whole "index" functionality does have problems as I said above...

---
edit: did you use portsnap? Cause I've just seen this on IRC:
04:01:58 < koobs> portsnap published a snapshot that couldnt build index
---

The build error you quote is more interesting:
Code: 
/usr/bin/install -c -m 0755 bdb.so /usr/ports/databases/ruby-bdb/work/stage/usr/local/lib/ruby/site_ruby/3.0/amd64-freebsd12
/usr/bin/strip /usr/ports/databases/ruby-bdb/work/stage/usr/local/lib/ruby/site_ruby/3.0/amd64-freebsd13/bdb.so
See how these two commands seem to disagree about the FreeBSD major version (12 vs 13)? I'm not sure how this can happen, but check the output of freebsd-version -kru. It should consistently say 13. Otherwise, another call to freebsd-update should probably fix it.

As for poudriere: It works on UFS, but that's not advisable. It builds each and every port in its own isolated jail, and constructing them would cause a LOT of disk I/O (and take forever) without the help of ZFS snapshots and clones. Still it is the most reliable and robust way of building ports, so, consider using ZFS...

It's not meant to directly build and install something, it creates a package repository and you install what you built using the normal pkg command. Of course this could be used for multiple machines, but it works just as well for a single machine.

If you only have very few config modifications, you could profit a lot from ports-mgmt/poudriere-devel, which has an option to fetch matching binary packages. So, you'd only build what is affected by your configs (or wasn't built on the official builders yet).
 
Assuming pkg info works, pkg query should also work. Use "pkg query -e '%#r = 0' %o" to create a list of "leaf" packages (no other package depends on these). You can also save the complete list with "pkg query %o" or "pkg query %n". Now if you want, you can blow away all packages. Next install the packages from the leaf list you made earlier. They will pull in any packages they depend on.

Also read /usr/ports/UPDATING. It shows you how to update from python38 to python39.

If you can live with the default ports for the most part, you will save a lot of time. I only build dri-kmod locally whenever I rebuild the kernel. In the past the few ports I wanted to use with different defaults, I built locally without much trouble.
 
It still seems like poudriere is trying to solve a problem I don't even understand. Maybe I shouldn't have brought it up, it seems to be confusing the issue. If anything, I'd just want to use some tool that lets you use more human-readable commands to interface with pkg.

freebsd-version -kru reports...
Code: 
13.1-RELEASE
13.1-RELEASE
13.1-RELEASE

So that looks good.

I did another portsnap and it seemed to update every single port, although it did not change the behavior of the errors.

I'm not ready to run the third freebsd-update install yet because it explicitly said not to do so until I update all the ports.

Thanks to bakul's advice, I am now messing around with pkg directly. It's going to take some re-thinking, but at least what it's doing makes sense, and I can just rebuild the non-default things I need manually. And I should have a lot more time to do that when I'm not recompiling the galaxy. I know from using portupgrade that the magical pkg system will figure out what changed so pkg info still outputs the correct list. (Manually maintaining a list for each server is not feasible.)

This is probably the kick in the pants I needed to drop portupgrade for good, so maybe this will ultimately be a good thing.
 
I'm making good progress, but for fun I reinstalled portupgrade and a portupgrade -a wants to upgrade a handful of ports, but if I do a pkg upgrade, it says there's nothing to do (after doing a forced upgrade of all packages).

What causes this discrepancy?
 
Oh, I see.

So, if portupgrade determines newness based on the last time I ran portsnap, how do I tell pkg to update its concept of newness? pkg update didn't seem to sync up with portsnap.
 
To keep ports and packages synced you can either switch the repository used by pkg and get latest packages, or use quarterly ports (which implies not using portsnap anymore but git instead).

See chapters 4.4.2 and 4.5.1 (the git method) of the handbook.
docs.freebsd.org

Chapter 4. Installing Applications: Packages and Ports

FreeBSD provides two complementary technologies for installing third-party software: the FreeBSD Ports Collection, for installing from source, and packages, for installing from pre-built binaries
docs.freebsd.org docs.freebsd.org
 
You said the default is to use quarterly ports...does the database magically update 4 times a year, or do you have to do it manually? What about in cases of security fixes?

My current experience is that if I do a pkg audit and see something come up, I do a portsnap and then (as long as a fix is available), a portupgrade will get me to the latest version.

In an all-package world, what's the procedure for dealing with security fixes? If I'm on the quarterly channel, will it pick up the fix as soon as it's available?

Also if a machine can function without specially-configured ports, then there's really no reason to install /usr/ports in the first the place, right?

And if I need to compile a particular port to get a different configuration on a server that already has /usr/ports, I should stop using portsnap, should use the git method to switch from quarterly to latest, and then do a standard make config in the standard /usr/ports/*/<port> directory, correct?

Sorry if these are basic questions, just trying to wrap my mind around the new lifestyle. Dropping portupgrade will probably slow my aging process a bit. Will feel weird to stop using portsnap. That one always feels refreshing to use.
 
Quarterly is default on RELEASE, latest is default on STABLE and CURRENT, but you can use whatever you prefer (except on CURRENT). I use latest on my PC but for a server I'd use quarterly unless I would specifically need a newer version of something, because version upgrades likely to break something will occur less often and getting the latest features is usually less of an urge on a server than on a PC. A quarterly branch doesn't get most version upgrades but it does get security fixes and some bug fixes. Packages are rebuilt when a port gets a fix and are available through pkg upgrade soon after.

When a new quarterly branch is out, the pkg repository is upgraded accordingly so you'll get version upgrades showing up for many of your packages in a regular pkg upgrade. The ports tree however won't automatically switch to the new branch so you'll have to run f.e. git switch 2022Q3 (2022Q3 is the current quarterly branch, then will come 2022Q4, 2023Q1, etc.).

No need for /usr/ports if you only use packages, indeed.

noodlefling said:
should use the git method to switch from quarterly to latest
Click to expand...
The opposite but yes :)

noodlefling said:
then do a standard make config in the standard /usr/ports/*/<port> directory, correct?
Click to expand...
Yes. Then you'll probably also want to use make install-missing-packages before make to get dependencies installed from pkg instead of built from ports, and pkg lock after it's installed to prevent your custom build from being automatically replaced by the default version through pkg upgrade. Then that port will need to be upgraded manually. If you plan to have many ports with custom options, setting up ports-mgmt/poudriere may be a good idea. For just a few ports, I prefer to keep it simple and take care of them manually.
 
So, where does /usr/ports/UPDATING (or the equivalent) come from with a package system? pkg update doesn't seem to update it. How would I know what tweaks are necessary when big things like python get updated?

Is git switch 2022Q3 then the replacement for portsnap? The point being that I should never type in that magic word again, because it will get out of sync from the package system?

I was thinking about switching to latest to deal with bleeding edge ports that get modified more frequently than 4 times a year, but I think what you're telling me is to keep the whole system on quarterly in the name of simplicity and not mixing incompatible worldviews.

Thanks for make install-missing-packages, I will give that a try. I don't have enough custom ports to make poudriere necessary, although now it's starting to become clearer why someone would recommend that to me. But still, no, I don't want that in my life.

Thanks again for all your help and patience. This should make the too-frequent OS upgrades less painful.
 
noodlefling said:
So, where does /usr/ports/UPDATING (or the equivalent) come from with a package system? pkg update doesn't seem to update it. How would I know what tweaks are necessary when big things like python get updated?
Click to expand...
There probably should be a similar reporting mechanism for packages. But at least you can access this file online, without having to download all of /usr/ports!
 
You must log in or register to reply here.
Back
Top