Solved Broken upgrade from 12.2 -> 13.0. libzfs.so.3 Undefined symbol "zcmd_ioctl_compat"

steveoc64

steveoc64

I obviously messed something up during an upgrade from 12.2 -> 13.0-RELEASE

Current state of the machine
- boot into multi-user mode, with ZFS on root, login as root
- mount shows zroot/ROOT/default mounted (obviously) of type zfs ... so there is basic ZFS functionality working in the kernel
- other zfs pools not mounting ?
- running "zfs list". / "zpool list" throws error :

ld-elf.so1: /lib/libzfs.so.3: Undefined symbol "zcmd_ioctl_compat"

Diagnostics / fixes attempted :

ls -l /lib/libzfs.so.3 shows it last updated at the current timestamp, size 312712. (amd64 build here), so it appears to be updated
freebsd-version
13.0-RELEASE-p10

kernel version = 13.0-RELEASE-p8. (not the same patchlevel as above ??)
running version = 13.0-RELEASE-p8

So Im assuming then that there is a mismatch between the running kernel version, and the contents of /lib/libzfs.so.3 ?

Question then :
- is it a mismatch between the running kernel and the lib ?
- what steps would you recommend for correcting the situation ?

Sorry for the noob mistake - I dont recall exactly where the upgrade went wrong :(
 
Update

Following the announcement about 10mins after I posted this, that 13.1-RC1 is avail, thought Id try upgrading to that, and see how that goes.

freebsd-update upgrade -r 13.1-RC1

Along the way, it found quite a few conf files that had merge conflicts to fix, so sorted all them out.
Then in the list of files to be added / upgraded, this included

/lib/libzfs.so.4

hmm !

complete the update, reboot, complete with 1 more pass of freebsd-update install

And ... ZFS userland utils are back

zpool import <each-pool> == success

So Im on 13.1-RC1 now, and that has indirectly fixed the problem.

I think the original issue may have been that my setup has "unbound" running in a jail to manage DNS, and that jail is mounted in a ZFS partition.
So on one of the reboot passes, DNS wasnt available, and therefore the freebsd-update would have failed to download updates. Im just guessing, but that seems a likely candidate for my messup.

Next time round - I will quick patch my /etc/resolv.conf to point to a vanilla router, and not rely on any more jailed svcs between update / reboots.
 
Next issue ...

pkg bootstrap is trying to fetch from the 12.x repo

# pkg boostrap
pkg is already bootstrapped at /usr/local/sbin/pkg
# file /usr/local/sbin/pkg
ELF 64-bit LSB Executable .... for FreeBSD 12.2. ...

That looks a bit severe.
 
steveoc64 said:
Next issue ...

pkg bootstrap is trying to fetch from the 12.x repo

# pkg boostrap
pkg is already bootstrapped at /usr/local/sbin/pkg
# file /usr/local/sbin/pkg
ELF 64-bit LSB Executable .... for FreeBSD 12.2. ...

That looks a bit severe.
Click to expand...
Pkgs aren't upgraded when you upgrade the base system, you still need to pkg-static upgrade -f.
 
Menelkir said:
Pkgs aren't upgraded when you upgrade the base system, you still need to pkg-static upgrade -f.
Click to expand...
So that command is going to look at the installed system and update the conf file to point at the 13.x repos?
I'm just double checking, it's been a while since I've done upgrades across versions.
 
steveoc64 said:
# pkg boostrap
pkg is already bootstrapped at /usr/local/sbin/pkg
# file /usr/local/sbin/pkg
ELF 64-bit LSB Executable .... for FreeBSD 12.2. ...
Click to expand...
Use pkg bootstrap -f

See pkg(7) (yes, that's a different man page from pkg(8)). pkg(7) refers to /usr/bin/pkg and pkg(8) refers to /usr/local/bin/pkg.

Code: 
    pkg [-4 | -6] bootstrap [-f] [-y]
            Attempt to bootstrap and do    not forward anything to    pkg(8)
            after it is    installed.  With -4 and    -6, pkg    will force
            IPv4 or IPv6 respectively to fetch pkg(8) and its signa-
            tures as needed.  If the -f    flag is    specified, then    pkg(8)
            will be fetched and    installed regardless if    it is already
            installed.    If the -y flag is specified, no    confirmation
            will be asked when bootstrapping pkg(8).

mer said:
So that command is going to look at the installed system and update the conf file to point at the 13.x repos?
Click to expand...
Configuration file refers to ${ABI}, which is interpreted at run-time. ${ABI} changes from FreeBSD:12:amd64 to FreeBSD:13:amd64. Thus it automatically switches between 12 and 13 repositories.

Code: 
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/quarterly",
 
That ${ABI} trick is quite useful. On my own repositories I added a couple of symlinks so it'll work there too. That way I don't have to worry about setting the repository correctly when upgrading my servers. It "automagically" switches to the right one.

Code: 
dice@molly:~ % ll /usr/local/poudriere/data/packages/
total 56
drwxr-xr-x  7 root  wheel  18 Oct  9 19:17 122-release-server
drwxr-xr-x  7 root  wheel  18 Mar 29 18:03 13-stable-desktop
drwxr-xr-x  7 root  wheel  18 Mar 28 18:57 13-stable-server
drwxr-xr-x  7 root  wheel  18 Mar 29 06:44 130-release-desktop
drwxr-xr-x  7 root  wheel  18 Mar 28 18:55 130-release-server
lrwxr-xr-x  1 root  wheel  18 Nov  7  2020 FreeBSD:12:amd64 -> 122-release-server
lrwxr-xr-x  1 root  wheel  18 Feb 15  2021 FreeBSD:13:amd64 -> 130-release-server
 
Sorted, found the culprit.

At some point in the past, I had modified PKG config, to point to a local pkg repo, and had hardcoded "12" as the abi version.

So all my pkg update runs, and pkg bootstraps were loading incorrect versioned binaries once I upgraded to 13.

Ouch!

By some miracle, BSD is robust enough to survive such a mistake. Machine is now repairing tself from a correct 13 pkg repo.

Moral of the story - don't go hacking config and forget what you changed

What an awesome system !
 
steveoc64 said:
Sorted, found the culprit.

At some point in the past, I had modified PKG config, to point to a local pkg repo, and had hardcoded "12" as the abi version.

So all my pkg update runs, and pkg bootstraps were loading incorrect versioned binaries once I upgraded to 13.

Ouch!

By some miracle, BSD is robust enough to survive such a mistake. Machine is now repairing tself from a correct 13 pkg repo.

Moral of the story - don't go hacking config and forget what you changed

What an awesome system !
Click to expand...
Because the base system will only breaks if you fiddle into it. If the base is stable and working well, in the worst case scenario, you can boot in single mode and fix the mess (that's one of the reasons you shouldn't use a shell from ports to your root account).
 
You must log in or register to reply here.
Back
Top