Solved error /lib/libkvm.so.7: Undefined symbol "fstat@FBSD_1.5" in jail following update to 12.0p5

byrnejb

byrnejb

The cause of all the grief described below was caused by forgetting to run ezjail-admin update -U -s 11.2-RELEASE on the jail host followed by pkg-static install -f pkg and pkg upgrade on each of the updated jails. 2019-06-28.

We use ezjail to manage our jails. We recently updated the host system to FreeBSD-12.0p5. We later updated a jail using
Code: 
ezjail-admin update -u
. Everything seemed to complete without error. However, some time later we had occasion to run uptime from within the jail and this produced the following error:
Code: 
/lib/libkvm.so.7: Undefined symbol "fstat@FBSD_1.5"

I have searched for this and the only report that I found which might apply is https://www.otoh.org/post/2019/02/undefined-symbol-statfbsd_1.5/ . However, when I check the sha512 hash I cannot find a mismatched library:
Code: 
# sha512 /lib/libkvm.so* /usr/jails/*/lib/libkvm.so*
SHA512 (/lib/libkvm.so.7) = ef20c02357e50b7f4366fefe5c678a6739ae38db270953fccb2badd11b201efcd6f4fbcde0fd5313069623551a76c47b3f7f8a02b8909b00a292d8b03f1df633
SHA512 (/usr/jails/basejail/lib/libkvm.so.7) = ef20c02357e50b7f4366fefe5c678a6739ae38db270953fccb2badd11b201efcd6f4fbcde0fd5313069623551a76c47b3f7f8a02b8909b00a292d8b03f1df633

I have not run across this problem before. This jail was successfully updated to FreeBSd-12.0 before. However, I cannot be certain that uptime was run on it previously and so the problem may have been there from the beginning.

How do I go about locating what the exact problem is and determine the remedy?
 
Last edited:
SirDice said:
Did you reboot afterwards?
Click to expand...

Yes. The host in this case is a bhyve vm running freebsd-12.0p5. The process was fetch, install, reboot, ezjail-admin update -u, reboot. There was a gap of several days between the update to the host system and applying the upgrade to the jails.

Code: 
. . .
2019-05-20 11:06:50: freebsd-update install
2019-05-20 11:10:26: shutdown -r now
. . .
2019-06-17 15:30:21: ezjail-admin update -u
. . .
2019-06-17 15:31:35: shutdown -r now
. . .
 
I am aware that there is some problem with FreeBSD FileStat but that bug is closed. Has the patch not made it into RELEASE as of yet or is that issue unrelated to what I am observing? I am seriously concerned about this because we run a lot of our public services inside dedicated jails.
 
I find this:
<CODE>
[root@mx31 ~]# service postfix status
/lib/libkvm.so.7: Undefined symbol "fstat@FBSD_1.5"
/lib/libkvm.so.7: Undefined symbol "fstat@FBSD_1.5"
postfix is not running.
</CODE>

What ever caused this it is not a ringing endorsement of FreeBSD quality control. I now have our principal MX server offline due to an OS induced error.
 
Update your packages too. If important structures change ports/packages that use them must be updated too. Or else they're linked to the old structures.

byrnejb said:
What ever caused this it is not a ringing endorsement of FreeBSD quality control. I now have our principal MX server offline due to an OS induced error.
Click to expand...
What you're actually saying is that you didn't test your updates before applying them to production?
 
Correction. The mail server is not offline. One simply cannot administer it from within the jail. Service reports the same error as above and says that the service is not running. But it is.
SirDice said:
Update your packages too. If important structures change ports/packages that use them must be updated too. Or else they're linked to the old structures.

What you're actually saying is that you didn't test your updates before applying them to production?
Click to expand...

It is a jail. I am used to experimenting on jails. I tested the p5 update on a vm and had no errors. I created a jail on that vm and had no errors. I then updated the vm host and checked for errors. There were none. I then updated each vm one at a time and had no errors. I also updated the jails one at a time and had no errors. We run our jails in ZFS and we run our vms the same way. Had an error arose at the time then we would have rolled back to the most recent snapshot, which is typically 2 minutes old. SO I am saying that I did not fail to carry out the updates in a prudent manner. I did not generally worry overmuch about pkg upgrades because I had never experienced this sort of problem arrising from a pkg upgrade. Jails are set up with different pkg sets which creates a rather burdensome maintenance problem for pkg updates. If it is indeed necessary to clone each jail, apply the upgrade, and then test before updating the service jail then we will have to abandon jails altogether.

I have no idea when or how this started but it has to have been very recently because I was maintaining the configuration on some of these jails last week. And the error is now evidenced on jails in vms which have not been modified since May 24 and whose jails were updated on that same date and have not been modified since. I have run the service utility on some of these jails in vms that actually run on different hardware as recently as June 11 and did not encounter this error. There was no update to the host operating system after May 31. There was no update to the jails after May 31 except in the case above which alerted me to the problem. So something else has introduced this error because it is occurring on different hardware with similar histories.

There were pkg upgrades performed on the host system after May 31:

24714 2019-06-12 10:31:44: pkg upgrade
24763 2019-06-14 10:40:20: pkg upgrade
24869 2019-06-18 11:24:57: pkg upgrade

So, one of these has introduced the problem. How do I find out what packages were updated on this host in each?
 
Ok, the packages updated on the host system since May 24 are these:
Code: 
May 31 15:19:56 <user.notice> vhost04 pkg[48811]: gdisk-1.0.4_4 installed
May 31 15:20:38 <user.notice> vhost04 pkg[48830]: freebsd-release-manifests upgraded: 20181211 -> 20190529
May 31 15:20:45 <user.notice> vhost04 pkg[48830]: firefox upgraded: 67.0_3,1 -> 67.0.1,1
May 31 15:20:46 <user.notice> vhost04 pkg[48830]: curl upgraded: 7.64.1_1 -> 7.65.0_1
May 31 15:32:45 <kern.crit> vhost04 kernel: drmn0: This code is obsolete abandonware. Install the graphics/drm-legacy-kmod pkg
May 31 15:32:45 <kern.crit> vhost04 kernel: drmn0: This code is obsolete abandonware. Install the graphics/drm-legacy-kmod pkg
Jun 10 15:12:27 <user.notice> vhost04 pkg[4162]: e2fsprogs-libuuid upgraded: 1.45.0 -> 1.45.2
Jun 10 15:12:27 <user.notice> vhost04 pkg[4162]: e2fsprogs-libss upgraded: 1.45.0 -> 1.45.2
Jun 10 15:12:27 <user.notice> vhost04 pkg[4162]: e2fsprogs-libblkid upgraded: 1.45.0 -> 1.45.2
Jun 10 15:12:28 <user.notice> vhost04 pkg[4162]: freebsd-release-manifests upgraded: 20190529 -> 20190607
Jun 10 15:12:34 <user.notice> vhost04 pkg[4162]: firefox upgraded: 67.0.1,1 -> 67.0.2,1
Jun 10 15:12:34 <user.notice> vhost04 pkg[4162]: e2fsprogs upgraded: 1.45.0 -> 1.45.2
Jun 10 15:12:35 <user.notice> vhost04 pkg[4162]: curl upgraded: 7.65.0_1 -> 7.65.0_2
Jun 12 10:32:28 <user.notice> vhost04 pkg[22763]: speexdsp upgraded: 1.2.r3_1 -> 1.2.0
Jun 12 10:32:34 <user.notice> vhost04 pkg[22763]: firefox upgraded: 67.0.2,1 -> 67.0.2_1,1
Jun 14 10:40:57 <user.notice> vhost04 pkg[45096]: libevent upgraded: 2.1.8_3 -> 2.1.10
Jun 14 10:40:58 <user.notice> vhost04 pkg[45096]: tor upgraded: 0.3.5.8 -> 0.3.5.8_1

I cannot see how any of these could give rise to the error I am seeing but there have been no other changes to the host and the jails did not evidence this problem on June 11,
 
One important question, are you using ports or packages? The packages on the official repositories are always built against the latest patch releases. So it's possible the packages are up to date but your jail for some reason isn't.

byrnejb said:
I have no idea when or how this started but it has to have been very recently because I was maintaining the configuration on some of these jails last week.
Click to expand...
Is it possible the base jail shifted somehow? And by that I mean that ezjail thinks it's /usr/jails/basejail but the jails are actually running off a different directory? So the update worked on the wrong directory?

Are you having that same issue on the host? Or does that appear to be working correctly?
 
byrnejb said:
No as a separate step. When freebsd-update install was run I was presented with several configuration files which recommended changes and had to accept or deny or alter these as I saw fit. I understood this to be the mergemaster step.
Click to expand...
It's not the exactly the same step. The problem is that freebsd-update only updates the files from the base jail. It does nothing for the files in /etc/ on each jail individually. It doesn't know they're there. That's where mergemaster(8) comes in.

14.6. Managing Jails with ezjail
 
SirDice said:
It's not the exactly the same step. The problem is that freebsd-update only updates the files from the base jail. It does nothing for the files in /etc/ on each jail individually. It doesn't know they're there. That's where mergemaster(8) comes in.

14.6. Managing Jails with ezjail
Click to expand...
I had not been doing that step, either from within or without the jail. I will apply this on one of the problem jails and report the result. I am about to leave for a surgical appointment so it may be a little while before I return to this matter.

Thanks for the help.
 
I have returned. I have discovered a work-around. It is brutal but it works 100% of the time. I stop all the jails. I archive them all. I destroy them all, together with their filesystems. I next remove the ezjail package. I then check for and delete, if found, any filesystem artifacts relating to ezjail, saving only /usr/local/etc/ezjail.conf.

I then
Code: 
pkg install ezjail
followed by
Code: 
ezjail-admin install
. Once complete I then simply re-created the jails from their archives and restarted them. The FSTAT error is gone from each.
 
Well, I was wrong. The fstat error is gone and PS and SERVICE now run. However, when I try to start named in a jail then I get this:
Code: 
ld-elf.so.1: Shared object "libcrypto.so.8" not found, required by "named-checkconf"
/usr/local/etc/rc.d/named: ERROR: named-checkconf for /usr/local/etc/namedb/named.conf failed

I do not know what was done to break jails in this fashion but it was done in a particularly through way.
 
P.S. mergemaster when run in a jail gives this:
Code: 
# mergemaster

*** /usr/src was not found.
    **** No suitable /usr/src found, exiting
 
And now I get this in a freshly created jail on a system that just had ezjail installed for the first time:
<pre>
pkg upgrade
ld-elf.so.1: Shared object "libssl.so.8" not found, required by "pkg"
</pre>

AmI to infer that somewhere between 12.0p3 and 12.0p5 something broke ezjail?
 
Last edited:
The host system for the jail referred to immediately above is a bhyve guest running 12.0-RELEASE-p6. The jail reports 12.0-RELEASE-p6.

When the bhyve guest boots it reports this:
Code: 
Starting syslogd.
syslogd: timed out waiting for child
/etc/rc: WARNING: failed to start syslogd

However, after the system has started service reports this:
Code: 
[root@inet13 ~]# service syslogd status
syslogd is running as pid 764.
[root@inet13 ~]# freebsd-version
12.0-RELEASE-p6

Something is seriously amiss here and I cannot see any reason to believe, given the number of hosts and vms so recently affected, that this is due to either hardware or any other local condition.
 
byrnejb said:
Pardon my ignorance, but does one install the sources into each and every jail or just on the host system?
Click to expand...
It depends on how you run it. If you run it from the host (which is what I typically do) then you only need it on the host. If you run it from within a jail then the jail would need to have access to it. But you can use nullfs(5) to mount the source tree read-only in a jail so you don't have to keep multiple copies of the source tree around.

But as I said, I always run this from the host side: mergemaster -U -D /path/to/jail and do this for each jail.
 
This is what a newly created jail on a bhyve guest running 12.0p6 displays as soon as one logs on via ezjail-admin console:
[root@dns03 ~]# service sshd status
/lib/libkvm.so.7: Undefined symbol "fstat@FBSD_1.5"
/lib/libkvm.so.7: Undefined symbol "fstat@FBSD_1.5"
Cannot 'status' sshd. Set sshd_enable to YES in /etc/rc.conf or use 'onestatus' instead of 'status'.

There is no update to apply. It is a freshly created jail using ezjail-admin create . . . . And this error arises. We did not have this problem until after applying the update to 12.0p5. Everything up to p3 worked fine. Now we cannot trust our jails to keep operating.
 
Installing the src archive and running mergemaster for each jail on the host has changed nothing. Creating a new jail using the ezjail-admin defaults results in exactly the same errors. This did not occur when we moved from 11 to 12.0. This is a problem recently introduced and it affects all of our platforms so it is doubtful if this is tied to the hardware.

I have found this: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221373#c8

(In reply to njh from comment #3)
Current packages have no chance for being usable with HEAD which is more than one year old. The ABI of the base system was updated, in particular, due to the ino64 project landing, which bumped the symbol versions for many important syscalls. If you use pre-build packages, you must use recent HEAD.
Click to expand...

We are on 12.0-RELEASE-p6 and we use binary packages exclusively. What is the comment above telling me to do? Do I have to switch FreeBSD versions from RELEASE to something else? STABLE perhaps?
 
Do NOT do this. 2019-06-28
All this is properly handled by running ezjail-admin update -U -s 11.2-RELEASE (or whatever release the basejail is at.)

These problems were caused by mismatched libraries in ezjail's basejail. Stopping the jails and copying the necessary libraries from /lib/ to /usr/jails/basejail/lib/ resolved the issue. I restarted the base host to restart the jails on the off chance that it might make a different in the outcome.

Code: 
ldd `which ps`
/bin/ps:
    libm.so.5 => /lib/libm.so.5 (0x80024e000)
    libkvm.so.7 => /lib/libkvm.so.7 (0x800280000)
    libjail.so.1 => /lib/libjail.so.1 (0x800293000)
    libxo.so.0 => /lib/libxo.so.0 (0x80029b000)
    libc.so.7 => /lib/libc.so.7 (0x8002bb000)
    libelf.so.2 => /lib/libelf.so.2 (0x8006ae000)
    libutil.so.9 => /lib/libutil.so.9 (0x8006c8000)

for L in libm.so.5 libkvm.so.7 libjail.so.1 libxo.so.0 libc.so.7 libelf.so.2 libutil.so.9 ;
   do diff /lib/$L /usr/jails/basejail/lib/$L;
   done
Binary files /lib/libm.so.5 and /usr/jails/basejail/lib/libm.so.5 differ
Binary files /lib/libjail.so.1 and /usr/jails/basejail/lib/libjail.so.1 differ
Binary files /lib/libxo.so.0 and /usr/jails/basejail/lib/libxo.so.0 differ
Binary files /lib/libc.so.7 and /usr/jails/basejail/lib/libc.so.7 differ
Binary files /lib/libelf.so.2 and /usr/jails/basejail/lib/libelf.so.2 differ
Binary files /lib/libutil.so.9 and /usr/jails/basejail/lib/libutil.so.9 differ

ezjail-admin stop

for L in libm.so.5 libkvm.so.7 libjail.so.1 libxo.so.0 libc.so.7 libelf.so.2 libutil.so.9 ;
  do
    chflags -R noschg /usr/jails/basejail/lib/$L
    cp -pf /lib/$L /usr/jails/basejail/lib/$L;
  done

ezjail-admin start
 
PS. If you have one library mismatch then you likely will find others. Use this code snippet to find them:
Code: 
for L in $(ls -1 /lib) ; do diff /lib/$L /usr/jails/basejail/lib/$L; done

But, all these problems are likely the result of me forgetting to do this after updating to 12.0:
Code: 
[root@inet19 ~]# file /usr/jails/basejail/bin/sh
/usr/jails/basejail/bin/sh: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD),
      dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 11.2, FreeBSD-style, stripped
[root@inet19 ~]# ezjail-admin update -U -s 11.2-RELEASE
 
You must log in or register to reply here.
Back
Top