Solved FreeBSD-12.0p6 and odd warnings in /var/log/messages

We continue to encounter anomalies with jails running on hosts with 12.0p6 and have just encountered these /var/log/message entries on a bhyve vm running 12.0p6 with two jails.

Code:
Jun 26 15:25:09 inet18 root[95378]: /usr/sbin/service: WARNING: $ is not set properly - see rc.conf(5).
Jun 26 15:25:09 inet18 root[44732]: /usr/sbin/service: WARNING: $tpmd_enable is not set properly - see rc.conf(5).
Jun 26 15:25:09 inet18 root[47106]: /usr/sbin/service: WARNING: $tcsd_enable is not set properly - see rc.conf(5).
Jun 26 15:25:10 inet18 root[32282]: /usr/sbin/service: WARNING: $unbound_enable is not set properly - see rc.conf(5).
Jun 26 15:25:10 inet18 root[33482]: /usr/sbin/service: WARNING: $spiped_enable is not set properly - see rc.conf(5).
Jun 26 15:25:10 inet18 root[37364]: /usr/sbin/service: WARNING: $rsyncd_enable is not set properly - see rc.conf(5).
Jun 26 15:25:10 inet18 root[39443]: /usr/sbin/service: WARNING: $postgrey_enable is not set properly - see rc.conf(5).
Jun 26 15:25:10 inet18 root[42168]: /usr/sbin/service: WARNING: $cyrus_imapd_enable is not set properly - see rc.conf(5).
Jun 26 15:25:10 inet18 root[43883]: /usr/sbin/service: WARNING: $milteropendkim_enable is not set properly - see rc.conf(5).
Jun 26 15:25:10 inet18 root[51362]: /usr/sbin/service: WARNING: $dbus_enable is not set properly - see rc.conf(5).
Jun 26 15:25:10 inet18 root[53292]: /usr/sbin/service: WARNING: $avahi_daemon_enable is not set properly - see rc.conf(5).
Jun 26 15:25:10 inet18 root[57600]: /usr/sbin/service: WARNING: $cups_browsed_enable is not set properly - see rc.conf(5).
Jun 26 15:25:10 inet18 root[58727]: /usr/sbin/service: WARNING: $avahi_dnsconfd_enable is not set properly - see rc.conf(5).
Jun 26 15:25:10 inet18 root[60552]: /usr/sbin/service: WARNING: $autossh_enable is not set properly - see rc.conf(5).

The bhyve host has these entries in rc.conf:

Code:
[root@inet18 ~]# grep enable /etc/rc.conf
rtsold_enable="YES"
gateway_enable="YES"              # Enable as ipv4 LAN gateway for guests/jails
ipv6_gateway_enable="YES"        # Enable as ipv6 LAN gateway
ezjail_enable="YES"               # Enable ezjail jail manager
cupsd_enable="YES"                # printers
local_unbound_enable="YES"        # DNS resolver
lpd_enable="YES"                  # system printers
moused_enable="NO"                # For desktop and console
named_enable="NO"                 # dns03 see INET13
ntpd_enable="YES"                 # Network time
sshd_enable="YES"                 # Secure SHell login server
#amavisd_enable="YES"              # Enable amavisd milter daemon
#clamav_clamd_enable="YES"         # Enable ClamAV daemon virus scanner
#clamav_freshclam_enable="YES"     # Enable virus signature auto-updates
#milteropendkim_enable="YES"       # Enable OpenDKIM SPF
postfix_enable="YES"              # Enable Postfix SMTP MTA
#postgrey_enable="YES"             # Enable Postgrey
saslauthd_enable="YES"            # Enable simple authentication service
sendmail_enable="NONE"            # Disable Sendmail SMTP MTA
#spamd_enable="YES"                # Enable spamassassin (sa-spamd)
#spiped_enable="YES"               # Enabled spiped daemon
#dbus_enable="YES"
#hald_enable="YES"
#kmd4_enable="YES         # Use startx with /usr/local/bin/startkde
clear_tmp_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
zfs_enable="YES"

The jails have these rc.conf entries:

Code:
[root@inet18 ~]# grep enable /usr/jails/mx31/etc/rc.conf
dbus_enable="YES"
local_unbound_enable="NO"         # Disable caching only DNS
named_enable="NO"                 # Enable authoritative DNS slave
ntpd_enable="NO""                 # Enable local network time server
sendmail_enable="NONE"            # Disable Sendmail SMTP MTA
sshd_enable="YES"                 # Enable ssh access
amavisd_enable="YES"              # Enable amavisd milter daemon
clamav_clamd_enable="YES"         # Enable ClamAV daemon virus scanner
clamav_freshclam_enable="YES"     # Enable virus signature auto-updates
milteropendkim_enable="YES"       # Enable OpenDKIM SPF
postfix_enable="YES"              # Enable Postfix SMTP MTA
postgrey_enable="YES"             # Enable Postgrey
saslauthd_enable="YES"            # Enable simple authentication service
spamd_enable="YES"                # Enable spamassassin (sa-spamd)
spiped_enable="YES"               # Enabled spiped daemon
webmin_enable="YES"               # System administration web interface


[root@inet18 ~]# grep enable /usr/jails/dns03/etc/rc.conf
named_enable="YES"                # Enable DNS service
ntpd_enable="YES"                 # Enable local network time server
postfix_enable="YES"              # Enable Postfix SMTP MTA
sendmail_enable="NONE"            # Disable Sendmail SMTP MTA
sshd_enable="NO"                  # disable ssh access use jail console

We are now in the process of moving things off 12.0 onto hosts running 11.2. 12.0 is definitely not ready for prime time and should problably never have made it to RELEASE in its current state. It has rendered jails completely unreliable and I have grave suspicions about how zfs has been affected as well.
 
There's something out of whack with your jail's /etc/ directories. They're not complete or not up to date. The reference to rc.conf is a red herring.
 
The two quotes next to ntpd_enable may be causing some problems. It does seem like the errors may be related to a syntax error in rc.conf.
 
There's something out of whack with your jail's /etc/ directories. They're not complete or not up to date. The reference to rc.conf is a red herring.

These are recently recreated jails. Both of them. Moved to a new bhyve guest because of problems that started on their original host, also encountered after 12.0p3. I likely have missed something I readily admit. But I am at a loss to know what that missing step is.

The two quotes next to ntpd_enable may be causing some problems. It does seem like the errors may be related to a syntax error in rc.conf.

Thanks, I never noticed. Both fixed.
 
No matter what I do I cannot get rid of this error when I try to run ps or netstat in a jail on 12.0p6.
/lib/libkvm.so.7: Undefined symbol "fstat@FBSD_1.5"

Further, on jails the service utility always reports a service as not running, even when it is possible to connect to it. This makes administering services on jails very difficult as there seems to be no way for a script to determine if one is running or not.
 
So I did this:
Code:
[root@mx31 ~]# ldd `which ps`
/bin/ps:
    libm.so.5 => /lib/libm.so.5 (0x800829000)
    libkvm.so.7 => /lib/libkvm.so.7 (0x800631000)
    libjail.so.1 => /lib/libjail.so.1 (0x800a56000)
    libxo.so.0 => /lib/libxo.so.0 (0x800c5b000)
    libc.so.7 => /lib/libc.so.7 (0x800e78000)
    libelf.so.2 => /lib/libelf.so.2 (0x801234000)
    libutil.so.9 => /lib/libutil.so.9 (0x80144b000)
[root@mx31 ~]# ps -auwx
/lib/libkvm.so.7: Undefined symbol "fstat@FBSD_1.5"



[root@inet18 ~]# ldd `which ps`
/bin/ps:
    libm.so.5 => /lib/libm.so.5 (0x80024e000)
    libkvm.so.7 => /lib/libkvm.so.7 (0x800280000)
    libjail.so.1 => /lib/libjail.so.1 (0x800293000)
    libxo.so.0 => /lib/libxo.so.0 (0x80029b000)
    libc.so.7 => /lib/libc.so.7 (0x8002bb000)
    libelf.so.2 => /lib/libelf.so.2 (0x8006ae000)
    libutil.so.9 => /lib/libutil.so.9 (0x8006c8000)
[root@inet18 ~]# diff /lib/libkvm.so.7 /usr/jails/basejail/lib/libkvm.so.7

[root@inet18 ~]# diff /lib/libm.so.5 /usr/jails/basejail/lib/libm.so.5
Binary files /lib/libm.so.5 and /usr/jails/basejail/lib/libm.so.5 differ

[root@inet18 ~]# diff /lib/libjail.so.1 /usr/jails/basejail/lib/libjail.so.1
Binary files /lib/libjail.so.1 and /usr/jails/basejail/lib/libjail.so.1 differ

[root@inet18 ~]# diff /lib/libkvm.so.7 /usr/jails/basejail/lib/libkvm.so.7

[root@inet18 ~]# export L='libxo.so.0' ; diff /lib/$L /usr/jails/basejail/lib/$L     
Binary files /lib/libxo.so.0 and /usr/jails/basejail/lib/libxo.so.0 differ

[root@inet18 ~]# export L='libc.so.7' ; diff /lib/$L /usr/jails/basejail/lib/$L
Binary files /lib/libc.so.7 and /usr/jails/basejail/lib/libc.so.7 differ

[root@inet18 ~]# export L='libelf.so.2' ; diff /lib/$L /usr/jails/basejail/lib/$L
Binary files /lib/libelf.so.2 and /usr/jails/basejail/lib/libelf.so.2 differ

[root@inet18 ~]# export L='libutil.so.9' ; diff /lib/$L /usr/jails/basejail/lib/$L
Binary files /lib/libutil.so.9 and /usr/jails/basejail/lib/libutil.so.9 differ

[root@inet18 ~]#

Now, I have no idea how these mismatches occurred. The Freebsd update proceedure was followed completely (fetch/install/reboot) and that was followed by ezjail-admin update -u. So, what I plan to do is to manually update the basejail since ezjail-admin does not seem to want to.
 
So, I copied all of these save one:
Code:
export L='libc.so.7' ; cp -p /lib/$L /usr/jails/basejail/lib/$L
overwrite /usr/jails/basejail/lib/libc.so.7? (y/n [n]) y
cp: /usr/jails/basejail/lib/libc.so.7: Operation not permitted

[root@inet18 ~]# ll /lib/$L /usr/jails/basejail/lib/$L
-r--r--r--  1 root  wheel  1949672 Mar 24 13:13 /lib/libc.so.7
-r--r--r--  1 root  wheel  1779544 Nov  8  2018 /usr/jails/basejail/lib/libc.so.7

Any ideas as to why `libc.so.7` cannot be copied?
 
The manual replacement of the libraries appears (so far) to have solved all the problems with ps and service in the jails. Why ezjail-admin update -u did not handle this I cannot fathom.

Code:
ezjail-admin stop
. . .
chflags -R noschg /usr/jails/basejail/lib/libc.so.7
cp -p /lib/libc.so.7 /usr/jails/basejail/lib/libc.so.7

ezjail-admin start
 
Back
Top