Trouble with Jailed Kerberized NFSv4 Server After Upgrade to 15.0

R

rss2706

I have had a jailed kerberized nfsv4 server running flawlessly on the FBSD 14.x (latest being 14.3) for quite a few months. Since I upgraded both the host and jail to 15.0 mounts fail on my linux clients with a "Permission denied" error.

I have checked the kerberos keytabs as well as the ticket caches and all seems in order. During my troubleshooting, I tried the same kerberized nfsv4 server configuration (same daemons running, same exports file, etc) on the host (not in jail) and the mounts work as expected. Starting gssd in verbose mode, I noticed no activity at all from the jailed server, but saw activity like getting user credentials, etc when running nfs on the host.

In looking at the source code for gssd, I noticed a change in strategy between 14.x and 15.0. The 14.x releases use a unix domain local socket for the upcalls from the gssapi kernel module while 15.0 uses a Netlink socket listening for multicast traffic.

I am wondering if I am missing a setting in my jail configuration to allow Netlink sockets and/or multicast traffic. In reading the man pages and doing Internet searches, I have not been able to identify setting that would help. I have tried multiple jail setting such as allow.raw_sockets and nothing has solved the issue. As I am currently configured, gssd does not appear to be getting upcalls for kerberos authentication.

Any help would be greatly appreciated.
 
Without detailed information about the configuration, it is not possible to determine a possible cause.

It would be helpful to see the host jail.conf, rc.conf, devfs.rules, the jails NFSv4, the kerberos server configuration (Heimdal or MIT).

Someone might spot what's missing. Those who are willing could try to reproduce the issue in a test setup.

If you’re concerned about exposing sensitive details, you can redact them from the configuration files before sharing.
 
rss2706 said:
Since I upgraded both the host and jail to 15.0 mounts fail on my linux clients with a "Permission denied" error.
Click to expand...
Isn't this the obvious thing? Linux tends to mount NFS from a random port, it will get you the same "permission denied" error.

Code: 
     -n      Allow non-root mount requests to be served.  This should only be
             specified if there are clients such as PC's, that require it.  It
             will automatically clear the vfs.nfsd.nfs_privport sysctl flag,
             which controls if the kernel will accept NFS requests from
             reserved ports only.
mountd(8)
 
SirDice, thank you for the reply, but I am already using the -n option for the mountd daemon, so I don't think that is the problem.

T-Daemon, also thank you for the reply ... Here are some details of my configuration:

I am using Bastille to administer the nfs server jail
Host: fsandbox.testdom.net
Jail: nfstest.testdom.net
KDC: dc0.testdom.net KDC is Samba 4.23.4 machine acting as Active Directory Domain Controller (Heimdal)
Client: rlaptop.testdom.com This is a linux box running Linux Mint with a 6.8.0 kernel

Host loader.conf
Code: 
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
zfs_load="YES"
kgssapi_load="YES"
kgssapi_krb5_load="YES"
Note that kldstat was used on the host to verify kgssapi and kgssapi_krb5 are loaded

Host rc.conf
Code: 
hostname="fsandbox.testdom.net"
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.81.43 netmask 0xffffff00 addm vtnet0 up"
ifconfig_vtnet0="up"
defaultrouter="192.168.81.1"
update_motd="NO"
sshd_enable="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
bastille_enable="YES"
nslcd_enable="YES"

Jail jail.conf from bastille
Code: 
nfstest {
  enforce_statfs = 1;
  devfs_ruleset = 13;
  exec.clean;
  exec.consolelog = /var/log/bastille/nfstest_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = nfstest.testdom.net;
  mount.devfs;
  mount.fstab = /bastille/jails/nfstest/fstab;
  path = /bastille/jails/nfstest/root;
  securelevel = 2;
  osrelease = 15.0-RELEASE;

  mount.fdescfs;
  allow.mount;
  allow.mount.fdescfs;
  zfs.mount_snapshot = 1;
  allow.nfsd;
  allow.sysvipc = 1;
  allow.raw_sockets = 1;

  vnet;
  vnet.interface = e0b_nfstest;
  exec.prestart += "epair0=\$(ifconfig epair create) && ifconfig \${epair0} up name e0a_nfstest && ifconfig \${epair0%a}b up name e0b_nfstest";
  exec.prestart += "ifconfig bridge0 addm e0a_nfstest";
  exec.prestart += "ifconfig e0a_nfstest description \"vnet0 host interface for Bastille jail nfstest\"";
  exec.poststop += "ifconfig e0a_nfstest destroy";
}
Note the the devfs_rulset=13 allows all devices to be mounted on the jail. Usually I am more restrictive, but wanted to open it up for debugging.

Jail rc.conf
Code: 
ifconfig_e0b_nfstest_name="vnet0"
ifconfig_vnet0="inet 192.168.81.44/24"
ifconfig_vnet0_descr="jail interface for bridge0"
ifconfig_vnet0_ipv6=""
defaultrouter="192.168.81.1"
ipv6_defaultrouter="NO"
syslogd_flags="-ss"
sshd_enable="YES"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
cron_flags="-J 60"
nslcd_enable="YES"
gssd_enable="YES"
gssd_flags="-v -h"
nfs_server_enable="YES"
nfs_server_flags="-t"
nfsv4_server_enable="YES"
nfsv4_server_only="YES"
nfsuserd_enable="YES"
nfs_reserved_port_only="NO"
mountd_enable="YES"
mountd_flags="-n"

Jail /etc/exports
Code: 
V4: / 
/userdata/common -sec=krb5i -maproot=root -network 192.168.0.0/16
Note that /userdata/common is a nullfs mounted zfs dataset from the host which will mount just fine to the client when sec=sys

Jail nsswitch.conf
Code: 
#
# nsswitch.conf(5) - name service switch configuration file
#
group: files ldap
hosts: files dns
netgroup: compat
networks: files
passwd: files ldap
shells: files
services: compat
services_compat: nis
protocols: files
rpc: files
Note that the Samba AD domain controller is the ldap server. User and group resolution has been tested using both getent and id commands.

Jail /etc/krb5.conf
Code: 
[libdefaults]
    default_realm = TESTDOM.NET
    ticket_lifetime = 24h
    renew_lifetime = 7d
    forwardable = yes
[domain_realm]
    testdom.net = TESTDOM.NET
    .testdom.net = TESTDOM.NET
[realms]
    TESTDOM.NET = {
       kdc = dc0.testdom.net
       default_domain = testdom.net
    }
[logging]
    default = SYSLOG:INFO:USER

Jail Service Principal Keys in /etc/krb5.keytab
Code: 
   2 NFSTEST$@TESTDOM.NET (aes256-cts-hmac-sha1-96) 
   2 NFSTEST$@TESTDOM.NET (aes128-cts-hmac-sha1-96) 
   2 NFSTEST$@TESTDOM.NET (DEPRECATED:arcfour-hmac) 
   2 host/nfstest@TESTDOM.NET (aes256-cts-hmac-sha1-96) 
   2 host/nfstest@TESTDOM.NET (aes128-cts-hmac-sha1-96) 
   2 host/nfstest@TESTDOM.NET (DEPRECATED:arcfour-hmac) 
   2 host/nfstest.testdom.net@TESTDOM.NET (aes256-cts-hmac-sha1-96) 
   2 host/nfstest.testdom.net@TESTDOM.NET (aes128-cts-hmac-sha1-96) 
   2 host/nfstest.testdom.net@TESTDOM.NET (DEPRECATED:arcfour-hmac) 
   2 RestrictedKrbHost/nfstest@TESTDOM.NET (aes256-cts-hmac-sha1-96) 
   2 RestrictedKrbHost/nfstest@TESTDOM.NET (aes128-cts-hmac-sha1-96) 
   2 RestrictedKrbHost/nfstest@TESTDOM.NET (DEPRECATED:arcfour-hmac) 
   2 RestrictedKrbHost/nfstest.testdom.net@TESTDOM.NET (aes256-cts-hmac-sha1-96) 
   2 RestrictedKrbHost/nfstest.testdom.net@TESTDOM.NET (aes128-cts-hmac-sha1-96) 
   2 RestrictedKrbHost/nfstest.testdom.net@TESTDOM.NET (DEPRECATED:arcfour-hmac) 
   2 nfs/nfstest@TESTDOM.NET (aes256-cts-hmac-sha1-96) 
   2 nfs/nfstest@TESTDOM.NET (aes128-cts-hmac-sha1-96) 
   2 nfs/nfstest@TESTDOM.NET (DEPRECATED:arcfour-hmac) 
   2 nfs/nfstest.testdom.net@TESTDOM.NET (aes256-cts-hmac-sha1-96) 
   2 nfs/nfstest.testdom.net@TESTDOM.NET (aes128-cts-hmac-sha1-96) 
   2 nfs/nfstest.testdom.net@TESTDOM.NET (DEPRECATED:arcfour-hmac)

Client krb5.conf
Code: 
[libdefaults]
default_realm = TESTDOM.NET
    dns_lookup_kdc = true
    dns_lookup_realm = true
    udp_preference_limit = 0

[realms]
    TESTDOM.NET = {
        kdc = dc0.testdom.net
        admin_server = dc0.testdom.net
        master_kdc = dc0.testdom.net
        default_domain = testdom.net
        }

[domain_realm]
    .testdom.net = TESTDOM.NET
    testdom.net = TESTDOM.NET

[logging]
    kdc = SYSLOG:INFO
    admin_server = FILE=/var/kadm5.log

Client Service Principal Keys in /etc/krb5.keytab
Code: 
  31 RLAPTOP@TESTDOM.NET (aes256-cts-hmac-sha1-96) 
  31 RLAPTOP@TESTDOM.NET (aes128-cts-hmac-sha1-96) 
  31 RLAPTOP@TESTDOM.NET (DEPRECATED:arcfour-hmac) 
  31 host/rlaptop@TESTDOM.NET (aes256-cts-hmac-sha1-96) 
  31 host/rlaptop@TESTDOM.NET (aes128-cts-hmac-sha1-96) 
  31 host/rlaptop@TESTDOM.NET (DEPRECATED:arcfour-hmac) 
  31 host/rlaptop.testdom.net@TESTDOM.NET (aes256-cts-hmac-sha1-96) 
  31 host/rlaptop.testdom.net@TESTDOM.NET (aes128-cts-hmac-sha1-96) 
  31 host/rlaptop.testdom.net@TESTDOM.NET (DEPRECATED:arcfour-hmac) 
  31 RestrictedKrbHost/rlaptop@TESTDOM.NET (aes256-cts-hmac-sha1-96) 
  31 RestrictedKrbHost/rlaptop@TESTDOM.NET (aes128-cts-hmac-sha1-96) 
  31 RestrictedKrbHost/rlaptop@TESTDOM.NET (DEPRECATED:arcfour-hmac) 
  31 RestrictedKrbHost/rlaptop.testdom.net@TESTDOM.NET (aes256-cts-hmac-sha1-96) 
  31 RestrictedKrbHost/rlaptop.testdom.net@TESTDOM.NET (aes128-cts-hmac-sha1-96) 
  31 RestrictedKrbHost/rlaptop.testdom.net@TESTDOM.NET (DEPRECATED:arcfour-hmac)

Client ticket cache shown by klist after kinit
Code: 
Default principal: Administrator@TESTDOM.NET

Valid starting       Expires              Service principal
04/09/2026 15:15:47  04/10/2026 01:15:47  krbtgt/TESTDOM.NET@TESTDOM.NET
    renew until 04/10/2026 15:15:42

This is the result of a mount attempt
Code: 
root@rlaptop:/etc# mount -vvv -t nfs4 nfstest:/userdata/common /mnt -o rw,sec=krb5i
mount.nfs4: timeout set for Thu Apr  9 15:29:11 2026
mount.nfs4: trying text-based options 'sec=krb5i,vers=4.2,addr=192.168.81.44,clientaddr=192.168.81.178'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5i,vers=4,minorversion=1,addr=192.168.81.44,clientaddr=192.168.81.178'
mount.nfs4: mount(2): Permission denied
mount.nfs4: trying text-based options 'sec=krb5i,vers=4,addr=192.168.81.44,clientaddr=192.168.81.178'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting nfstest:/userdata/common

This configuration has worked reliably on 14.3-RELEASE for many months. Updrading to 15.0-RELEASE has precipitated these errors. Also, this nfs configuration mounts without issue and works fine when served from a non-jailed host. When looking at the daemon.log file that is produced, gssd on the jailed 15.0 server shows no output at all as opposed to the 14.3 server or the non-jailed 15.0 server which shows the expected output from upcalls.

It almost seems like a communication issue between the kernel and the userspace gssd, since no kerberos authentication upcalls seem to be happening. Again, while examining source code I noticed the kernel/gssd communication strategy changed between 14.x and 15.0 from a unix socket to a Netlink socket. Perhaps that has somthing to do with the issue.

Any help in resolving this would be greatly appreciated.
 
You must log in or register to reply here.
Back
Top