jails NFSv4 Server won't start inside a FreeBSD 14.2 VNET Jail | error nfsd : nfssvc stablerestart failed: allow.nfsd might not be configured

R

rai.mohammed

Hello everybody ;

I'm trying to set up a NFSv4 Server inside a FreeBSD 14.2 vnet jail, after the necessary configuration and parameters, I started the service from the consol, unfortunately I get a negative result and the nfsd server won't start, like decribed below :
Code: 
Jail : [root@odoo18erp01:~] # service nfsd restart
nfsd not running?
Starting nfsd.

So I jumped into the log messages to understand the issue, and I finded an error saying that allow.nfsd is not configured,
even thought my jail.conf file has this option.
Code: 
Jail : [root@odoo18erp01:~] # cat /var/log/messages
Feb  6 18:42:15 odoo18erp01 syslogd: exiting on signal 15
Feb  6 18:43:26 odoo18erp01 syslogd: kernel boot file is /boot/kernel/kernel
Feb  6 18:43:26 odoo18erp01 mountd[41593]: Can't delete exports for V4:
Feb  6 18:43:26 odoo18erp01 mountd[41593]: Exporting V4: failed
Feb  6 18:43:26 odoo18erp01 mountd[41593]: bad exports list line 'V4: /                         -sec'
Feb  6 18:43:26 odoo18erp01 mountd[41593]: Warning: exporting /usr/local/odoo_erp/odoo18ce exports entire / file system
Feb  6 18:43:26 odoo18erp01 mountd[41593]: bad opt rw
Feb  6 18:43:26 odoo18erp01 mountd[41593]: bad exports list line '/usr/local/odoo_erp/odoo18ce  -sec'
Feb  6 18:43:26 odoo18erp01 nfsd[45357]: nfssvc stablerestart failed: allow.nfsd might not be configured

The next step for me, is it to verify the security jail params, suprised by allow.nfsd is not authorized,
contrary to the allow.raw_sockets param as exposed in these commands :
Code: 
Jail : [root@odoo18erp01:~] # sysctl security.jail.param.allow.nfsd
security.jail.param.allow.nfsd: 0
Jail : [root@odoo18erp01:~] # sysctl -a |grep -i security.jail.allow
security.jail.allow_raw_sockets: 1

Plus the host dmesg throw the error [mountd/nfsd requires a separate file system] :
Code: 
root@desktopfreebsd:~ # dmesg
[15567] Warning jail jid=10: mountd/nfsd requires a separate file system
[15567] lo0: link state changed to UP
[15567] arp: 192.168.20.233 moved from 02:d9:63:f9:cf:0b to 02:11:b2:67:b9:0b on epair234b

root@desktopfreebsd:~ # jls
   JID  IP Address      Hostname                      Path
     1                  pg17db01.itlinker.local       /usr/local/jails/containers/pg17db01
     3                  apache24web01.itlinker.local  /usr/local/jails/containers/apache24web01
    10                  odoo18erp01.itlinker.local    /usr/local/jails/containers/odoo18erp01
root@desktopfreebsd:~ #

My setup and configuration are summerized like so :
Code: 
root@desktopfreebsd:~ # cat /etc/jail.conf.d/odoo18erp01.conf
odoo18erp01 {
  # Summarized configuration :

# PERMISSIONS

  allow.raw_sockets;
  # NFSD/VNET
  allow.nfsd;

  allow.mount;
  allow.mount.devfs;
  allow.mount.fdescfs;
  mount.devfs;
  allow.mount.zfs;
  enforce_statfs = 1;
}

Jail : [root@odoo18erp01:~] # cat /etc/rc.conf
# Summarized configuration :

# DAEMONS
rpcbind_enable="YES"
rpcbind_flags="-h 192.168.20.231"
rpc_lockd_enable="YES"
rpc_lockd_flags="-h 192.168.20.231"
rpc_lockd_enable="YES"
rpc_lockd_flags="-h 192.168.20.231"
rpc_statd_enable="YES"
rpc_statd_flags="-h 192.168.20.231"
mountd_enable="YES"
mountd_flags="-h 192.168.20.231 -r -S"

nfs_server_enable="YES"
nfsv4_server_enable="YES"
nfsv4_server_only="YES"
nfs_server_flags="-h 192.168.20.231 -u -t -n 8"

Jail : [root@odoo18erp01:~] # cat /etc/sysctl.conf
# VNET/NFSD
vfs.nfs.enable_uidtostring=1
vfs.nfsd.enable_stringtouid=1

Jail : [root@odoo18erp01:~] # cat /etc/exports
V4: /                        -sec=sys                    -network 192.168.20.0/24
/usr/local/odoo_erp/odoo18ce -sec=sys -maproot=1001:1001 -network 192.168.20.0/24

Please, can someone tell me, is there any problem in my configuration, or did I forget a parameter somewhere ?!
 
Ole said:
where is your
Code: 
vnet
option?
Click to expand...
As I said the config above is a summarized one.
I use a jail with multiple vnet interfaces, please see below :
Code: 
cat /etc/jail.conf.d/odoo18erp01.conf
...
....
.....

# NETWORKS/INTERFACES - The order must be respected
  $id1 = "231";
  $ip1 = "192.168.20.${id1}/24";

  $id2 = "233";
  $ip2 = "192.168.20.${id2}/24";

  $gateway = "192.168.20.1";
  $bridge = "bridge0";

  $epair1 = "epair${id1}";
  $epair2 = "epair${id2}";

  # VNET/VIMAGE - The order must be respected
  vnet;
  vnet.interface += "${epair2}b";
  vnet.interface += "${epair1}b";

...
...
...
 
I see that you have nfs flags:
Code: 
nfs_server_flags="-h 192.168.20.231 -u -t -n 8"
Per:

nfsd

man.freebsd.org man.freebsd.org

Code: 
When nfsd is run in an appropriately configured vnet jail, the server is
restricted to TCP transport and no pNFS service. Therefore, the -t option must
be specified and none of the -u, -p and -m options can be specified when run
in a vnet jail. See jail(8) for more information.

So, could it be the U switch?
 
JohnK said:
I see that you have nfs flags:
Code: 
nfs_server_flags="-h 192.168.20.231 -u -t -n 8"
Per:

nfsd

man.freebsd.org man.freebsd.org

Code: 
When nfsd is run in an appropriately configured vnet jail, the server is
restricted to TCP transport and no pNFS service. Therefore, the -t option must
be specified and none of the -u, -p and -m options can be specified when run
in a vnet jail. See jail(8) for more information.

So, could it be the U switch?
Click to expand...
I tested the config with :
nfs_server_flags="-h 192.168.20.231 -t -n 8"

The result was the same.
 
I'm about out of ideas but then, wouldn't it be (/etc/rc.conf):
Code: 
rpcbind_enable="YES"
rpcbind_flags="-h 192.168.20.231"

rpc_lockd_enable="YES"
rpc_lockd_flags="-h 192.168.20.231"

rpc_statd_enable="YES"
rpc_statd_flags="-h 192.168.20.231"

mountd_enable="YES"
mountd_flags="-h 192.168.20.231 -r -S"

nfs_server_enable="YES"

# In a jail, only TCP, no pNFS.
#nfsv4_server_enable="YES"
#nfsv4_server_only="YES"
#nfs_server_flags="-h 192.168.20.231 -u -t -n 8"
I assume the other options/and flags have been looked at.

And then...
Code: 
# service nfsd start
# service mountd reload
 
JohnK said:
I'm about out of ideas but then, wouldn't it be (/etc/rc.conf):
Code: 
rpcbind_enable="YES"
rpcbind_flags="-h 192.168.20.231"

rpc_lockd_enable="YES"
rpc_lockd_flags="-h 192.168.20.231"

rpc_statd_enable="YES"
rpc_statd_flags="-h 192.168.20.231"

mountd_enable="YES"
mountd_flags="-h 192.168.20.231 -r -S"

nfs_server_enable="YES"

# In a jail, only TCP, no pNFS.
#nfsv4_server_enable="YES"
#nfsv4_server_only="YES"
#nfs_server_flags="-h 192.168.20.231 -u -t -n 8"
I assume the other options/and flags have been looked at.

And then...
Code: 
# service nfsd start
# service mountd reload
Click to expand...
Thank you for your help, and I'm sorry if I disturbed you, your replaying was very appreciated. 👍
 
rai.mohammed said:
Jail : [root@odoo18erp01:~] # cat /etc/exports
V4: / -sec=sys -network 192.168.20.0/24

Please, can someone tell me, is there any problem in my configuration, or did I forget a parameter somewhere ?!
Click to expand...
I use zfs as well.

The only thing I have in my /etc/exports file is:

V4: /

and nothing else. The rest is all done with several

zfs set sharenfs= ...

lines.

Using FreeBSD v14.1p7 presently but had this setup since v13.x
 
Bucky said:
I use zfs as well.

The only thing I have in my /etc/exports file is:

V4: /

and nothing else. The rest is all done with several

zfs set sharenfs= ...

lines.

Using FreeBSD v14.1p7 presently but had this setup since v13.x
Click to expand...
Is your NFSv4 setup inside or outside a jail ?
 
rai.mohammed said:
I'm trying to set up a NFSv4 Server inside a FreeBSD 14.2 vnet jail, after the necessary configuration and parameters, I started the service from the consol, unfortunately I get a negative result and the nfsd server won't start, like decribed below :
Click to expand...
I have a working NFS server in a jail on a test VM, bhyve(8). I can mount the NFS share from the jail on the VM's host.

Code: 
root@jailHost: ~ # jexec 142NFS service nfsd status
nfsd is running as pid 1407 1408.

Basic configuration, documentations from handbook 17.5.3. Creating a VNET Jail and https://people.freebsd.org/~rmacklem/nfsd-vnet-prison-setup.txt.

I haven't checked of unnecessary settings.

Host:

/etc/rc.conf
Code: 
defaultrouter="192.168.2.1"
cloned_interfaces="bridge0"
ifconfig_bridge0="192.168.2.20/24 up addm vtnet0"

/etc/sysctl.conf
Code: 
security.jail.param.allow.nfsd=1

/etc/jail.conf.d/142NFS.conf
Code: 
142NFS {
        exec.start = "/bin/sh /etc/rc";
        exec.stop  = "/bin/sh /etc/rc.shutdown";
        exec.consolelog = "/var/log/jail_console_142NFS";

        allow.raw_sockets;
        exec.clean;
        mount.devfs;
        devfs_ruleset = 5;

        path = "/usr/local/jails/14.2-RELEASE";
        host.hostname = "142NFS";

        vnet;
        vnet.interface = "${epair}b";

        $id = "20";
        $ip = "192.168.2.${id}/24";
        $gateway = "192.168.2.1";
        $bridge = "bridge0";
        $epair = "epair${id}";

        exec.prestart = "/sbin/ifconfig ${epair} create up";
        exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:142NFS";
        exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";
        exec.start += "/sbin/ifconfig ${epair}b ${ip} up";
        exec.start += "/sbin/route add default ${gateway}";
        exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a";
        exec.poststop += "/sbin/ifconfig ${epair}a destroy";

        allow.nfsd;
        enforce_statfs = "1";
}


Jail:

/etc/rc.conf
Code: 
nfs_server_enable="YES"
nfsv4_server_enable="YES"
nfsv4_server_only="YES"
nfs_server_flags="-t"
nfsuserd_enable="YES"

/etc/sysctl.conf
Code: 
vfs.nfs.enable_uidtostring=1
vfs.nfsd.enable_stringtouid=1

/etc/exports
Code: 
V4: /usr/nfs

/usr/nfs/Share1
 
T-Daemon said:
I have a working NFS server in a jail on a test VM, bhyve(8). I can mount the NFS share from the jail on the VM's host.

Code: 
root@jailHost: ~ # jexec 142NFS service nfsd status
nfsd is running as pid 1407 1408.

Basic configuration, documentations from handbook 17.5.3. Creating a VNET Jail and https://people.freebsd.org/~rmacklem/nfsd-vnet-prison-setup.txt.

I haven't checked of unnecessary settings.

Host:

/etc/rc.conf
Code: 
defaultrouter="192.168.2.1"
cloned_interfaces="bridge0"
ifconfig_bridge0="192.168.2.20/24 up addm vtnet0"

/etc/sysctl.conf
Code: 
security.jail.param.allow.nfsd=1

/etc/jail.conf.d/142NFS
Code: 
142NFS {
        exec.start = "/bin/sh /etc/rc";
        exec.stop  = "/bin/sh /etc/rc.shutdown";
        exec.consolelog = "/var/log/jail_console_142NFS";

        allow.raw_sockets;
        exec.clean;
        mount.devfs;
        devfs_ruleset = 5;

        path = "/usr/local/jails/14.2-RELEASE";
        host.hostname = "142NFS";

        vnet;
        vnet.interface = "${epair}b";

        $id = "20";
        $ip = "192.168.2.${id}/24";
        $gateway = "192.168.2.1";
        $bridge = "bridge0";
        $epair = "epair${id}";

        exec.prestart = "/sbin/ifconfig ${epair} create up";
        exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:142NFS";
        exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";
        exec.start += "/sbin/ifconfig ${epair}b ${ip} up";
        exec.start += "/sbin/route add default ${gateway}";
        exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a";
        exec.poststop += "/sbin/ifconfig ${epair}a destroy";

        allow.nfsd;
        enforce_statfs = "1";
}


Jail:

/etc/rc.conf
Code: 
nfs_server_enable="YES"
nfsv4_server_enable="YES"
nfsv4_server_only="YES"
nfs_server_flags="-t"
nfsuserd_enable="YES"

/etc/sysctl.conf
Code: 
vfs.nfs.enable_uidtostring=1
vfs.nfsd.enable_stringtouid=1

/etc/exports
Code: 
V4: /usr/nfs

/usr/nfs/Share1
Click to expand...
Thanks for your reply, the config of my jail is the same as yours, except I use a multi vnet interfaces (the first setup was with one vnet interface), but the weird thing is the param "security.jail.param.allow.nfsd", even configured in "/etc/sysctl.conf" file and rebooting the host, it's always equal to "0", see below :
Code: 
root@desktopfreebsd:~ #
root@desktopfreebsd:~ # sysctl security.jail.param.allow.nfsd
security.jail.param.allow.nfsd: 0
root@desktopfreebsd:~ #
root@desktopfreebsd:~ # cat /etc/sysctl.conf | grep security.jail.param.allow.nfsd
security.jail.param.allow.nfsd=1
root@desktopfreebsd:~ # sysctl security.jail.param.allow.nfsd=1
security.jail.param.allow.nfsd: 0 -> 0
root@desktopfreebsd:~ #
 
rai.mohammed said:
the param "security.jail.param.allow.nfsd", even configured in "/etc/sysctl.conf" file and rebooting the host, it's always equal to "0",
Click to expand...
I got that too. Apparently it's not critical for nfsd to run. I haven't checked for non-useful settings.

Perhaps start with a minimal configuration, which is working, then add one configuration after the other.
 
You must log in or register to reply here.
Back
Top