Solved FreeBSD 14-2 Jail cannot access mount point

[rai.mohammed]

My objective is to install PostgreSQL 17 inside FreeBSD 14.2 Jail, and that by separating and tweaking 3 datasets inside the jail.
My FreeBSD 14-2 Host-Managed zfs datasets are like below :

My PostgreSQL /etc/jail.conf.d/pgdb01.conf :


rai.mohammed@desktopfb:~ $ cat /etc/jail.conf.d/pgdb01.conf
pgdb01 {
# STARTUP/LOGGING
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jails/jail_console_${name}.log";

# PERMISSIONS
allow.raw_sockets;
exec.clean;
mount.devfs;

allow.mount;
allow.mount.devfs;
allow.mount.zfs;
enforce_statfs = 1;

# /etc/defaults/devfs.rules
devfs_ruleset = 5;

# To run PostgreSQL in a FreeBSD jail
sysvmsg=new ;
sysvsem=new ;
sysvshm=new ;

# HOSTNAME/PATH
$domain = "itlinker.lan";
host.hostname = "${name}.${domain}";
path = "/usr/local/jails/containers/${name}";
mount.fstab="/etc/fstab.$name";

# VNET/VIMAGE
vnet;
vnet.interface = "${epair}b";

# NETWORKS/INTERFACES
$id = "232";
$ip = "192.168.20.${id}/24";
$gateway = "192.168.20.1";
$bridge = "bridge0";
$epair = "epair${id}";

# ADD TO bridge INTERFACE
exec.prestart = "/sbin/ifconfig ${epair} create up";
exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:${name}";
exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";
exec.start += "/sbin/ifconfig ${epair}b ${ip} up";
exec.start += "/sbin/route add default ${gateway}";
exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a";
exec.poststop += "/sbin/ifconfig ${epair}a destroy";

# ADD TO show ZFS dataset in jail
exec.poststart += "/sbin/zfs jail pgdb01 zroot/jails/containers/pgdb01/data17";
# exec.poststart += "/sbin/zfs jail pgdb01 zroot/jails/containers/pgdb01/data17/base";
# exec.poststart += "/sbin/zfs jail pgdb01 zroot/jails/containers/pgdb01/data17/pg_wal";


exec.poststart += "/sbin/zfs mount zroot/jails/containers/pgdb01/data17";
# exec.poststart += "/sbin/zfs mount zroot/jails/containers/pgdb01/data17/base";
# exec.poststart += "/sbin/zfs mount zroot/jails/containers/pgdb01/data17/pg_wal";

# exec.poststop += "/sbin/zfs unmount zroot/jails/containers/pgdb01/data17/pg_wal";
# exec.poststop += "/sbin/zfs unmount zroot/jails/containers/pgdb01/data17/base";
exec.poststop += "/sbin/zfs unmount zroot/jails/containers/pgdb01/data17";

# exec.poststop += "/sbin/zfs unjail pgdb01 zroot/jails/containers/pgdb01/data17/pg_wal";
# exec.poststop += "/sbin/zfs unjail pgdb01 zroot/jails/containers/pgdb01/data17/base";
exec.poststop += "/sbin/zfs unjail pgdb01 zroot/jails/containers/pgdb01/data17";
}


My problem is when I'm inside the jail, I cannot access the mount point /var/db/postgres/ as shown in the picture bellow :


# uname -a
FreeBSD pgdb01.itlinker.lan 14.2-RELEASE FreeBSD 14.2-RELEASE releng/14.2-n269506-c8918d6c7412 GENERIC amd64
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
zroot 33.8G 412G 96K /zroot
zroot/jails 714M 412G 104K /usr/local/jails
zroot/jails/containers 517M 412G 517M /usr/local/jails/containers
zroot/jails/containers/pgdb01 192K 412G 96K /usr/local/jails/containers/pgdb01
zroot/jails/containers/pgdb01/data17 96K 412G 96K /var/db/postgres/data17
# ls -al /
total 135
drwxr-xr-x 18 root wheel 22 Dec 17 10:12 .
drwxr-xr-x 18 root wheel 22 Dec 17 10:12 ..
-rw-r--r-- 2 root wheel 1011 Nov 29 10:58 .cshrc
-rw-r--r-- 2 root wheel 495 Nov 29 10:58 .profile
drwxr-xr-x 2 root wheel 49 Nov 29 10:58 bin
drwxr-xr-x 14 root wheel 68 Nov 29 11:42 boot
-r--r--r-- 1 root wheel 6109 Nov 29 11:42 COPYRIGHT
dr-xr-xr-x 14 root wheel 512 Dec 17 13:50 dev
drwxr-xr-x 30 root wheel 107 Dec 17 10:57 etc
drwxr-xr-x 4 root wheel 78 Nov 29 11:20 lib
drwxr-xr-x 3 root wheel 5 Nov 29 10:57 libexec
drwxr-xr-x 2 root wheel 2 Nov 29 10:54 media
drwxr-xr-x 2 root wheel 2 Nov 29 10:54 mnt
drwxr-xr-x 2 root wheel 2 Nov 29 10:54 net
dr-xr-xr-x 2 root wheel 2 Nov 29 10:54 proc
drwxr-xr-x 2 root wheel 150 Nov 29 11:15 rescue
drwxr-x--- 2 root wheel 8 Dec 17 14:06 root
drwxr-xr-x 2 root wheel 150 Nov 29 11:23 sbin
lrwxr-xr-x 1 root wheel 11 Nov 29 10:54 sys -> usr/src/sys
drwxrwxrwt 6 root wheel 6 Dec 17 14:01 tmp
drwxr-xr-x 14 root wheel 14 Nov 29 10:54 usr
drwxr-xr-x 24 root wheel 24 Dec 17 13:50 var
# ls -al /var/db/postgres/
ls: /var/db/postgres/: No such file or directory


 

[rafael_grether]

Note that having a hierarchy in your ZFS dataset does not mean that you will have that same directory hierarchy.
So zroot/jails/containers/pgdb01/data17 is pointed to the HOST Machine in /var/db/postgres/data17 , not inside jail. So the jail won't have access.

If your postgresql database is located on the host machine (inside /var/db/postgres/data17), you can use nullfs(5) to share this directory to jail.

As you already mounting fstab.pgdb01 on pgdb01.conf, you can create /etc/fstab.pgdb01 file with content:

/var/db/postgres/data17/ /usr/local/jails/containers/pgdb01/var/db/postgres/data17/ nullfs rw 0 0


And make sure that directory /usr/local/jails/containers/pgdb01/var/db/postgres/data17/ exists.

 

[rai.mohammed]

First of all thank you for your replay M. rafael_grether .
As a reminder and to be concise, the installation is composed of 4 datasets. The first is the dataset pgdb01 is used to install FreeBSD 14.2 in jail,
the 3 separates datasets (data17, base, pg_wal) are used as a destination to install and initialize PostgreSQL DB inside the jail pgdb01.
And those are 4 different datasets with their mounted points in the host, as you can see below :

zroot/jails/containers/pgdb01 --------------------------------------#. mounted at /usr/local/jails/containers/pgdb01
zroot/jails/containers/pgdb01/data17 -------------------------------#. mounted at /var/db/postgres/data17 #(PG_DATA)
zroot/jails/containers/pgdb01/data17/base --------------------------#. mounted at /var/db/postgres/data17/base
zroot/jails/containers/pgdb01/data17/pg_wal ------------------------#. mounted at /var/db/postgres/data17/pg_wal


01 - This is the summery for the config of the jailed PostgreSQL, the file /etc/jail.conf.d/pgdb01.conf :

02 - The mount points on the host when the jail pgdb01 is not started, and where you can see the directory /var/db/postgres/data17 :

03 - Inside the jail pgdb01 - zfs list dataset and mount, where I can't access the directory /var/db/postgres/data17:

04 - From the Freebsd 14-2 host - Mount on the host and Jail pgdb01 is started, where you can see the mounted data17 dataset, the nullfs share and the directory /var/db/postgres/data17 :

At the end the problem persist and I cannot access the postgreSQL dataset.

 

[rafael_grether]

rai.mohammed,​

It seems your zroot/jails/containers/pgdb01/data17 is empty. Is that right?
Do you already have a database, or are you going to initialize one? I imagine it's the second case.

When using nullfs, you don't need to "zfs jail". Either you use one or the other.
I prefer use nullfs, since I don't consider it interesting to provide so many allow.mount.whatever permissions to use zfs jail.
So, remove "zfs jail" commands from your pgdb01.conf, since you're using nullfs.

The origin and dest on your nullfs command was the same. You need to set the full path, as I mentioned before.
So, mount dataset and sub datasets on your main host (if not mounted yet):

zfs mount zroot/jails/containers/pgdb01/data17
zfs mount zroot/jails/containers/pgdb01/data17/base
zfs mount zroot/jails/containers/pgdb01/data17/pg_wal


This dataset will be mounted on /var/db/postgres/data17 on your main host (Empty in you case, it seems, prepared to be initialized from the pgdb01 jail)

Create /var/db/postgres/data17/ directory inside the jail:
mkdir -p /usr/local/jails/containers/pgdb01/var/db/postgres/data17/

Start pgdb01 jail, and still on the host, mount nullfs (full path):
mount_nullfs -o rw /var/db/postgres/data17/ /usr/local/jails/containers/pgdb01/var/db/postgres/data17/
OR do it on exec.poststart OR use mount.fstab as I mentioned before.

Note that inside the jail, now you can initialize postgres database, and database will appear on /var/db/postgres/data17/ and subdirectories inside the jail. On main host you will see the databases also on /var/db/postgres/data17/ , making sure nullfs worked.

Also note that on pgdb01 jail, zfs list won't return you nothing, since you're using nullfs, not "zfs jail".
So your main host will manage these 3 datasets. And nullfs will share them to postgres jail to use it, create databases and store the data.

 

[rai.mohammed]

Thanks for your replay M. rafael_grether.

This post is a summery after resolving the issue and making some modification in my setup.
There are some miss understanding and confusion for my part in a certain topics, please let me know if someone have a solution.

The solution that works for me is a dataset with a nullfs mount point.
A dataset having the mount point directing to the jail's mount point, causing me a problem I will expose it later.

The installation is composed of 4 datasets. The first is the dataset pgdb01 is used to install FreeBSD 14.2 in jail,
the 3 separates datasets (data17, base, pg_wal) with nullfs mount (where I have a problem with nested nullfs mount,
that I will explain it later).
the 3 datasets are used as a destination to install and initialize PostgreSQL DB inside the jail pgdb01.
And those are 4 different datasets with their mounted points in the host, are as you can see below :

zroot/jails/containers/pgdb01 -----------------------------#. mounted at /usr/local/jails/containers/pgdb01
zroot/jails/containers/pgdb01/data17 ----------------------#. mounted at /usr/local/jails/containers/pgdb01/data17 #(PG_DATA)
zroot/jails/containers/pgdb01/data17/base -----------------#. mounted at /usr/local/jails/containers/pgdb01/data17/base
zroot/jails/containers/pgdb01/data17/pg_wal ---------------#. mounted at /usr/local/jails/containers/pgdb01/data17/pg_wal


My PostgreSQL /etc/jail.conf.d/pgdb01.conf :

rai.mohammed@desktopfb:~ $ cat /etc/jail.conf.d/pgdb01.conf
pgdb01 {
# STARTUP/LOGGING
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jails/jail_console_${name}.log";

# PERMISSIONS
allow.raw_sockets;
exec.clean;
mount.devfs;

allow.mount;
allow.mount.devfs;
allow.mount.zfs;
allow.mount.nullfs;
allow.mount.procfs;
enforce_statfs = 1;

# /etc/defaults/devfs.rules
devfs_ruleset = 5;

# To run PostgreSQL in a FreeBSD jail
sysvmsg=new ;
sysvsem=new ;
sysvshm=new ;

# HOSTNAME/PATH
$domain = "itlinker.lan";
host.hostname = "${name}.${domain}";
path = "/usr/local/jails/containers/${name}";

# Not used because the mount points are not auto unmounted where stopping or restaring a Jail
# mount.fstab="/etc/fstab.$name";

# VNET/VIMAGE
vnet;
vnet.interface = "${epair}b";

# NETWORKS/INTERFACES
$id = "232";
$ip = "192.168.20.${id}/24";
$gateway = "192.168.20.1";
$bridge = "bridge0";
$epair = "epair${id}";

# ADD TO bridge INTERFACE
exec.prestart = "/sbin/ifconfig ${epair} create up";
exec.prestart += "/sbin/ifconfig ${epair}a up descr jail:${name}";
exec.prestart += "/sbin/ifconfig ${bridge} addm ${epair}a up";
exec.start += "/sbin/ifconfig ${epair}b ${ip} up";
exec.start += "/sbin/route add default ${gateway}";
exec.poststop = "/sbin/ifconfig ${bridge} deletem ${epair}a";
exec.poststop += "/sbin/ifconfig ${epair}a destroy";

# ADD TO show ZFS dataset in jail
exec.created += "/sbin/zfs set jailed=on zroot/jails/containers/pgdb01/data17";
exec.poststart += "/sbin/zfs jail pgdb01 zroot/jails/containers/pgdb01/data17";

exec.created += "/sbin/zfs set jailed=on zroot/jails/containers/pgdb01/data17/mount_dataset_in_jail";
exec.poststart += "/sbin/zfs jail pgdb01 zroot/jails/containers/pgdb01/data17/mount_dataset_in_jail";

exec.created += "/sbin/zfs set jailed=on zroot/jails/containers/pgdb01/data17/base";
exec.poststart += "/sbin/zfs jail pgdb01 zroot/jails/containers/pgdb01/data17/base";

exec.created += "/sbin/zfs set jailed=on zroot/jails/containers/pgdb01/data17/pg_wal";
exec.poststart += "/sbin/zfs jail pgdb01 zroot/jails/containers/pgdb01/data17/pg_wal";

exec.poststop += "/sbin/zfs unjail pgdb01 zroot/jails/containers/pgdb01/data17/pg_wal";
exec.release += "/sbin/zfs set jailed=off zroot/jails/containers/pgdb01/data17/pg_wal";

exec.poststop += "/sbin/zfs unjail pgdb01 zroot/jails/containers/pgdb01/data17/base";
exec.release += "/sbin/zfs set jailed=off zroot/jails/containers/pgdb01/data17/base";

exec.poststop += "/sbin/zfs unjail pgdb01 zroot/jails/containers/pgdb01/data17/mount_dataset_in_jail";
exec.release += "/sbin/zfs set jailed=off zroot/jails/containers/pgdb01/data17/mount_dataset_in_jail";
exec.release += "/sbin/zfs unmount zroot/jails/containers/pgdb01/data17/data17/mount_dataset_in_jail";

exec.poststop += "/sbin/zfs unjail pgdb01 zroot/jails/containers/pgdb01/data17";
exec.release += "/sbin/zfs set jailed=off zroot/jails/containers/pgdb01/data17";

# Testing nullfs mount - working
mount += "/usr/local/jails/containers/pgdb01/data17 /var/db/postgres/data17 nullfs rw,local,noatime,nfsv4acls 0 0";

# With nested nullfs share, one Target and Mount Point must be specified, othewise you will get this error bellow :
# Starting jails: cannot start jail "pgdb01":
# mount_nullfs: /var/db/postgres/data17/base: Resource deadlock avoided

# mount +="${path}/data17/base /var/db/postgres/data17/base nullfs rw,local,noatime,nfsv4acls 0 0";
# mount +="${path}/data17/pg_wal /var/db/postgres/data17/pg_wal nullfs rw,local,noatime,nfsv4acls 0 0";
}


FreeBSD 14.2 Host - Datasets on the host and the jail pgdb01 started :

FreeBSD 14.2 Jail - Datasets on the jail pgdb01 and PostgreSQL started :

Problem of a dataset having the mount point directing to the jail's mount point :​

I have successfully created a dataset from the host and mounted inside the jail mount point :

zfs create zroot/jails/containers/pgdb01/data17/mount_dataset_in_jail
zfs set mountpoint=/usr/local/jails/containers/pgdb01/var/db/mount_dataset_in_jail zroot/jails/containers/pgdb01/data17/mount_dataset_in_jail
zfs set jailed=on zroot/jails/containers/pgdb01/data17/mount_dataset_in_jail

FreeBSD 14.2 Jail - Dataset mount_dataset_in_jail on the jail pgdb01 filled with 2 files for a total size of 120M,
but I noticed the used space listed by zfs list is only 96K, comparing that to the dataset data17 mounted
with nullfs on /var/db/postgres/data17 on the jail is using 392K of space.

The problem of 3 separates datasets (data17, base, pg_wal) with nested nullfs mount points :​

With nested nullfs share, one Target and Mount Point must be specified, otherwise you will get this error bellow :

Starting jails: cannot start jail "pgdb01":
mount_nullfs: /var/db/postgres/data17/base: Resource deadlock avoided


So because of this problem I didn't achieve the installation of PostgreSQL with a separation of 3 datasets (data17, base, pg_wal), and all the database reside on the dataset data17.