Solved VNET Jails (via Bastille) no longer connect after upgrade from FreeBSD 12 to 13.

I'm using Bastille to manage several jails, and things have been working great for a while now. But recently I decided to upgrade the server and its jails from 12.2-RELEASE to 13.0-RELEASE. The upgrade on the server went without a hitch, and upgrading the base jail through Bastille was also not an issue. But I had some trouble connecting to pkg.freebsd.org when trying to upgrade the jails themselves. Eventually, I was able to get all the jails using a loopback interface to connect and now they don't seem to have any issues.

But I have a couple of jails with a VNET connection and they are giving me problems. One uses a tun interface (transmission), but the other doesn't (radarr). And for troubleshooting, I created a new VNET jail through Bastille, but that also could not connect. For example, in a VNET jail, this is an error I get:
Code:
root@transmission:~ # host pkg.freebsd.org
;; connection timed out; no servers could be reached


I'll include the relevant config files.

/etc/rc.conf
Code:
clear_tmp_enable="YES"
sendmail_enable="NONE"
hostname="miniserver"
ifconfig_re0="inet 192.168.0.250 netmask 255.255.255.0"
defaultrouter="192.168.0.1"
sshd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"

#NTP Server
ntpd_enable="YES"
# ntp time synchronization at startup
ntpd_sync_on_start="YES"
bastille_enable="YES"
cloned_interfaces="lo1"
ifconfig_lo1_name="bastille0"
pf_enable="YES"
gateway_enable="YES"

#Allow OpenVPN in jails
cloned_interfaces+="tun"
#ifconfig_tun="10.8.0.0/24"
jail_sysvipc_allow="YES"

/etc/sysctl.conf
Code:
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.see_jail_proc=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0
vfs.zfs.min_auto_ashift=12
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_member=0

/etc/devfs.rules
Code:
[bastille_vnet=13]
add path 'bpf*' unhide

/usr/local/bastille/jails/radarr/jail.conf
Code:
radarr {
  devfs_ruleset = 13;
  enforce_statfs = 2;
  exec.clean;
  exec.consolelog = /var/log/bastille/radarr_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = radarr;
  mount.devfs;
  mount.fstab = /usr/local/bastille/jails/radarr/fstab;
  path = /usr/local/bastille/jails/radarr/root;
  securelevel = 2;

  allow.mlock = 1;
  allow.raw_sockets = 1;

  vnet;
  vnet.interface = e0b_bastille1;
  exec.prestart += "jib addm bastille1 re0";
  exec.poststop += "jib destroy bastille1";
}

/usr/local/bastille/jails/transmission/jail.conf
Code:
transmission {
  devfs_ruleset = 13;
  enforce_statfs = 2;
  exec.clean;
  exec.consolelog = /var/log/bastille/transmission_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = transmission;
  mount.devfs;
  mount.fstab = /usr/local/bastille/jails/transmission/fstab;
  path = /usr/local/bastille/jails/transmission/root;
  securelevel = 2;

  vnet;
  vnet.interface = e0b_bastille0;
  exec.prestart += "jib addm bastille0 re0";
  exec.poststop += "jib destroy bastille0";
}

/etc/pf.conf
Code:
ext_if="re0"
tun_if="tun0"
caddy_ip="10.0.0.100"
transmission_ip="192.168.0.251"
tun_net="10.8.0.0/24"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

transmission_jail_udp = "{ openvpn }" # is in /etc/services
nat on $ext_if from $tun_net to any -> $ext_if

## inbound to container example
## rdr pass inet proto tcp from any to any port {80, 443} -> 10.17.89.45
#rdr pass inet proto tcp from any to any port {80, 443} -> $caddy_ip
#rdr pass inet proto tcp from any to any port {9091, 51413} -> $transmission_ip

pass quick on $tun_if

block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA keep state

The two rdr rules are commented out as I'd added the rdr-anchor rule after the upgrade. Apparently, I hadn't upgraded Bastille in a while. As it happens, adding the rdr rule dynamically was no problem for caddy, but when I tried the same with transmission, I got this error:
Code:
❯ sudo bastille rdr transmission tcp 9091 9091
stdin:2: syntax error
pfctl: Syntax error in config file: pf rules not loaded

Another message I'm getting on stdout when updating the pf rules is this:
Code:
❯ sudo service pf status
/etc/rc.conf: cloned_interfaces+=tun: not found
/etc/rc.conf: cloned_interfaces+=tun: not found

I'm afraid I'm not sure why either of those occur.

I have a pretty good feeling that my problem is with my pf config though, because if I disable pf, I can then access the internet from within the VNET jails.
 
Regarding your last code example, take a second look at devfs.rules. I had a similar problem after upgrading to 13.0 and I noticed they added a separate section for the VNET jails. In 12.2 section 4 (jails) was the last one but in 13.0 there is section 5 for vnet jails. See if you are including the sections properly because this would explain your problem with the " tun: not found".
Regarding pf I can't help sorry. I am an ipfw user.
 
Regarding your last code example, take a second look at devfs.rules. I had a similar problem after upgrading to 13.0 and I noticed they added a separate section for the VNET jails. In 12.2 section 4 (jails) was the last one but in 13.0 there is section 5 for vnet jails. See if you are including the sections properly because this would explain your problem with the " tun: not found".
Regarding pf I can't help sorry. I am an ipfw user.
While there was nothing in the Jails chapter of the Handbook, nor anything in the devfs man pages about this change, I did see this at the end of the default rules file:
/etc/defaults/devfs.rules
Code:
...
# Devices usually found in a jail.
#
[devfsrules_jail=4]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path fuse unhide
add path zfs unhide

[devfsrules_jail_vnet=5]
add include $devfsrules_jail
add path pf unhide

So it would seem that this ruleset 5 adds something for pf. That sounds promising. So I incorporated that, and some of the other rules, into my own file:

/etc/devfs.rules
Code:
[bastille_vnet=13]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path fuse unhide
add path zfs unhide
add path pf unhide
add path 'bpf*' unhide

Unfortunately, this did not help. The VNET jails still cannot connect to the internet and the tun: not found error remains.
 
This is a failure in rc.conf. Change the one cloned_interfaces you have to:
Code:
cloned_interfaces="lo1 tun"
Thanks. That has fixed the issue of tun not being found. By the way, was that something changed recently? It was working in FreeBSD 12. Just curious.

Unfortunately, the VNET jails still can't access the internet. Also, when starting the one that uses tun for openvpn, errors related to the pf.conf file are reported:
Code:
❯ sudo bastille start transmission
[transmission]:
re0bridge
e0a_bastille0
e0b_bastille0
transmission: created
stdin:2: syntax error
pfctl: Syntax error in config file: pf rules not loaded
stdin:2: syntax error
pfctl: Syntax error in config file: pf rules not loaded

And it's a similar story if I try to add a dynamic rdr rule for the jail:
Code:
❯ sudo bastille rdr transmission tcp 9091 9091
stdin:2: syntax error
pfctl: Syntax error in config file: pf rules not loaded

I have to imagine that this is related somehow. But I don't know enough to say more. The pf.conf file (included in the original post) was working in FreeBSD 12 and I don't really know enough about pf to figure out what might be the problem. I tried checking the logs for pf, but they don't seem very helpful:

/var/log/pf.today
Code:
block return in all [ Evaluations: 8912 Packets: 4756 Bytes: 854807 States: 0 ]
block drop in on ! re0 inet from 192.168.0.0/24 to any [ Evaluations: 6567 Packets: 1773 Bytes: 301709 States: 0 ]
 
Code:
❯ sudo bastille rdr transmission tcp 9091 9091 
stdin:2: syntax error pfctl: Syntax error in config file: pf rules not loaded
Check the rdr.conf of that jail. I suspect the error is actually in there, not in your system's pf.conf (that seems to be fine at a quick glance).
 
Check the rdr.conf of that jail. I suspect the error is actually in there, not in your system's pf.conf (that seems to be fine at a quick glance).

Thank you for taking the time to help.

/usr/local/bastille/jails/transmission/rdr.conf
Code:
tcp 9091 9091
tcp 51413 51413

For comparison, this is the one that works fine with my jail for caddy:

/usr/local/bastille/jails/caddy/rdr.conf
Code:
tcp 80 80
tcp 443 443

They seem to have the same format, so that would suggest it's okay, right?
 
Yeah, those look fine, I have something similar in my rdr.conf files.
 
Dang. I was hoping that I was missing something obvious. Looks like I'll probably end up rolling back to the snapshot I took before the upgrade.
 
I don't think it has anything to do with the version of FreeBSD.
Perhaps, but things were working fine before the upgrade. So if I can go back to that known working state and start over, perhaps I can do the upgrade again without messing up whatever it is I apparently messed up.
 
What is pfctl -nf /etc/pf.conf telling you? It should just return to the prompt, no errors, nothing.

Remove this one from /etc/rc.conf though:
Code:
jail_sysvipc_allow="YES"
Those jail_ variables have been deprecated a long time ago. And this particular setting was changed in favor of a per jail setting (allow.sysvipc), instead of an all or nothing setting.
 
What is pfctl -nf /etc/pf.conf telling you? It should just return to the prompt, no errors, nothing.
That's what I get. Back to the prompt.

Remove this one from /etc/rc.conf though:
Code:
jail_sysvipc_allow="YES"
Those jail_ variables have been deprecated a long time ago. And this particular setting was changed in favor of a per jail setting (allow.sysvipc), instead of an all or nothing setting.
Thanks. I'm assuming that wouldn't really have an effect on the jails in question, though. I believe that was something I enabled for PostgreSQL in another jail.
 
I believe that was something I enabled for PostgreSQL in another jail.
Yeah, PostgreSQL seems to need it. Most applications don't though. So best to leave it off and only turn it on for a specific jail that needs it.
 
So there is some progress, but still problems.

I decided to recreate the VNET jail for Transmission and start from scratch with that one. And actually, things worked. I had to remember the net.link.bridge sysctl settings mentioned in the Bastille docs, but following that guide, I could connect to the internet from the jail.

But I still can't get OpenVPN working. I remember this being a pain the first time around, but past me didn't feel the need to document the problem and solution at the time.

Here is the relevant bit from the logs (some info redacted):
/var/log/messages
Code:
Oct 19 02:25:08 transmission openvpn[98436]: DEPRECATED OPTION: --cipher set to 'aes-128-cbc' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM). Future OpenVPN version will ignore --cipher for cipher negotiations. Add 'aes-128-cbc' to --data-ciphers or change --cipher 'aes-128-cbc' to --data-ciphers-fallback 'aes-128-cbc' to silence this warning.
Oct 19 02:25:08 transmission openvpn[98436]: OpenVPN 2.5.3 amd64-portbld-freebsd13.0 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Oct  5 2021
Oct 19 02:25:08 transmission openvpn[98436]: library versions: OpenSSL 1.1.1k-freebsd  24 Aug 2021, LZO 2.10
Oct 19 02:25:08 transmission openvpn[98583]: CRL: loaded 1 CRLs from file -----BEGIN X509 CRL-----XXXXX-----END X509 CRL-----
Oct 19 02:25:09 transmission openvpn[98583]: TCP/UDP: Preserving recently used remote address: [AF_INET]xxx.xxx.xxx.xxx:1198
Oct 19 02:25:09 transmission openvpn[98583]: UDP link local: (not bound)
Oct 19 02:25:09 transmission openvpn[98583]: UDP link remote: [AF_INET]xxx.xxx.xxx.xxx:1198
Oct 19 02:25:09 transmission openvpn[98583]: [XXXXX] Peer Connection Initiated with [AF_INET]xxx.xxx.xxx.xxx:1198
Oct 19 02:25:09 transmission openvpn[98583]: GDG6: problem writing to routing socket: No such process (errno=3)
Oct 19 02:25:09 transmission openvpn[98583]: TUN/TAP device /dev/tun0 opened
Oct 19 02:25:09 transmission openvpn[98583]: /sbin/ifconfig tun0 10.22.112.233 10.22.112.1 mtu 1500 netmask 255.255.255.0 up
Oct 19 02:25:09 transmission openvpn[98583]: FreeBSD ifconfig failed: external program exited with error status: 1
Oct 19 02:25:09 transmission openvpn[98583]: Exiting due to fatal error
Oct 19 02:25:10 transmission transmission-daemon[97517]: UDP Failed to set receive buffer: No buffer space available (/wrkdirs/usr/ports/net-p2p/transmission-daemon/work/transmission-3.00/libtransmission/tr-udp.c:68)
Oct 19 02:25:10 transmission transmission-daemon[97517]: UDP Failed to set receive buffer: requested 4194304, got 42080 (/wrkdirs/usr/ports/net-p2p/transmission-daemon/work/transmission-3.00/libtransmission/tr-udp.c:97)

So it looks like it's not having a problem getting access to the tun0 device, which was an issue in many of the other posts about OpenVPN in a jail. But there's the line
Code:
GDG6: problem writing to routing socket: No such process (errno=3)
, which made me think about allowing raw sockets in the jail. So I did that, but the same error repeats.

Since I started this jail over, here is the config file:
/usr/local/bastille/jails/transmission/jail.conf
Code:
transmission {
  devfs_ruleset = 13;
  enforce_statfs = 2;
  exec.clean;
  exec.consolelog = /var/log/bastille/transmission_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = transmission;
  mount.devfs;
  mount.fstab = /usr/local/bastille/jails/transmission/fstab;
  path = /usr/local/bastille/jails/transmission/root;
  securelevel = 2;

  allow.raw_sockets = 1;

  vnet;
  vnet.interface = e0b_bastille3;
  exec.prestart += "jib addm bastille3 re0";
  exec.poststop += "jib destroy bastille3";
}
 
But I still can't get OpenVPN working. I remember this being a pain the first time around, but past me didn't feel the need to document the problem and solution at the time.
It mostly comes back to haunt you. :)

Let me post what kind of configuration I use, this is tested with 13.0. I use raw jails without bastille.
I omitted the fstab because it is irrelevant how you mount your jail root.

My devfs.rules is the 13.0 stock one, plus I add only one line to ruleset 5 (vnet jails):
Bash:
add path 'tun*' unhide

jail.conf
Bash:
vpn_test_domain {
    host.hostname = "vpn.test.domain";
    path = "/containers/test.domain/vpn.test.domain/mnt";

    vnet;
    vnet.interface = "";

    allow.dying;
    allow.raw_sockets;
    #mount.devfs;
    devfs_ruleset="5";
    #mount.fdescfs;
    #

    sysvmsg = "new";
    sysvsem = "new";
    sysvshm = "new";

    securelevel = "2";
    enforce_statfs = "2";
    children.max = "0";

    #mount.fstab = "$path/../fstab";

    exec.system_user = "root";
    exec.jail_user = "root";

    exec.consolelog = "$path/../log/jail_${name}_console.log";

    # Executed on the host
    exec.prestart  += "$path/../exe/hooks/prestart.sh  $name \"$path\"";
    exec.created   += "$path/../exe/hooks/created.sh   $name \"$path\"";
    exec.poststart += "$path/../exe/hooks/poststart.sh $name \"$path\"";
    exec.prestop   += "$path/../exe/hooks/prestop.sh   $name \"$path\"";
    exec.poststop  += "$path/../exe/hooks/poststop.sh  $name \"$path\"";

    # executed in the jail
    exec.start += "/bin/sh /etc/rc";
    exec.stop  =  "/bin/sh /etc/rc.shutdown";

    exec.clean;
    exec.timeout = "60";
    stop.timeout = "30";

    # taken from iocage
    #exec.created = "/usr/bin/true";

    # Use this parameter if jail needs to be created without any processes
    #persist;
}

poststart.sh is actually empty. I discovered stuff needs to be initialized very early, so created.sh fits better.

created.sh:
Bash:
#!/bin/sh -x

jail=$1
mntDir=$2

ifconfig bridge99 >> /dev/null 2>&1
if [ $? -ne 0 ]; then
    bridge="$(ifconfig bridge create)" || { exit 1; }
    ifconfig $bridge name bridge99 up || { ifconfig $bridge destroy; exit 1; }
fi
ifconfig epairh9903 >> /dev/null 2>&1
if [ $? -ne 0 ]; then
    epairA="$(ifconfig epair create)" || { exit 1; }
    epairB="${epairA%?}b"
    ifconfig $epairA name epairh9903 up || { ifconfig $epairA destroy; exit 1; }
    ifconfig $epairB name epairj9903 up || { ifconfig $epairA destroy; exit 1; }
else
    ifconfig epairh9903 up || { exit 1; }
    ifconfig epairj9903 up || { exit 1; }
fi
ifconfig bridge99 addm epairh9903 up || true

# move to the jail and set address
ifconfig epairj9903 vnet $jail
jexec $jail ifconfig epairj9903 10.0.99.3/24 up

# routes
jexec $jail route add default 10.0.99.2

In your case you might need to add your external interface to the bridge. I route through another jail so I don't need that, so pay attention to the proper routing through your gateway.

poststop.sh:
Bash:
#!/bin/sh -x

jail=$1
mntDir=$2

[ -n "$mntDir" ] || { exit 3; }

echo "Unmounting all under $mntDir"

# unmount fdescfs
/sbin/umount "$mntDir/dev/fd"  || /sbin/umount -f "$mntDir/dev/fd"    || true

# unmount devfs
/sbin/umount "$mntDir/dev"    || /sbin/umount -f "$mntDir/dev"      || true

cat $mntDir/../fstab \
    | grep -v '^\s*#.*' \
    | sort -r -k 6 \
    | awk 'NF { print "/sbin/umount -t " $3 " " $2 " || /sbin/umount -f -t " $3 " " $2 " || true"; }' \
    | /bin/sh \
    #
prestart.sh:
Bash:
#!/bin/sh -x

name=$1
mntDir=$2

isMounted=$(zfs get -H -o value mounted "zroot/containers/test.domain/vpn.test.domain/mnt")
if [ "$isMounted" = "no" ]; then
        zfs mount "zroot/containers/test.domain/vpn.test.domain/mnt"
fi

/sbin/mount -a -F "$mntDir/../fstab"

/sbin/mount -t devfs -oruleset=4 . "$mntDir/dev"
/sbin/mount -t fdescfs . "$mntDir/dev/fd"
prestop.sh:
Bash:
#!/bin/sh -x

jail=$1
mntDir=$2

# bridged
ifconfig epairj9903 -vnet $jail || true
ifconfig epairj9903 destroy || true
ifconfig epairh9903 destroy || true

# routes

openvpn.conf:
Bash:
dev tun
proto tcp
server 10.0.100.0 255.255.255.0
port 1194
local 10.0.99.3
ifconfig-pool-persist ipp.txt

management localhost 7505

push "route 10.0.99.0 255.255.255.0 vpn_gateway 3"
push "route 10.0.100.0 255.255.255.0 vpn_gateway 3"
push "dhcp-option DNS 10.0.99.11"
push "dhcp-option DOMAIN test.domain"

# server security
tls-server
ca /usr/local/etc/pki/ssl/crt/ca.crt
cert /usr/local/etc/pki/ssl/crt/vpn_server.crt
key /usr/local/etc/pki/ssl/key/vpn_server.key
dh /usr/local/etc/pki/ssl/dh/vpn.dh

# Downgrade UID and GID to "nobody" after initialization for extra security.
; user nobody
; group nobody

comp-lzo

# logging
verb 3
status openvpn-status.log

keepalive 10 120

# https://www.unixtutorial.org/multiple-openvpn-clients-sharing-the-same-certificate
# Allow multiple connections with same cert.
duplicate-cn
 
Let me post what kind of configuration I use, this is tested with 13.0. I use raw jails without bastille.
Thanks a lot for sharing your config. I'm not sure if those options would help my case. I'm hoping to keep extra configuration to a minimum if possible, to keep troubleshooting simple (too late, perhaps).

But actually, I have resolved the above problem with the routing socket, though another has been revealed.

I realized that some of the config changes I'd already made could have required restarting some system services, so I rebooted the system. And that seems to have resolved the routing socket problem. Now OpenVPN is running in the jail, but it doesn't connect to the VPN server. I'm getting a resolve error.

/var/log/messages
Code:
Oct 24 17:15:34 transmission syslogd: kernel boot file is /boot/kernel/kernel
Oct 24 17:15:34 transmission openvpn[3327]: DEPRECATED OPTION: --cipher set to 'aes-128-cbc' but missing in --data-ciphers (AES-256-GCM:AES-128-GCM). Future OpenVPN version will ignore --cipher for cipher negotiations. Add 'aes-128-cbc' to --data-ciphers or change --cipher 'aes-128-cbc' to --data-ciphers-fallback 'aes-128-cbc' to silence this warning.
Oct 24 17:15:34 transmission openvpn[3327]: OpenVPN 2.5.3 amd64-portbld-freebsd13.0 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Oct  5 2021
Oct 24 17:15:34 transmission openvpn[3327]: library versions: OpenSSL 1.1.1k-freebsd  24 Aug 2021, LZO 2.10
Oct 24 17:15:34 transmission openvpn[3605]: CRL: loaded 1 CRLs from file -----BEGIN X509 CRL----- XXXX -----END X509 CRL-----
Oct 24 17:15:36 transmission transmission-daemon[2463]: UDP Failed to set receive buffer: No buffer space available (/wrkdirs/usr/ports/net-p2p/transmission-daemon/work/transmission-3.00/libtransmission/tr-udp.c:68)
Oct 24 17:15:36 transmission transmission-daemon[2463]: UDP Failed to set receive buffer: requested 4194304, got 42080 (/wrkdirs/usr/ports/net-p2p/transmission-daemon/work/transmission-3.00/libtransmission/tr-udp.c:97)
Oct 24 17:16:34 transmission openvpn[3605]: RESOLVE: Cannot resolve host address: japan.privacy.network:1198 (Name does not resolve)
Oct 24 17:17:35 transmission syslogd: last message repeated 1 times
Oct 24 17:17:35 transmission openvpn[3605]: Could not determine IPv4/IPv6 protocol
Oct 24 17:17:35 transmission openvpn[3605]: SIGUSR1[soft,init_instance] received, process restarting
Oct 24 17:18:40 transmission openvpn[3605]: RESOLVE: Cannot resolve host address: japan.privacy.network:1198 (Name does not resolve)
Oct 24 17:19:40 transmission syslogd: last message repeated 1 times
Oct 24 17:19:40 transmission openvpn[3605]: Could not determine IPv4/IPv6 protocol
Oct 24 17:19:40 transmission openvpn[3605]: SIGUSR1[soft,init_instance] received, process restarting
Oct 24 17:20:45 transmission openvpn[3605]: RESOLVE: Cannot resolve host address: japan.privacy.network:1198 (Name does not resolve)

I'm using the standard OpenVPN config file provided by the VPN service and resolving the VPN's domain name from the host is no problem.

I'm also getting that pfctl error again when I start the jail:
Code:
stdin:2: syntax error
pfctl: Syntax error in config file: pf rules not loaded

For reference, here is my current pf configuration on the host:

/etc/pf.conf
Code:
ext_if="re0"
tun_if="tun0"
transmission_jail_ip="192.168.0.251"
tun_net="10.8.0.0/24"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

transmission_jail_udp = "{ openvpn }" # is in /etc/services
nat on $ext_if from $tun_net to any -> $ext_if

pass quick on $tun_if

block in all
pass out quick keep state
antispoof for $ext_if inet
pass in inet proto tcp from any to any port ssh flags S/SA keep state

The rdr.conf for the jail:

/usr/local/bastille/jails/transmission/rdr.conf
Code:
tcp 9091 9091

The jail configuration:

/usr/local/bastille/jails/transmission/jail.conf
Code:
transmission {
  devfs_ruleset = 13;
  enforce_statfs = 2;
  exec.clean;
  exec.consolelog = /var/log/bastille/transmission_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = transmission;
  mount.devfs;
  mount.fstab = /usr/local/bastille/jails/transmission/fstab;
  path = /usr/local/bastille/jails/transmission/root;
  securelevel = 2;

  allow.raw_sockets = 1;

  vnet;
  vnet.interface = e0b_bastille3;
  exec.prestart += "jib addm bastille3 re0";
  exec.poststop += "jib destroy bastille3";
}

And the rc.conf on the host:

/etc/rc.conf
Code:
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
hostname="miniserver"
ifconfig_re0="inet 192.168.0.250 netmask 255.255.255.0"
defaultrouter="192.168.0.1"
sshd_enable="YES"
ntpdate_enable="YES"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
bastille_enable="YES"
bastille_list="caddy www wikijs"
cloned_interfaces="lo1 tun0"
ifconfig_lo1_name="bastille0"
pf_enable="YES"
gateway_enable="YES"

# VPN
ifconfig_tun0="10.8.0.0/24"

Additionally, when I try to stop the OpenVPN service from within the jail to test connections without it, it hangs. And trying to stop the jail with Bastille also hangs, though I can stop it with the system jail -r command. But then I can't restart the jail from within Basitlle. This is fixed with a reboot of the host.
 
I'm using the standard OpenVPN config file provided by the VPN service and resolving the VPN's domain name from the host is no problem.
The jail has its own independent DNS resolver configuration. Make sure the domain name can be resolved within the VPN jail. Compare /etc/resolv.conf between the host and the jail. If you use hosts entries, make sure they are in the jail too. And the DNS server must be reachable from the jail.

Regarding the pf config error I can't help. Maybe try commenting the lines one by one until you find the wrong one.
 
After much banging of my head against the keyboard and late night hours googling away, I've figured it out.

First, the issue of not getting an outside connection had to do with these sysctl values:
Code:
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_member=0

Apparently, something changed in FreeBSD 13 and they were no longer getting set from sysctl on boot. But adding if_bridge_load="YES" to /boot/loader.conf fixed that. This is why it was intermittent. I would set them manually and things would work, but then on a reboot, they would reset and fail to take my sysctl.conf settings because no bridge interface had been loaded yet.

But I still couldn't access TUN properly from within the jails. It then occurred to me that since it was working with my old rc.conf settings, then that cloned_interfaces problem wasn't actually a problem for me. And fixing it, which then cloned the tun0 interface at boot, was the issue. I should be letting the jail clone the interface itself. So I removed tun0 from the list of cloned interfaces. And OpenVPN now works in the jail. Oy. There are still some other issues with other jails that got a bit borked by the update, but those (I hope) will be less torturous to troubleshoot.

Thanks all for the help!
 
Back
Top