How do I route all Jail traffic through OpenVPN on FIB1

Rob J

New Member


Messages: 1

Hello FreeBSD users, this is my fist post, I'm still learning so please be gentle :)

I am trying to setup a jail (using ioCage) so that all of its traffic goes through our VPN. I have already setup OpenVPN which connects to our VPN service without issue.

However the I seem unable 😕 to successfully connect the jail to the VPN on FIB 1.


So far I have the following: (FreeBSD 12.2):

/boot/loader.conf
Code:
net.fibs=2

/etc/rc.conf
Code:
gateway_enable="YES"
pf_enable="YES"
defaultrouter="192.168.1.1"

/etc/rc.local
Code:
route add default 192.168.1.1 -fib 1
setfib -F 1 /usr/local/sbin/openvpn --config /root/myconfig.ovpn --daemon[/I]

This all seems to work OK, at least when I test it using:

# curl http://ipecho.net/plain

Which returns my public IP address 👍.

# setfib -F 1 curl http://ipecho.net/plain

This returns my VPNs public IP address 👍 .


Next up I have created jail using ioCage:

# iocage create -n myjail -r LATEST vnet=on dhcp=on bpf=on allow_raw_sockets=on boot=on -T

myjail successfully created!
No default gateway found for ipv6.
* Starting myjail
+ Started OK
+ Using devfs_ruleset: 1002 (iocage generated default)
+ Configuring VNET OK
+ Using IP options: vnet
+ Starting services OK
+ Executing poststart OK
+ DHCP Address: 192.168.1.202/24


Next I set tun=1 and fib=1:

# iocage set allow_tun=1 myjail
# iocage set exec_fib=1 myjail


Restarted the jail:

# iocage restart myjail


Then accessed the jail using:

# iocage console myjail


The jail seems ok and is accessible on the network, however despite setting the jail fib=1 the jail does not seem utilize our VPN.

So from within our jail:

# curl http://ipecho.net/plain

Returns our normal public IP address 😕.

Again from within the jail:
Code:
# netstat -nr

Routing tables (fib: 1)

Internet:
Destination         Gateway         Flags         Netif         Expire
default                192.168.1.1     UGS         epair0b
127.0.0.1            link#1              UH            lo0
192.168.1.0/24   link#2              U               epair0b

Internet6:
Destination         Gateway         Flags         Netif         Expire
::/96                    ::1                   UGRS         lo0
::1                       link#1              UH             lo0
::ffff:0.0.0.0/96    ::1                   UGRS         lo0
fe80::/10             ::1                   UGRS         lo0
fe80::%lo0/64     link#1             U                 lo0
fe80::%epair0b/64     link#2      U                 epair0b
ff02::/16             ::1                   UGRS          lo0


Any help and advice to get this working would be greatly appreciated...

Thanks in advance.
 
Last edited by a moderator:

Nemo1024

Member

Reaction score: 5
Messages: 25

This might be because of missing pf configuration. I have just spent several days trying to get a service running in a jail and have its in- and outbound traffic routed through VPN using FreeBSD 13-STABLE, OpenVPN and regular jail command (/etc/jail.conf). Finally I have made it work!

First, a list of all the resources that I used during my research. I first tried going the cloned interface way, but ended up with setting up a new FIB for the jail.

(This ended up being the main line of investigation)

The intended solution is pretty simple: a single service should run within the jail and listen to a range of ports. The jail should bypass the local network and talk to the outside world through VPN.

My setup is similar to OP's:

/boot/loader.conf
Code:
net.fibs=2

/etc/sysctl.conf
Code:
# Without this OpenVPN fails to initialise, even though the default route is set on FIB 1
net.add_addr_allfibs=1
This would be the error during OpenVPN initialization if the above is not set to 1 on FreeBSD 13. Maybe someone has a clue as to why and how to mitigate this:
Code:
2021-12-08 15:42:56 TUN/TAP device /dev/tun0 opened
2021-12-08 15:42:56 /sbin/ifconfig tun0 10.27.152.126 10.27.152.1 mtu 1500 netmask 255.255.255.0 up
2021-12-08 15:42:56 /sbin/route add -net 10.27.152.0 10.27.152.1 255.255.255.0
route: writing to routing socket: Network is unreachable
add net 10.27.152.0: gateway 10.27.152.1 fib 1: Network is unreachable
2021-12-08 15:42:56 ERROR: FreeBSD route add command failed: external program exited with error status: 1[/COLOR]
/etc/rc.conf
Code:
jail_enable="YES"
# I launch the jail later from a script after the VPN is up - thus empty list
jail_list=" "
gateway_enable="YES"
pf_enable="YES"
defaultrouter="192.168.1.1"
/etc/pf.conf
Code:
 set skip on { lo0 }
scrub in all
jail_ip = "192.168.1.20"
jail_service_range = "15000:15100"
nat pass on tun0 from $jail_ip to any -> ( tun0 )
rdr pass on tun0 proto tcp from any to any port $jail_service_range -> $jail_ip port $jail_service_range
** Run: service pf restart to effectuate the rules above **

/etc/jail.conf
Code:
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
mount.fstab = "/etc/fstab.$name";

servicejail {
     host.hostname = "servicejail";
     path = "/usr/local/jails/servicejail";
     interface = "em0";
     ip4.addr = 192.168.1.20;
     exec.fib=1;
}
/etc/rc.d/local
Code:
route add 192.168.1.0/24 -iface em0 -fib 1
route add default 192.168.1.1 -fib 1
setfib 1 /usr/local/sbin/openvpn --config /usr/local/etc/openvpn/config.ovpn --daemon
** Reboot **

** Run: jail -c servicejail once all the IPs are assigned and the DDNS services are updated.

Now, if I SSH into the jail and run nc -l 15000 to listen to the port in range, then do an external connectivity check to the VPN IP or DDNS name, I would see output from the nc(1) command. Running the same nc command on the host under both fib 0 and fib 1 does not give any result - the traffic is routed to the jail as expected.
 
Top