Setting up a router in a jail to use openvpn client

I know, this appears to be answered all over the place. But none of the answers are working for me so I must have misunderstood something somewhere along the way.

Background: I'm trying to set up a FreeBSD jail that acts as a router. Inside that jail, I'd like to run OpenVPN. I would like all traffic I send to the FreeBSD jail to route through the OpenVPN connection and not through any other interface.

I'm running FreeBSD 11.2-RELEASE-p2. I've installed iocage to assist with managing the jail environments. After creating a new jail, I can console into the jail and touch the internet. The settings for the jail are:


Code:
{
    "CONFIG_VERSION": "9",
    "allow_chflags": "0",
    "allow_mount": "0",
    "allow_mount_devfs": "0",
    "allow_mount_nullfs": "0",
    "allow_mount_procfs": "0",
    "allow_mount_tmpfs": "0",
    "allow_mount_zfs": "0",
    "allow_quotas": "0",
    "allow_raw_sockets": "1",
    "allow_set_hostname": "1",
    "allow_socket_af": "1",
    "allow_sysvipc": "1",
    "available": "readonly",
    "basejail": "no",
    "boot": "off",
    "bpf": "no",
    "children_max": "0",
    "cloned_release": "11.2-RELEASE",
    "comment": "none",
    "compression": "lz4",
    "compressratio": "readonly",
    "coredumpsize": "off",
    "count": "1",
    "cpuset": "off",
    "cputime": "off",
    "datasize": "off",
    "dedup": "off",
    "defaultrouter": "192.168.11.1",
    "defaultrouter6": "none",
    "depends": "none",
    "devfs_ruleset": "4",
    "dhcp": "off",
    "enforce_statfs": "2",
    "exec_clean": "1",
    "exec_fib": "0",
    "exec_jail_user": "root",
    "exec_poststart": "/usr/bin/true",
    "exec_poststop": "/usr/bin/true",
    "exec_prestart": "/usr/bin/true",
    "exec_prestop": "/usr/bin/true",
    "exec_start": "/bin/sh /etc/rc",
    "exec_stop": "/bin/sh /etc/rc.shutdown",
    "exec_system_jail_user": "0",
    "exec_system_user": "root",
    "exec_timeout": "60",
    "host_domainname": "none",
    "host_hostname": "vpn_router",
    "host_hostuuid": "vpn_router",
    "host_time": "yes",
    "hostid": "00000000-0000-0000-0000-002590f006d6",
    "interfaces": "none",
    "ip4": "inherit",
    "ip4_addr": "igb0|192.168.11.5/24",
    "ip4_saddrsel": "1",
    "ip6": "new",
    "ip6_addr": "none",
    "ip6_saddrsel": "1",
    "jail_zfs": "off",
    "jail_zfs_dataset": "iocage/jails/vpn_router/data",
    "jail_zfs_mountpoint": "none",
    "last_started": "2018-09-08 13:32:34",
    "login_flags": "-f root",
    "mac_prefix": "02ff60",
    "maxproc": "off",
    "memorylocked": "off",
    "memoryuse": "off",
    "mount_devfs": "1",
    "mount_fdescfs": "1",
    "mount_linprocfs": "0",
    "mount_procfs": "0",
    "mountpoint": "readonly",
    "msgqqueued": "off",
    "msgqsize": "off",
    "nmsgq": "off",
    "notes": "none",
    "nsemop": "off",
    "nshm": "off",
    "nthr": "off",
    "openfiles": "off",
    "origin": "readonly",
    "owner": "root",
    "pcpu": "off",
    "priority": "99",
    "pseudoterminals": "off",
    "quota": "none",
    "release": "11.2-RELEASE-p2",
    "reservation": "none",
    "resolver": "/etc/resolv.conf",
    "rlimits": "off",
    "securelevel": "2",
    "shmsize": "off",
    "stacksize": "off",
    "stop_timeout": "30",
    "swapuse": "off",
    "sync_state": "none",
    "sync_target": "none",
    "sync_tgt_zpool": "none",
    "sysvmsg": "new",
    "sysvsem": "new",
    "sysvshm": "new",
    "template": "no",
    "type": "jail",
    "used": "readonly",
    "vmemoryuse": "off",
    "vnet": "off",
    "vnet0_mac": "none",
    "vnet1_mac": "none",
    "vnet2_mac": "none",
    "vnet3_mac": "none",
    "wallclock": "off"
}

/etc/defaults/devfs.rules

Code:
[devfsrules_jail=4]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'tun*' unhide
add path zfs unhide

Code:
iocage start vpn_router                                                                                                                     
* Starting vpn_router
  + Started OK
  + Starting services OK

Log in to the console.
Code:
iocage console vpn_router
Last login: Sat Sep  8 09:04:09 on pts/2
FreeBSD 11.2-RELEASE-p2 (GENERIC) #0:

Check to see it works.
Code:
fetch -qo - http://ifconfig.co
***.***.***.200

What networks do I have inside the jail?

Code:
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:25:90:f0:06:d6
        hwaddr 00:25:90:f0:06:d6
        inet 192.168.11.5 netmask 0xffffff00 broadcast 192.168.11.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active

So far so good. However, if I set my default route (on another host: 192.168.11.3) to 192.168.11.5, no traffic routes from that host. Trying to ping google from 192.168.11.3 with 192.168.11.5 as the default route. tcpdump'ing from the host shows:

Code:
tcpdump -i igb0 host 192.168.11.5
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:23:12.083933 ARP, Request who-has 192.168.11.5 tell 192.168.11.3, length 46
09:23:12.083965 ARP, Reply 192.168.11.5 is-at 00:25:90:f0:06:d6 (oui Unknown), length 28
09:23:12.112283 ARP, Request who-has 192.168.11.5 tell 192.168.11.3, length 46
09:23:12.112304 ARP, Reply 192.168.11.5 is-at 00:25:90:f0:06:d6 (oui Unknown), length 28
09:23:12.114072 IP 192.168.11.3.61798 > 192.168.11.5.domain: 35187+ PTR? db._dns-sd._udp.0.11.168.192.in-addr.arpa. (59)
09:23:12.114078 IP 192.168.11.3.55078 > 192.168.11.5.domain: 64239+ PTR? b._dns-sd._udp.0.11.168.192.in-addr.arpa. (58)
09:23:12.114098 IP 192.168.11.5 > 192.168.11.3: ICMP 192.168.11.5 udp port domain unreachable, length 36
09:23:12.114100 IP 192.168.11.5 > 192.168.11.3: ICMP 192.168.11.5 udp port domain unreachable, length 36

The host has:
Code:
gateway_enable="YES"
cloned_interfaces="tun"

I can't set that on the jail.

Code:
root@vpn_router:~ # sysctl net.inet.ip.forwarding=1
net.inet.ip.forwarding: 1
sysctl: net.inet.ip.forwarding=1: Operation not permitted

At this point, what am I doing wrong? Most of the places that detail how to fix this type of issue show something like cloning the loopback and then utilizing pf to perform NAT. I could probably copy that, but I am trying to understand what I'm doing wrong here.

Of course, the VPN doesn't work either. I can start the vpn but it doesn't stay connected. It fails as it cannot set the IP on the tun interface. I will ask more about this later, but I wanted to understand my issue with normal routing first. But for those that are easily able to understand and fix this I'll explain.

On the host I create a tun interface.
Code:
ifconfig tun0 create

Both the host and the jail see tun0. However, when starting OpenVPN the following error happens.
Code:
Sat Sep  8 09:39:02 2018 us=495288 TUN/TAP device /dev/tun0 opened
Sat Sep  8 09:39:02 2018 us=495308 do_ifconfig, tt->did_ifconfig_ipv6_setup=1
Sat Sep  8 09:39:02 2018 us=495360 /sbin/ifconfig tun0 10.***.***.*** 10.***.***.1 mtu 1500 netmask 255.255.255.0 up
ifconfig: ioctl SIOCSIFMTU (set mtu): Operation not permitted
Sat Sep  8 09:39:02 2018 us=498640 FreeBSD ifconfig failed: external program exited with error status: 1
Sat Sep  8 09:39:02 2018 us=498694 Exiting due to fatal error
Thanks in advance for any help you can offer.
 
No, don't. There's no way you're going to manipulate the routing table of the host system from the jail, it just won't work and for obvious reasons.
 
No, don't. There's no way you're going to manipulate the routing table of the host system from the jail, it just won't work and for obvious reasons.

OK. Then how should this be done, is there another path I should be investigating? Should I be using setfib on the host to create a separate routing table for the jail? The goal I'm after seems to be a fairly common request by others.
 
You still run into the limitation of a jail not being able to manipulate the separate routing table at run time. Why can't you just run the VPN service on the host like everyone else does?
 
Separation of services.

If the openvpn is set on the host, and a separate routing table is established then the jail would be able to use that routing table I would think, right? The goal was not to manipulate the routing table from within the jail. I wanted a jail to act as a router. The routing table does not have to be dynamic.
 
What exactly you want to route? Traffic originating from where and going to where? A jail is just collection of userspace processes under the same restricting chroot(8) type environment and it relies on the kernel for all networking related tasks including routing.
 
As kpa mentioned, you cannot manipulate the host routing table from a jail that is already running. I know what you are trying to achieve. You will only be able to set/manipulate the routing table when your jail is starting. In other words, you need set the routes in your jail config file. So, you will have to do as shown below (Note that I am using ezjail):

Code:
export jail_JAILNAME_parameters="allow.raw_sockets=1"
export jail_JAILNAME_allow_raw_sockets="YES"
export jail_JAILNAME_devfs_ruleset="5"
export jail_JAILNAME_exec_prestart0="/sbin/ifconfig tun0 inet 10.8.0.1/32 10.8.0.2"
export jail_JAILNAME_exec_prestart2="/sbin/route add -net 10.8.0.0/24 10.8.0.2"

The first two export commands show reduncancy. You choose what works for you. Sometimes, these instructions vary from FBSD versions or so.

Anyway, you will then use "tun0" as your vpn interface. You need enter it in your openvpn config file and perhaps your pf rules. Here is another related guide.
 
Separation of services.
Other than what kpa already said I also think you may be putting too much value into this. A jail is definitely a good way to keep processes separated from the host, but it's not the holy grail. And sometimes it's simply more trouble that it's worth.

I see what you're trying to do: not necessarily routing but more so shielding the OpenVPN process in a separate environment. However, in this case that's somewhat pointless because you're not actually shielding anything. That's because jails use the same kernel as the host and it is the kernel which is responsible for handling all the network traffic.

The problem is basically that you can't compare a service such as OpenVPN with any regular networking service such as Apache, or Sendmail, etc. Those last two basically open a port and then wait for incoming data to process, whereas OpenVPN does a whole lot more. This is somewhat comparable to trying to run NFS inside a jail, that's also pretty much impossible due to the low level access that is required. Even running Samba inside a jail can be quite problematic.

A jail isn't the best approach for everything.
 
Even running Samba inside a jail can be quite problematic.

A jail isn't the best approach for everything.
You are very right! I have realized a jail is not the best approach on many occasion after losing several productive hours to troubleshooting simple tasks with no fix. But then using the host showed to be the best approach in a breeze.
Samba, OpenVPN and some other packages have been successfully running in a jail though. squirtle should get OpenVPN running in a jail with the above suggestions.
 
I run samba in a jail without any problems, you just need to pay a bit attention to make sure the services bind only to the jail IP address and you disable things like netbios that rely on broadcasts.
 
Would this be a call for using fibs?
I wondered about this too. I've not used it before, so I'm not familiar with exactly how it works. I started reading up on it a bit to see if I could leverage it. But I'm also reviewing what Lamia wrote above.
 
Back
Top