Wireguard in iocage jail: Failed to create TUN device: open /dev/tun: no such file or directory

Hi,

I'm having issues with running wireguard in an iocage jail on FreeBSD 13.0-RELEASE-p10.
The weird thing is, I also had issues with this on my TrueNAS Core system (12.2-RELEASE-p15), but after enabling the allow_tun option for the jail it now runs just fine there.

Here's some config data and the console output I get when I (try to) start wireguard from both systems:

FreeBSD System (currently not working)
Jail's config.json:
Code:
{
    "allow_raw_sockets": 1,
    "allow_tun": 1,
    "basejail": 1,
    "boot": 1,
    "devfs_ruleset": "4",
    "host_hostname": "webserver",
    "host_hostuuid": "webserver",
    "ip4_addr": "lo1|10.0.0.2/24",
    "jail_zfs_dataset": "iocage/jails/webserver/data",
    "last_started": "2022-03-31 10:59:09",
    "release": "13.0-RELEASE-p10"
 }

Console output:
Code:
root@vpn:~ # ls /dev/
crypto    fd    null    ptmx    pts    random    stderr    stdin    stdout    tun0    urandom    zero    zf

root@vpn:~ # service wireguard start
[#] ifconfig wg create name wg0
[!] Missing WireGuard kernel support (ifconfig: SIOCIFCREATE2: Invalid argument). Falling back to slow userspace implementation.
[#] wireguard-go wg0
┌──────────────────────────────────────────────────────┐
│                                                      │
│   Running wireguard-go is not required because this  │
│   kernel has first class support for WireGuard. For  │
│   information on installing the kernel module,       │
│   please visit:                                      │
│         https://www.wireguard.com/install/           │
│                                                      │
└──────────────────────────────────────────────────────┘
[#] wg setconf wg0 /dev/stdin
Warning: AllowedIP has nonzero host part: 10.0.10.1/24
[#] ifconfig wg0 inet 10.0.10.2/32 alias
[#] ifconfig wg0 mtu 8920
[#] ifconfig wg0 up
[#] route -q -n add -inet 10.0.10.0/24 -interface wg0
[+] Backgrounding route monitor
[#] wg set wg0 private-key /usr/local/etc/wireguard/privatekey

TrueNAS Core System (currently working)
Jail's config.json:
Code:
{
    "allow_tun": 1,
    "basejail": 1,
    "boot": 1,
    "host_hostname": "vpn",
    "host_hostuuid": "vpn",
    "ip4_addr": "vnet0|10.0.0.17/24",
    "jail_zfs_dataset": "iocage/jails/vpn/data",
    "last_started": "2022-03-27 17:13:49",
    "mac_prefix": "0202c9",
    "release": "12.2-RELEASE-p15",
    "vnet": 1,
    "vnet0_mac": "0202c9363831 0202c9363832"
}

Console output:
Code:
root@test:~ # ls /dev/
crypto    fd    null    pts    random    stderr    stdin    stdout    tun0    urandom    zero    zfs

root@test:~ # service wireguard start
[#] ifconfig wg create name wg0
[!] Missing WireGuard kernel support (ifconfig: SIOCIFCREATE2: Operation not permitted). Falling back to slow userspace implementation.
[#] wireguard-go wg0
┌──────────────────────────────────────────────────────┐
│                                                      │
│   Running wireguard-go is not required because this  │
│   kernel has first class support for WireGuard. For  │
│   information on installing the kernel module,       │
│   please visit:                                      │
│         https://www.wireguard.com/install/           │
│                                                      │
└──────────────────────────────────────────────────────┘
ERROR: (wg0) 2022/03/31 13:16:49 Failed to create TUN device: open /dev/tun: no such file or directory
[#] ifconfig wg0 destroy
ifconfig: interface wg0 does not exist

Does anyone here know what might be the issue and how to fix this?
 
I am not familiar with iocage or TrueNAS, but I see two things that look wrong with your broke config:
  1. It uses a fixed IP. According to this post, /dev/tun requires vnet.
  2. (maybe) devfs_ruleset is set to 4, but if you look in /etc/defaults/devfs.rules you'll see the vnet config is ruleset 5.
I spent a few hours last night banging my head against the wall trying the same thing you are, and I think it just doesn't work. Fortunately vnet is a solution that makes sense if you understand a bit of networking (and if you don't, you'll learn :)

The underlying principle is that your vnet jail has its own networking stack, so you need to configure that and ultimately set up a route to the internet. There are several ways of configuring that (again it's "just" networking, but you effectively have virtual devices. Here's the config that I came up with last night that I'm happy with:
  • bridge interface on the host, assigned a private IP and treated as the network gateway
  • epair interfaces, with host (a) side configured with no IP and added as members to the bridge, and jail (b) side configured with an IP in the same subnet as the gateway / bridge
  • jails configured with the bridge IP as default gateway
  • pf on the host to nat the jail network to external
Note: You can configure some aspects of those within each jail's /etc/rc.conf or you can configure them when setting up the jail. I opted for the latter because each jail is a specific service that fits into a bigger system, rather than being something individual that should configure itself.

relevant host /etc/rc.conf:

Code:
cloned_interfaces="bridge0"
ifconfig_bridge0="192.168.42.1/24"
gateway_enable="YES"
jail_enable="YES"
pf_enable="YES"
host /etc/jail.conf:
Code:
host.hostname = "gcp-$name";
vnet;
mount.devfs;
exec.clean;
exec.start = "sh /etc/rc";
exec.stop = "sh /etc/rc.shutdown";
path = "/jail/$name";
devfs_ruleset = 5;
vnet.interface = "epair${epair}b";
exec.prestart = "ifconfig epair${epair} create up";
exec.prestart += "ifconfig bridge0 addm epair${epair}a";
exec.start += "ifconfig epair${epair}b ${ip}/24";
exec.start += "route add default 192.168.42.1";
exec.poststop = "ifconfig epair${epair}a destroy";

jail1 {
  $epair = 1;
  $ip = "192.168.42.100";
}

jail2 {
  $epair = 2;
  $ip = "192.168.42.101";
}

host /etc/pf.conf:
note: interface is vtnet0 because this is running in a cloud provider (at least I think that's why)

Code:
nat on vtnet0 from bridge0:network to any -> (vtnet0)
pass all

With this setup, the jails don't actually have anything in /etc/rc.conf. The host effectively configs the network topology, and then the jails can configure whatever services they want.

Don't forget /etc/resolv.conf in the jails!

I'm happy to answer any questions or help however I can.
 
It uses a fixed IP. According to this post, /dev/tun requires vnet.
Thanks, I completely missed that!

I actually tried setting up a VNET configuration first, but unfortunately I didn't get it to work. Whatever configuration I tried so far, I always ended up with "Network is unreachable" when trying to ping the gateway or whatever outside IP.

What I actually find a little weird is, if i try to set the external interface to "up" (ifconfig_vtnet0="up" in /etc/rc.conf) it crashes my whole networking and I can only acces the system via KVM to bring it back to live. Though that might be an artifact from the virtual hosting setup. I'm indeed running this host on a cloud VPS.

My hosts /etc/pf.conf:
Code:
ext_if = "vtnet0"    # external interface
ext_net = $ext_if:network
jail_if = "bridge0"        # internal interface
jail_net = $jail_if:network

nginx_ports = "{ http, https }"
nginx_ip = "10.0.0.2"

grafana_port = "{ 3000 }"
grafana_ip = "10.0.0.3"

bind_port = "{ 53 }"
bind_ip = "10.0.0.4"

# Do not filter loopback
set skip on lo0
set loginterface lo1
set block-policy return
set fail-policy return

# Sanitize incoming data
scrub in on $ext_if all

# Route HTTP(S) to nginx jail
nat on $ext_if from $jail_net to any -> ($ext_if)
rdr pass on $ext_if proto tcp from any to ($ext_if) port $nginx_ports -> $nginx_ip

# Route port 3000 to grafana jail
rdr pass on $ext_if proto tcp from any to ($ext_if) port $grafana_port -> $grafana_ip

# Route DNS to BIND jail
rdr pass on $ext_if proto { tcp, udp } from any to ($ext_if) port $bind_port -> $bind_ip

# Block incoming by default
block in

# Allow outgoing by default
pass out

# Prevent spoofing attacks
antispoof for $ext_if

# Allow SSH
pass in on $ext_if proto tcp from any to ($ext_if) port 2222

# Allow ICMP
pass in on $ext_if inet proto icmp to ($ext_if) icmp-type { unreach, redir, timex, echoreq }
pass in on $jail_if inet proto icmp to ($jail_if) icmp-type { unreach, redir, timex, echoreq }

My hosts /etc/sysctl.conf:
Code:
security.jail.allow_raw_sockets=1
net.inet.ip.forwarding=1
net.link.bridge.pfil_onlyip=0
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_member=0

I tried configuring it according to iocage documentation:
host /etc/rc.conf:
Code:
## Enable Firewall
pf_enable="YES"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"

## Allow host to act as a gateway
gateway_enable="YES"

## Basic Networking
hostname="hosting01"
ifconfig_vtnet0="DHCP"
defaultrouter="172.31.1.1"

## VNET/Bridge Networking
cloned_interfaces="bridge0"
ifconfig_bridge0="addm vtnet0 up"

## Enable other services
iocage_enable="YES"

jails config.json:
Code:
{
    "allow_raw_sockets": 1,
    "allow_tun": 1,
    "basejail": 1,
    "boot": 0,
    "defaultrouter": "10.0.0.1",
    "devfs_ruleset": "5",
    "host_hostname": "webserver",
    "host_hostuuid": "webserver",
    "ip4_addr": "vnet0|10.0.0.2/24",
    "jail_zfs_dataset": "iocage/jails/webserver/data",
    "last_started": "2022-04-12 09:14:10",
    "release": "13.0-RELEASE-p10",
    "vnet": 1,
    "vnet0_mac": "960001170c38 960001170c39"
}

I also tried configuring it based on your suggestions:
host /etc/rc.conf:
Code:
## Enable Firewall
pf_enable="YES"
pflog_enable="YES"
pflog_logfile="/var/log/pflog"

## Allow host to act as a gateway
gateway_enable="YES"

## Basic Networking
hostname="hosting01"
ifconfig_vtnet0="DHCP"
defaultrouter="172.31.1.1"

## VNET/Bridge Networking
cloned_interfaces="bridge0 epair1"
ifconfig_bridge0="10.0.0.1/24"

## JAIL interfaces
# webserver
ifconfig_epair1="up"
ifconfig_bridge0="addm epair1a"

## Enable other services
iocage_enable="YES"

jails config.json:
Code:
{
    "allow_raw_sockets": 1,
    "allow_tun": 1,
    "basejail": 1,
    "boot": 0,
    "defaultrouter": "10.0.0.1",
    "devfs_ruleset": "5",
    "exec_start": "/bin/sh /etc/rc && ifconfig epair1b 10.0.0.2/24",
    "host_hostname": "webserver",
    "host_hostuuid": "webserver",
    "interfaces": "epair1b:bridge0",
    "ip4_addr": "epair1b|10.0.0.2/24",
    "jail_zfs_dataset": "iocage/jails/webserver/data",
    "last_started": "2022-04-12 09:14:10",
    "release": "13.0-RELEASE-p10",
    "vnet": 1,
    "epair1b_mac": "960001170c38 960001170c39"
}

Don't forget /etc/resolv.conf in the jails!
I'm not sure what exactly to set there. I added nameserver 10.0.0.1 below the existing entries, is that correct?

Thanks for your help!
 
It seems like iocage configures the epair interface itself.
This is the ifconfig output for the vnet0 interface created by iocage:

Code:
vnet0.1: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: webserver as nic: epair0b
    options=8<VLAN_MTU>
    ether 96:00:01:17:0c:38
    hwaddr 02:48:35:53:3c:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

The weird thing is: no epair0b is showing up in the jail via ifconfig. Only lo0 and pflog0.
 
I just realised, that the vnet interface created by iocage also doesn't show up as member on bridge0 :/.
Code:
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:ff:d2
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vtnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
 
I encourage you to set aside the port forwarding stuff for the time being, as well as iocage. Get a simple setup working, and then introduce new parts, or convert to iocage, or whatever. So in your shoes I would:
  1. Make a single /etc/jail.conf based jail
  2. Get networking going so you can ping, curl, etc from it
  3. Convert to iocage (if you want iocage), get the network working there
  4. Add other jails one at a time, and your pf rules for port-forwarding
Whatever configuration I tried so far, I always ended up with "Network is unreachable" when trying to ping the gateway or whatever outside IP.
There are a couple reasons this could be. First is just that the routes aren't set up. Your host machine needs to have a route to the outside world, which the cloud provider should have configured. If you can curl from the host, you're good there. Then you need a route from the jail to the host, as well as a default gateway. Setting up the ipair b end on the same subnet as the host bridge will let the jail see the bridge, and then setting up default gateway on the jail will route any external requests through the bridge.

In my example, you can see that the bridge and epair-b are on the same subnet (192.168.42.0/24) and that the jail's default gateway is the bridge (192.168.42.1). The host has its own default gateway preconfigured by GCP, so when I `curl http://forums.freebsd.org` the sequence is roughly 1) jail says "I don't know how to get there, I'll ask 192.168.42.1" (the bridge) 2) bridge says "I don't know how to get there either, I'll ask 12.34.56.78" (whatever the provider has set up) 3) the request goes out the external NIC.

There's a wrinkle with cloud providers: you may need to enable an "allow packet / IP forwarding option" in the cloud provider console in addition to gateway_enable. For example, GCP calls it IP forwarding, and I think if you don't have it set up, then nothing you do on the host will matter.
 
What I actually find a little weird is, if i try to set the external interface to "up" (ifconfig_vtnet0="up" in /etc/rc.conf) it crashes my whole networking and I can only acces the system via KVM to bring it back to live. Though that might be an artifact from the virtual hosting setup. I'm indeed running this host on a cloud VPS.
I found out where the weirdness with my default network interface came from: My cloud provider, for whatever reason, uses a point-to-point connection for the VPS.
Thanks to this FreeBSD forum thread I was able to configure the interface statically and also change its name with this configuration in /etc/rc.conf:
Code:
hostname="hosting0"
ifconfig_vtnet0_name="jailether"
ifconfig_jailether="inet vps.public.ip.addr/32 mtu 1450"
defaultrouter="a.b.c.1"
static_routes="defgw"
route_defgw="-host a.b.c.1 -iface jailether"

I also spent some time yesterday and today reading "FreeBSD Mastery: Jails" by Michael W Lucas—not only a very informative lecture but also quite an amusing one.
It's back to trying some stuff, now. I'll give another update once I've figured more of it out.
 
This behavior is pretty weird …

I extended the configuration above for vnet:
Code:
## VNET settings
cloned_interfaces="bridge0"
ifconfig_bridge0_name="jailetherbridge"
ifconfig_jailetherbridge="192.168.42.1/24"
gateway_enable="YES"

## Enable user services
jail_enable="YES"
jail_list=""

On boot, the jail now starts automatically, even though according to "FreeBSD Mastery: Jails" the empty jail_list parameter should prevent automatic startup. That was at least my understanding.

My current /etc/jail.conf:
Code:
exec.timeout=90;
stop.timeout=30;

host.hostname="$name";
path="/jail/jails/$name/root";
mount.fstab="/jail/jails/$name/fstab";
mount.devfs;
devfs_ruleset=5;
exec.clean;
exec.start="sh /etc/rc";
exec.stop="sh /etc/rc.shutdown";
exec.consolelog="/var/tmp/$name";

vnet;
vnet.interface="epair${epair}b";
exec.prestart="ifconfig epair${epair} create up";
exec.prestart+="ifconfig jailetherbridge addm epair${epair}a";
exec.start+="ifconfig epair${epair}b ${ip}/24";
exec.start+="route add default 192.168.42.1";
exec.poststop="ifconfig epair${epair}a destroy";

webserver {
  $epair=0;
  $ip="192.168.42.100";
}
How important is actually the order of these entries?

As I said, on boot, the jail starts but it goes down after a bit (I actually got a "Killed" output while I was logged into the jails console via jexec -l webserver).
Afterwards, the epair0a interface remains and I cannot destroy it manually. I tried multiple times and always the console hung up. Which also prevents me from restarting the jail because the interface already exists.

Here's a sanitized output from ifconfig after the jail got killed:
Code:
jailether: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1450
    options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
    ether 96:00:01:37:d7:61
    inet a.b.c.d netmask 0xffffffff broadcast a.b.c.d
    media: Ethernet autoselect (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
jailetherbridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 58:9c:fc:10:38:5e
    inet 192.168.42.1 netmask 0xffffff00 broadcast 192.168.42.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 4 priority 128 path cost 2000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:0a:db:3f:dc:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Edit: I forgot to mention, when I was in the still running jail, there was also no epair0b interface showing up, even though the epair0a was present on the host and correctly assigned to the bridge.

Edit 2: Changing the /etc/jail.conf script to use another epair interface after booting leads to an error on jail startup:

Code:
service jail start webserver
Starting jails: cannot start jail  "webserver":
epair1a
ifconfig: BRDGADD epair1a: Invalid argument
jail: webserver: ifconfig jailetherbridge addm epair1a: failed
.
Although epair1a and epair1b were created.
 
Okay, next step solved.
I did a fully manual setup and was able to:
- create a bridge interface
- create an epair interface
- add jailether (renamed vtnet0) to the bridge interface
- add epair0a to the bridge interface (the culprit here was the weird mtu setting of the cloud interface, that was the origin of the ifconfig: BRDGADD epair1a: Invalid argument error. After setting the mtu of the epair interface accordingly it worked.)

All of this was done with a running jail where I only set the vnet option and devfs_ruleset=5. No interface setup or anything in /etc/jail.conf.

The problem left is adding the other end of the epair interface to the jail.
Running ifconfig epair0b vnet webserver leads to the console hanging up. From a new console I can see that epair0b is now missing from ifconfig on the host but doesn't appear on the jail.
 
I created a new thread under "Networking" as I felt like my current issue doesn't really fit the original topic and space anymore. Here it is.
I'll come back to this thread as soon as the other issue is resolved.
 
Top