Can't seem to get vnet (bridge + epair) jail networking connected

T

tjpcc

I'm building a jails host and can't for the life of me get bridged networking to work. I've currently got an ip-sharing jail working (can ping the host and outside world) so that's always an option, but I'd like to find out what I'm doing wrong in the vnet jail. After I bring up this vnet jail, jexec vnet_test ping 192.168.0.131 (131 being the host's IP on epair3a) shows one packet being delivered, then the rest are dropped. Subsequent pings without restarting the jail drop all packets.

/etc/rc.conf snippet:
Code: 
ifconfig_vtnet0="DHCP"
cloned_interfaces="bridge0 lo1"
ifconfig_bridge0="addm vtnet0 up"

/etc/jail.conf:
Code: 
exec.system_user = "root";
exec.jail_user = "root";
exec.consolelog = "/var/log/jail_console_${name}.log";


allow.raw_sockets;
exec.clean;
mount.devfs;
devfs_ruleset = "5";

host.hostname = "${name}";
path = "/jails/roots/${name}";

mount += "/jails/templates/14.1-RELEASE /jails/roots/${name}    nullfs  ro              0       0";
mount += "/jails/layers/${name}         /jails/roots/${name}    unionfs rw,noatime      0       0";

#www {
#       $id = "1";
#       mount += "/data/cert    /jails/roots/www/root/.acme.sh  nullfs  ro      0       0";
#}

#acme {
#       persist;
#       $id = "2";
#       mount += "/data/cert    /jails/roots/acme/data/cert     nullfs  rw,noatime      0       0";
#}

vnet_test {
        persist;
        $id = "3";

        $brg = "bridge0";
        $epa = "epair${id}a";
        $epb = "epair${id}b";
        $jip = "192.168.0.$id";
        $hip = "192.168.0.$(($id + 128))";

        vnet;
        vnet.interface = "$epb";

        exec.prestart = "ifconfig epair${id} create";
        exec.prestart += "ifconfig $epa descr vnet-${name}";
        exec.prestart += "ifconfig $epa inet $hip/24 up";
        exec.prestart += "ifconfig $brg addm $epa";

        exec.start = "ifconfig $epb inet $jip/24 up";
        exec.start += "route add default $hip";
        exec.start += "/bin/sh /etc/rc";

        exec.prestop = "ifconfig $epb -vnet ${name}";

        exec.stop = "/bin/sh /etc/rc.shutdown";

        exec.poststop = "ifconfig $brg deletem $epa";
        exec.poststop += "ifconfig $epa destroy";
}

classic_test {
        persist;
        $id = "4";

        interface = "lo1";
        ip4.addr = "192.168.0.$id";
}

/etc/pf.conf:
Code: 
ext_if="vtnet0"
jails_nw="192.168.0.0/24"

nat pass on $ext_if from $jails_nw to any -> ($ext_if)

ifconfig:
Code: 
vtnet0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE>
        ether 56:00:05:08:24:dc
        inet 207.148.7.164 netmask 0xfffffe00 broadcast 207.148.7.255
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 58:9c:fc:00:27:6b
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair3a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        member: vtnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo1: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 192.168.0.4 netmask 0xffffffff
        inet6 fe80::1%lo1 prefixlen 64 scopeid 0x4
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair3a: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: vnet-vnet_test
        options=8<VLAN_MTU>
        ether 02:72:1a:2d:e2:0a
        inet 192.168.0.131 netmask 0xffffff00 broadcast 192.168.0.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

jexec vnet_test ifconfig:
Code: 
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair3b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:72:1a:2d:e2:0b
        inet 192.168.0.3 netmask 0xffffff00 broadcast 192.168.0.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

What am I missing?
 
Don't bridge the jail interface with your external interface if you plan on using NAT. Bridge it with your lo1 instead.
 
I definitely mixed up suggestions from different sources there, thanks SirDice.

That (removing the bridge to vtnet0) restores the connection between the host and jail's epair IPs, but I still for some reason can't ping the outside internet from the vnet_test jail.
 
What gateway address is the jail using ( netstat -4rn)?
 
It adds the default route in exec.start.
Code: 
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.0.131      UGS     epair3b
127.0.0.1          link#6             UH          lo0
192.168.0.0/24     link#5             U       epair3b
192.168.0.3        link#6             UHS         lo0
 
192.168.0.131 is the IP address of the jail itself. So your packets get routed back to itself. In other words, there's nowhere for your packets to go when a destination is outside the 192.168.0.0/24 subnet.

I would suggest setting 192.168.0.1/24 on bridge0 and use that address as the default gateway in your VNET jails.
 
I've made things confusing here by doing two things at once. The current state of this machine is for testing both approaches: jail shared networking with a host lo1 on which each jail takes a unique IP, and vnet jails with a bridge on the host and an epair for each jail, with the b side going into the vnet. If I can get vnet working, I'll drop lo1 entirely (unless there's a compelling reason to keep it?).

192.168.0.131 is definitely the IP address of the "a" side still on the host. ifconfig epair3a:
Code: 
epair3a: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: vnet-vnet_test
        options=8<VLAN_MTU>
        ether 02:fd:d8:40:0b:0a
        inet 192.168.0.131 netmask 0xffffff00 broadcast 192.168.0.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

while 192.168.0.3 is the "b" side in the jail. jexec vnet_test ifconfig epair3b:
Code: 
epair3b: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:fd:d8:40:0b:0b
        inet 192.168.0.3 netmask 0xffffff00 broadcast 192.168.0.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

This was by design, so that reaching jails from the host or each other is a matter of reaching 192.168.0.$id.

I tried it anyway and 192.168.0.3 doesn't work any better as the jail's default route.

The intention is that the jail will send public internet-bound packets to 131, which is in the host so that pf then sees it, translates it, and sends it out on the public interface.
Code: 
ext_if="vtnet0"
jails_cidr="192.168.0.0/24"
nat pass on $ext_if from $jails_cidr to any -> $ext_if
 
This doesn't seem super productive. It's not really fair to take you down my debugging rabbit hole, so let me take a step back.

I'm configuring a VM with a single, publicly-routable IP address, so I think I need to use NAT to network jails on this host. I've gotten this to work with shared-network jails, but I'd like to get vnet jails working instead. How should I go about that?
 
SirDice said:
I would suggest setting 192.168.0.1/24 on bridge0 and use that address as the default gateway in your VNET jails.
Click to expand...
SirDice's suggestion is just what I'm using for my working VNET jail.
In my configuration, host's rc.conf has something like:
Code: 
ifconfig_igb0="SYNCDHCP"
gateway_enable="YES"
pf_enable="YES"
...
cloned_interfaces="bridge0 epair0"
ifconfig_bridge0="inet 192.168.0.1/24 addm epair0a up"
ifconfig_epair0a="up"
The VNET jail uses epair0b with 192.168.0.2/24 and set its default gateway to 192.168.0.1 (host's bridge0 address).
With the host's IP forwarding enabled, the host simply routes packets between bridge0 (internal I/F) and igb0 (external I/F) so NAT on the external interface should work.
 
You must log in or register to reply here.
Back
Top