Solved Trouble with VNET within jail

[kzxc8]

Hello,

I am attempting to use vnet (plus epair and bridge) within a jail but I am unable to make the jail reachable on my network.

I'm relatively new to FreeBSD and jails and most certainly doing something incorrectly.

The process I am following:

1. Setup networking on host using interface hn0​

Relevant portion of /etc/rc.conf:

ifconfig_hn0="inet 192.168.1.124 netmask 255.255.248.0"
defaultrouter="192.168.1.1"

This interface works as expected, I can SSH into the host and download packages from the internet.

2. Create epair and bridge interfaces​

# ifconfig epair0 create up
# ifconfig bridge create up
# ifconfig bridge0 inet 192.168.1.224 netmask 255.255.248.0 addm hn0 addm epair0a

The resulting bridge looks like:

# ifconfig bridge0
bridge0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether 58:9c:fc:10:ff:86
        inet 192.168.1.224 netmask 0xfffff800 broadcast 192.168.7.255
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 3 priority 128 path cost 2000
        member: hn0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 2000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>

2. Create a thick jail at /opt/jail/test​

The jail config at /etc/jail.conf.d/test.conf looks like:

test {
  exec.start = "/bin/sh /etc/rc";
  exec.stop = "/bin/sh /etc/rc.shutdown";
  exec.consolelog = "/var/log/jail_console_test.log";
  allow.raw_sockets;
  exec.clean;
  mount.devfs;
  host.hostname = "testjail";
  path = "/opt/jail/test";
  vnet;
  vnet.interface = "epair0b";
  allow.set_hostname = 1;
}

2. Start jail and configure network​

# service jail start test
# jexec test ifconfig epair0b inet 192.168.1.210 netmask 255.255.248.0 up
# jexec test ifconfig epair0b inet
# jexec test route add default 192.168.1.1

At this point, I can ping the bridge from within the jail:

# jexec test ping -c 1 192.168.1.224
PING 192.168.1.224 (192.168.1.224): 56 data bytes
64 bytes from 192.168.1.224: icmp_seq=0 ttl=64 time=0.394 ms

--- 192.168.1.224 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.394/0.394/0.394/0.000 ms

But cannot ping the default router:

# jexec test ping -c 1 192.168.1.1
PING 192.168.1.1 (192.168.1.1): 56 data bytes
--- 192.168.1.1 ping statistics ---
1 packets transmitted, 0 packets received, 100.0% packet loss

 

[SKull]

I think you need to:

sysrc gateway_enable=YES

and

sysctl net.ip.forwarding=1

I am not sure about the correct grammar of the forwarding parameter. Do a

sysctl -a | grep forward

to get the correct spelling.

 

[kzxc8]

SKull thank you for the response, that looked very promising, however I am still seeing the same behavior.

I ran:

# sysrc gateway_enable=YES
# sysctl net.inet.ip.forwarding=1
# service netif restart (on both host and jail)

I also tried restarted the host with forwarding enabled and setup the epair and bridge again but still cannot ping the gateway.

 

[Emrion]

That works here. Except, I don't use Hyper-V. On a physical interface, there is no problem with such a setting, even without defining a default route.

Is hn0 in promiscuous mode when you set up this jail? It should after addm this interface to the bridge.
like:

ifconfig em0
em0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP>


If not, look for a setting to allow this.

PS: no need to define an ip address for the bridge.

 

[kzxc8]

Emrion thank you for the response. I was trying in Hyper-V solely to try and reduce variables (and ironically it sounds like I created a new one due to the adapter not having promiscuous mode enabled.)

I'll try again on hardware and report back.

It does make me wonder though, if I'll encounter a similar issue when I eventually try this on AWS (though I can leverage their virtual NICs probably, and perhaps just use a different configuration without needing promiscuous mode.)

 

[kzxc8]

Emrion , real hardware worked.

I didn't need to enable net.inet.ip.forwarding, or add a default route. So the issue was with my hypervisor's NIC settings.

I'm now good to go. Thank you!

 

[ccammack]

A few times on here I have seen SirDice recommend against using sysctl net.inet.ip.forwarding=1 in favor of using only gateway_enable="YES".

 

[tommiie]

If I remember correctly, the sysctl command enables IP forwarding right now while the gateway_enable setting in rc.conf enables IP forwarding automatically at reboot. So if you want it right now and you want it to happen at every reboot, you need both.