vnet jails on VLAN subinterfaces

[jandras]

Hello everyone,

Searching the web for vnet jails mostly shows examples using one epair per jail, which is connected with a bridge to the physical interface in the host:
E.g. em0 - bridge0 - epair0a - epair0b, where the last one belongs to the jail's network stack.

Mixing in tagged 802.1Q VLANs, one VLAN per jail, I've tried the following setup, which I also saw on the web:
em0 - em0.99 (host)
em0 - em0.100 - bridge0 - epair0a - epair0b (jail0)
em0 - em0.101 - bridge1 - epair1a - epair1b (jail1)
Here epair[01]b belong to the jails' network stacks, and all other interfaces to the base system.

This setup worked mostly on 12.1-RELEASE-p5 GENERIC amd64, but there was a strange throughput problem: Everything worked fine until I enabled pf on the host. The problem remained even with an empty /etc/pf.conf, or a pf.conf containing only "set skip on xx" for every interface. When pf was enabled on the host, IPv4 throuthput in jails dropped in the incoming direction dramatically.
iperf3 -s runs on another host in a gigabitEthernet network. With iperf3 -c running in jail0 I measured above 900 Mbit/s in both directions on IPv6 and on IPv4 outbound, but only 50-100 kbit/s inbound on IPv4. All four measurements from the host system were OK (IPv4 and IPv6, in- and outbound, all above 900 Mbit/s). If pf is disabled on the host, then all four measurements are OK from jail0 too. Enabling pf in jail0 is OK too, there's no throughput degradation.

Then I tried a simpler setup, assigned the VLAN subinterfaces directly to the jails' network stacks:
em0 - em0.99 (host)
em0 - em0.100 (jail0)
em0 - em0.101 (jail1)
Here em0 and em0.99 belong to the base system while em0.10[01] belong to the jails' network stacks.

This works perfectly, using pf both on the host and in jails is OK, throughput is fine. But I didn't see this setup mentioned anywhere. Is this a "valid" setup?

András

 

[SirDice]

Everything worked fine until I enabled pf on the host. The problem remained even with an empty /etc/pf.conf, or a pf.conf containing only "set skip on xx" for every interface. When pf was enabled on the host, IPv4 throuthput in jails dropped in the incoming direction dramatically.

One issue you might run into with this combination, TSO/LRO and/or checksum offloading on the em0 interface. Try turning those off.

In another thread there's also a small patch for the bridge(4) interfaces, apparently it can dramatically improve the throughput between two bridge members.

Edit: this is the thread: https://forums.freebsd.org/threads/backport-of-if_bridge-patch-for-better-performance-to-12-1.75483/

 

[jandras]

One issue you might run into with this combination, TSO/LRO and/or checksum offloading on the em0 interface. Try turning those off.

Disabling LRO on em0 solved the incoming IPv4 throughput problem. Thank you SirDice.

Anyway, as the setup without bridges and epairs is much simpler, and there's also no throughput problem with the default LRO setting, I'm still curious whether it's OK to simply assign em0.100 directly to jail0's network stack without using a bridge and an epair.