[VLAN] Can't get jail/bhyve in vlan working

pprocacci · Mar 17, 2020

Setup:

Physical Machine w/ two nics.
One of the nics is passed through to bhyve pfsense instance.

Bhyve pfsense instance has four interfaces. 1 WAN and 1 LAN and 2 vlans.

In addition to the bhyve pfsense instance, there is a vnet jail with epair70a and epair70b accordingly.
A fully running FreeBSD host, jail, and bhyve pfsense looks as follows:

Host:

Code:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
        ether d0:50:99:d4:b9:fe
        inet 192.168.0.2 netmask 0xffffff00 broadcast 192.168.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-customswitch
        ether 02:b4:bd:ea:4e:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: epair70a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000000
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000000
        groups: bridge vm-switch viid-cc582@
        nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-pfsense-0-customswitch
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:ff:91
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 1446
epair70a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:67:41:86:99:0a
        inet6 fe80::67:41ff:fe86:990a%epair70a prefixlen 64 scopeid 0x6
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Bhyve PFsense:
-------------------------

Code:

vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 58:9c:fc:06:47:08
        hwaddr 58:9c:fc:06:47:08
        inet6 fe80::5a9c:fcff:fe06:4708%vtnet0 prefixlen 64 scopeid 0x1
        inet 192.168.0.1 netmask 0xffffff00 broadcast 192.168.0.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
vtnet0.70: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 58:9c:fc:06:47:08
        inet6 fe80::5a9c:fcff:fe06:4708%vtnet0.70 prefixlen 64 scopeid 0x7
        inet 192.168.70.1 netmask 0xffffff00 broadcast 192.168.70.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        vlan: 70 vlanpcp: 0 parent interface: vtnet0
        groups: vlan
vtnet0.71: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 58:9c:fc:06:47:08
        inet6 fe80::5a9c:fcff:fe06:4708%vtnet0.71 prefixlen 64 scopeid 0x8
        inet 192.168.71.1 netmask 0xffffff00 broadcast 192.168.71.255
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
        vlan: 71 vlanpcp: 0 parent interface: vtnet0
        groups: vlan

Jail:
-----------

Code:

Simply failes to DHCP

What I'm trying to do is get the jail on vlan 70 but all attempts to do so failed. I can create a vlan interface from tap0 (tap0.70) add that the bridge1 (for instance) with epair70a on the host .... and when doing so the pfsense bhyve instance receives the request from the jail ... sends the response, but the response is never seen on tap0.70 nor the jail interface.

The only thing out of place given the above is the epair70a device in bridge0 as that was the only way for DHCP to work properly albeit on the wrong VLAN and my lack of inclusion for the WAN and loopback interfaces as no one would care.

Does anyone have a working example of a vlan (call it 70) between a jail and bhyve instance working?

pprocacci · Mar 17, 2020

... and thanks in advance for the responses.

sol289 · Mar 17, 2020

It seems that Vlans and VNET and bridges doesn't work as we want them to work on FreeBSD for now. There is a thread about similar problem here:

Solved - Bridge/epair not passing through tagged VLAN traffic between host and VNET jail

I have worked with FreeBSD and/or VLANs for quite a while now, but now I have something I cannot figure out. Hope someone can tell me what I am doing wrong. I want to set up a FreeBSD box with VNET enabled jails where I can put each jail into a specific VLAN, all accessible via a single...

forums.freebsd.org

And there's a PR 240106 (you will find comment #3 there made by me telling my part of story, do follow the links from that comment to mailing list messages, I think you will see that your problem is more or less have the same root). Maybe you could add your comment to that PR to bring more attention to this problem.

I decided for myself in future jails deployment to avoid trio of VLANs, bridges and VNET in the same time, for they're not working now as we wanted to them to work

VladiBG · Mar 17, 2020

When you are giving entire physical interface to the VM then the 802.1q is done on the VM otherwise the 802.1q should be done on the Hypervisor and presented to the VM as separate interfaces.

genneko · Mar 17, 2020

Hi, what you want is something like this? (addresses/IF names were slightly changed)

Code:

                (192.168.70.1/24)                              (192.168.70.2/24)
                vtnet0.70 (vlan70)         Host    epair0a.70  epair0b.70 (vlan70)
--- WAN [bhyve] vtnet0 ------------ tap0 [bridge1] epair0a --- epair0b [jail1]
                vtnet0.71 (vlan71)                 epair1a --- epair1b [jail2]
                (192.168.71.1/24)                  epair1a.70  epair1b.71 (vlan71)
                                                               (192.168.71.2/24)

For using vlans, it seems enough to bridge only parent interfaces.
It also looks like an epair should have a vlan sub-interface on both ends.
So, to build the above topology, the host(bridge1) config would be:

Code:

ifconfig tap0 create
sysctl net.link.tap.up_on_open=1
ifconfig epair0 create up
ifconfig epair1 create up
ifconfig epair0a.70 create vlan 70 vlandev epair0a up
ifconfig epair1a.71 create vlan 71 vlandev epair1a up
ifconfig bridge1 create up
ifconfig bridge1 addm tap0 addm epair0a addm epair1a

The bhyve VM would have (You already have this):

Code:

ifconfig vtnet0 up
ifconfig vtnet0.70 create vlan 70 vlandev vtnet0 inet 192.168.70.1/24 up
ifconfig vtnet0.71 create vlan 71 vlandev vtnet0 inet 192.168.71.1/24 up

Jail1:

Code:

ifconfig epair0b up
ifconfig epair0b.70 create vlan 70 vlandev epair0b inet 192.168.70.2/24 up

Jail2:

Code:

ifconfig epair1b up
ifconfig epair1b.71 create vlan 71 vlandev epair1b inet 192.168.71.2/24 up

Please make sure all the interfaces are up.
Host: tap0, epair0a, epair0a.70, epair1a, epair1a.71, bridge1
Bhyve VM: vtnet0, vtnet0.70, vtnet0.71
Jail1: epair0b, epair0b.70
Jail2: epair1b, epair1b.71

Although it's a very quick testing, with those setup, I can see vlan tagged traffic between the bhyve VM and the jail1/jail2 on the host's bridge1.

Code:

$ sudo tcpdump -velni bridge1                            
tcpdump: listening on bridge1, link-type EN10MB (Ethernet), capture size 262144 
bytes                 
22:33:32.861699 00:a0:98:11:f8:25 > 02:2c:4b:b0:e1:0b, ethertype 802.1Q (0x8100)
, length 102: vlan 70, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 52474, offset 0
, flags [none], proto ICMP (1), length 84)                                      
    192.168.70.1 > 192.168.70.2: ICMP echo request, id 15365, seq 0, length 64  
22:33:32.861721 02:2c:4b:b0:e1:0b > 00:a0:98:11:f8:25, ethertype 802.1Q (0x8100)
, length 102: vlan 70, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 53992, offset 0
, flags [none], proto ICMP (1), length 84)                                      
    192.168.70.2 > 192.168.70.1: ICMP echo reply, id 15365, seq 0, length 64    
                                                                                
22:33:41.750062 00:a0:98:11:f8:25 > 02:4f:db:0d:cc:0b, ethertype 802.1Q (0x8100)
, length 102: vlan 71, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 52475, offset 0
, flags [none], proto ICMP (1), length 84)                                      
    192.168.71.1 > 192.168.71.2: ICMP echo request, id 16389, seq 0, length 64  
22:33:41.750084 02:4f:db:0d:cc:0b > 00:a0:98:11:f8:25, ethertype 802.1Q (0x8100)
, length 102: vlan 71, p 0, ethertype IPv4, (tos 0x0, ttl 64, id 28558, offset 0
, flags [none], proto ICMP (1), length 84)                                      
    192.168.71.2 > 192.168.71.1: ICMP echo reply, id 16389, seq 0, length 64

pprocacci · Mar 17, 2020

Thanks for all the responses. I will read / try this after work.

The one thing that I'd like to comment on before trying things tonight is a comment by genneko.
It never occured to me to create a vlan in the jail with epair70b as it's parent.
The reason why this is the case is because generally it's never necessary for a "client machine" to know anything about the vlan it's on.

I will certainly play around some more tonight. Again, thanks for the responses...I've got some more reading to do.

genneko · Mar 17, 2020

pprocacci said:
It never occured to me to create a vlan in the jail with epair70b as it's parent.
The reason why this is the case is because generally it's never necessary for a "client machine" to know anything about the vlan it's on.

Me too. Totally agree.
I configured that way because I couldn't find a way to define "untagged" or "acccess" ports for a specific vlan on if_bridge. Maybe there are better ways...

SirDice · Mar 17, 2020

genneko said:
I configured that way because I couldn't find a way to define "untagged" or "acccess" ports for a specific vlan on if_bridge.

You typically create a bridge for each vlan and connect the bridge to the vlan interfaces:

Code:

dice@hosaka:~ % ifconfig vm-servers
vm-servers: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        ether 8a:bf:c5:8e:8f:d7
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 18 priority 128 path cost 2000000
        member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 13 priority 128 path cost 2000000
        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000000
        member: lagg0.11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 55
        groups: bridge vm-switch viid-d5539@
        nd6 options=1<PERFORMNUD>
dice@hosaka:~ % ifconfig lagg0.11
lagg0.11: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: vm-vlan-servers-lagg0.11
        ether 00:25:90:f1:58:39
        inet6 fe80::225:90ff:fef1:5839%lagg0.11 prefixlen 64 scopeid 0x8
        groups: vlan vm-vlan viid-8bf4d@
        vlan: 11 vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Code:

dice@hosaka:~ % ifconfig vm-public
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        ether 9e:25:0d:17:5b:51
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 23 priority 128 path cost 2000000
        member: tap11 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 22 priority 128 path cost 2000000
        member: tap10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 21 priority 128 path cost 2000000
        member: tap9 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 20 priority 128 path cost 2000000
        member: tap8 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 19 priority 128 path cost 2000000
        member: tap6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 17 priority 128 path cost 2000000
        member: tap5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 16 priority 128 path cost 2000000
        member: tap4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 15 priority 128 path cost 2000000
        member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 14 priority 128 path cost 2000000
        member: lagg0.10 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 55
        groups: bridge vm-switch viid-4c918@
        nd6 options=1<PERFORMNUD>
dice@hosaka:~ % ifconfig lagg0.10
lagg0.10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: vm-vlan-public-lagg0.10
        ether 00:25:90:f1:58:39
        inet6 fe80::225:90ff:fef1:5839%lagg0.10 prefixlen 64 scopeid 0xa
        groups: vlan vm-vlan viid-bdfd6@
        vlan: 10 vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

The lagg(4) interface itself is the "trunk" and connected to trunked switch port.

Code:

dice@hosaka:~ % ifconfig lagg0.10
lagg0.10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: vm-vlan-public-lagg0.10
        ether 00:25:90:f1:58:39
        inet6 fe80::225:90ff:fef1:5839%lagg0.10 prefixlen 64 scopeid 0xa
        groups: vlan vm-vlan viid-bdfd6@
        vlan: 10 vlanpcp: 0 parent interface: lagg0
        media: Ethernet autoselect
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Code:

vlan 10
   name "vlan10"
   untagged 2-3,5-10,13-23,25-44
   tagged 1,24,Trk1
   no ip address
   jumbo
   exit
vlan 11
   name "vm-servers"
   tagged 1,Trk1
   no ip address
   jumbo
   exit

genneko · Mar 17, 2020

SirDice said:
You typically create a bridge for each vlan and connect the bridge to the vlan interfaces:

Thank you for the clarification.

At first I thought the VMs should also have corresponding VLAN interfaces such as vtnet0.10, but then I realized it's wrong.

On your bridge "vm-public", lagg0.10 is a tagged port for vlan10 while tap3-6,8-12 are untagged.
Is it right?

pprocacci · Mar 17, 2020

So I think my problem is adding "main interfaces" to the bridge with another interface on the system.
I haven't tried it yet, gotta wait once things die down here but I've essentially done:

bridge0
---------------
igb0 <- physical nic w/ lan behind it
tap0 <- bhyve pfsense instance

bridge70
--------------
igb0.70
epair70a <- jail
tap0.70

The first bridge was to get machines on the network on whatever default vlan they could out to the internet via tap0 (and this works).
bridge70 however was my attempt (a playground of sorts) at moving some of that traffic onto it's own vlan. (this doesn't work).

Given SirDice's reply, it would seem it's best to leave main interfaces (like igb0/tap0) alone, not adding them to any bridge at all.
So what I'll try tonight is the following on the host:

igb0 (left alone)
tap0 (left alone)

bridge10
----------------
igb0.10
tap0.10

bridge70
----------------
igb0.70
tap0.70
epair70a

Making vlan 10 the "default" for everything and 70 for that pet project I'm trying to get separated. If anyone sees any problems with this I'd like to know why ahead of time.

pprocacci · Mar 18, 2020

Well, I gave up.

You would think you can assign a vlan to the hosts' jail side of the epair and stick that in a bridge but you can't (easily) without hurdles.

I instead created a vlan dev from within the jail and just added the hosts' side of the epair to the main bridge.

I wanted to avoid having to do anything from within the jail, but something super simple isn't quite simple at all.

I personally think sol289 is right and this is quite simply a bug.

What I had prior to this endevor which worked:

Code:

                            HOST
                 +--------------------------+
                 |                          |
                 |         BRIDGE0          |
                 |  +-------------------+   |
     LAN TRUNK   |  |                   |   |
<----------------|--|igb0               |   |
                 |  |              tap0 |   |
                 |  +-------------------+   |
                 |                       \  |
                 +------------------------\-+
                                           \
                                            \    BHYVE / PFSENSE
                                             \  +---------------+
                                              \ |               |      WAN
                                               \|vtnet0         |------------->
                                                |vtnet0.70      |
                                                |vtnet0.71      |
                                                +---------------+

What I wanted (which didn't work):

Code:

                            HOST                     JAIL
                 +--------------------------+  +---------------+
                 |                 epair70a |  |               |
                 |         BRIDGE0          |  |               |
                 |  +-------------------+   |  |               |
     LAN TRUNK   |  |       epair70a.70 |---+--+epair70b       |
<----------------|--|igb0               |   |  |               |
                 |  |              tap0 |   |  |               |
                 |  +-------------------+   |  +---------------+
                 |                       \  |
                 +------------------------\-+
                                           \
                                            \    BHYVE / PFSENSE
                                             \  +---------------+
                                              \ |               |      WAN
                                               \|vtnet0         |------------->
                                                |vtnet0.70      |
                                                |vtnet0.71      |
                                                +---------------+

What I ended up with:

Code:

                            HOST                     JAIL
                 +--------------------------+  +---------------+
                 |                          |  |               |
                 |         BRIDGE0          |  |               |
                 |  +-------------------+   |  |epair70b       |
     LAN TRUNK   |  |          epair70a |---+--+epair70b.70    |
<----------------|--|igb0               |   |  |               |
                 |  |              tap0 |   |  |               |
                 |  +-------------------+   |  +---------------+
                 |                       \  |
                 +------------------------\-+
                                           \
                                            \    BHYVE / PFSENSE
                                             \  +---------------+
                                              \ |               |      WAN
                                               \|vtnet0         |------------->
                                                |vtnet0.70      |
                                                |vtnet0.71      |
                                                +---------------+

SirDice · Mar 18, 2020

genneko said:
On your bridge "vm-public", lagg0.10 is a tagged port for vlan10 while tap3-6,8-12 are untagged.
Is it right?

That is correct. The VMs all get untagged traffic.

sol289 · Apr 12, 2020

I tried in VM jail scheme with vlans in bridges, and I liked it. Host creates bridge for every VLAN, and every jail creates it's own vlan interfaces and bridge them together with host's bridges.
In my network I have management VLAN which is usually connected to every host, and some auxilliary VLANs for specific applications. So I created two jails, on jail one I 've added also VLAN22 with default VLAN4, and on jail two only default VLAN4. I've add IP addresses to both vlan's jails interfaces and to vlan's host interface, and all of them can ping each other.

My /etc/jail.conf:

Code:

#
exec.clean;
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
# Verify bridge0 is present (we need it for spinning up jails)
# if not, create it and then add our uplink interface (ix0 in this example)

$uplinkdev = "em0";
$bridge4 = "bridge4";
$bridge22 = "bridge22";
$j4 = "v4";
$j22 = "v22";
vnet.interface = "$j4";
vnet;
exec.prestart     = "ifconfig $bridge4 > /dev/null 2> /dev/null || ( ifconfig $bridge4 create up && ifconfig $bridge4 addm vlan4 )";
exec.prestart    += "ifconfig $epair create up                 || echo 'Skipped creating epair (exists?)'";
exec.prestart    += "ifconfig $bridge4 addm ${epair}a           || echo 'Skipped adding bridge member (already member?)'";
exec.created      = "ifconfig ${epair}b name $j4             || echo 'Skipped renaming ifdev to $j4 (looks bad...)'";
exec.prestop      = "ifconfig $j4 -vnet $name";
exec.poststop     = "ifconfig $bridge4 deletem ${epair}a";
exec.poststop    += "ifconfig ${epair}a destroy";

allow.raw_sockets;
allow.mount;
allow.mount.nullfs;
allow.sysvipc;

enforce_statfs = 1;
#exec.system_user = "root";
#exec.jail_user = "root";
exec.consolelog = "/var/log/jail_${host.hostname}.log";

path = "/mnt/jails/$name";
host.hostname = $name.zato.local;
mount.fstab = "/mnt/jails/$name.fstab";
mount.devfs;

# added by mkjail
one {
    $epair = "epair14";
    $epair22 = "epair122";

    vnet.interface = $j4, $j22;

    exec.created     += "ifconfig ${epair22}b name $j22             || echo 'Skipped renaming ifdev to $j22 (looks bad...)'";
    exec.prestart    += "ifconfig $bridge22 > /dev/null 2> /dev/null || ( ifconfig $bridge22 create up && ifconfig $bridge22 addm vlan22 )";
    exec.prestart    += "ifconfig $epair22 create up                 || echo 'Skipped creating epair (exists?)'";
    exec.prestart    += "ifconfig $bridge22 addm ${epair22}a           || echo 'Skipped adding bridge member (already member?)'";
    exec.prestop     += "ifconfig $j22 -vnet $name";
    exec.poststop    += "ifconfig $bridge22 deletem ${epair22}a";
    exec.poststop    += "ifconfig ${epair22}a destroy";
}
# added by mkjail
two {
  $epair = "epair24";
}

For this config to work you must first create you VLAN interfaces on host, for example in /etc/rc.conf:

Code:

cloned_interfaces="vlan4 vlan22"
ifconfig_vlan4="up vlandev em0 vlan 4"
ifconfig_vlan22="up vlandev em0 vlan 22"

vonbeitthia · Jul 22, 2020

check my project https://github.com/vonbeitthia/bhyveWindows7/blob/master/vm windows v7+ iocage.pdf

Drizzt321 · Nov 20, 2020

So this seems exactly like what I have (current thread https://forums.freebsd.org/threads/vlans-with-bhyve-guests-not-getting-dhcp.77647/), although LAN side only, not dealing with any kind of routing/etc. Just 1 trunk port, and only VMs, no jails (for now). And currently just 1 VM.

After reading this thread, my understanding is this is what I have:
em0 -- vm-public (bridge) -- VM home-assistant
em0.30 -- vm-ha-iot (bridge) -- VM home-assistant

With the em0 getting the traffic destined for VM home-assistant (based on MAC).

Code:

                            HOST                     Bridge vm-public (no VLAN, untagged)
                 +--------------------------+  +---------------+
                 |                          |  |               |
                 |                          |  |         tap 2 | ------ vmnet-home-assistant-0-public -- VM home-assistant
                 |  +-------------------+   |  |               |                                         |
     LAN TRUNK   |  |  em0              |---+--+ em0           |                                         --> enp0s5f0
<----------------|--|                   |   |  |         tap 0 | --|
                 |  |  em0.30           |   |  |         tap 1 | --|---> 2 other VMs
                 |  +-------------------+   |  +---------------+
                 |                       \  |
                 +------------------------\-+
                                           \
                                            \    Bridge  vm-ha-iot (VLAN 30)
                                             \  +---------------+
                                              \ |               |
                                               \| em0.30        |
                                                |          tap3 | ------ vmnet-home-assistant-1-ha-iot -- VM home-assistant
                                                |               |                                         |
                                                +---------------+                                         --> enp0s5f1

And what I need is:

em0 -- vm-public (bridge) -- VM home-assistant
em0 -- bridge public -- tap3.30 (VLAN on tap interface) -- VM home-assistant

Code:

                            HOST                     Bridge vm-public (no VLAN, untagged)
                 +--------------------------+  +---------------+
                 |                          |  |               |
                 |                          |  |         tap 2 | ------ vmnet-home-assistant-0-public -- VM home-assistant
                 |  +-------------------+   |  |               |                                         |
     LAN TRUNK   |  |  em0              |---+--+ em0           |                                         --> enp0s5f0
<----------------|--|                   |   |  |         tap 0 | --|
                 |  |                   |   |  |         tap 1 | --|---> 2 other VMs
                 |  +-------------------+   |  |               |
                 |                          |  |       tap3.30 | ------ vmnet-home-assistant-1-public -- VM home-assistant
                 |                          |  +---------------+                                         |
                 |                          |                                                            --> enp0s5f1
                 +--------------------------+

sol289 · Nov 20, 2020

Drizzt321 said:
And what I need is:

em0 -- vm-public (bridge) -- VM home-assistant
em0 -- bridge public -- tap3.30 (VLAN on tap interface) -- VM home-assistant

I think you better detrunk your trunk link on host, make VLAN interfaces for each VLAN you need, make multiple bridges for each VLAN, and add those VLAN interfaces to corresponding bridge, and connect each bridge as switch to your VMs.

Drizzt321 · Nov 20, 2020

How do you mean detrunk? I have all my normal network/internet traffic (this is my NAS/home server) on em0, untagged. I'm trying to setup IoT stuff, and want it isolated, thus VLAN.

Do you mean create a LAN VLAN (e.g. VLAN 10) for all the normal network traffic, NAS serving, etc, and bridge from that to my LAN/public VMs, and then VLAN 30 (IoT VLAN) with a bridge to the VMs which need access to that?

So:
em0 (host)
em0.10 (VLAN 10, LAN) -- vm-public bridge
em0.30 (VLAN 30, IoT) -- vm-ha-iot bridge

With no bridge on em0

Code:

                            HOST                     Bridge vm-public (VLAN 10)
                 +--------------------------+   +---------------+
                 |                          |   |               |
                 |                          |   |         tap 2 | ------ vmnet-home-assistant-0-public -- VM home-assistant
                 |  +-------------------+   |   |               |                                         |
     LAN TRUNK   |  |  em0              |---+---+ em0.10        |                                         --> enp0s5f0
<----------------|--|  em0.10 (LAN)     |   |   |         tap 0 | --|
                 |  |  em0.30 (IoT)     |   |   |         tap 1 | --|---> 2 other VMs
                 |  +-------------------+   |   +---------------+
                 |                       \  |
                 +------------------------\-+
                                           \
                                            \    Bridge  vm-ha-iot (VLAN 30)
                                             \  +---------------+
                                              \ |               |
                                               \| em0.30        |
                                                |          tap3 | ------ vmnet-home-assistant-1-ha-iot -- VM home-assistant
                                                |               |                                         |
                                                +---------------+                                         --> enp0s5f1

I started down the road with the "what I have" diagram above because that's how churchers vm-bhyve, which is what I'm using for my VM management, set it up that way based on the examples to create a 'public' virtual switch (their term) and a VLAN virtual switch.

I'd have to route on my router (opnSense) between untagged and VLAN10 (LAN) then, right? Of course I could just move most everything else onto VLAN10 as well, leave untagged as having almost no traffic at all.

sol289 · Nov 20, 2020

Drizzt321 said:
How do you mean detrunk? I have all my normal network/internet traffic (this is my NAS/home server) on em0, untagged. I'm trying to setup IoT stuff, and want it isolated, thus VLAN.

Your "normal network traffic" goes in unspoken VLAN 1 (which is default native VLAN for untagged traffic). So traffic ... on em0, untagged is better read as traffic of native VLAN in trunk port connected to em0 (which is by definition native is untagged).

And you should beware of native traffic handling in FreeBSD, when you're using trunk, VLANs and bridges. If your switch is capable to configure native VLAN on trunk, do so, configure some dummy VLAN as native (like 999) and do not use it. Then VLAN 1 will be tagged in your trunk, and then you can decouple traffic in VLAN 1 as other VLANs to separate interface.

But maybe it's complicated for you, or your switch don't have native VLAN configuration (or you don't have no switch at all), then - yes, choose another VLAN for you normal traffic, like VLAN 10.

Code:

                                 ROUTER                                           HOST
               +----------------------------------------+               +--------------------------+
               |                                        |               |                          |
               |              +----------------------+  |               |                          |
 INTERNET      |              |     em1              |  |               |  +-------------------+   |
               | em0: DHCP    |                      |  |   LAN TRUNK   |  |   em0             |   |
<-----------------+           |  Vlan10: 192.168.1.1 +---------------------+  em0.10 (LAN)     |   |
               |              |  vlan30: 10.1.1.1    |  |               |  |  em0.30 (IoT)     |   |
               |              |                      |  |               |  +-------------------+   |
               |              +----------------------+  |               |                          |
               +----------------------------------------+               +--------------------------+

Drizzt321 · Nov 20, 2020

I've got Ubiquiti Unifi US-8-60 8 port switches.

I'll revisit this tomorrow, need to get to bed. I'll work to get a more exhaustive diagram of what the setup is, what I've tried, what's working, what's not.

[VLAN] Can't get jail/bhyve in vlan working

pprocacci

pprocacci

sol289

Solved - Bridge/epair not passing through tagged VLAN traffic between host and VNET jail

VladiBG

genneko

pprocacci

genneko

SirDice

Administrator

genneko

pprocacci

pprocacci

SirDice

Administrator

sol289

vonbeitthia

Drizzt321

sol289

Drizzt321

sol289

Drizzt321