Trouble with VLANs & VM-Bhyve

vorell

vorell

Backstory:
A long time linux user, I tried FreeBSD on digitalocean a few years back. Now I'm to the point where my entire cloud & home infrastructure is FreebSD based- pfsense, FreeNAS, and a FreeBSD application server.

Server Specifications:
Code: 
Release: FreeBSD 11.1-RELEASE-p4
CPU: Intel(R) Xeon(R) CPU E31230 @ 3.20GHz (3192.82-MHz K8-class CPU)
Memory: 32GB ECC DDR3
Network: 2x Onboard Broadcom GB NICs 
              1x Mellanox Connect-X2 10GB SFP+
Storage: 1TB(4x500GB) SSD RAIDZ10

The Problem:
My application server is the current source of my problem. Networking, right from the get go has been kind of finicky. Wouldn't pull down freebsd-updates, occasionally had to manually restart netif & routing via cli, and more recently the bce0 & bce1 interfaces puked completely.

My original virtual switch setup was as follows:
vm switch list
Code: 
NAME            TYPE       IDENT       VLAN      NAT          PORTS
DMZ            auto       bridge0         128         -            bce1
LAN             auto       bridge1           -           -            mlxen0
VOIP            auto       bridge2         32          -            bce0
NVR             auto       bridge3          -            -            -

Pretty straightforward, I assign the virtual switch a vlan, assign the switch to an interface which is trunked on the physical switch, and then traffic is tagged all the way to the firewall for dhcp & routing.

Since the broadcom interfaces are such a problem I went ahead & disabled them in the BIOS & removed their config from /etc/rc.conf. I then went ahead and recreated all my virtual switches on the one mlxen0 interface like below:
vm switch list
Code: 
NAME            TYPE       IDENT       VLAN      NAT          PORTS
LAN             auto       bridge0          -            -            mlxen0
NVR             auto       bridge1       128          -            mlxen0
VOIP            auto       bridge2        32           -            mlxen0
VPN             auto       bridge3         8            -            mlxen0
GUEST         auto       bridge4        64           -            mlxen0

I've verified that the SFP+ port that mlxen0 is connected to is a trunk port on the physical switch, however any VMs which are supposed to be on a vlan have no networking now. Any that were on the LAN / untagged have networking as expected.

Below is my ifconfig output:
Code: 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
mlxen0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=ad00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6>
        ether 00:02:c9:4f:b4:70
        hwaddr 00:02:c9:4f:b4:70
        inet 192.168.50.50 netmask 0xffffff00 broadcast 192.168.50.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-LAN
        ether 02:73:6f:35:8e:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 0 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
        member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 16 priority 128 path cost 2000000
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000000
        member: mlxen0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 2000
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-NVR
        ether 02:73:6f:35:8e:01
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 0 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 14 priority 128 path cost 2000000
        member: vlan0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
vlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-vlan-mlxen0-128
        options=280001<RXCSUM,LINKSTATE,RXCSUM_IPV6>
        ether 00:02:c9:4f:b4:70
        inet6 fe80::202:c9ff:fe4f:b470%vlan0 prefixlen 64 scopeid 0x5
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        vlan: 128 vlanpcp: 0 parent interface: mlxen0
        groups: vlan
bridge2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-VOIP
        ether 02:73:6f:35:8e:02
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 0 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
        member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 15 priority 128 path cost 2000000
        member: vlan1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000
vlan1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-vlan-mlxen0-32
        options=280001<RXCSUM,LINKSTATE,RXCSUM_IPV6>
        ether 00:02:c9:4f:b4:70
        inet6 fe80::202:c9ff:fe4f:b470%vlan1 prefixlen 64 scopeid 0x7
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        vlan: 32 vlanpcp: 0 parent interface: mlxen0
        groups: vlan
bridge3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-VPN
        ether 02:73:6f:35:8e:03
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 0 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
        member: vlan2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000
vlan2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-vlan-mlxen0-8
        options=280001<RXCSUM,LINKSTATE,RXCSUM_IPV6>
        ether 00:02:c9:4f:b4:70
        inet6 fe80::202:c9ff:fe4f:b470%vlan2 prefixlen 64 scopeid 0x9
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        vlan: 8 vlanpcp: 0 parent interface: mlxen0
        groups: vlan
bridge4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-GUEST
        ether 02:73:6f:35:8e:04
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 0 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 0 ifcost 0 port 0
        member: vlan3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000
vlan3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vm-vlan-mlxen0-64
        options=280001<RXCSUM,LINKSTATE,RXCSUM_IPV6>
        ether 00:02:c9:4f:b4:70
        inet6 fe80::202:c9ff:fe4f:b470%vlan3 prefixlen 64 scopeid 0xb
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        vlan: 64 vlanpcp: 0 parent interface: mlxen0
        groups: vlan
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-unifi-0-LAN
        options=80000<LINKSTATE>
        ether 00:bd:ec:0a:f7:00
        hwaddr 00:bd:ec:0a:f7:00
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        groups: tap
        Opened by PID 42198
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
        ether 00:24:9b:1e:84:73
        hwaddr 00:24:9b:1e:84:73
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (none)
        status: no carrier
tap1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-nvr-0-NVR
        options=80000<LINKSTATE>
        ether 00:bd:ee:1e:f7:01
        hwaddr 00:bd:ee:1e:f7:01
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        groups: tap
        Opened by PID 38873
tap2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-pbx-0-VOIP
        options=80000<LINKSTATE>
        ether 00:bd:b8:33:f7:02
        hwaddr 00:bd:b8:33:f7:02
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        groups: tap
        Opened by PID 72882
tap3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet-dc01-0-LAN
        options=80000<LINKSTATE>
        ether 00:bd:65:47:f7:03
        hwaddr 00:bd:65:47:f7:03
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        groups: tap
        Opened by PID 28005

Any pointers or assistance will be much appreciated. I think what vm-bhyve has done to my ifconfig is a bit ugly but based on my reading- in theory this should be working. One possibility which has occurred to me is the need to make the network interfaces inside of the VMs vlan aware, but since this wasn't a necessary step in the past... why would it be now?

If there's any further command output you'd like posted, let me know.
 
You don't need to make the vm vlan aware.

Did you tell your vm to use the proper interface ?
Code: 
# [MAN=8]vm[/MAN] switch list
NAME            TYPE       IDENT       VLAN      NAT          PORTS
manage          auto       bridge0     10        -            lagg0
internet        auto       bridge1     -         -            ix0
public          auto       bridge2     100       -            lagg0
private         auto       bridge3     200       -            lagg0
phone           auto       bridge4     400       -            lagg0
video           auto       bridge5     300       -            lagg0
machine         auto       bridge6     500       -            lagg0

vm(8) configure test

Code: 
network0_type="virtio-net"
network0_switch="private"

needs to contain which switch it's supposed to use. For example for vlan 200 it is switch private in my case. Don't forget to change your switch ports to TRUNK and add the vlans you need.

A mistake I made was however I had created vlans on the host system. You can't make vlans on the host system if you want to passthrough the vlan to the bridge, if you want to use the vlan on the host system then set the IP to the bridge interface, that should work, not sure if that's recommended however, but remove all vlans from the interface you put on the bridge interface.

Basically mlxen0 can't have any vlans on the host system.
 
You must log in or register to reply here.
Back
Top