I've ran OPNSense in the past virtualized under Proxmox which worked well. I moved OPNSense to a repurposed Checkpoint appliance, and recently moved OPNSense over to be virtualized under FreeBSD 13.1 bhyve.
The problem I am facing is the WAN network will seemingly randomly become unresponsive for a period of time anywhere from 30 seconds to a few minutes. It happens mostly when above 100Mbps but has happened as low as an idle connection. It also seems that packet loss occurs often when loaded up beyond more than a few Mbps (internet is 350/10Mbps). This did not occur in the previously described setups.
It only happens on the WAN port vtnet1 which is a tap evice to em0 while vtnet0 is a tap device on a lagg between 2 mlxen devices (LAN). The modem is connected directly to em0 on the host (vtnet1 in the guest).
I tried using virtio-net and e1000 which made no difference.
VM config:
From /etc/rc.conf:
The problem I am facing is the WAN network will seemingly randomly become unresponsive for a period of time anywhere from 30 seconds to a few minutes. It happens mostly when above 100Mbps but has happened as low as an idle connection. It also seems that packet loss occurs often when loaded up beyond more than a few Mbps (internet is 350/10Mbps). This did not occur in the previously described setups.
It only happens on the WAN port vtnet1 which is a tap evice to em0 while vtnet0 is a tap device on a lagg between 2 mlxen devices (LAN). The modem is connected directly to em0 on the host (vtnet1 in the guest).
I tried using virtio-net and e1000 which made no difference.
VM config:
Code:
tommy@ecorp:~ % cat /vm/bob/bob.conf
loader="bhyveload"
priority=-1
cpu=4
memory=4G
network0_type="virtio-net"
network0_switch="lanxfer"
network1_type="virtio-net"
network1_switch="modem"
disk0_type="nvme"
disk0_name="disk0.img"
uuid="3b52dad1-c916-11ed-a8a9-002590247e86"
network0_mac="58:9c:fc:06:7b:8f"
network1_mac="58:9c:fc:06:b1:1f"
Code:
tommy@ecorp:~ % doas vm swi list
NAME TYPE IFACE ADDRESS PRIVATE MTU VLAN PORTS
lanxfer standard vm-lanxfer - no - 2309 lagg0
modem standard vm-modem - no - - em0
Code:
tommy@ecorp:~ % ifconfig em0
em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4812098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
ether 00:25:90:24:7e:86
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
tommy@ecorp:~ % ifconfig lagg0
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ed00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,L
INKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:02:c9:1c:46:80
laggproto lacp lagghash l2,l3,l4
laggport: mlxen0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: mlxen1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>From /etc/rc.conf:
Code:
ifconfig_em0="up -txcsum -rxcsum -tso4 -tso6 -lro"
ifconfig_mlxen0="up -txcsum -rxcsum -tso -lro"
ifconfig_mlxen1="up -txcsum -rxcsum -tso -lro"