Solved Windows 10 bhyve VM loses network after 1-3 seconds

subnetspider

subnetspider

Hello everyone,

I am trying to get Windows 10 to work on my FreeBSD bhyve Test server, but for some strage reason, it just doesn't want to work.
The problem I am facing is, that after Windows starts the network interface, the VM completely loses network access after 1-3 seconds.

Context

The Windows 10 VM (10.3.70.205) is connected with it's e1000 tap interface tap1 to the public switch, that is using bridge70 which is connected to vlan70.
On the same public switch, there is a also Debian 12 VM (10.3.70.203) and FreeBSD 14.3 VM (10.3.70.201), which both can reach the Internet and the rest of the LAN.

The bhyve host is connected to a stack of switches via LACP, which is represented as the lagg0 interface, where vlan70 is connected to.
All VMs receive their IP addresses via DHCP on a OPNsense Firewall, which has one interface that is connected to VLAN 70 on the same switch as the bhyve host.
This FreeBSD bhyve host now runs 15.0-RELEASE-p1, upgrading it from 14.2-RELEASE-p2 didn't help.
It uses a Chelsio T520-CR dual port SFP+ 10Gbit/s network card, which are connected to two stacked Brocade ICX6450 switches.

Here is a little overview:

[---------------------------FreeBSD bhyve host----------------------------------------]+++++++++++++++[------------------OPNsense Firewal----------------------]
[Windows 10 VM]---[tap1]---[public / bridge70]---[vlan70]---[lagg0]---[cxl0 + cxl1]---[Network Switch]---[igb0 + igb1]---[lagg0]---[vlan70]---[OPNsense Firewall]

When I tcpdump the traffic of the Windows 10 tap1 interface on the bhyve host, I can clearly see that the tap interface sends and receives traffic:
admin@srv01.local:~ % doas tcpdump ether host 58:9c:fc:00:d2:5c -i tap1 and host 10.3.70.203
...
14:31:21.935693 IP 10.3.70.205 > 10.3.70.203: ICMP echo request, id 1, seq 82, length 40
14:31:21.936010 IP 10.3.70.203 > 10.3.70.205: ICMP echo reply, id 1, seq 82, length 40
14:31:26.739050 ARP, Request who-has 10.3.70.203 (58:9c:fc:00:0f:7c (oui Unknown)) tell 10.3.70.205, length 28
14:31:26.739189 ARP, Reply 10.3.70.203 is-at 58:9c:fc:00:0f:7c (oui Unknown), length 28
14:31:26.754852 IP 10.3.70.205 > 10.3.70.203: ICMP echo request, id 1, seq 83, length 40
14:31:26.754957 IP 10.3.70.203 > 10.3.70.205: ICMP echo reply, id 1, seq 83, length 40
14:31:27.192884 ARP, Request who-has 10.3.70.205 tell 10.3.70.203, length 28
...

For some reason, the Windows 10 VM stops receiving the return traffic, and it completely loses network access.
After this point, I can still see traffic on the tap interface, but it's almost completely ARP and NDP traffic, which never arrives:

admin@srv01.local:~ % doas tcpdump ether host 58:9c:fc:00:d2:5c -i tap1
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on tap1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:55:52.755311 ARP, Request who-has 10.3.70.203 tell 10.3.70.205, length 28
14:55:52.755501 ARP, Reply 10.3.70.203 is-at 58:9c:fc:00:0f:7c (oui Unknown), length 28
14:55:53.301764 ARP, Request who-has 10.3.70.1 tell 10.3.70.205, length 28
14:55:53.302559 ARP, Reply 10.3.70.1 is-at 00:00:5e:00:01:0d (oui IANA), length 42
14:55:53.739076 ARP, Request who-has 10.3.70.203 tell 10.3.70.205, length 28
14:55:53.739198 ARP, Reply 10.3.70.203 is-at 58:9c:fc:00:0f:7c (oui Unknown), length 28
14:55:54.239071 ARP, Request who-has 10.3.70.1 tell 10.3.70.205, length 28
14:55:54.239784 ARP, Reply 10.3.70.1 is-at 00:00:5e:00:01:0d (oui IANA), length 42

Some time ago, I had a similar issue with VNET jails, were some of them could only send, but not receive traffic.

At this point, I am out of Ideas, as everything else works without any problems.
Does someone know what could cause this?

Config files

Contents of /etc/rc.conf:
# Network
ifconfig_igb0="inet 10.1.20.59/24"
ifconfig_igb0_ipv6="inet6 accept_rtadv"
cloned_interfaces="lagg0 vlan10 vlan60 vlan70 bridge10 bridge60 bridge70"
ifconfig_cxl0="up -rxcsum"
ifconfig_cxl1="up -rxcsum"
ifconfig_lagg0="laggproto lacp laggport cxl0 laggport cxl1"
ifconfig_vlan10="vlan 10 vlandev lagg0"
ifconfig_vlan60="vlan 60 vlandev lagg0"
ifconfig_vlan70="vlan 70 vlandev lagg0"
ifconfig_bridge10="addm vlan10"
ifconfig_bridge60="addm vlan60"
ifconfig_bridge70="addm vlan70"

# Services
sshd_enable="NO"
openssh_enable="YES"
pflog_enable="YES"
pf_enable="NO"
bastille_enable="NO"

# Bhyve
vm_enable="YES"
vm_dir="/usr/local/bhyve"
vm_list=""
vm_delay="3"

Contents of /usr/local/bhyve/.config/system.conf:
switch_list="public"
type_public="manual"
bridge_public="bridge70"


Contents of /usr/local/bhyve/windows10/windows10.conf:
loader="uefi"
graphics="yes"
xhci_mouse="yes"
cpu="4"
memory="8G"

# put up to 8 disks on a single ahci controller.
# without this, adding a disk pushes the following network devices onto higher slot numbers,
# which causes windows to see them as a new interface
#ahci_device_limit="8"

# ideally this should be changed to virtio-net and drivers installed in the guest
# e1000 works out-of-the-box
network0_type="e1000"
network0_switch="public"

disk0_type="nvme"
disk0_name="disk0.img"

# windows expects the host to expose localtime by default, not UTC
utctime="no"
uuid="02914350-42cb-11f0-8cd3-00262d08da0a"
network0_mac="58:9c:fc:00:d2:5c"
 
Okay, so I just gave up with e1000 and just switched to VirtIO instead, which now works.
No Idea if this is a FreeBSD or Microsoft Problem, but I feel it's probably the latter.

Anyways, this is what I did to solve the problem:

Stop the Windows VM:
doas vm stop windows10


Download the VirtIO drivers:
fetch https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.118-2/virtio-win-0.1.118.iso


Edit the Windows VM configuration:
doas vm configure windows10


Replace network0_type="e1000" with network0_type="virtio-net" and save the changes.

Attach the VirtIO driver iso file and start the Windows VM:
doas vm install windows10 virtio-win-0.1.118.iso


Install the drivers for the VirtIO network interface. Done.
 
You must log in or register to reply here.
Back
Top