Hi,
after decades of using Linux I decided to set up my new vm server on FreeBSD.
I have almost no experience with FreeBSD and utterly failed on getting a Debian 13 guest installed in a bhyve VM because I just can't get the network running with a passed-through VF.
After fiddling with this for days and googling as much as I could, I resign and come here for help.
I am running FreeBSD 15.0-RELEASE on an AsRock Rack X570D4U with an AMD Ryzen 7 5700X and 64 GB ECC RAM.
The NIC in question is an Intel X710 dual-port SFP+ PCIe card in a PCIe 4 x8 slot (connected directly to the CPU, not the chipset).
My Debian 13 guest complains about this:
On the FreeBSD host dmesg tells me
I've stumbled across
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270966
which looks a lot like my issue and is AMD IOMMU-related, but should be fixed be now (?).
My configuration:
I am running bhyve with
I checked my BIOS settings, AMD-SVM, IOMMU, SR-IOV, ARI and the other suspects should be enabled.
What may I be overlooking or missing here?
Any help would be much appreciated!
Peter
after decades of using Linux I decided to set up my new vm server on FreeBSD.
I have almost no experience with FreeBSD and utterly failed on getting a Debian 13 guest installed in a bhyve VM because I just can't get the network running with a passed-through VF.
After fiddling with this for days and googling as much as I could, I resign and come here for help.
I am running FreeBSD 15.0-RELEASE on an AsRock Rack X570D4U with an AMD Ryzen 7 5700X and 64 GB ECC RAM.
The NIC in question is an Intel X710 dual-port SFP+ PCIe card in a PCIe 4 x8 slot (connected directly to the CPU, not the chipset).
My Debian 13 guest complains about this:
Code:
iavf 0000:00:06.0: Admin queue command never completedOn the FreeBSD host dmesg tells me
Code:
ivhd0: EVT INTR 1 Status:0x1a EVT Head:0x10 Tail:0x20]
[CMD Total 0x5b] Tail:0x5b0, Head:0x5b0.
ivhd0: [Event0: Head:0x10 Tail:0x20]
[IO_PAGE_FAULT EVT: devId:0x2d10 DomId:0x3 Addr:0x1086fc000 0x0]
ivhd0: EVT INTR 2 Status:0x1a EVT Head:0x20 Tail:0x30]
[CMD Total 0x67] Tail:0x670, Head:0x670.
ivhd0: [Event0: Head:0x20 Tail:0x30]
[IO_PAGE_FAULT EVT: devId:0x2d10 DomId:0x4 Addr:0x1a02ce000 0x0]I've stumbled across
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270966
which looks a lot like my issue and is AMD IOMMU-related, but should be fixed be now (?).
My configuration:
Code:
# cat /etc/rc.conf
hostname="bumblebee"
keymap="de.kbd"
ifconfig_ixl0="DHCP"
#ifconfig_ixl0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
ntpd_enable="YES"
powerd_enable="YES"
moused_nondefault_enable="NO"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
iovctl_files="/etc/iov/ixl0.conf"
# Power saving
performance_cx_lowest="Cmax"
economy_cs_lowest="Cmax"
Code:
# cat /boot/loader.conf
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
zfs_load="YES"
vmm_load="YES"
pptdevs="43/0/0 38/0/0 39/0/0"
hw.vmm.amdvi.enable=1
Code:
# cat /etc/iov/ixl0.conf
PF {
device : "ixl0";
num_vfs : 2;
}
DEFAULT {
allow-set-mac : true;
}
VF-0 {
mac-addr : "xx:xx:xx:xx:xx:f0"; [mac edited]
passthrough : true;
}
VF-1 {
mac-addr : "xx:xx:xx:xx:xx:f1"; [mac edited]
passthrough: true;
}
Code:
# pciconf -l
hostb0@pci0:0:0:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1480 subvendor=0x1849 subdevice=0x1480
amdviiommu0@pci0:0:0:2: class=0x080600 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1481 subvendor=0x1022 subdevice=0x1481
hostb1@pci0:0:1:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1482 subvendor=0x0000 subdevice=0x0000
pcib1@pci0:0:1:2: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x1483 subvendor=0x1849 subdevice=0x1453
hostb2@pci0:0:2:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1482 subvendor=0x0000 subdevice=0x0000
hostb3@pci0:0:3:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1482 subvendor=0x0000 subdevice=0x0000
pcib10@pci0:0:3:2: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x1483 subvendor=0x1849 subdevice=0x1453
hostb4@pci0:0:4:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1482 subvendor=0x0000 subdevice=0x0000
hostb5@pci0:0:5:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1482 subvendor=0x0000 subdevice=0x0000
hostb6@pci0:0:7:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1482 subvendor=0x0000 subdevice=0x0000
pcib11@pci0:0:7:1: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x1484 subvendor=0x1022 subdevice=0x1484
hostb7@pci0:0:8:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1482 subvendor=0x0000 subdevice=0x0000
pcib12@pci0:0:8:1: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x1484 subvendor=0x1022 subdevice=0x1484
intsmb0@pci0:0:20:0: class=0x0c0500 rev=0x61 hdr=0x00 vendor=0x1022 device=0x790b subvendor=0x1849 subdevice=0x790b
isab0@pci0:0:20:3: class=0x060100 rev=0x51 hdr=0x00 vendor=0x1022 device=0x790e subvendor=0x1849 subdevice=0x790e
hostb8@pci0:0:24:0: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1440 subvendor=0x0000 subdevice=0x0000
hostb9@pci0:0:24:1: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1441 subvendor=0x0000 subdevice=0x0000
hostb10@pci0:0:24:2: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1442 subvendor=0x0000 subdevice=0x0000
hostb11@pci0:0:24:3: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1443 subvendor=0x0000 subdevice=0x0000
hostb12@pci0:0:24:4: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1444 subvendor=0x0000 subdevice=0x0000
hostb13@pci0:0:24:5: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1445 subvendor=0x0000 subdevice=0x0000
hostb14@pci0:0:24:6: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1446 subvendor=0x0000 subdevice=0x0000
hostb15@pci0:0:24:7: class=0x060000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1447 subvendor=0x0000 subdevice=0x0000
pcib2@pci0:32:0:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x57ad subvendor=0x0000 subdevice=0x0000
pcib3@pci0:33:4:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x57a3 subvendor=0x1849 subdevice=0x57a3
pcib4@pci0:33:5:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x57a3 subvendor=0x1849 subdevice=0x57a3
pcib5@pci0:33:6:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x57a3 subvendor=0x1849 subdevice=0x57a3
pcib7@pci0:33:8:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x57a4 subvendor=0x1022 subdevice=0x1484
pcib8@pci0:33:9:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x57a4 subvendor=0x1022 subdevice=0x1484
pcib9@pci0:33:10:0: class=0x060400 rev=0x00 hdr=0x01 vendor=0x1022 device=0x57a4 subvendor=0x1022 subdevice=0x1484
ppt0@pci0:38:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x1849 subdevice=0x1533
ppt1@pci0:39:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x1849 subdevice=0x1533
pcib6@pci0:40:0:0: class=0x060400 rev=0x04 hdr=0x01 vendor=0x1a03 device=0x1150 subvendor=0x1849 subdevice=0x1150
vgapci0@pci0:41:0:0: class=0x030000 rev=0x41 hdr=0x00 vendor=0x1a03 device=0x2000 subvendor=0x1a03 subdevice=0x2000
none0@pci0:42:0:0: class=0x130000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1485 subvendor=0x1022 subdevice=0x1485
xhci0@pci0:42:0:1: class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1022 device=0x149c subvendor=0x1022 subdevice=0x1486
xhci1@pci0:42:0:3: class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1022 device=0x149c subvendor=0x1022 subdevice=0x148c
ppt2@pci0:43:0:0: class=0x010601 rev=0x51 hdr=0x00 vendor=0x1022 device=0x7901 subvendor=0x1022 subdevice=0x7901
ahci0@pci0:44:0:0: class=0x010601 rev=0x51 hdr=0x00 vendor=0x1022 device=0x7901 subvendor=0x1022 subdevice=0x7901
ixl0@pci0:45:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1572 subvendor=0x8086 subdevice=0x0008
ixl1@pci0:45:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1572 subvendor=0x8086 subdevice=0x0000
none1@pci0:47:0:0: class=0x130000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x148a subvendor=0x1022 subdevice=0x148a
none2@pci0:48:0:0: class=0x130000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1485 subvendor=0x1022 subdevice=0x1485
none3@pci0:48:0:1: class=0x108000 rev=0x00 hdr=0x00 vendor=0x1022 device=0x1486 subvendor=0x1022 subdevice=0x1486
xhci2@pci0:48:0:3: class=0x0c0330 rev=0x00 hdr=0x00 vendor=0x1022 device=0x149c subvendor=0x1849 subdevice=0x149c
ppt3@pci0:45:0:16: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x154c subvendor=0x8086 subdevice=0x0000
ppt4@pci0:45:0:17: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x154c subvendor=0x8086 subdevice=0x0000I am running bhyve with
Code:
bhyve -AHPSDu \
-c cpus=4,sockets=1,cores=2,threads=2 \
-m 6144M \
-s 0,amd_hostbridge \
-s 2,nvme,/dev/zvol/zroot/vm/nas \
-s 6,passthru,45/0/16 \
-s 8,ahci-cd,/mnt/usb/debian-13.3.0-amd64-netinst.iso \
-s 28,fbuf,tcp=0.0.0.0:5900,w=1024,h=768,wait \
-s 29,xhci,tablet \
-s 30,virtio-rnd \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd,/vm/nas/BHYVE_UEFI_VARS.fd \
nasI checked my BIOS settings, AMD-SVM, IOMMU, SR-IOV, ARI and the other suspects should be enabled.
What may I be overlooking or missing here?
Any help would be much appreciated!
Peter