general/other VT-D supporting CPU and chipset not able to use VT-D

[soupbowl]

HP Z420 Workstation: https://img2.insight.com/graphics/uk/media/pdf/hp-workstation-z420-datasheet.pdf
FreeBSD 13,
CPU: E5-1620 V3,
Chip set: Intel C612


Hello,

I have an HP Z420 workstation, I have VT-X and VT-D enabled in the UEFI, I installed FreeBSD 13 in UEFI mode.
dmesg shows that I have VT-X support working but nothing about VT-D, any input on how to troubleshoot this?
I did see people having issues with bhyve passthrough using intel c6xx chip sets but it was patched in an earlier version of
FreeBSD, I am not sure if that is related.


Dmesg shows nothing to do with VT-d

# dmesg
CPU: Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz (3491.99-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306f2  Family=0x6  Model=0x3f  Stepping=2
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Structured Extended Features=0x37ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,NFPUSG>
  Structured Extended Features3=0x9c000400<MD_CLEAR,IBPB,STIBP,L1DFL,SSBD>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics

acpidump returns nothing for DMAR support

# acpidump -t | grep DMAR

Thanks for looking!

 

[grehan@]

Without a DMAR table, bhyve isn't able to locate the IOMMU h/w or determine details about what busses are connected etc.

It does seem strange that the BIOS allows that to be enabled, but then doesn't generate the table

 

[sko]

Running 12.2 on a Supermicro X11DPU /w C621 chipset (seems to be reported as C610) where I already tested passthrough and confirmed it to be working:

# uname -a
FreeBSD srv1 12.2-RELEASE-p3 FreeBSD 12.2-RELEASE-p3 GENERIC  amd64

# lspci | grep chipset
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)
00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05)
00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)

# # acpidump -t | grep DMAR
  DMAR: Length=384, Revision=1, Checksum=159,

But dmesg doesn't explicitly show VT-D either:

# dmesg | grep -i vt-
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr

Same goes for several other systems (running 12.2-R and 13.0-R) where VT-D is enabled - so it seems VT-d isn't reported explicitly by dmesg. But from my (very basic) understanding, the DMAR table is the important indicator and requirement for VT-d to work.

Have a look at the specs of the HP board/system - like Dell they often disable some features the chipset/CPU are perfectly capable of and only enable them on higher spec variants (e.g. like PCIe bifurcation on older Dell servers...)

 

[soupbowl]

Thank you for the info! I tried windows on a spare HDD and VT-D does work, I guess this is a problem with the HP firmware ACPI tables or how FreeBSD interprets them.

 

[kr0m]

I'm in the same situation, I have a CPU that supports VT-d, but DMAR is empty and vm-bhyve complains about it:

dmesg
CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz (4007.99-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
AMD Features2=0x121<LAHF,ABM,Prefetch>
Structured Extended Features=0x29c6fbf<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
Structured Extended Features3=0xbc002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
IA32_ARCH_CAPS=0xc04<RSBA>
VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
TSC: P-state invariant, performance statistics

dmidecode:
Base Board Information
Manufacturer: ASUSTeK COMPUTER INC.
Product Name: MAXIMUS VIII HERO
Version: Rev 1.xx
Serial Number: 160469690300850
Asset Tag: Default string
Features:
Board is a hosting board
Board is replaceable
Location In Chassis: Default string
Chassis Handle: 0x0003
Type: Motherboard
Contained Object Handles: 0

tail -f /zroot/vm/ubuntu-cloud/vm-bhyve.log
Oct 25 21:58:36: fatal; pci passthrough not supported on this system (no VT-d or amdvi)

I checked that virtualization is enabled in BIOS and the BIOS version is updated to last available.
In my rc.conf I have microcode_update I don't know if it's related to the topic, but just in case:

microcode_update_enable="YES"

OS/software versions:

uname -a
FreeBSD Garrus.alfaexploit.com 13.2-RELEASE-p4 FreeBSD 13.2-RELEASE-p4 GENERIC amd64

freebsd-version -kru
13.2-RELEASE-p4
13.2-RELEASE-p4
13.2-RELEASE-p4

pkg info|grep bhyve
bhyve-firmware-1.0_1 Collection of Firmware for bhyve
edk2-bhyve-g202308_3 EDK2 Firmware for bhyve
grub2-bhyve-0.40_10 Grub-emu loader for bhyve
uefi-edk2-bhyve-csm-0.2_4,1 UEFI EDK2 firmware for bhyve with CSM (16-bit BIOS)
vm-bhyve-1.5.0 Management system for bhyve virtual machines

Any idea?

 

[kr0m]

I have found the solution, the VT-d is in another configuration part of the BIOS:

Advanced -> System Agent Configuration -> VT-D: Enabled

Once enabled, I get DMAR output and vm-bhyve doesn't complain about it.