Bhyve passthrough Ryzen x370

pestslent1

pestslent1

Hope your not sick of my questions yet.
I have bhyve up and running on my "FreeBSD 11.2 rc3 running kde5 desktop with: "ryzen 7 1700 (overclocked to 3700Mhz, if that matters) 32 GB of ram on a MSI x370 Gaming Plus motherboard with the latest bios". I have a windows 2012r2 vm (config below) up and running, it has 4 of my physical hard-drives passed through, all is working. I am trying to passthrough the below USB 3 controller.

Code: 
ppt0@pci0:40:0:0:       class=0x0c0330 card=0x7a331462 chip=0x21421b21 rev=0x00 hdr=0x00
    vendor     = 'ASMedia Technology Inc.'
    class      = serial bus
    subclass   = USB
    cap 05[50] = MSI supports 8 messages, 64 bit 
    cap 01[78] = powerspec 3  supports D0 D3  current D0
    cap 10[80] = PCI-Express 2 legacy endpoint max data 128(512) RO NS
                 link x2(x2) speed 8.0(8.0) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
    ecap 0019[200] = PCIe Sec 1 lane errors 0
    ecap 0018[300] = LTR 1
It shows up:
Code: 
vm passthru
DEVICE     BHYVE ID     READY        DESCRIPTION
hostb0     0/0/0        No           Family 17h (Models 00h-0fh) Root Complex
none0      0/0/2        No           Family 17h (Models 00h-0fh) I/O Memory Management Unit
hostb1     0/1/0        No           Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
pcib1      0/1/1        No           Family 17h (Models 00h-0fh) PCIe GPP Bridge
pcib2      0/1/3        No           Family 17h (Models 00h-0fh) PCIe GPP Bridge
hostb2     0/2/0        No           Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
hostb3     0/3/0        No           Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
pcib13     0/3/1        No           Family 17h (Models 00h-0fh) PCIe GPP Bridge
hostb4     0/4/0        No           Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
hostb5     0/7/0        No           Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
pcib14     0/7/1        No           Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
hostb6     0/8/0        No           Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge
pcib15     0/8/1        No           Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B
none1      0/20/0       No           FCH SMBus Controller
isab0      0/20/3       No           FCH LPC Bridge
hostb7     0/24/0       No           Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0
hostb8     0/24/1       No           Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1
hostb9     0/24/2       No           Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2
hostb10    0/24/3       No           Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3
hostb11    0/24/4       No           Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4
hostb12    0/24/5       No           Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5
hostb13    0/24/6       No           Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6
hostb14    0/24/7       No           Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7
nvme0      1/0/0        No           NVMe SSD Controller SM961/PM961
xhci0      3/0/0        No           -
ahci0      3/0/1        No           -
pcib3      3/0/2        No           -
pcib4      22/0/0       No           300 Series Chipset PCIe Port
pcib5      22/1/0       No           300 Series Chipset PCIe Port
pcib6      22/2/0       No           300 Series Chipset PCIe Port
pcib7      22/3/0       No           300 Series Chipset PCIe Port
pcib8      22/4/0       No           300 Series Chipset PCIe Port
pcib9      22/5/0       No           300 Series Chipset PCIe Port
pcib10     22/6/0       No           300 Series Chipset PCIe Port
pcib11     22/7/0       No           300 Series Chipset PCIe Port
pcib12     22/8/0       No           300 Series Chipset PCIe Port
re0        33/0/0       No           RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
ahci1      36/0/0       No           88SE9123 PCIe SATA 6.0 Gb/s controller
atapci0    36/0/1       No           88SE912x IDE Controller
siis0      38/0/0       No           SiI 3132 Serial ATA Raid II Controller
ppt0       40/0/0       Yes  
vgapci0    41/0/0       No           Ellesmere [Radeon RX 470/480/570/570X/580/580X]
hdac0      41/0/1       No           Ellesmere [Radeon RX 580]
none2      42/0/0       No           -
none3      42/0/2       No           Family 17h (Models 00h-0fh) Platform Security Processor
xhci1      42/0/3       No           Family 17h (Models 00h-0fh) USB 3.0 Host Controller
none4      43/0/0       No           -
ahci2      43/0/2       No           FCH SATA Controller [AHCI mode]
hdac1      43/0/3       No           Family 17h (Models 00h-0fh) HD Audio Controller

acpidump -t | grep DMAR shows nothing, I have IOMMU enabled in bios or uefi.
Code: 
Jun 27 08:10:31: fatal; pci passthrough not supported on this system (no VT-d) is listed in the "vm-bhyve.log"
My loader.conf
Code: 
kern.vty=vt
zfs_load="YES"
fuse_load="YES"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
vmm_load="YES"
nmdm_load="YES"
if_bridge_load="YES"
if_tap_load="YES"
hw.syscons.disable="1"
hw.vmm.amdvi.enable="1"
hw.vmm.topology.cores_per_package=4
#hw.vmm.topology.threads_per_core=2
pptdevs="40/0/0"

My VM config file:
Code: 
uefi="yes"
cpu=4
memory=6G
graphics="yes"
xhci_mouse="yes"
graphics_res="1600x900"
network0_type="virtio-net"
network0_switch="public"
#ahci_device_limit="6"
debug="yes"
disk0_opts="sectorsize=512"
disk0_type="ahci-hd"
disk0_name="BackupR7.img"
disk1_dev="custom"
disk1_opts="sectorsize=512"
disk1_type="ahci-hd"
disk1_name="/dev/ada0,hd:/dev/ada1"
disk2_dev="custom"
disk2_opts="sectorsize=512"
disk2_type="ahci-hd"
disk2_name="/dev/ada2,hd:/dev/ada3"
uuid="762f78e9-270b-11e8-87a4-309c230c9abc"
network0_mac="02:15:5d:45:32:01"
passthru0="40/0/0"

Did I forget or mis-configure something?

Not sure if it matters, the IOMMU was set to "auto" in the bios/uefi when I installed the OS.
It is now set to "enable"
 
Last edited by a moderator:
Oh sorry, seems like the check in vm-bhyve is done in a very awkward way. Try hw.vmm.amdvi instead.
It seems you like have enable set in loader.conf although I would expect it to be read-only.

Ok scratch all that. Looking at vmm source it seems amdvi.enable is a user setting. I assumed this was set by the system to indicate amdvi support but that doesn't seem to be the case.

Anyway, vm-bhyve-1.2 checks for a DMAR table or this sysctl which would likely avoid the "not supported" error on your system. That check isn't in the current ports version though. You could try downloading a copy of 1.2 or master branch from GitHub to see if that works.
 
Well that's strange. I would expect you to have hw.vmm.amdvi entries in the sysctl list if you have an AMD cpu with vt-d support, especially as you seem to be setting them in loader.conf
 
usdmatt said:
Oh sorry, seems like the check in vm-bhyve is done in a very awkward way. Try hw.vmm.amdvi instead.
It seems you like have enable set in loader.conf although I would expect it to be read-only.

Ok scratch all that. Looking at vmm source it seems amdvi.enable is a user setting. I assumed this was set by the system to indicate amdvi support but that doesn't seem to be the case.

Anyway, vm-bhyve-1.2 checks for a DMAR table or this sysctl which would likely avoid the "not supported" error on your system. That check isn't in the current ports version though. You could try downloading a copy of 1.2 or master branch from GitHub to see if that works.
Click to expand...

Thanks for the info. Will I have to remove the present version and start from scratch or will it upgrade what I already have?

I read a thread and forgot to bookmark it, it said something about commenting out the check for the IOMMU (or whatever it checks for that is stopping me) in a bhyve start up or config file. I wanted to try it but cannot remember what file it said to alter or where it is. Would you know by what I stated?
 
You could try removing the check for iommu. The newer versions check for vmm.amdvi but if you don't have that sysctl either, the check will still fail even if you install a newer copy.

There's a function called util::check_bhyve_iommu in /usr/local/lib/vm-bhyve/vm-util that tries to verify IOMMU support. Just stick a return 0 at the top of that function.
I'd be intrigued to see if pass-through works on your system if you don't have any of the amdvi sysctls.
 
Nope, Bhyve log:
Code: 
rdmsr to register 0xc0010114 on vcpu 0
rdmsr to register 0xc0010114 on vcpu 1
wrmsr to register 0x10(0) on vcpu 1
rdmsr to register 0xc0010114 on vcpu 2
wrmsr to register 0x10(0) on vcpu 2
rdmsr to register 0xc0010114 on vcpu 3
wrmsr to register 0x10(0) on vcpu 3
wrmsr to register 0x10(0x17431a3d2c4) on vcpu 3
wrmsr to register 0x10(0x17431a3d2c4) on vcpu 0
wrmsr to register 0x10(0x17431a3d2c4) on vcpu 1
wrmsr to register 0x10(0x17431a3d2c4) on vcpu 2
rdmsr to register 0xc0010114 on vcpu 0
rdmsr to register 0xc0010114 on vcpu 1
wrmsr to register 0x10(0) on vcpu 1
rdmsr to register 0xc0010114 on vcpu 2
wrmsr to register 0x10(0) on vcpu 2
rdmsr to register 0xc0010114 on vcpu 3
wrmsr to register 0x10(0) on vcpu 3
wrmsr to register 0x10(0x2fa57b463c) on vcpu 3
wrmsr to register 0x10(0x2fa57b463c) on vcpu 2
wrmsr to register 0x10(0x2fa57b463c) on vcpu 1
wrmsr to register 0x10(0x2fa57b463c) on vcpu 0
fbuf frame buffer base: 0x9c7000000 [sz 16777216]
bhyve: PCI device at 40/0/0 is not using the ppt(4) driver

When I " dmesg | egrep -i 'ivrs|ivh|amd-vi|mmu'"
Code: 
root@vmbsd:/usr/home/pete #   dmesg | egrep -i 'ivrs|ivh|amd-vi|mmu'
pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached)
pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached)
pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached)
amd_iommu_init: not implemented

Can I some how install "vmm.amdvi"?
Is there a log file for loader.conf so I could see the message when it tries to enable it?
I need this in loader.conf "hw.syscons.disable="1"" for my rx580, might that interfere?

anyway, thanks for taking the time to try and assist me, I appreciate it.
 
I wouldn't of thought the syscons sysctl would have any effect. The amdvi entries should appear automatically. I have no experience with pass-through though, I have one machine I can run bhyve on and it doesn't support it so I've no way of trying it myself.

Strange that it says the device is not using the ppt driver.

You'll probably get a more definitive answer from the freebsd-virtualization@freebsd.org mailing list.
 
usdmatt said:
I wouldn't of thought the syscons sysctl would have any effect. The amdvi entries should appear automatically. I have no experience with pass-through though, I have one machine I can run bhyve on and it doesn't support it so I've no way of trying it myself.

Strange that it says the device is not using the ppt driver.

You'll probably get a more definitive answer from the freebsd-virtualization@freebsd.org mailing list.
Click to expand...

Thanks for pointing me to the mailing list. I was helped by "Anish". (helped is a understatement) It seems to work as far as I've tested.

I've included the following in case someone stumbles across this with a similar problem.


He told me to tcopy these 4 files from a install of 12 CURRENT to my running 11.2 . (I installed on a HyperV box)


/usr/src/sys/amd64/vmm/amd/amdvi_hw.c
/usr/src/sys/amd64/vmm/amd/amdvi_priv.h
/usr/src/sys/amd64/vmm/amd/ivrs_drv.c
/usr/src/sys/modules/vmm/Makefile


run make -C /usr/src/sys/modules/vmm

make -j8 buildkernel installkernel

I still get

Code: 
root@vmbsd:/usr/home/pete # acpidump -t -v | grep DMAR
acpidump: loading RSD PTR from /dev/mem
acpidump: printing various SDT tables
root@vmbsd:/usr/home/pete #

And a error in my bhyve.log "PCI device at 40/0/0 is not using the ppt(4) driver"

The rest seems ok

Code: 
root@vmbsd:/usr/home/pete # sysctl hw.vmm.amdvi
hw.vmm.amdvi.domain_id: 0
hw.vmm.amdvi.disable_io_fault: 0
hw.vmm.amdvi.ptp_level: 4
hw.vmm.amdvi.host_ptp: 1
hw.vmm.amdvi.enable: 1
hw.vmm.amdvi.count: 1

Code: 
root@vmbsd:/usr/home/pete # dmesg | egrep -i 'ivrs | ivh | and-vi | mmu' 
AMD-Vi: IVRS Info VAsize = 64 PAsize = 48 GVAsize = 2 flags:0
AMD-Vi: IVRS Info VAsize = 64 PAsize = 48 GVAsize = 2 flags:0
AMD-Vi: IVRS Info VAsize = 64 PAsize = 48 GVAsize = 2 flags:0

pciconf -v -l
Shows ppt
Code: 
ppt0@pci0:40:0:0:       class=0x0c0330 card=0x7a331462 chip=0x21421b21 rev=0x00 hdr=0x00
    vendor     = 'ASMedia Technology Inc.'
    class      = serial bus
    subclass   = USB

vm passthru
Shows (I think this is the only relevant line
Code: 
ppt0       40/0/0       Yes          -

loader.conf
Code: 
kern.vty=vt
zfs_load="YES"
fuse_load="YES"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
vmm_load="YES"
nmdm_load="YES"
if_bridge_load="YES"
if_tap_load="YES"
hw.vmm.amdvi.enable=1
hw.vmm.topology.cores_per_package=4
#hw.vmm.topology.threads_per_core=2
pptdevs="40/0/0"
hw.syscons.disable="1"

rc.conf
Code: 
zfs_enable="YES"
clear_tmp_enable="YES"
hostname="vmbsd"
ifconfig_re0="DHCP"
ifconfig_re0="inet 192.168.1.69 netmask 255.255.255.0"
defaultrouter="192.168.1.1"
sendmail_enable="NONE"
kld_list="/boot/modules/amdgpu.ko"
dbus_enable="YES"
hald_enable="YES"
sddm_enable="YES"
xrdp_enable="YES"
xrdp_sesman_enable="YES"
samba_server_enable="YES"
fuse_enable="YES"
vm_enable="YES"
vm_dir="/mnt/bhyve/HyperV"
vm_list="2012r2 vistadl 7bubsd"
vm_delay="10"
 
It would appear I hit a snag, The USB passthru works great but, when I have "passthru0="40/0/0" in my VM config it makes the 4 physical hard drives I have passing thru show up as Not Initialized in my VM disk management (2012r2). When I comment out the "passthru0="40/0/0" in my config, the drives return to Initialized with all data but, no USB.

My Config:

Code: 
uefi="yes"
cpu=4
memory=6G
graphics="yes"
xhci_mouse="yes"
graphics_res="1600x900"
network0_type="virtio-net"
network0_switch="public"
#ahci_device_limit="8"
debug="yes"
disk0_opts="sectorsize=512"
disk0_type="ahci-hd"
disk0_name="BackupR7.img"
disk1_dev="custom"
disk1_opts="sectorsize=512"
disk1_type="ahci-hd"
disk1_name="/dev/ada0,hd:/dev/ada1"
disk2_dev="custom"
disk2_opts="sectorsize=512"
disk2_type="ahci-hd"
disk2_name="/dev/ada2,hd:/dev/ada3"
uuid="762f78e9-270b-11e8-87a4-309c230c9abc"
network0_mac="02:15:5d:45:32:01"
passthru0="40/0/0"

Any Ideas?
 
Hi pestslent1. I wish I saw this a month ago, I could have helped you right away. For anyone else stumbling across this thread, the above in plain english: amdvi is not in the RELEASE kernel yet. You either have to patch your kernel like above, or run CURRENT. It is not in STABLE. Here is the commit. pestslent1 : can I ask what version of vm-bhyve you have? the newest, which supports amdvi, should say "(no VT-d or amdvi)" to clue that amdvi indeed isn't present. Its probably not in ports yet.

As for your latest problem, you're passing those disks weird. why not 'disk1_name="/dev/ada0"'? Whatever you're doing there I've not seen before. Also, I see you have quite a few disk controllers plugged into that machine. Move some cards around, perhaps some slots just don't play nice with the usb controller. I've had an integrated usb controller share part of its address with stuff and wipe out half the motherboard when i try to pass it through. Though, the raw disk passthrough feature has nothing to do with pci; you should start up that machine with USB passed, and check your HOST to see if any of those disks / controller disappeared. They should'nt. Worst case, just pass through a whole controller, instead of individual disks.
 
I apologize for slow reply, life got in the way.
(1) The version is vm-bhyve 1.2.2. Where would I look for "(no VT-d or amdvi)", I do not see it in pkg info vm-bhyve?
(2) For "As for your latest problem, you're passing those disks weird. why not 'disk1_name="/dev/ada0"'? " If you mean why are there 2 disks per line "disk1_name="/dev/ada0,hd:/dev/ada1"", it was the only way to get all 4 of my disks to show up in the vm. It only shows me two at a time if I list them separately. Is my above config file mis-configured? The 4 drives are all attached to sata ports on the motherboard.
(3) I have, "pptdevs="40/0/0" (my usb controler) in loader.conf. All my drives are there. When I put, "passthru0="40/0/0" in my vm config file, the drives show up but, are listed as need to be initialized in the disk management section server manager in the 2012r2 VM, but the usb works. If i try to initialize the drives in the VM the host and VM freeze.
 
The vm will crash and write "(no VT-d or amdvi)" in its log if neither are present. Yes, the pkg version is still only aware of vt-d.

I'm sorry, I dont know what to think about your disks. I've read mentions about ahci limitations in bhyve, but I don't know enough about it.

If I were you, I would definitely boot that exact VM with a live linux or BSD iso using the "install" command (all install does is attach an iso as the first disk) and have a look at pci, dmesg, and disks to see if there is anything obviously weird going on. Perhaps windows 2012 just plain doesnt like bhyve - I know I had trouble until about windows version 1800+.

You might also have better luck over at the vm-bhyve github issues board.
Just keep in mind, when I said "supports amdvi", I should have said "preemptively allows it".
 
Hey, I'm sorry that have to dig out this really old thread, but right now I'm having exactly the same problem as the op.
When i look through sysctl the amdvi entries are also missing and I can't start vm with the passthru option, though vm-bhyve knows about amd-vi according to the error message.
I am using freenas 11.3 which is based on freebsd 11.3.
I can't really connect the linked commit to a freebsd release, so my question is, which freebsd version do I need so that I can use the amd vi features with byhve?
 
You must log in or register to reply here.
Back
Top