bhyve pci passthrough

I just wasted some time on trying to get pci-passthrough to work. I tried a lot, and configured the stuff according to the wiki. pciconf -lv shows
Code:
ppt1@pci0:3:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet

and trying to run a system with "sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 2048M -p "3/0/0" -d disk0.img -w freebsd" just aborts while booting with:
Code:
Loading kernel...
/boot/kernel/kernel text=0x168fdf1 data=0x1d0a68+0x768d80 syms=[0x8+0x178bc0+0x8+0x1969d5]
Loading configured modules...
/boot/entropy size=0x1000
bhyve: PCI device at 3/0/0 is not using the ppt(4) driver
device emulation initialization error: Device busy

tried the same with vm-bhyve, which dies with "bhyve exited with status 4". System is 12.1-RELEASE, and CPU: Intel(R) Celeron(R) CPU J1900 @ 1.99GHz. as mentioned in the wiki, all looks great except for the "acpidump -t | grep DMAR" which does not find anything, but neither on my ryzen 3700X where I also tried to passthrough the network card. Any hints?
 
the system I am talking about has 4 '82583V Gigabit Network Connection' Adapters, one using the host. could it be that the driver being used by the host is somehow interfering?
 
Looking here:

I see no VT-d support for your CPU.

Maybe the Ryzen has IOMMU support?
 
Here is one of my 4 port network cards:
Code:
42857-ppt0@pci0:6:0:0:    class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
42939-    vendor     = 'Intel Corporation'
42976:    device     = '82571EB/82571GB Gigabit Ethernet Controller (Copper)'
43048:    class      = network
43073:    subclass   = ethernet
43099-ppt1@pci0:6:0:1:    class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
43181-    vendor     = 'Intel Corporation'
43218:    device     = '82571EB/82571GB Gigabit Ethernet Controller (Copper)'
43290:    class      = network
43315:    subclass   = ethernet
43341-ppt2@pci0:7:0:0:    class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
43423-    vendor     = 'Intel Corporation'
43460:    device     = '82571EB/82571GB Gigabit Ethernet Controller (Copper)'
43532:    class      = network
43557:    subclass   = ethernet
43583-ppt3@pci0:7:0:1:    class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
43665-    vendor     = 'Intel Corporation'
43702:    device     = '82571EB/82571GB Gigabit Ethernet Controller (Copper)'
43774:    class      = network
43799:    subclass   = ethernet

could it be that the driver being used by the host is somehow interfering?
No. You can pass through whatever you want. Each port does use a seperate PCI address.
So you can pass through one or all four ports.
 
thanks for the reply! here is my pciconf of all 4 ports (now again ppt disabled):
Code:
em0@pci0:1:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em1@pci0:2:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em2@pci0:3:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em3@pci0:4:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet

well, bad it does not work on that machine. but will try on the ryzen, just out of interest.
I wanted to run the host, and 3 vms each with an ip-address of my homeoffice network ... any suggestion how to accomplish a similar thing? maybe a private network and a 1:1 mapping from one of the ips for the vms to the private ip, or a bridge?
 
I wanted to run the host, and 3 vms each with an ip-address of my homeoffice network ... any suggestion how to accomplish a similar thing?
I use DHCP on my network so it is simple.
On the host assign the Intel Interface DHCP.
Then in the VM assign the passed-thru interface DHCP.
To assign each ppt ethernet interface I use plain old bhyve. No helper programs.
I setup each bhyve instance to use UEFI-Bhyve firmware. I use a /etc/rc.local script to start my VM's.
It looks like a tangled mess but I feel comfortable with the method.
Here is a snippet of my script.
Code:
#!/bin/sh
sleep 6
/usr/local/bin/tmux new-session -d -s freebsd1 'exec bhyve -S -c 8 -m 8G -AHP -u -s 0:0,hostbridge -s 1:0,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd1.img -s 7:0,passthru,6/0/0 -s 30:0,xhci,tablet -l com1,stdio -l bootrom,/vm/freebsd/BHYVE_UEFI.fd freebsd1'
sleep 6
/usr/local/bin/tmux new-session -d -s freebsd2 'exec bhyve -S -c 4 -m 4G -AHP -u -s 0:0,hostbridge -s 1:0,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd2.img -s 7:0,passthru,6/0/1 -s 30:0,xhci,tablet -l com1,stdio -l bootrom,/vm/freebsd/BHYVE_UEFI.fd freebsd2'
So I use tmux to execute my bhyve instances.
I use four amd64 VM's and one i386 VM. There is no i386 UEFI so that one has a bit more to it.

I really think it is wise to learn raw bhyve with no helpers. It will give you a good foundation to build off of.
There are great examples in the handbook. It is not that hard.
So skip the tap and bridge crap. With passthru it is not needed.
Also skip the vmrun.sh method and start bhyve from the command prompt.
Then you can use command line history to build your startup script.

One word of advice. You can pass through 2 or more disks/images to a VM.
The VM will always boot off the first image presented.
So for initial setup use image #1 as the FreeBSD installer image(memstick image) and make image #2 the new empty image for a VM.

I like to make a /vm directory to hold all my VM's and installers.
I also copy the UEFI-bhyve firmware file to this directory for a shorter file path..(BHYVE_UEFI.fd)
 
I am a GUI queen so I also have Virt-Manager running on one experimental box.
This is harder as it requires XML file for setup of libvirtd and Bhyve.
Here are some quick screenshots.
screenshot14.png
 
Last edited:
cool, thanks! Did not expect virt-manager works with bhyve ... I have used it for years in all my corporate linux environments. Do you have a suggestion for the machine where passthrough is not possible to have a similar working setup? Is there a kind of "map-interface-on-host-to-virtual-interface-in-vm"? (looking for a more performant and elegant solution than bridging which comes to my mind)
 
I had the same error, with the following in my vmname.conf for vm-bhyve:
...
passthru0="2/0/0=00:08.0"

Result: Exit status 4

Changed to:
passthru0="2/0/0=08.0"

That change, along with having:
pptdevs="2/0/0"
in /boot/loader.conf works for me.

Good luck.
 
Which software are you using to get Bhyve with GUI? It looks pretty cool :)
Comparing the terminal ...
Code:
root@freebsd:~ # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC  AUTOSTART  STATE
centos7        default    grub       2    2048M   -    No         Stopped
debian-docker  default    grub       2    1024M   -    No         Stopped
freebsd12      default    bhyveload  1    1024M   -    No         Stopped
ubuntu16       default    grub       2    2048M   -    No         Stopped
Cheers
 
Bhyve GUI with libvirtd requires this:
Install devel/libvirt

Configure libvirt for bhyve by making an XML file.
Start up bhyve and test your configuration with virsh.

Then you can install the GUI facility on your host anywhere you want and log into Virtual Machine Manager.
deskutils/virt-manager
From here you can now create new VM instances and manage them with a GUI.
 
Last edited:
I just came here again via searching the internet for the same error, just to note for future users: if nmdm kernel module is not loaded and you want to use vm-bhyve with an uefi VM with vnc graphics it stops with "bhyve exited with status 4". So just load the kernel module before. Time travelling is great, I will probably stumble upon this in a few months again haha
 
Back
Top