bhyve pci passthrough

rootbert

Well-Known Member

Reaction score: 176
Messages: 429

I just wasted some time on trying to get pci-passthrough to work. I tried a lot, and configured the stuff according to the wiki. pciconf -lv shows
Code:
ppt1@pci0:3:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet

and trying to run a system with "sh /usr/share/examples/bhyve/vmrun.sh -c 1 -m 2048M -p "3/0/0" -d disk0.img -w freebsd" just aborts while booting with:
Code:
Loading kernel...
/boot/kernel/kernel text=0x168fdf1 data=0x1d0a68+0x768d80 syms=[0x8+0x178bc0+0x8+0x1969d5]
Loading configured modules...
/boot/entropy size=0x1000
bhyve: PCI device at 3/0/0 is not using the ppt(4) driver
device emulation initialization error: Device busy

tried the same with vm-bhyve, which dies with "bhyve exited with status 4". System is 12.1-RELEASE, and CPU: Intel(R) Celeron(R) CPU J1900 @ 1.99GHz. as mentioned in the wiki, all looks great except for the "acpidump -t | grep DMAR" which does not find anything, but neither on my ryzen 3700X where I also tried to passthrough the network card. Any hints?
 
OP
R

rootbert

Well-Known Member

Reaction score: 176
Messages: 429

the system I am talking about has 4 '82583V Gigabit Network Connection' Adapters, one using the host. could it be that the driver being used by the host is somehow interfering?
 

Phishfry

Beastie's Twin

Reaction score: 2,752
Messages: 5,685

CPU: Intel(R) Celeron(R) CPU J1900
I wonder if your problem is the host. I doubt that a J1900 supports IOMMU
IOMMU is required for PCI passthrough.

I am using two very similar cards with no issues. You must passthrough 4 PCI addresses for them to work with all 4 ports.
Like this:
pptdevs="3/0/0 3/0/1 4/0/0 4/0/1"
That will get you 4 ethernet devices to passthrough.
 

Phishfry

Beastie's Twin

Reaction score: 2,752
Messages: 5,685

Looking here:

I see no VT-d support for your CPU.

Maybe the Ryzen has IOMMU support?
 

Phishfry

Beastie's Twin

Reaction score: 2,752
Messages: 5,685

Here is one of my 4 port network cards:
Code:
42857-ppt0@pci0:6:0:0:    class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
42939-    vendor     = 'Intel Corporation'
42976:    device     = '82571EB/82571GB Gigabit Ethernet Controller (Copper)'
43048:    class      = network
43073:    subclass   = ethernet
43099-ppt1@pci0:6:0:1:    class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
43181-    vendor     = 'Intel Corporation'
43218:    device     = '82571EB/82571GB Gigabit Ethernet Controller (Copper)'
43290:    class      = network
43315:    subclass   = ethernet
43341-ppt2@pci0:7:0:0:    class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
43423-    vendor     = 'Intel Corporation'
43460:    device     = '82571EB/82571GB Gigabit Ethernet Controller (Copper)'
43532:    class      = network
43557:    subclass   = ethernet
43583-ppt3@pci0:7:0:1:    class=0x020000 card=0x11bc8086 chip=0x10bc8086 rev=0x06 hdr=0x00
43665-    vendor     = 'Intel Corporation'
43702:    device     = '82571EB/82571GB Gigabit Ethernet Controller (Copper)'
43774:    class      = network
43799:    subclass   = ethernet

could it be that the driver being used by the host is somehow interfering?
No. You can pass through whatever you want. Each port does use a seperate PCI address.
So you can pass through one or all four ports.
 
OP
R

rootbert

Well-Known Member

Reaction score: 176
Messages: 429

thanks for the reply! here is my pciconf of all 4 ports (now again ppt disabled):
Code:
em0@pci0:1:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em1@pci0:2:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em2@pci0:3:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet
em3@pci0:4:0:0:    class=0x020000 card=0x00008086 chip=0x150c8086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82583V Gigabit Network Connection'
    class      = network
    subclass   = ethernet

well, bad it does not work on that machine. but will try on the ryzen, just out of interest.
I wanted to run the host, and 3 vms each with an ip-address of my homeoffice network ... any suggestion how to accomplish a similar thing? maybe a private network and a 1:1 mapping from one of the ips for the vms to the private ip, or a bridge?
 

Phishfry

Beastie's Twin

Reaction score: 2,752
Messages: 5,685

I wanted to run the host, and 3 vms each with an ip-address of my homeoffice network ... any suggestion how to accomplish a similar thing?
I use DHCP on my network so it is simple.
On the host assign the Intel Interface DHCP.
Then in the VM assign the passed-thru interface DHCP.
To assign each ppt ethernet interface I use plain old bhyve. No helper programs.
I setup each bhyve instance to use UEFI-Bhyve firmware. I use a /etc/rc.local script to start my VM's.
It looks like a tangled mess but I feel comfortable with the method.
Here is a snippet of my script.
Code:
#!/bin/sh
sleep 6
/usr/local/bin/tmux new-session -d -s freebsd1 'exec bhyve -S -c 8 -m 8G -AHP -u -s 0:0,hostbridge -s 1:0,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd1.img -s 7:0,passthru,6/0/0 -s 30:0,xhci,tablet -l com1,stdio -l bootrom,/vm/freebsd/BHYVE_UEFI.fd freebsd1'
sleep 6
/usr/local/bin/tmux new-session -d -s freebsd2 'exec bhyve -S -c 4 -m 4G -AHP -u -s 0:0,hostbridge -s 1:0,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd2.img -s 7:0,passthru,6/0/1 -s 30:0,xhci,tablet -l com1,stdio -l bootrom,/vm/freebsd/BHYVE_UEFI.fd freebsd2'
So I use tmux to execute my bhyve instances.
I use four amd64 VM's and one i386 VM. There is no i386 UEFI so that one has a bit more to it.

I really think it is wise to learn raw bhyve with no helpers. It will give you a good foundation to build off of.
There are great examples in the handbook. It is not that hard.
So skip the tap and bridge crap. With passthru it is not needed.
Also skip the vmrun.sh method and start bhyve from the command prompt.
Then you can use command line history to build your startup script.

One word of advice. You can pass through 2 or more disks/images to a VM.
The VM will always boot off the first image presented.
So for initial setup use image #1 as the FreeBSD installer image(memstick image) and make image #2 the new empty image for a VM.

I like to make a /vm directory to hold all my VM's and installers.
I also copy the UEFI-bhyve firmware file to this directory for a shorter file path..(BHYVE_UEFI.fd)
 

Phishfry

Beastie's Twin

Reaction score: 2,752
Messages: 5,685

I am a GUI queen so I also have Virt-Manager running on one experimental box.
This is harder as it requires XML file for setup of libvirtd and Bhyve.
I use a minimal Xorg and XForwarding on the host to have remote access from my laptops..
Here are some quick screenshots.
screenshot14.png
 
OP
R

rootbert

Well-Known Member

Reaction score: 176
Messages: 429

cool, thanks! Did not expect virt-manager works with bhyve ... I have used it for years in all my corporate linux environments. Do you have a suggestion for the machine where passthrough is not possible to have a similar working setup? Is there a kind of "map-interface-on-host-to-virtual-interface-in-vm"? (looking for a more performant and elegant solution than bridging which comes to my mind)
 

joeafterdinner

New Member


Messages: 4

I had the same error, with the following in my vmname.conf for vm-bhyve:
...
passthru0="2/0/0=00:08.0"

Result: Exit status 4

Changed to:
passthru0="2/0/0=08.0"

That change, along with having:
pptdevs="2/0/0"
in /boot/loader.conf works for me.

Good luck.
 

diego

Member

Reaction score: 34
Messages: 93

Which software are you using to get Bhyve with GUI? It looks pretty cool :)
Comparing the terminal ...
Code:
root@freebsd:~ # vm list
NAME           DATASTORE  LOADER     CPU  MEMORY  VNC  AUTOSTART  STATE
centos7        default    grub       2    2048M   -    No         Stopped
debian-docker  default    grub       2    1024M   -    No         Stopped
freebsd12      default    bhyveload  1    1024M   -    No         Stopped
ubuntu16       default    grub       2    2048M   -    No         Stopped
Cheers
 
Top