In bhyve, the PCI-usb 2.0 controller is passthrough to the Ubuntu guest

odin32 · Apr 27, 2020

Inserted a pci-usb 2.0 controller into the PC. pci-usb devices in the Freebsd system:
# pciconf -lv | grep -B2 USB

Code:

xhci0@pci0:0:20:0:    class=0x0c0330 card=0x50071458 chip=0x8c318086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB xHCI'
    class      = serial bus
    subclass   = USB
--
ehci0@pci0:0:26:0:    class=0x0c0320 card=0x50061458 chip=0x8c2d8086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB EHCI'
    class      = serial bus
    subclass   = USB
--
ehci2@pci0:0:29:0:    class=0x0c0320 card=0x50061458 chip=0x8c268086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB EHCI'
    class      = serial bus
    subclass   = USB
--
uhci0@pci0:5:0:0:    class=0x0c0300 card=0x30381106 chip=0x30381106 rev=0x61 hdr=0x00
    vendor     = 'VIA Technologies, Inc.'
    device     = 'VT82xx/62xx UHCI USB 1.1 Controller'
    class      = serial bus
    subclass   = USB
uhci1@pci0:5:0:1:    class=0x0c0300 card=0x30381106 chip=0x30381106 rev=0x61 hdr=0x00
    vendor     = 'VIA Technologies, Inc.'
    device     = 'VT82xx/62xx UHCI USB 1.1 Controller'
    class      = serial bus
    subclass   = USB
ehci1@pci0:5:0:2:    class=0x0c0320 card=0x31041106 chip=0x31041106 rev=0x63 hdr=0x00
    vendor     = 'VIA Technologies, Inc.'
    device     = 'USB 2.0'
    class      = serial bus
    subclass   = USB

Device ehci1@pci0:5:0:2: not defined as ppt. When adding the parameter -s 7:0, passthru, 5/0/2 to the bhyve startup script, the script does not start!
I want to throw pci-usb 2.0 into guest Ubuntu. Please tell me what I'm doing wrong. I will provide the necessary commands conclusions.

Phishfry · Apr 28, 2020

What happens when you pass thru all the PCI addresses for the USB card? Like this:
/boot/loader.conf
pptdevs="5/0/0 5/0/1 5/0/2"

In your Bhyve script:
-s 7:0,passthru,5/0/0
-s 7:1,passthru,5/0/1
-s 7:2,passthru,5/0/2

This should result in the whole device being passed thru.
In this case you must pass thru the 'Parent' device along with the child device.

odin32 · Apr 28, 2020

I add the line to the /boot/loader.conf file: pptdevs="5/0/0 5/0/1 5/0/2"

Code:

# vi /boot/loader.conf
fuse_load="YES"
cd9660_iconv_load="YES"
msdosfs_iconv_load="YES"
kern.vty=vt
tmpfs_load="YES"
vmm_load="YES"
nmdm_load="YES"
if_bridge_load="YES"
if_tap_load="YES"
# pci passthrough
pptdevs="5/0/0 5/0/1 5/0/2"

Interface ehci1@pci0:5:0:2 is not determined, as, ppt
Added to the script start.ubuntu18.sh lines: - s 7:0, passthru, 5/0/0 -s 7:1, passthru, 5/0/1 -s 7:2, passthru, 5/0/2
# cd /usr/vm/bhyve/ubuntu18
# vi start.ubuntu18.sh

Code:

#!/bin/sh

grub-bhyve -m ./device.map -r hd0,msdos1 -M 2048M ubuntu18

bhyve -w -A -H -P \
-s 0:0,hostbridge \
-s 1:0,lpc \
-s 2:0,virtio-net,tap0 \
-s 3:0,virtio-blk,./disk0.img \
-s 7:0,passthru,5/0/0 \
-s 7:1,passthru,5/0/1 \
-s 7:2,passthru,5/0/2 \
-l com1,stdio -c 2 -m 2048M ubuntu18

# cd /usr/vm/bhyve/ubuntu18
# ./start.ubuntu18.sh
The script does not run, writes a message:

Code:

vm_create: Invalid argument
./start.ubuntu18.sh: -s: not found

SirDice · Apr 28, 2020

odin32 said:
The script does not run, writes a message:

I'm fairly certain you forgot a backslash at the end of a line somewhere.

Phishfry · Apr 28, 2020

odin32 said:
bhyve -w -A -H -P \

I think your problem is with your options.
-S is needed with your bhyve command.

Guest memory must be wired using the -S option when a pass-through device is configured.

bhyve(8)

Phishfry · Apr 28, 2020

Here is my VM startup script. It is a bit messy with each VM being on a single line but I just deal with it.
I start my VM's automatically with /etc/rc.local, It requires some sleep time to allow for interfaces to come up.

Code:

cat /etc/rc.local
#!/bin/sh
sleep 5
bhyve -S -c 4 -m 4G -A -H -P -u -s 0:0,hostbridge -s 3:0,ahci-hd,/vm/freebsd-vm1.img -s 5:0,passthru,131/0/0 -s 10:0,passthru,3/0/0 -s 11:0,passthru,137/0/0 -s 12:0,passthru,137/0/1 -s 31:0,lpc -l com1,/dev/nmdm1A -l bootrom,/vm/BHYVE_UEFI.fd freebsd-vm1 &
sleep 6
bhyve -S -c 4 -m 4G -A -H -P -u -s 0:0,hostbridge -s 3:0,ahci-hd,/vm/freebsd-vm2.img -s 5:0,passthru,131/0/1 -s 10:0,passthru,1/0/0 -s 11:0,passthru,134/0/0 -s 12:0,passthru,134/0/1 -s 31:0,lpc -l com1,/dev/nmdm2A -l bootrom,/vm/BHYVE_UEFI.fd freebsd-vm2 &
sleep 5
bhyve -S -c 4 -m 4G -A -H -P -u -s 0:0,hostbridge -s 3:0,ahci-hd,/vm/freebsd-vm3.img -s 5:0,passthru,132/0/0 -s 31:0,lpc -l com1,/dev/nmdm3A -l bootrom,/vm/BHYVE_UEFI.fd freebsd-vm3 &
sleep 5
/usr/local/sbin/grub-bhyve -m /vm/device.map -r hd0,msdos1 -M 2048M -S devuan1 -d /boot/grub/ &
sleep 6
bhyve -S -c 4 -m 2G -A -H -P -u -s 0:0,hostbridge -s 3:0,ahci-hd,/vm/devuan1.img -s 5:0,passthru,132/0/1 -s 31:0,lpc -l com1,/dev/nmdm4A devuan1 &

Please notice that you need the full path for grub-bhyve.
You will probably need a sleep delay between grub-bhyve and bhyve commands to allow for the grub menu to timeout and proceed.
The sleep time will depend on Ubuntu's grub menu timeout plus a second or two..
You could probably set grub timeout to 0 if you wanted.

Phishfry · Apr 28, 2020

I bought a Supermicro X10DRX for virtulization. It has 11 slots and it can become very confusing what PCI bus is what.
So I annotate my /boot/loader.conf

Code:

pptdevs="1/0/0 3/0/0 133/0/0 137/0/0 137/0/1 134/0/0 134/0/1 135/0/0 135/0/1 131/0/0 131/0/1 132/0/0 132/0/1 7/0/1"
#pptdevs2="1/0/0"   ### USB Controller ### SLOT 2 ###
#pptdevs3="3/0/0"   ### USB Controller ### SLOT 3 ###
#pptdevs4="4/0/0"   ### USB Controller ### SLOT 4 ###
#pptdevs5="133/0/0" ### USB Controller ### SLOT 5 ###
#pptdevs8="5/0/0"   ### SLOT 8 ###
#pptdevs11="131/0/0 131/0/1 132/0/0 132/0/1" ### Intel Quad NIC ### SLOT 11 ###
#pptdevs9="134/0/0" ### MSI   VID=1462 PID=8c93 GT710 VGA passthru ### SLOT 9 ###
#pptdevs10="135/0/0" ### Zotac VID=19da PID=5360 GT710 VGA passthru ### SLOT 10 ###
#pptdevs7="137/0/0" ### Dell  VID=1028 PID=1083 GT730 VGA passthru ### SLOT 7 ###

tingo · Apr 29, 2020

This subject is interesting. Does anyone know if pci passthru for bhyve work on AMD cpus / chipsets too?

odin32 · Apr 30, 2020

I provide an updated script: added the full path to bhyve, parameter -S, delay of 5 seconds
# cd /usr/vm/bhyve/ubuntu18
# ./start/ubuntu18.sh

Code:

#!/bin/sh

grub-bhyve -m /usr/vm/bhyveubuntu18/device.map -r hd0,msdos1 -M 2048M ubuntu18
sleep 5
/usr/sbin/bhyve -S -c 2 -m 2048M -w -A -H -P -l com1,stdio \
-s 0:0,hostbridge -s 1:0,lpc \
-s 2:0,virtio-net,tap0 \
-s 3:0,virtio-blk,/usr/vm/bhyve/ubuntu18/ubuntu18.img \
-s 7:0,passthru,5/0/0 -s 7:1,passthru,5/0/1 -s 7:2,passthru,5/0/2 ubuntu18

Run the script, appears 30 seconds menu: Ubuntu, Advance options for Ubuntu, when you select any of them outputs a message

Code:

vm_create: Invalid argument
./start.ubuntu18.sh: -s: not found
./start.ubuntu18.sh: -s: not found

I will try to reinstall the virtual machine, it is interesting that the script was run before (before pci-passthrough), now it does not run at all.

SirDice · Apr 30, 2020

odin32 said:
# ./start/ubuntu18.sh

Run this as # sh -x ./start/ubuntu18.sh. That will show what it's executing, that will make it easier to find the error.

Phishfry · Apr 30, 2020

tingo said:
Does anyone know if pci passthru for bhyve work on AMD cpus / chipsets too?

According to the wiki they now work.

[base] Revision 317600

svnweb.freebsd.org

By default AMD-Vi passthrough support is disabled, set hw.vmm.amdvi.enable and reload vmm.ko to enable it.

bhyve/pci_passthru - FreeBSD Wiki

Phishfry · Apr 30, 2020

I think the nmdm(4) console works good for discovering issues with a Bhyve VM.

odin32 · May 1, 2020

Reinstalled bhyve using the vm snap-in. See # vm help
# cd /usr/vm/bhyve/ubuntu18 # ./start.ubuntu18.sh
The script starts the virtual machine in normal mode.
If I add parameters (pci-passthrough) to the script, I get a message

Code:

#!/bin/sh

grub-bhyve -m ./device.map -r hd0,msdos1 -M 2048M ubuntu18

bhyve -w -A -H -P -S \
-s 0:0,hostbridge \
-s 1:0,lpc \
-s 2:0,virtio-net,tap0 \
-s 3,virtio-blk,./disk0.img \
-s 7:0,passthru,5/0/0 -s 7:1,passthru,5/0/1 -s 7:2,passthru,5/0/2 \
-l com1,stdio -c 2 -m 2048M ubuntu18

Message

Code:

Unable to setup memory(17)

Entered # sh -x ./start.ubuntu18.sh

Code:

+ bhyve -w -A -H -P -S -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3,virtio-blk,./disk0.img -s 7:0,passthru,5/0/0 -s 7:1,passthru,5/0/1 -s 7:2,passthru,5/0/2 -l com1,stdio -c 2 -m 2048M ubuntu18    
Unable to setup memory(17)

Without parameters (pci-passthrough), the script runs.

SirDice · May 1, 2020

odin32 said:
Reinstalled bhyve using the vm snap-in. See # vm help

You mean sysutils/vm-bhyve? That doesn't install bhyve(8), bhyve(8) is part of the OS. That vm-bhyve is a tool to manage VMs. You're not even using it as you're creating your own scripts to launch bhyve VMs.

Quickstart

Shell based, minimal dependency bhyve manager. Contribute to churchers/vm-bhyve development by creating an account on GitHub.

github.com

Phishfry · May 1, 2020

You need the -S switch for both grub-bhyve and bhyve.
Look at my example.

odin32 · May 2, 2020

You don't understand me, or rather I did not understand what I wrote. All necessary packages installed:
# pkg install sysutils/vm-bhyve
# pkg install sysutils/grub2-bhyve
# pkg install sysutils/bhyve-firmware
# pkg install sysutils/uefi-edk2-bhyve
In addition to interfaces: re0 (network), bridge0, tap0 added
# vm switch create -t standard -i re0 -a 192.168.88.0/24 -p public
Template-based configuration: /usr/vm/bhyve/.templates/ubuntu.conf
# vm create -t ubuntu_gui -s 10G -m 2048M -c 2 ubuntu18
With all the necessary settings, see # vm help

As for the parameter -S and -s 7:0, passthru, 5/0/0 -s 7:1, passthru, 5/0/1 -s 7:2, passthru,5/0/2
so they are added to the script. I don't understand which version of the script you want to offer?

Phishfry · May 2, 2020

The port vm-bhyve is a method to create VM's easily.
The script you are showing above is not using sysutils/vm-bhyve. You are using a manual method with a bhyve script.
Also sysutils/bhyve-firmware provides sysutils/uefi-edk2-bhyve and some others , So you are double loading for no reason. Not that it hurts but it is unnecessary.

All you really need is these - sysutils/uefi-edk2-bhyve and sysutils/grub2-bhyve. for manually loading bhyve.
I see you mentioning vm-create and /usr/vm/bhyve/.templates/ubuntu.conf. These are for vm-bhyve ONLY.
You are showing two different methods here. vm-bhyve will handle PCI passthru differently and you must read their manual for that.

Here are the relevant lines needed for passthru on vm-bhyve.

vm-bhyve/sample-templates/config.sample at master · churchers/vm-bhyve

Shell based, minimal dependency bhyve manager. Contribute to churchers/vm-bhyve development by creating an account on GitHub.

github.com

odin32 · May 3, 2020

It's weird.
Learn "bhyve" from official FreeBSD information sources (handbook, mans, wiki...)
As a result, away.
I will try to understand this homework.
Thank you all for your help!

Phishfry · May 3, 2020

I also wanted to mention that I was not able to use EFI with grub-bhyve.
I am using legacy bios method.(As you can see from my example)

Lamia · May 4, 2020

And "vm console ubuntu18" will get you into the OS console.

Phishfry:
1) What IP address would Ubuntu have?
2) Would you have to set the IP address in the netwrkinf conf for the guest OS after logging into it? In addition, would you have to set the routes, etc?
2) Can you ping the guest IP address from the host/baseOS via the tap0/vm-public without the traffic going out of the host interface?

I've got CentOS as vm-bhyve guest. I use the 192.168.88.0/24 for the vm-public and set the guest IP to 192.168.88.1 for the eth0 interface.
Despite that, I couldn't ping the 192.168.88.1 from the host. I then changed the vm-public to 192.168.88.1/24 and eth0 to 192.168.88.2. Of course, I can ping the IP .1 now but not .2; the traffic isn't reaching the guest. The eth0 is of no use even when it is assigned the hardware or MAC address for the passthru Ethernet card. I had to create a /etc/sysconfig/network-scripts/ifcfg-ix0.conf , assign the MAC address of the passthru Ethernet card and set a public IP. It is then that the VM guest is reachable via the Public IP on ix1 inf but not via the Private IP on eth0 inf. The eth0 in the config of the VM-bhyve guest does nothing. That's why I asked you if you can ping the guest from the host. I might have been doing something wrong.

odin32 · May 4, 2020

Tell me if there are instructions for installing the uefi-edk2-bhyve-UDK2014.SP1 package.
Or where to find it in the package itself

odin32 · May 4, 2020

I have no problems with the network. Ping is from host to guest, and from guest to host. The Internet is also available.

In bhyve, the PCI-usb 2.0 controller is passthrough to the Ubuntu guest

odin32

Phishfry

odin32

SirDice

Administrator

Phishfry

Phishfry

Phishfry

tingo

odin32

SirDice

Administrator

Phishfry

[base] Revision 317600

Phishfry

odin32

SirDice

Administrator

Quickstart

Phishfry

odin32

Phishfry

vm-bhyve/sample-templates/config.sample at master · churchers/vm-bhyve

odin32

Phishfry

Lamia

odin32

odin32