In bhyve, the PCI-usb 2.0 controller is passthrough to the Ubuntu guest

odin32

Member

Reaction score: 1
Messages: 24

Inserted a pci-usb 2.0 controller into the PC. pci-usb devices in the Freebsd system:
# pciconf -lv | grep -B2 USB
Code:
xhci0@pci0:0:20:0:    class=0x0c0330 card=0x50071458 chip=0x8c318086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB xHCI'
    class      = serial bus
    subclass   = USB
--
ehci0@pci0:0:26:0:    class=0x0c0320 card=0x50061458 chip=0x8c2d8086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB EHCI'
    class      = serial bus
    subclass   = USB
--
ehci2@pci0:0:29:0:    class=0x0c0320 card=0x50061458 chip=0x8c268086 rev=0x05 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB EHCI'
    class      = serial bus
    subclass   = USB
--
uhci0@pci0:5:0:0:    class=0x0c0300 card=0x30381106 chip=0x30381106 rev=0x61 hdr=0x00
    vendor     = 'VIA Technologies, Inc.'
    device     = 'VT82xx/62xx UHCI USB 1.1 Controller'
    class      = serial bus
    subclass   = USB
uhci1@pci0:5:0:1:    class=0x0c0300 card=0x30381106 chip=0x30381106 rev=0x61 hdr=0x00
    vendor     = 'VIA Technologies, Inc.'
    device     = 'VT82xx/62xx UHCI USB 1.1 Controller'
    class      = serial bus
    subclass   = USB
ehci1@pci0:5:0:2:    class=0x0c0320 card=0x31041106 chip=0x31041106 rev=0x63 hdr=0x00
    vendor     = 'VIA Technologies, Inc.'
    device     = 'USB 2.0'
    class      = serial bus
    subclass   = USB
Device ehci1@pci0:5:0:2: not defined as ppt. When adding the parameter -s 7:0, passthru, 5/0/2 to the bhyve startup script, the script does not start!
I want to throw pci-usb 2.0 into guest Ubuntu. Please tell me what I'm doing wrong. I will provide the necessary commands conclusions.
 
Last edited by a moderator:

Phishfry

Beastie's Twin

Reaction score: 2,842
Messages: 5,842

What happens when you pass thru all the PCI addresses for the USB card? Like this:
/boot/loader.conf
pptdevs="5/0/0 5/0/1 5/0/2"

In your Bhyve script:
-s 7:0,passthru,5/0/0
-s 7:1,passthru,5/0/1
-s 7:2,passthru,5/0/2

This should result in the whole device being passed thru.
In this case you must pass thru the 'Parent' device along with the child device.
 
OP
O

odin32

Member

Reaction score: 1
Messages: 24

I add the line to the /boot/loader.conf file: pptdevs="5/0/0 5/0/1 5/0/2"
Code:
# vi /boot/loader.conf
fuse_load="YES"
cd9660_iconv_load="YES"
msdosfs_iconv_load="YES"
kern.vty=vt
tmpfs_load="YES"
vmm_load="YES"
nmdm_load="YES"
if_bridge_load="YES"
if_tap_load="YES"
# pci passthrough
pptdevs="5/0/0 5/0/1 5/0/2"
Interface ehci1@pci0:5:0:2 is not determined, as, ppt
Added to the script start.ubuntu18.sh lines: - s 7:0, passthru, 5/0/0 -s 7:1, passthru, 5/0/1 -s 7:2, passthru, 5/0/2
# cd /usr/vm/bhyve/ubuntu18
# vi start.ubuntu18.sh
Code:
#!/bin/sh

grub-bhyve -m ./device.map -r hd0,msdos1 -M 2048M ubuntu18

bhyve -w -A -H -P \
-s 0:0,hostbridge \
-s 1:0,lpc \
-s 2:0,virtio-net,tap0 \
-s 3:0,virtio-blk,./disk0.img \
-s 7:0,passthru,5/0/0 \
-s 7:1,passthru,5/0/1 \
-s 7:2,passthru,5/0/2 \
-l com1,stdio -c 2 -m 2048M ubuntu18
# cd /usr/vm/bhyve/ubuntu18
# ./start.ubuntu18.sh
The script does not run, writes a message:
Code:
vm_create: Invalid argument
./start.ubuntu18.sh: -s: not found
 

Phishfry

Beastie's Twin

Reaction score: 2,842
Messages: 5,842

Here is my VM startup script. It is a bit messy with each VM being on a single line but I just deal with it.
I start my VM's automatically with /etc/rc.local, It requires some sleep time to allow for interfaces to come up.

Code:
cat /etc/rc.local
#!/bin/sh
sleep 5
bhyve -S -c 4 -m 4G -A -H -P -u -s 0:0,hostbridge -s 3:0,ahci-hd,/vm/freebsd-vm1.img -s 5:0,passthru,131/0/0 -s 10:0,passthru,3/0/0 -s 11:0,passthru,137/0/0 -s 12:0,passthru,137/0/1 -s 31:0,lpc -l com1,/dev/nmdm1A -l bootrom,/vm/BHYVE_UEFI.fd freebsd-vm1 &
sleep 6
bhyve -S -c 4 -m 4G -A -H -P -u -s 0:0,hostbridge -s 3:0,ahci-hd,/vm/freebsd-vm2.img -s 5:0,passthru,131/0/1 -s 10:0,passthru,1/0/0 -s 11:0,passthru,134/0/0 -s 12:0,passthru,134/0/1 -s 31:0,lpc -l com1,/dev/nmdm2A -l bootrom,/vm/BHYVE_UEFI.fd freebsd-vm2 &
sleep 5
bhyve -S -c 4 -m 4G -A -H -P -u -s 0:0,hostbridge -s 3:0,ahci-hd,/vm/freebsd-vm3.img -s 5:0,passthru,132/0/0 -s 31:0,lpc -l com1,/dev/nmdm3A -l bootrom,/vm/BHYVE_UEFI.fd freebsd-vm3 &
sleep 5
/usr/local/sbin/grub-bhyve -m /vm/device.map -r hd0,msdos1 -M 2048M -S devuan1 -d /boot/grub/ &
sleep 6
bhyve -S -c 4 -m 2G -A -H -P -u -s 0:0,hostbridge -s 3:0,ahci-hd,/vm/devuan1.img -s 5:0,passthru,132/0/1 -s 31:0,lpc -l com1,/dev/nmdm4A devuan1 &
Please notice that you need the full path for grub-bhyve.
You will probably need a sleep delay between grub-bhyve and bhyve commands to allow for the grub menu to timeout and proceed.
The sleep time will depend on Ubuntu's grub menu timeout plus a second or two..
You could probably set grub timeout to 0 if you wanted.
 

Phishfry

Beastie's Twin

Reaction score: 2,842
Messages: 5,842

I bought a Supermicro X10DRX for virtulization. It has 11 slots and it can become very confusing what PCI bus is what.
So I annotate my /boot/loader.conf
Code:
pptdevs="1/0/0 3/0/0 133/0/0 137/0/0 137/0/1 134/0/0 134/0/1 135/0/0 135/0/1 131/0/0 131/0/1 132/0/0 132/0/1 7/0/1"
#pptdevs2="1/0/0"   ### USB Controller ### SLOT 2 ###
#pptdevs3="3/0/0"   ### USB Controller ### SLOT 3 ###
#pptdevs4="4/0/0"   ### USB Controller ### SLOT 4 ###
#pptdevs5="133/0/0" ### USB Controller ### SLOT 5 ###
#pptdevs8="5/0/0"   ### SLOT 8 ###
#pptdevs11="131/0/0 131/0/1 132/0/0 132/0/1" ### Intel Quad NIC ### SLOT 11 ###
#pptdevs9="134/0/0" ### MSI   VID=1462 PID=8c93 GT710 VGA passthru ### SLOT 9 ###
#pptdevs10="135/0/0" ### Zotac VID=19da PID=5360 GT710 VGA passthru ### SLOT 10 ###
#pptdevs7="137/0/0" ### Dell  VID=1028 PID=1083 GT730 VGA passthru ### SLOT 7 ###
 

tingo

Son of Beastie

Reaction score: 653
Messages: 2,566

This subject is interesting. Does anyone know if pci passthru for bhyve work on AMD cpus / chipsets too?
 
OP
O

odin32

Member

Reaction score: 1
Messages: 24

I provide an updated script: added the full path to bhyve, parameter -S, delay of 5 seconds
# cd /usr/vm/bhyve/ubuntu18
# ./start/ubuntu18.sh
Code:
#!/bin/sh

grub-bhyve -m /usr/vm/bhyveubuntu18/device.map -r hd0,msdos1 -M 2048M ubuntu18
sleep 5
/usr/sbin/bhyve -S -c 2 -m 2048M -w -A -H -P -l com1,stdio \
-s 0:0,hostbridge -s 1:0,lpc \
-s 2:0,virtio-net,tap0 \
-s 3:0,virtio-blk,/usr/vm/bhyve/ubuntu18/ubuntu18.img \
-s 7:0,passthru,5/0/0 -s 7:1,passthru,5/0/1 -s 7:2,passthru,5/0/2 ubuntu18
Run the script, appears 30 seconds menu: Ubuntu, Advance options for Ubuntu, when you select any of them outputs a message
Code:
vm_create: Invalid argument
./start.ubuntu18.sh: -s: not found
./start.ubuntu18.sh: -s: not found
I will try to reinstall the virtual machine, it is interesting that the script was run before (before pci-passthrough), now it does not run at all.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 13,109
Messages: 39,700

# ./start/ubuntu18.sh
Run this as # sh -x ./start/ubuntu18.sh. That will show what it's executing, that will make it easier to find the error.
 
OP
O

odin32

Member

Reaction score: 1
Messages: 24

Reinstalled bhyve using the vm snap-in. See # vm help
# cd /usr/vm/bhyve/ubuntu18 # ./start.ubuntu18.sh
The script starts the virtual machine in normal mode.
If I add parameters (pci-passthrough) to the script, I get a message
Code:
#!/bin/sh

grub-bhyve -m ./device.map -r hd0,msdos1 -M 2048M ubuntu18

bhyve -w -A -H -P -S \
-s 0:0,hostbridge \
-s 1:0,lpc \
-s 2:0,virtio-net,tap0 \
-s 3,virtio-blk,./disk0.img \
-s 7:0,passthru,5/0/0 -s 7:1,passthru,5/0/1 -s 7:2,passthru,5/0/2 \
-l com1,stdio -c 2 -m 2048M ubuntu18
Message
Code:
Unable to setup memory(17)
Entered # sh -x ./start.ubuntu18.sh
Code:
+ bhyve -w -A -H -P -S -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 -s 3,virtio-blk,./disk0.img -s 7:0,passthru,5/0/0 -s 7:1,passthru,5/0/1 -s 7:2,passthru,5/0/2 -l com1,stdio -c 2 -m 2048M ubuntu18    
Unable to setup memory(17)
Without parameters (pci-passthrough), the script runs.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 13,109
Messages: 39,700

Phishfry

Beastie's Twin

Reaction score: 2,842
Messages: 5,842

You need the -S switch for both grub-bhyve and bhyve.
Look at my example.
 
OP
O

odin32

Member

Reaction score: 1
Messages: 24

You don't understand me, or rather I did not understand what I wrote. All necessary packages installed:
# pkg install sysutils/vm-bhyve
# pkg install sysutils/grub2-bhyve
# pkg install sysutils/bhyve-firmware
# pkg install sysutils/uefi-edk2-bhyve
In addition to interfaces: re0 (network), bridge0, tap0 added
# vm switch create -t standard -i re0 -a 192.168.88.0/24 -p public
Template-based configuration: /usr/vm/bhyve/.templates/ubuntu.conf
# vm create -t ubuntu_gui -s 10G -m 2048M -c 2 ubuntu18
With all the necessary settings, see # vm help

As for the parameter -S and -s 7:0, passthru, 5/0/0 -s 7:1, passthru, 5/0/1 -s 7:2, passthru,5/0/2
so they are added to the script. I don't understand which version of the script you want to offer?
 

Phishfry

Beastie's Twin

Reaction score: 2,842
Messages: 5,842

The port vm-bhyve is a method to create VM's easily.
The script you are showing above is not using sysutils/vm-bhyve. You are using a manual method with a bhyve script.
Also sysutils/bhyve-firmware provides sysutils/uefi-edk2-bhyve and some others , So you are double loading for no reason. Not that it hurts but it is unnecessary.

All you really need is these - sysutils/uefi-edk2-bhyve and sysutils/grub2-bhyve. for manually loading bhyve.
I see you mentioning vm-create and /usr/vm/bhyve/.templates/ubuntu.conf. These are for vm-bhyve ONLY.
You are showing two different methods here. vm-bhyve will handle PCI passthru differently and you must read their manual for that.

Here are the relevant lines needed for passthru on vm-bhyve.
 
OP
O

odin32

Member

Reaction score: 1
Messages: 24

It's weird.
Learn "bhyve" from official FreeBSD information sources (handbook, mans, wiki...)
As a result, away.
I will try to understand this homework.
Thank you all for your help!
 

Phishfry

Beastie's Twin

Reaction score: 2,842
Messages: 5,842

I also wanted to mention that I was not able to use EFI with grub-bhyve.
I am using legacy bios method.(As you can see from my example)
 

Lamia

Aspiring Daemon

Reaction score: 240
Messages: 806

And "vm console ubuntu18" will get you into the OS console.

Phishfry:
1) What IP address would Ubuntu have?
2) Would you have to set the IP address in the netwrkinf conf for the guest OS after logging into it? In addition, would you have to set the routes, etc?
2) Can you ping the guest IP address from the host/baseOS via the tap0/vm-public without the traffic going out of the host interface?

I've got CentOS as vm-bhyve guest. I use the 192.168.88.0/24 for the vm-public and set the guest IP to 192.168.88.1 for the eth0 interface.
Despite that, I couldn't ping the 192.168.88.1 from the host. I then changed the vm-public to 192.168.88.1/24 and eth0 to 192.168.88.2. Of course, I can ping the IP .1 now but not .2; the traffic isn't reaching the guest. The eth0 is of no use even when it is assigned the hardware or MAC address for the passthru Ethernet card. I had to create a /etc/sysconfig/network-scripts/ifcfg-ix0.conf , assign the MAC address of the passthru Ethernet card and set a public IP. It is then that the VM guest is reachable via the Public IP on ix1 inf but not via the Private IP on eth0 inf. The eth0 in the config of the VM-bhyve guest does nothing. That's why I asked you if you can ping the guest from the host. I might have been doing something wrong.
 
OP
O

odin32

Member

Reaction score: 1
Messages: 24

Tell me if there are instructions for installing the uefi-edk2-bhyve-UDK2014.SP1 package.
Or where to find it in the package itself
 
OP
O

odin32

Member

Reaction score: 1
Messages: 24

I have no problems with the network. Ping is from host to guest, and from guest to host. The Internet is also available.
 
Top