Bhyve GPU pass-through

I saw on Reddit via Twitter that a user has GPU passthrough working with Bhyve.

So he is claiming that you can pass-through a Nvidia video card and assign it to an X Windows session using the nvidia-config application.
You must add the video device to your VM's Xorg config beforehand. Supposedly one line to add.
I have been saving old monitors for just this purpose. I want 4 headed VM each with their own display. Yipee.
Now we are getting closer to Xen functionality.

That way sounds like it might just work. Has anybody tried anything similar?
 
Well I have a bhyve testing machine up checking this out. Using GeForce GTX650 half height card.
So far so good. I loaded Nvidia driver 390 and linux compat. on a FreeBSD-12 VM with Xorg and Xfce
So I am configuring it now.
I passed through the device and its child. Wrote a nvidia.conf with BusID included.
Code:
vgapci0@pci0:0:5:0:    class=0x030000 card=0x129119da chip=0x0fc610de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    device     = 'GK107 [GeForce GTX 650]'
    class      = display
    subclass   = VGA
hdac0@pci0:0:5:1:    class=0x040300 card=0x129119da chip=0x0e1b10de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    device     = 'GK107 HDMI Audio Controller'
    class      = multimedia
    subclass   = HDA

I might have to ditch tmux to test this out.
Currently I start all my VM's from a rc.local script with a tmux session.
 
Well I have to say, I have one VM with video passthru working.
I used a single EFI bhyve VM and I have xfce desktop over stdio on a HDMI monitor coming from my GTX650.
Now, will that same setup work for multiple video cards passed through.

I did nothing special. Normal nvidia driver and didn't use nvidia-xconfig or nvidia-settings. Just my own conf with BusID.
/usr/local/etc/X11/xorg.conf.d/20-nvidia.conf
Code:
        Identifier      "Nvidia Card"
        Driver          "nvidia"
        VendorName      "NVIDIA Corporation"
        BoardName       "GeForce GTX 650"
        BusID           "PCI:0:5:0"
EndSection

Section "Screen"
        Identifier     "Screen0"
        Device         "Device0"
        Monitor        "Monitor0"
        Option         "metamodes" "nvidia-auto-select +0+0 {ForceCompositionPipeline=On, ForceFullCompositionPipeline=On}"
        Option         "AllowIndirectGLXProtocol" "off"
        Option         "TripleBuffer" "on"
EndSection

I have no mouse or keyboard control on the HDMI screen. I might need to add to /x11/xorg.conf.d/ file for input devices.
 
Here is my /etc/rc.local I use to start up my VM. Device 5:0 and 5:1 are my video card.
Code:
 # cat /etc/rc.local
#!/bin/sh
bhyve -S -c 4 -m 3G -AHP -u -s 0:0,hostbridge -s 1:0,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd1.img -s 3:0,passthru,2/0/0 -s 5:0,passthru,5/0/0 -s 5:1,passthru,5/0/1 -s 30:0,xhci,tablet -l com1,stdio -l bootrom,/vm/freebsd/BHYVE_UEFI.fd freebsd1 &
Notice with the ampersand it sends the process to the backround.
With no ampersand it boots to the VM and I can run startxfce4.
Host is unreachable when ran this way. This way is hacky but it does work. Probably a better way.

Still working on mouse input. Isn't that what xhci,tablet does?
 
For the mouse- xhci,tablet pass-thru ends up as /dev/ums0 on the VM. I tried assigning that a xorg input with no luck.
This was more a vanity exercise. No mouse or keyboard all but makes this useless. A nice picture on a HDMI screen.
Wondering about passing through a USB mouse/keyboard for each desktop.
 
I was able to get two NVidia video cards working with two monitors and two seperate VM's.
The hangup is input. No mouse or keyboard.
For testing setup I bought some USB3.0 PCIe cards and plan on passing the card thru. NEC USB3 chipset.
Curious to see how USB disks work and mouse and kb in a VM with their own dedicated host device.
 
I am happy to say I have passed thru the USB3 controller and I have a ums0 mouse working on a VM's Xorg hdmi screen.
Bought 2 USB3 cards so checking on the second one now in a second VM.
 
Awe shucks that won't work. When I mentioned above I have 2 video cards on 2 VM's, notice I did not use the word 'simultaneous'.
The only way I can get this to work is launching bhyve --in the foreground-- and with that method I can launch Xorg in for the foreground VM mapped onto a dedicated HDMI monitor..

To do this for testing maybe I run VM in the backround and do autologin and startxfce script to bring it up kiosk style.
Sure is hacky. I could do multiple Xorg VM's to multiple monitors that way believe.
I also have a virtio-console method in mind to test.
 
I am happy to say I have passed thru the USB3 controller and I have a ums0 mouse working on a VM's Xorg hdmi screen.
That's really cool! I never had luck to pass through a USB controller, always got errors with various controllers and PCs (reported a couple of times on these forums). I don't think I tried with 12.0 though.
 
I was only testing with mouse. So I opened mousepad text editor and no keyboard, so I hotplugged a keyboard in my passthru USB3 controller and I had text in mousepad.
Mission complete on VM1

Time for the real test.
 
I think the problem with using onboard USB is you would lose the xhci controller to the host system.
I need that (mouse/kybd) for host management.
The onboard USB controllers all hang off of one pci address.
 
I have not tried that port yet but it sounds like something that might work.
Right now I like using a separate USB3 controller for each Xorg VM.
Problem is I only have 7 slot motherboard and I am running out of slots.
Trying to use 2 NVMe, Chelsio 10G, and 2 video cards and 2 USB cards. Maxed out at 2 VM's.
I could bifurcate my NVMe as a last resort.
Noticed that my onboard USB uses two PCI addresses so I need to suss that out.
It appears that the USB pin headers are on a separate PCI bus. So I can use those.

Does anybody know of any Video cards with USB inputs?
Hardware virtualization cards that work in FreeBSD?
Here is what I have seen on ebay;
TERADICI PCoIP cards:
I doubt these would work but I like the idea. Multiple functions per slot.
 
Just tested a USB-over-Ethernet adapter BB-UE204 with Windows bhyve guest as a keyboard and mouse interface. Works fine. The drawback is its high price ($200+). I got one at work place, but wouldn't buy if had to pay own money.
 
Ahh yes. BlackBox makes adapter for everything (at a steep price).

What I was just wondering is 'does PCIe contain USB natively'.
I bought a 90 degree PCIe riser from an IBM box and it had a USB jack on it.
The thing I worry about with that approach (like USB with video card) is that it would probably not use separate PCI bus ID.
I dunno. All new to me.
EDIT: I see nothing in PCIe slot signaling for USB. Only miniPCIe adds USB2 for cellular modems and their sims.
Probably a special IBM X36xx riser for certain slot in that server.
 
Has been crickets from that guy..

As far as I can tell this works with FreeBSD no special sauce needed. Just make a xorg.conf with the BusID.
Surprised no one has tried it before.
I am now launching my two Xorg VM's with tmux. So no issues there. I though maybe tmux would be an issue.
I am copying in some MPG videos now to test HDMI audio passthru to separate monitors.
 
No HDMI audio without any work.
I did test USB Memory sticks and found that device discovery happens quick as normal, but the CAM support lags by a minute.
No device in /dev until CAM.
 
Back
Top