Bhyve GPU pass-through

Phishfry

Son of Beastie

Reaction score: 1,422
Messages: 4,188

I saw on Reddit via Twitter that a user has GPU passthrough working with Bhyve.

So he is claiming that you can pass-through a Nvidia video card and assign it to an X Windows session using the nvidia-config application.
You must add the video device to your VM's Xorg config beforehand. Supposedly one line to add.
I have been saving old monitors for just this purpose. I want 4 headed VM each with their own display. Yipee.
Now we are getting closer to Xen functionality.

That way sounds like it might just work. Has anybody tried anything similar?
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,422
Messages: 4,188

Well I have a bhyve testing machine up checking this out. Using GeForce GTX650 half height card.
So far so good. I loaded Nvidia driver 390 and linux compat. on a FreeBSD-12 VM with Xorg and Xfce
So I am configuring it now.
I passed through the device and its child. Wrote a nvidia.conf with BusID included.
Code:
vgapci0@pci0:0:5:0:    class=0x030000 card=0x129119da chip=0x0fc610de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    device     = 'GK107 [GeForce GTX 650]'
    class      = display
    subclass   = VGA
hdac0@pci0:0:5:1:    class=0x040300 card=0x129119da chip=0x0e1b10de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    device     = 'GK107 HDMI Audio Controller'
    class      = multimedia
    subclass   = HDA
I might have to ditch tmux to test this out.
Currently I start all my VM's from a rc.local script with a tmux session.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,422
Messages: 4,188

Well I have to say, I have one VM with video passthru working.
I used a single EFI bhyve VM and I have xfce desktop over stdio on a HDMI monitor coming from my GTX650.
Now, will that same setup work for multiple video cards passed through.

I did nothing special. Normal nvidia driver and didn't use nvidia-xconfig or nvidia-settings. Just my own conf with BusID.
/usr/local/etc/X11/xorg.conf.d/20-nvidia.conf
Code:
        Identifier      "Nvidia Card"
        Driver          "nvidia"
        VendorName      "NVIDIA Corporation"
        BoardName       "GeForce GTX 650"
        BusID           "PCI:0:5:0"
EndSection

Section "Screen"
        Identifier     "Screen0"
        Device         "Device0"
        Monitor        "Monitor0"
        Option         "metamodes" "nvidia-auto-select +0+0 {ForceCompositionPipeline=On, ForceFullCompositionPipeline=On}"
        Option         "AllowIndirectGLXProtocol" "off"
        Option         "TripleBuffer" "on"
EndSection
I have no mouse or keyboard control on the HDMI screen. I might need to add to /x11/xorg.conf.d/ file for input devices.
 

roccobaroccoSC

Well-Known Member

Reaction score: 97
Messages: 438

This is really cool, could you share your bhyve arguments please? How do you do the passthrough?
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,422
Messages: 4,188

Here is my /etc/rc.local I use to start up my VM. Device 5:0 and 5:1 are my video card.
Code:
 # cat /etc/rc.local
#!/bin/sh
bhyve -S -c 4 -m 3G -AHP -u -s 0:0,hostbridge -s 1:0,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd1.img -s 3:0,passthru,2/0/0 -s 5:0,passthru,5/0/0 -s 5:1,passthru,5/0/1 -s 30:0,xhci,tablet -l com1,stdio -l bootrom,/vm/freebsd/BHYVE_UEFI.fd freebsd1 &
Notice with the ampersand it sends the process to the backround.
With no ampersand it boots to the VM and I can run startxfce4.
Host is unreachable when ran this way. This way is hacky but it does work. Probably a better way.

Still working on mouse input. Isn't that what xhci,tablet does?
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,422
Messages: 4,188

For the mouse- xhci,tablet pass-thru ends up as /dev/ums0 on the VM. I tried assigning that a xorg input with no luck.
This was more a vanity exercise. No mouse or keyboard all but makes this useless. A nice picture on a HDMI screen.
Wondering about passing through a USB mouse/keyboard for each desktop.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,422
Messages: 4,188

I was able to get two NVidia video cards working with two monitors and two seperate VM's.
The hangup is input. No mouse or keyboard.
For testing setup I bought some USB3.0 PCIe cards and plan on passing the card thru. NEC USB3 chipset.
Curious to see how USB disks work and mouse and kb in a VM with their own dedicated host device.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,422
Messages: 4,188

I am happy to say I have passed thru the USB3 controller and I have a ums0 mouse working on a VM's Xorg hdmi screen.
Bought 2 USB3 cards so checking on the second one now in a second VM.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,422
Messages: 4,188

Awe shucks that won't work. When I mentioned above I have 2 video cards on 2 VM's, notice I did not use the word 'simultaneous'.
The only way I can get this to work is launching bhyve --in the foreground-- and with that method I can launch Xorg in for the foreground VM mapped onto a dedicated HDMI monitor..

To do this for testing maybe I run VM in the backround and do autologin and startxfce script to bring it up kiosk style.
Sure is hacky. I could do multiple Xorg VM's to multiple monitors that way believe.
I also have a virtio-console method in mind to test.
 

aragats

Daemon

Reaction score: 524
Messages: 1,255

I am happy to say I have passed thru the USB3 controller and I have a ums0 mouse working on a VM's Xorg hdmi screen.
That's really cool! I never had luck to pass through a USB controller, always got errors with various controllers and PCs (reported a couple of times on these forums). I don't think I tried with 12.0 though.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,422
Messages: 4,188

I was only testing with mouse. So I opened mousepad text editor and no keyboard, so I hotplugged a keyboard in my passthru USB3 controller and I had text in mousepad.
Mission complete on VM1

Time for the real test.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,422
Messages: 4,188

I think the problem with using onboard USB is you would lose the xhci controller to the host system.
I need that (mouse/kybd) for host management.
The onboard USB controllers all hang off of one pci address.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,422
Messages: 4,188

I have not tried that port yet but it sounds like something that might work.
Right now I like using a separate USB3 controller for each Xorg VM.
Problem is I only have 7 slot motherboard and I am running out of slots.
Trying to use 2 NVMe, Chelsio 10G, and 2 video cards and 2 USB cards. Maxed out at 2 VM's.
I could bifurcate my NVMe as a last resort.
Noticed that my onboard USB uses two PCI addresses so I need to suss that out.
It appears that the USB pin headers are on a separate PCI bus. So I can use those.

Does anybody know of any Video cards with USB inputs?
Hardware virtualization cards that work in FreeBSD?
Here is what I have seen on ebay;
TERADICI PCoIP cards:
I doubt these would work but I like the idea. Multiple functions per slot.
 

aragats

Daemon

Reaction score: 524
Messages: 1,255

Just tested a USB-over-Ethernet adapter BB-UE204 with Windows bhyve guest as a keyboard and mouse interface. Works fine. The drawback is its high price ($200+). I got one at work place, but wouldn't buy if had to pay own money.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,422
Messages: 4,188

Ahh yes. BlackBox makes adapter for everything (at a steep price).

What I was just wondering is 'does PCIe contain USB natively'.
I bought a 90 degree PCIe riser from an IBM box and it had a USB jack on it.
The thing I worry about with that approach (like USB with video card) is that it would probably not use separate PCI bus ID.
I dunno. All new to me.
EDIT: I see nothing in PCIe slot signaling for USB. Only miniPCIe adds USB2 for cellular modems and their sims.
Probably a special IBM X36xx riser for certain slot in that server.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,422
Messages: 4,188

Has been crickets from that guy..

As far as I can tell this works with FreeBSD no special sauce needed. Just make a xorg.conf with the BusID.
Surprised no one has tried it before.
I am now launching my two Xorg VM's with tmux. So no issues there. I though maybe tmux would be an issue.
I am copying in some MPG videos now to test HDMI audio passthru to separate monitors.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,422
Messages: 4,188

No HDMI audio without any work.
I did test USB Memory sticks and found that device discovery happens quick as normal, but the CAM support lags by a minute.
No device in /dev until CAM.
 

shkhln

Aspiring Daemon

Reaction score: 235
Messages: 752

Does OpenGL actually work? And after multiple VM reboots?

So I'd wager he got it working and now is deep in some red dead redemption session ;)
That game is kind of infamous for being console exclusive. (Am I ruining the joke?)
 
Top