Bhyve GPU pass-through

i thought you could onl connect to byhve sessions with VNC
that would not be suitable for gaming it hought?

Since X11 is network aware you can also forward that to the host. It will be indirect rendering however so quite a bit slower than direct because it is going through the network (by design). In my experience, for retained stuff (i.e vertex data uploaded to the GPU) it is pretty good but for old OpenGL immediate mode usage (i.e the games I want to play!!!), it is not great performance.

If you do try this; don't forward through ssh (even though that is easier). The extra encryption and compression is wasteful. Instead use xhost to allow the guest OS X11 programs to access your running X11 server.

Off topic ramble:
Luckily the older games don't do *that* much in terms of rendering so a software rasteriser is feasible (Xvnc + llvmpipe is OK). Unfortunately the platforms that they were designed for (Windows 98) have no real software rendering support and the third party stuff (i.e Google's SwiftShader) are implemented in unportable ways and only support Windows 7+ which my good old Windows 98 games don't support. I just can't seem to win. It is like the world doesn't want me to play my Windows 98 games! ;)
 
Try it and report your results. Everybody is happy to talk about GPU passthrough, but what we really need are tests with different hardware configurations.
I mean, I would try win 7/10 but anyone got it running? Linux passthrough is possible, windows as far as I know - nope
 
i thought you could onl connect to byhve sessions with VNC
that would not be suitable for gaming it hought?

That's completely irrelevant for GPU passthrough. What's relevant is having displays for host and guest GPUs respectively.

Off topic ramble:
Luckily the older games don't do *that* much in terms of rendering so a software rasteriser is feasible (Xvnc + llvmpipe is OK). Unfortunately the platforms that they were designed for (Windows 98) have no real software rendering support and the third party stuff (i.e Google's SwiftShader) are implemented in unportable ways and only support Windows 7+ which my good old Windows 98 games don't support. I just can't seem to win. It is like the world doesn't want me to play my Windows 98 games! ;)

SwiftShader definitely supported Win XP at some time in the past. Why would you want SwiftShader on Win 98 anyway? It only works with d3d 8 and 9.
 
SwiftShader definitely supported Win XP at some time in the past.

Possibly when it was still managed by the TransGaming guys? I think it could very well be that Google are linking it against a recent C/C++ runtime that prevent its usage on anything below Windows 7. I know I tried compiling it on an older OS but I made very little progress.

Why would you want SwiftShader on Win 98 anyway? It only works with d3d 8 and 9.

Yeah, that is true. I suppose I just haven't seen a DirectX 3 software renderer so I can run Interstate '76 haha. Admittedly looking at it now; I might have a bit more luck with some sort of voodoo/glide emulation.
 
i thought you could onl connect to byhve sessions with VNC
that would not be suitable for gaming it hought?
It is possible to directly passthru some NVidia video cards. I used 7xx series and had 3 cards attached to different VM's with passthru.
I also passed thru USB cards for user inputs. This was using FreeBSD only. I did not test Linux VM's.
GLXgears showed normal performance.
 
Just a side note on this very interesting thread. I've just decommissioned my 4 years old multi-head virtualized workstation built on top of ESXi. It used PCI pass-through extensively: 1 Win7 (later Win10) VM with Radeon GPU + full USB controller + digital audio, 1 OSX VM with Radeon GPU + full USB controller, 1 Linux VM with full USB controller.
Building parts are of paramount importance. I've chosen at the time a Supermicro X10SRA-F motherboard that sports 3 distinct USB controllers, allowing me to use one per VM (as I don't need one for ESXi). Passing through of a complete USB controller allows you to plug absolutely what you want: any keyboard/mouse of course, but also thumdrives, CF card reader, USB hub, yubikey, DVD burner, Nooelec NESDR SMArt, USB-serial adapter, USB Creative Sound card, USB keylogger (yes I did) etc.
I used a USB switch (Aten US424-AT) in order to present a set of devices to a unique VM at the same time (keyboard, mouse, Creative sound card, Yubikey): one press on a button and those devices would be unplugged from USB controller 1 and plugged on USB controller 2, etc. Every GPU was hooked on the same display: one press of a button on the display to switch inputs. Very convenient :)
I really hope you'll overcome all obstacles and limitation and get a nice multihead workstation built on top of FreeBSD+bhyve!
 
Heads up guys.

Corvin Köhne has created a patch for bhyve, it allows passing through an Intel IGD to VM. Windows guest and Linux guest both work.
You can check following URL

D26209
 
It is a strange use case. Passing through the IGD. I guess you would need to use ssh to control the host.
I am glad someone is working on different options for VM graphics.
 
Using FreeBSD as a headless hypervisor can be a very nice idea, giving up its integrated GPU to a guest VM. You got ZFS as storage and that alone is a killing feature.
 
Here is my /etc/rc.local I use to start up my VM. Device 5:0 and 5:1 are my video card.
Code:
 # cat /etc/rc.local
#!/bin/sh
bhyve -S -c 4 -m 3G -AHP -u -s 0:0,hostbridge -s 1:0,lpc -s 2:0,ahci-hd,/vm/freebsd/freebsd1.img -s 3:0,passthru,2/0/0 -s 5:0,passthru,5/0/0 -s 5:1,passthru,5/0/1 -s 30:0,xhci,tablet -l com1,stdio -l bootrom,/vm/freebsd/BHYVE_UEFI.fd freebsd1 &
Notice with the ampersand it sends the process to the backround.
With no ampersand it boots to the VM and I can run startxfce4.
Host is unreachable when ran this way. This way is hacky but it does work. Probably a better way.

Still working on mouse input. Isn't that what xhci,tablet does?

why you passed thru 5/0/0 and 5/0/1 if your system has :

vgapci0@pci0:0:5:0: class=0x030000 card=0x129119da chip=0x0fc610de rev=0xa1 hdr=0x00
vendor = 'NVIDIA Corporation'
device = 'GK107 [GeForce GTX 650]'
class = display
subclass = VGA
--------------------------------------------------------------------------------------------------------------------------------------------
hdac0@pci0:0:5:1: class=0x040300 card=0x129119da chip=0x0e1b10de rev=0xa1 hdr=0x00
vendor = 'NVIDIA Corporation'
device = 'GK107 HDMI Audio Controller'
class = multimedia
subclass = HDA

shouldn't you have passed 0/5/0 and 0/5/1 ?
 
anyway I did the same. I created a xorg.conf file and I've placed inside the ubuntu VM and then I've tried to boot it in bhyve. On my part it didn't work. The VM does not boot at all. Did u install also the nvidia drivers inside the linux vm before to launch it with bhyve ? (I'm trying with the RTX 2080 ti and the Gtx 1060)
 
Well I think shkhln made a good point above.
We need to document where this works and where it doesn't.

I am wondering about some things.

Is anybody trying this on Dual CPU boards? Single socket only? E3 Xeon or E5 Xeon?

As stated LGA2011 SuperMicro E5- X9/X10 was my testbed.

My tested platforms were GT710 and GT730. Some 1 Gig some 2 Gig video memory.
All different brands. All x8 slot PCIe cards due to lack of x16 slots on my board.

And like patpro I used a Belkin 16 port KVM connected to my 3 video cards.
One port used for host too..
 
Hi Guys!

I'm try to up that topic.

Is there any options to get Win7/10 with VGA (Ryzen 5700g video) passthrought to guest?
I have installed Win10 on Bhyve, and install RedHat VirtIO drivers and get device called "Red Hat VirtIO GPU DOD Controller", but that device stoped with "Code 43".

And simple Win 3D games not works. Not throught RDP, nor VNC

What solutions i have?
 
That gpu smells to behave like the nvidia gpu on a Windows OS. I suspect that it does not work because the lack of the "line interrupts support for passed through devices". Nothing that you can do because the developers does not want to fix it.
 
Some results:

7000 series AMD CPU with AMD-VI.
I'm not sure if there are security concerns with giving out a motherboard model, so omitting that.
Monitor is a TV from around 2007 or so. I'm using one monitor.
This is on 13.1, and I think I'm up-to-date.

On the Host FreeBSD Machine:
in /boot/loader.conf
Code:
#Reserve device for Bhyve passthrough
hw.vmm.amdvi.enable=1
vmm_load="YES"
pptdevs="3/0/0"

Code:
ppt0@pci0:3:0:0:        class=0x030000 rev=0xc7 hdr=0x00 vendor=0x1002 device=0x73ff subv>
    vendor     = 'Advanced Micro Devices, Inc. [AMD/ATI]'
    device     = 'Navi 23 [Radeon RX 6600/6600 XT/6600M]'
    class      = display
    subclass   = VGA

I set this up for Linux Mint beforehand by following the Handbook.
I'm not sure what the differences are between the uefi-firmware files, but I saw this one mentioned.
/path/to/initialize/bhyve.sh
Code:
#! /bin/sh

bhyve -AHPS \
-s 0:0,hostbridge \
-s 1:0,lpc \
-s 2:0,virtio-net,tap0 \
-s 3:0,passthru,3/0/0 \
-s 5:0,virtio-blk,./mint.img \
-s 6:0,hda,play=/dev/dsp,rec=/dev/dsp \
-s 30,xhci,tablet \
-s 29,fbuf,tcp=0.0.0.0:5900,wait \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
-c 8 \
-m 8G linuxguest

On the linux guest:
In the file /etc/X11/xorg.conf
Code:
Section "Device"
        Identifier "AMD Card"
        Driver "amdgpu"
        BusID  "PCI:0:3:0"
EndSection

Section "Screen"
        Identifier "Screen0"
        Device "AMD Card"
EndSection

Boot host, log in, and start X.
Start two terminal emulators.
Running the above script in a terminal emulator in the foreground gives a short response about the framebuffer being active.

The next few lines prevent me from going back to the FreeBSD host without a reboot.
Run remmina in the unused terminal (VNC viewer), select VM -> it crashes on the first run, creating a core file.
This next line is the point of no return.
Run remmina again, select the VM. -> this time it succeeds.
Without GPU passthrough - it creates a window and Linux will show up normally.
With GPU passthrough - The Display gets captured.

If the xorg.conf file does not exist or used PCI:3:0:0 instead of PCI:0:3:0, I get a black screen.
Audio on the host still works, but the host is now unreachable.
Using 14-CURRENT, and without running X, I would get the Linux Mint log-in screen, without a usable mouse or keyboard. X's log mentions PCI:0:3:0.

If the file used "PCI:0:3:0" :
At this point the Display goes black, with a thin green line at the bottom of the monitor.
The Linux Mint logo comes up, followed by the login screen.
The mouse pointer: works on the guest.
The keyboard: works on the guest.
Audio: works on the guest.

At this point I attempted to install some programs to test it.
GlxGears: works on the guest, and reports the GPU as being a 6600.
radeontop: works on the guest.
Steam: works on the guest, as I can play Factorio. This fixes some minor screen tearing.

Getting machine learning running seems to be fiddly.
Add the user to the render and video groups.
Install packages.
In python:
Code:
import torch; torch.cuda.is_available() returns True.
clinfo runs.
rocminfo runs.
I haven't gone much further.
 
Back
Top