Bhyve GPU pass-through

Crivens

Moderator
Staff member
Moderator

Reaction score: 977
Messages: 1,984

That game is kind of infamous for being console exclusive. (Am I ruining the joke?)
No, you just expose my complete ignorance of the gaming subuniverse from the last 10 years. ;)
 
Last edited:
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,505
Messages: 4,350

Does OpenGL actually work? And after multiple VM reboots?
glxgears -info
Code:
303 frames in 5.0 seconds = 60.520 FPS
297 frames in 5.0 seconds = 59.379 FPS
301 frames in 5.0 seconds = 60.002 FPS
300 frames in 5.0 seconds = 59.828 FPS
300 frames in 5.0 seconds = 59.978 FPS
301 frames in 5.0 seconds = 60.002 FPS
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,505
Messages: 4,350

glxinfo |grep render
Code:
direct rendering: Yes
OpenGL renderer string: GeForce GT 730/PCIe/SSE2
    GL_ARB_conditional_render_inverted, GL_ARB_conservative_depth,
    GL_KTX_buffer_region, GL_NVX_conditional_render, GL_NVX_gpu_memory_info,
    GL_NV_compute_program5, GL_NV_conditional_render,
    GL_NV_path_rendering, GL_NV_pixel_data_range, GL_NV_point_sprite,
    GL_ARB_compute_variable_group_size, GL_ARB_conditional_render_inverted,
    GL_KTX_buffer_region, GL_NVX_conditional_render, GL_NVX_gpu_memory_info,
    GL_NV_compute_program5, GL_NV_conditional_render,
    GL_NV_path_rendering, GL_NV_pixel_data_range, GL_NV_point_sprite,
    GL_EXT_multisample_compatibility, GL_EXT_multisampled_render_to_texture,
    GL_EXT_multisampled_render_to_texture2, GL_EXT_occlusion_query_boolean,
    GL_EXT_render_snorm, GL_EXT_robustness, GL_EXT_sRGB,
    GL_NV_blend_equation_advanced, GL_NV_conditional_render,
    GL_NV_packed_float_linear, GL_NV_path_rendering,
    GL_OES_fbo_render_mipmap, GL_OES_geometry_point_size,
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,505
Messages: 4,350

I have went as far as I can. With a 7 slot motherboard I can only get 2 maybe 3 Xorg VM desktops running from it.
If money was no object I found a USB3 controller with dual channels. Probably uses 2 Bus ID's.
With that I could possibly serve 4 independant Xorg desktops. Each with a monitor(multiple possible too) and USB.
I must say I had little success passing through a subset of onboard USB PCI address.
eHCI, xHCI- hands off - USB Enabled, Auto, Smart Auto. So many BIOS settings and I ran them all.
No way could I split the two USB PCI devices. Either all on host or all passed through. The only way I could get onboard USB.

Considered trying Linux VM Xorg but I don't need that. I keep a Devuan VM on the side for building OpenWRT images.

I could see a use case for a single Xorg VM just for isolation and security purpose. Similar to running X in a jail.

I would like to hear others experiences. I never bothered to nail down the lack of audio, but I see the device in guest VM dmesg.
Maybe an Xorg setting or mixer.

Somehow I was hoping this TwitterGuy was using the bhyve framebuffer VNC-EFI and pass that through for console too.
Was thinking why can't you redirect VNC-EFI console video-out to a video card instead of over an IP.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,505
Messages: 4,350

And after multiple VM reboots?
My dirty little bhyve scripting doesn't do VM reboots I have to take down the whole rig.
I use a shutdown.rc.local script from host to issue SIGTERM events..
I like freewheeling it.
Learning scripting for launching exec, daemon and finallly tmux sessions has been great learning.

I am now launching my two Xorg VM's with tmux. So no issues there.
Actually this was slightly wrong..
With tmux launching I lost my ability to autostart x/autologin via rc.local script I was using.
Manually running via startxfce4 in the tmux session worked fine. Tried extra sleep time in script with no success.

For hardware virt I look alot at this picture.
 

shkhln

Aspiring Daemon

Reaction score: 295
Messages: 897

Ok, so I finally decided to give bhyve a try:

Code:
% sudo sh /usr/share/examples/bhyve/vmrun.sh -c 1 -d FreeBSD-12.0-RELEASE-amd64.raw -p 1/0/0 -p 1/0/1 -p 1/0/2 -p 1/0/3 vm0

...

root@freebsd:~ # kldload nvidia-modeset
nvidia0: <Unknown> on vgapci0
vgapci0: child nvidia0 requested pci_enable_io
vgapci0: child nvidia0 requested pci_enable_io
root@freebsd:~ # X -retro

X.Org X Server 1.18.4
Release Date: 2016-07-19
X Protocol Version 11, Revision 0
Build Operating System: FreeBSD 12.0-RELEASE-p5 amd64
Current Operating System: FreeBSD freebsd 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC amd64
Build Date: 16 May 2019  09:00:46AM

Current version of pixman: 0.34.0
        Before reporting problems, check http://wiki.x.org
        to make sure that you have the latest version.
Markers: (--) probed, (**) from config file, (==) default setting,
        (++) from command line, (!!) notice, (II) informational,
        (WW) warning, (EE) error, (NI) not implemented, (??) unknown.
(==) Log file: "/var/log/Xorg.0.log", Time: Tue May 28 17:11:27 2019
(==) Using config file: "/etc/X11/xorg.conf"
NVRM: GPU 0000:00:04.0: RmInitAdapter failed! (0x26:0x65:1133)
nvidia0: NVRM: rm_init_adapter() failed!
(EE)
Fatal server error:
(EE) no screens found(EE)
(EE)
Please consult the The X.Org Foundation support
         at http://wiki.x.org
for help.
(EE) Please also check the log file at "/var/log/Xorg.0.log" for additional information.
(EE)
(EE) Server terminated with error (1). Closing log file.
Am I missing something obvious?
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,505
Messages: 4,350

Did you make a video card config file in VM's /usr/local/etc/X11/xorg.d/ with the BusID of passed-thru video card?
 

shkhln

Aspiring Daemon

Reaction score: 295
Messages: 897

Did you make a video card config file in /usr/local/etc/X11/xorg.d/ with the video card BusID ?
Yes, the relevant section:
Code:
Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID          "PCI:0:4:0"
EndSection
Code:
root@freebsd:~ # pciconf -lv
...
vgapci0@pci0:0:4:0:     class=0x030000 card=0x218410de chip=0x218410de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    device     = 'TU116 [GeForce GTX 1660]'
    class      = display
    subclass   = VGA
hdac0@pci0:0:5:0:       class=0x040300 card=0x218410de chip=0x1aeb10de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    class      = multimedia
    subclass   = HDA
xhci0@pci0:0:6:0:       class=0x0c0330 card=0x218410de chip=0x1aec10de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    class      = serial bus
    subclass   = USB
none0@pci0:0:7:0:       class=0x0c8000 card=0x218410de chip=0x1aed10de rev=0xa1 hdr=0x00
    vendor     = 'NVIDIA Corporation'
    class      = serial bus
I also had to set kern.vty=sc in boot/loader.conf because of the very bizarre "xf86OpenConsole: No console driver found" error message from Xorg.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,505
Messages: 4,350

I was using FreeBSD 12-STABLE on host and FreeBSD 12-RELEASE on guest VM's.
Hardware is SuperMicro X9SRL and then when messing with onboard USB ports I moved the setup to X10SRI.
PCIe x8 PNY GT710 and GT730 Video cards. The GTX650 I started with was too wide for my slots arrangement.
Started with a pair of 960GB PM983 NVMe then downsized to one to free up PCIe slots..
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,505
Messages: 4,350

How do you have everything in pciconf as NVIDIA?? For example:: NVIDIA Serial Bus - USB

Is this a Nvidia chipset motherboard?? AMD board? LGA2011?

I saw no such errors as you have.

Maybe I need to try this on a Skylake Xeon too. I thought it was easy breezy.
Just like an ordinary Xorg-Nvidia setup.
 

shkhln

Aspiring Daemon

Reaction score: 295
Messages: 897

How do you have everything in pciconf as NVIDIA?? For example:: NVIDIA Serial Bus - USB Is this a Nvidia chipset motherboard?? AMD board? LGA2011?
Let's have a look: besides the GPU itself there is some kind of an audio device, a USB device and some mysterious serial bus. The first one is used for HDMI / Display Port audio, the second is probably for USB-C port present on some Turing GPUs (I don't have one — it's a bit of a premium feature and GTX 1660 is a bottom of the middle range GPU, but whatever) and I have no clue about the third device.

Oh, it's not "everything" — I trimmed uninteresting entries, as usual.

I saw no such errors as you have.
Eh… As far as loading in UEFI mode is concerned, it's just a broken image. Reinstalling everything from ISO works for me. Still the same RmInitAdapter error, though.
 

shkhln

Aspiring Daemon

Reaction score: 295
Messages: 897

A bit more info:
Code:
root@attempt2:~ # sysctl hw.nvidia.registry.ResmanDebugLevel=0
hw.nvidia.registry.ResmanDebugLevel: 4294967295 -> 0
root@attempt2:~ # echo 123 > /dev/nvidia0
NVRM: GPU 0000:00:04.0: RmInitAdapter
NVRM: GPU 0000:00:04.0: RmSetupRegisters for 0x10de:0x2184
NVRM: GPU 0000:00:04.0: pci config info:
NVRM: GPU 0000:00:04.0:    registers look  like: 0xc1000000 0x1000000NVRM: GPU 0000:00:04.0:    fb        looks like: 0xd000000000 0x10000000NVRM: GPU 0000:00:04.0: Successfully mapped framebuffer and registers
NVRM: GPU 0000:00:04.0: final mappings:
NVRM: GPU 0000:00:04.0:     regs: 0xc1000000 0x1000000 0x0xfffff800c1000000
NVRM: nv_acpi_dsm_method: failed to evaluate _DSM method!
NVRM: nv_acpi_dsm_method: failed to evaluate _DSM method!
NVRM: nv_acpi_dsm_method: failed to evaluate _DSM method!
NVRM: nv_acpi_dsm_method: failed to evaluate _DSM method!
NVRM: nv_acpi_dsm_method: failed to evaluate _DSM method!
NVRM: nv_acpi_dsm_method: failed to evaluate _DSM method!
NVRM: nv_acpi_dsm_method: failed to evaluate _DSM method!
NVRM: nv_acpi_dod_method: failed to evaluate _DOD method!
NVRM: GPU 0000:00:04.0: RM reports GPU is not primary VGA
NVRM: GPU 0000:00:04.0: PCI config space reports GPU is not primary VGA
NVRM: GPU 0000:00:04.0:  is not primary VGA
NVRM: nv_acpi_dod_method: failed to evaluate _DOD method!
NVRM: GPU 0000:00:04.0: Tearing down registers
NVRM: GPU 0000:00:04.0: RmInitAdapter failed! (0x26:0x65:1133)
nvidia0: NVRM: rm_init_adapter() failed!
/dev/nvidia0: Input/output error.
(Any attempt to access /dev/nvidia* device node is enough to trigger that behavior, so Xorg is not really necessary for testing.)
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,505
Messages: 4,350

Do you have any older Nvidia cards? I had this working with nvidia 340 driver with a legacy card too. GT2xx based card.
How about Radeon. I don't think there is anything special here. I should have tried that.
 

shkhln

Aspiring Daemon

Reaction score: 295
Messages: 897

I gifted my old Kepler card away a few days ago. I'm mostly interested in the newer drivers, hence it wouldn't be very useful even if it does work.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,505
Messages: 4,350

If there is anything I can do my bhyve VM setup NVMe is on a shelf. Can revist.

UEFI actually fails with "Boot Failed. EFI Misc Device", which is not something I expected from an official bhyve image.
Yes this is strange. I have ran the bhyve Tiano UEFI firmware on both Legacy host install as well as EFI host install.
Running package system 'latest' for the firmware. Copy it out to VM directory for convenience.
The Tiano -UEFI-CSM module is broke but I saw DianeB. investigating that in mailing list post.

On my test machines BIOS I had all the PCIe settings jammed to EFI. Every setting. No legacy. EFI only.
With that my newer LSI Controller Cards actually show setup screen in BIOS and you can configure them there.
EFI Payload I assume.

Can you please list your motherboard and CPU. The Skylake Xeon needs bhyve patch for IOMMU to work correctly.
 

shkhln

Aspiring Daemon

Reaction score: 295
Messages: 897

Yes this is strange. I have ran the bhyve Tiano UEFI firmware on both Legacy host install as well as EFI host install.
Running package system 'latest' for the firmware. Copy it out to VM directory for convenience.
The Tiano -UEFI-CSM module is broke but I saw DianeB. investigating that in mailing list post.
To be precise, I mean specifically this VM image: https://download.freebsd.org/ftp/releases/VM-IMAGES/12.0-RELEASE/amd64/Latest/FreeBSD-12.0-RELEASE-amd64.raw.xz. I have no complaints about UEFI firmware.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,505
Messages: 4,350

Ahha. The official raw disk image for VM. I still use a proper "installed" VM but I am working to just generating custom images.
The raw disk images made me feel hemmed-in by the pre-determined size.
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,505
Messages: 4,350

I ended up getting a Startech 4 port USB3 card WITH 4 Channels. Yehawwwww.
Code:
pcib12@pci0:8:9:0:    class=0x060400 card=0x860810b5 chip=0x860810b5 rev=0xba hdr=0x01
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch'
    class      = bridge
    subclass   = PCI-PCI
xhci0@pci0:9:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
ppt0@pci0:10:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
xhci1@pci0:11:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
xhci2@pci0:12:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
In ONE slot I have 4 seperate USB devices I can pass.
PLX controller was recognized and it looks groovy.
I did get a different model than the one I talked about above.
I found Startech has 4 separate bus USB3 cards and had to try that. 40 Bucks used.
Looks like it was recognised liked I hoped it would. Each port is on its own BUS-ID.

Still need to verify it all works and re-arrange my PCI stuff for passthru..
 
OP
OP
Phishfry

Phishfry

Son of Beastie

Reaction score: 1,505
Messages: 4,350

OK Short and dirty on this USB3 card.
PEXUSB3S400 STARTECH
https://www.startech.com/support/PEXUSB3S400
It has everything I wanted but does not work for virt.
One VM I connect my physical KVM cable and bingo, it shows up.
Move to another VM and connect my USB KVM cable and I see usbhub errors spewing across screen.
I also tested a memstick in the assigned USB slot and that caused error spew as well and console was unusable.
So great card for ordinary use but not for VM use. Unless you pass all through to one VM.

I kinda knew this was coming as I read EVERYTHING before buying.
Luckily people left virt comments when purchasing. KVM on Linux had the same issue.

These cards do have a PLX chip for control. It shows up as 4 separate PLX devices in pciconf.
So I tried pairing these with a physical USB device. The sibling PLX chips will not passthrough. At all. Ever.
The structure seems to be like this
13:0:0 = Parent PLX address
14:1:0=Sibling PLX address
14:5:0=Sibling PLX address
14:7:0=Sibling PLX address
14:9:0=Sibling PLX address
15:0:0=USB controller
16:0:0=USB controller
17:0:0=USB controller
18:0:0=USB controller

Code:
pcib13@pci0:13:0:0:    class=0x060400 card=0x860810b5 chip=0x860810b5 rev=0xba hdr=0x01
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib14@pci0:14:1:0:    class=0x060400 card=0x860810b5 chip=0x860810b5 rev=0xba hdr=0x01
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib15@pci0:14:5:0:    class=0x060400 card=0x860810b5 chip=0x860810b5 rev=0xba hdr=0x01
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib16@pci0:14:7:0:    class=0x060400 card=0x860810b5 chip=0x860810b5 rev=0xba hdr=0x01
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch'
    class      = bridge
    subclass   = PCI-PCI
pcib17@pci0:14:9:0:    class=0x060400 card=0x860810b5 chip=0x860810b5 rev=0xba hdr=0x01
    vendor     = 'PLX Technology, Inc.'
    device     = 'PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch'
    class      = bridge
    subclass   = PCI-PCI
ppt6@pci0:15:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
ppt7@pci0:16:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
ppt8@pci0:17:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
ppt9@pci0:18:0:0:    class=0x0c0330 card=0x00151912 chip=0x00151912 rev=0x02 hdr=0x00
    vendor     = 'Renesas Technology Corp.'
    device     = 'uPD720202 USB 3.0 Host Controller'
    class      = serial bus
    subclass   = USB
 
Top