bhyve What could keep in use the Intel GPU in my system (based on FreeBSD 13.1-RELEASE)

[ziomario]

Hello.

I've tried to passthru my CoffeeLake-S GT2 [UHD Graphics 630] on a Linux VM but it failed. This graphic card :

ppt4@pci0:0:2:0:        class=0x038000 rev=0x02 hdr=0x00 vendor=0x8086 device=0x3e98 subvendor=0x1458 subdevice=0xd000
vendor     = 'Intel Corporation'
device     = 'CoffeeLake-S GT2 [UHD Graphics 630]'
class      = display

below there are the bhyve parameters that I've used :

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,nvme,/dev/$vmdisk0,bootindex=1 \
-s 2,ahci-hd,/dev/$vmdisk2 \
-s 3,ahci-hd,/dev/$vmdisk4 \
-s 4,passthru,0/2/0 \
-s 7,virtio-net,tap4 \
-s 10,hda,play=/dev/dsp,rec=/dev/dsp \
-s 29,fbuf,tcp=0.0.0.0:5904,w=1500,h=950 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm4 < /dev/null & sleep 2 && vncviewer 0:4

this is the error I get :

bhyve: Warning: Unable to reuse host address of Graphics Stolen Memory. GPU passthrough might not work properly. 
bhyve: gvt_d_setup_opregion: Unable to get OpRegion base and length ; 
bhyve: gvt_d_init: Unable to setup OpRegion ; device emulation initialization error: Operation not supported by device

Someone already tried to pass an Intel integrated gpu with bhyve ? Someone already got that error ? Should I apply some patches ? I know for sure that I can pass through this kind of gpu,using the Corvin patches,but I don't know if I have found a bug that should be fixed. Let me know.

 

[ziomario]

A bhyve developer replied to me in this way :

I thought in general we couldn't pass the GPU if it was being used by the host? (Thus you would need to have two GPUs, one to allow the host to display its output and one to passthrough. Unless the host is completely headless).

At this point I think that the intel GPU is actually in use,but I don't know what could keep it in use. Infact :

1) I've removed the xf86-video-intel driver from the system with :

pkg remove xf86-video-intel

2) I haven't any i915kms or legacy intel module inside the memory :

# kldstat | grep i915 : nothing
# kldstat | grep kms : nothing
# kldstat | grep intel : nothing

3) this is my xorg.conf (where is there is no trace of intel driver)

Section "ServerLayout"
Identifier     "X.org Configured"
Screen      0  "Screen0" LeftOf "Screen1"
Screen      1  "Screen1"
InputDevice    "Mouse0" "CorePointer"
InputDevice    "Keyboard0" "CoreKeyboard"
EndSection

Section "Files"
ModulePath   "/usr/local/lib/xorg/modules"
FontPath     "/usr/local/share/fonts/misc/"
FontPath     "/usr/local/share/fonts/TTF/"
FontPath     "/usr/local/share/fonts/OTF/"
FontPath "/usr/local/share/fonts/Type1/"
FontPath "/usr/local/share/fonts/100dpi/"
FontPath "/usr/local/share/fonts/75dpi/"
FontPath "catalogue:/usr/local/etc/X11/fontpath.d"
EndSection

Section "Module"
Load  "glx"
EndSection

Section "InputDevice"
Identifier  "Keyboard0"
Driver      "kbd"
EndSection

Section "InputDevice"
Identifier  "Mouse0"
Driver      "mouse"
Option        "Protocol" "auto"
Option        "Device" "/dev/sysmouse"
Option        "ZAxisMapping" "4 5 6 7"
EndSection

Section "Monitor"
Identifier   "Monitor0"
VendorName   "Monitor Vendor"
ModelName    "Monitor Model"
EndSection

Section "Monitor"
Identifier   "Monitor1"
VendorName   "Monitor Vendor"
ModelName    "Monitor Model"
EndSection

Section "Device"
Identifier  "Card0"
Driver      "nvidia"
BusID       "PCI:1:0:0"
EndSection

Section "Screen"
Identifier "Screen0"
Device     "Card0"
Monitor    "Monitor0"
SubSection "Display"
Viewport   0 0
Depth     1
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     4
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     8
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     15
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     16
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     24
EndSubSection
EndSection

Section "Screen"
Identifier "Screen1"
Device     "Card0"
Monitor    "Monitor1"
SubSection "Display"
Viewport   0 0
Depth     1
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     4
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     8
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     15
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     16
EndSubSection
SubSection "Display"
Viewport   0 0
Depth     24
EndSubSection
EndSection

 

[SirDice]

The vt(4) console uses it. Actually all video consoles would. Unless you make your system truly "headless" by configuring a serial console.

 

[ziomario]

I'm not sure. If this was true,even the Geforce RTX 2080 ti would not work. Even because the nvidia driver (nvidia and nvidia-modesetting are loaded in memory),so it should be considered in use. But I've just booted a linux vm passing thru the RTX 2080 ti using the same parameters used before and it worked. I've detached the HDMI cable from the 1060 and I've attached it to the 2080 ti and I've got what I wanted from the beginning : to choose between virtualize an OS with bhyve or more easily,only using the second monitor without bhyve. Yes,I want everything. I want also pass thru the intel GPU.

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 1,nvme,/dev/$vmdisk0,bootindex=1 \
-s 2,ahci-hd,/dev/$vmdisk2 \
-s 3,ahci-hd,/dev/$vmdisk4 \
-s 8:0,passthru,2/0/0,rom=TU102.rom \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \
-s 7,virtio-net,tap4 \
-s 10,hda,play=/dev/dsp,rec=/dev/dsp \
-s 29,fbuf,tcp=0.0.0.0:5904,w=1500,h=950 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm4 < /dev/null & sleep 2 && vncviewer 0:4

 

[ziomario]

SirDice : can you elaborate more ? I didn't understand if you have suggested to me to change some parameter or not. If yes,which parameter ?

 

[SirDice]

@SirDice : can you elaborate more ?

What video card do you think provides the image you're looking at on your monitor?

 

[ziomario]

1) monitor AOC + monitor Samsung -> 1 HDMI cable (for the samsung) + 1 DP cable (for the AOC) ---> geforce 1060 - without a bhyve-Linux VM

2) monitor AOC + monitor Samsung -> 1 DP cable attached to the AOC and 1 hdmi cable attached to the RTX 2080 ti - if I want to use a bhyve / Linux vm passing the RTX 2080 ti GPU

3) monitor AOC + monitor Samsung -> 1 DP cable attached to the AOC and 1 HDMI cable attached to the Intel GPU - if I want to use a bhyve / Linux VM passing the Intel GPU

The Intel GPU is always kept free as well as the RTX 2080 ti. Even in the BIOS,since I've chosen the 1060 as default GPU,but the Intel GPU is enabled.

 

[ziomario]

I have 3 GPUs in my PC. Actually the BIOS uses the geforce 1060 to start it. So,two GPUs are out and kept free. The Geforce RTX 2080 ti and the Intel CoffeeLake-S GT2 [UHD Graphics 630]. I'm able to pass through the RTX 2080 ti to a linux vm,but I'm not able to do the same with the UHD Graphics 630. As I have explained in the post,there is nothing that keeps it busy. I'm not using it. Both my monitors are connected to the 1060 using the twinview mode. One is using the DisplayPort,the other one is using the HDMI port. When I want to pass the RTX 2080 inside a Linux VM and I need to use my 2 monitor,I detach the HDMI cable from the 1060 and I attach it to the RTX 2080 ti and it's done. I can use it inside my bhyve VM. If I don't want to use bhyve,I connect the HDMI cable to the 1060 and in this case I can use two screens at the same time without running a VM. Cool. I can do this only with the RTX 2080 ti,but not with the Intel GPU. If I'm obliged to use it even if I don't use it,I can't imagine how and in which kind of situation I can pass it through.

 

[SirDice]

As I have explained in the post,there is nothing that keeps it busy. I'm not using it.

Ever wondered how vt(4) shows information on your screen?

 

[ziomario]

Finally I've found the solution for this problem. Or,at least I'm on the right path. I've reinstalled FreeBSD 13.1-RELEASE without applying the Corvin patches and voila' :

my GVT-d intel GPU is recognized by Windows 11. So,the problem is inside the Corvin's repository. Unfortunately he doesn't reply to my clarification messages about what to do (it has been the first operation I did,but he does not reply from days,so I've looked for some help elsewhere). Anyway I can't use the bhyve code that has been upstreamed on the FreeBSD servers,because it lacks of the nvidia patches.