bhyve How to use the framebuffer as primary video device instead of the nvidia passed-through graphic card in a bhyve/linux vm

Hello to everyone.

what Im trying to do is to set the framebuffer video adapter as primary graphic card on my bhyve-ubuntu vm instead of the nvidia RTX 2080 ti card that I have passed through. What I want to do really is to use both the graphic adapters,but the primary should be the framebuffer and the secondary the nvidia. I tried different Xorg configurations,but what Ive got has been that Xorg failed displaying some errors. So,the controller that you see below should be used as primary inside the ubuntu vm :

Code:
00:1d.0 VGA compatible controller: Device fb5d:40fb

while the ones you see below as secondary :

Code:
08:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1)
08:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Controller (rev a1)
08:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (rev a1)
08:00.3 Serial bus controller: NVIDIA Corporation TU102 USB Type-C UCSI Controller (rev a1)

The script that I use to launch the vm is the following :

Code:
#!/bin/sh
setxkbmap it
vms="$(ls /dev/vmm/*)"
vncs="vm$(ps ax | awk '/vncviewer [0]/{print $6}')"

for vm in $vms; do
                session="${vm##*/}"
                echo "bhyve session = $session"
                echo "vnc session = $vncs"
                if ! printf '%s\n' "${vncs}" | grep "${session}"; then
                                printf 'VNC session not found,destroying ghost vms\n'
                                bhyvectl --vm=$session --destroy                          
                else
                                printf 'Found VNC session %s\n' "${session},no ghost vms found,not destroying them"
                fi
done

vmdisk1=`geom disk list | awk '/^Geom name: /{d=$NF} /^ *ident: (2015020204055E)/ && d{print d}'`
echo "TOSHIBA External USB 3.0 1.8 TB ; $vmdisk1"

mount -t ufs /dev/$vmdisk1'p2' /mnt/$vmdisk1'p2'

bhyve -S -c sockets=1,cores=2,threads=2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 2,virtio-blk,/mnt/$vmdisk1'p2'/bhyve/img/Linux/ubuntu2210.img,bootindex=1 \
-s 3,virtio-blk,/dev/$vmdisk4 \
-s 4,virtio-blk,/dev/$vmdisk2 \
-s 8:0,passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \
-s 10,virtio-net,tap19 \
-s 11,virtio-9p,sharename=/ \
-s 29,fbuf,tcp=0.0.0.0:5919,w=1600,h=950,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:19 < /dev/null & sleep 2 && vncviewer 0:19

For sure ,on /boot/loader.conf Ive added :

Code:
/boot/loader.conf

pptdevs="2/0/0 2/0/1 2/0/2 2/0/3"

As I told before,I tried two xorg conf files to achieve the goal. On the first one I tried to add only the framebuffer,like this :

Code:
Section "ServerLayout"
    Identifier     "X.org Configured"
    Screen      0  "Screen0" 0 0
    InputDevice    "Mouse0" "CorePointer"
    InputDevice    "Keyboard0" "CoreKeyboard"
EndSection

Section "Files"
    ModulePath   "/usr/lib/xorg/modules"
    FontPath     "/usr/share/fonts/X11/misc"
    FontPath     "/usr/share/fonts/X11/cyrillic"
    FontPath     "/usr/share/fonts/X11/100dpi/:unscaled"
    FontPath     "/usr/share/fonts/X11/75dpi/:unscaled"
    FontPath     "/usr/share/fonts/X11/Type1"
    FontPath     "/usr/share/fonts/X11/100dpi"
    FontPath     "/usr/share/fonts/X11/75dpi"
    FontPath     "built-ins"
EndSection

Section "Module"
    Load  "vnc"
    Load  "glx"
EndSection


Section "InputDevice"
    Identifier  "Keyboard0"
    Driver      "kbd"
EndSection

Section "InputDevice"
    Identifier  "Mouse0"
    Driver      "mouse"
    Option        "Protocol" "auto"
    Option        "Device" "/dev/input/mice"
    Option        "ZAxisMapping" "4 5 6 7"

EndSection

Section "Monitor"
    Identifier   "Monitor0"
    VendorName   "Monitor Vendor"
    ModelName    "Monitor Model"
EndSection

Section "Device"
   Identifier  "Card0"
   Driver      "modesetting"
   BusID       "PCI:0:29:0"

EndSection

Section "Screen"
    Identifier "Screen0"
    Device     "Card0"
    Monitor    "Monitor0"
    SubSection "Display"
        Viewport   0 0
        Depth     1
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     4
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     8
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     15
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     16
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     24
    EndSubSection
EndSection

but it didnt work. This is the log file that shows the errors reported : https://pastebin.ubuntu.com/p/Gv7wgsDR4K/

I have also removed the xorg.conf file,but it didnt work also. This is the log file that shows the errors reported :


Can someone give me some suggestion that can help me to understand where is the mistake,please,thanks.
 
Last edited:
I still don't get it why you hard power off vms that have no vnc viewer session, but that is not your original question, it just seems very weird.
 
According to the logs it looks like it selects the framebuffer device as primary graphics in the first place:

Code:
(--) PCI:*(0@0:7:0) ...
(--) PCI: (0@0:8:0) ...

but it does not work. If I use ONLY the framebuffer argument,it works : the desktop manager is loaded within the vm window,but if between the bhyve parameters I declare the framebuffer AND the nvidia slots,on the vm window I see the blinking pointer because the output is redirected to the screen that I have attached to the nvidia HDMI connector. The problem is that at the moment I can't use that monitor,since I'm using my PC in a different room. What I want to do is to use the framebuffer as primary video device and the nvidia adapter only if a 3D application detects automatically that it needs more performance.
 
I still don't get it why you hard power off vms that have no vnc viewer session, but that is not your original question, it just seems very weird.

The reason is that often I need to stop a Linux VM when it is not fully loaded,but it's still on the grub menu. How to close it when it is on that situation ? If I close the vnc session,the VM remains hybernated on the memory. If I close it pressing CTRL+C on the terminal,the same happens. And if there are more hybernated VMs I saw that they can conflict, generating errors. And anyway that VM is no longer functional and it wastes only memory. Also I can live without the higher speed offered by an hibernated VM.
 
The reason is that often I need to stop a Linux VM when it is not fully loaded,but it's still on the grub menu. How to close it when it is on that situation ? If I close the vnc session,the VM remains hybernated on the memory. If I close it pressing CTRL+C on the terminal,the same happens. And if there are more hybernated VMs I saw that they can conflict, generating errors. And anyway that VM is no longer functional and it wastes only memory. Also I can live without the higher speed offered by an hibernated VM.
There may be edge cases, but normally when you close the VNC session, the VM just continues running normally. VNC is just the equivalent to a monitor. So if you disconnect VNC it is like switching of the monitor. The VM is just up an running and you can reconnect at any time.
 
To close it, you need to open a terminal, and just need to leave bhyve destroy it: bhyvectl --destroy --vm=linuxguest,or any other name you gave to the vm, like reported on chapter 23 freeBSD handbook . P.S. when you are on grub menù, just quit it,CTRL + C, no need to destroy the vm.Read this chapter, if you need some suggestions about it. If I correctly intended your purpose, you need a switcher to permit the vm to use both VGA when you need it.
[ 643.533] (EE) Screen 0 deleted because of no matching config section.
[ 643.533] (II) UnloadModule: "modesetting"
[ 643.533] (EE)
Fatal server error:
[ 643.533] (EE) Cannot run in framebuffer mode. Please specify busIDs for all framebuffer devices
this looks the culprit in your case, if you want use a specific card you need to backlist one and specify inside /etc/default/grub whitch one to use if you read the the wiki suggested by me about Passthrou and KVM,so need to use iommu specific configuration with a script to abilify a card like normal use like primary and for a specific use the mentioned Nvidia card that you pasthrou to the vm.PCI Passthrou
look this :

X does not start after enabling vfio_pci​


This is related to the host GPU being detected as a secondary GPU, which causes X to fail/crash when it tries to load a driver for the guest GPU. To circumvent this, a Xorg configuration file specifying the BusID for the host GPU is required. The correct BusID can be acquired from lspci -n or the Xorg log [7]. Note that the value from the lspci output is hexadecimal and should be converted to decimal in the .conf file.

/etc/X11/xorg.conf.d/10-intel.conf
Section "Device"
Identifier "Intel GPU"
Driver "modesetting"
BusID "PCI:0:2:0"
EndSection
maybe you can found the correct busID and create a a specific configuration like this

:-/ for your framebuffer device and another one for Nvidia VGA to permit HDMI output inside the vm

depending on your needs.
 
There may be edge cases, but normally when you close the VNC session, the VM just continues running normally. VNC is just the equivalent to a monitor. So if you disconnect VNC it is like switching of the monitor. The VM is just up an running and you can reconnect at any time.

Sure,but I never close the vnc session,usually I minimize only its window because I find uncomfortable to reopen a new vnc session. For me its faster to maximize its window :P
 
To close it, you need to open a terminal, and just need to leave bhyve destroy it: bhyvectl --destroy --vm=linuxguest,or any other name you gave to the vm...

I know man,but do you realize that "your" method is slower than mine ? you forgot that I should also find the name of the VM that is running,doing a "ps ax | grep bhyve" ; because I use more VM at the same time and I gave to each VM a specific label.

when you are on grub menù, just quit it,CTRL + C, no need to destroy the vm....

I wrote on my comment that I tried to press CTRL + C. Point is that it is a dead VM. It should not stay inside the memory,because I don't need it anymore. And since I use several VMs at the same time,leaving a VM that I don't need anymore may cause conflicts with the other VMs that I need to run and it takes some of the memory wthout a reason.
 
If I correctly intended your purpose, you need a switcher to permit the vm to use both VGA when you need it.

this looks the culprit in your case, if you want use a specific card you need to backlist one and specify inside /etc/default/grub whitch one to use if you read the the wiki suggested by me about Passthrou and KVM,so need to use iommu specific configuration with a script to abilify a card like normal use like primary and for a specific use the mentioned Nvidia card that you pasthrou to the vm.PCI Passthrou
look this :

maybe you can found the correct busID and create a a specific configuration like this

:-/ for your framebuffer device and another one for Nvidia VGA to permit HDMI output inside the vm

depending on your needs.

I tried to assign a busID to the framebuffer. It didn't work. At this point I've started a post on the nvidia forum to understand why it does not work and an nvidia developer replied. You can read his answer going here :


To make it work I should passthru one of the other graphic card that I have on my PC (intel igvt or nvidia 1060). The simple fbdev does not work.
 
Back
Top