bhyve Bhyve can't pass thru any of my NVIDIA graphic cards to an Ubuntu bhyved os : the vm freezes before recognizing the disk

Again, that assertion check failure doesn't have anything to do with passthrough. Most likely your /dev/dsp8 simply doesn't refer to anything, since now you have fewer host visible audio devices (after masking the video card ones — that changes order quite a bit).
 
I'm not sure if I have understood well,but this parameter ----> -s 20,hda,play=/dev/dsp8,rec=/dev/dsp8 \ works good without adding the passthru arguments. It allows me to hear the audio coming from the youtube videos,for example. So,maybe it interferes with the passing thru lines.
 
ok. it's partially true. I'm trying to boot Ubuntu with these arguments :

Code: 
bhyve -S -c 4 -m 8G -w -H \
        -s 0,hostbridge \
        -s 1,virtio-blk,/mnt/da1p1/vms/os/ubuntu-budgie-gpu/ubuntu-2104-gpu.img \
        -s 6,virtio-net,tap0 \
        -s 2,passthru,1/0/0 \
        -s 2:1,passthru,1/0/1 \
        -s 2:2,passthru,1/0/2 \
        -s 2:3,passthru,1/0/3 \
        -s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \
        -s 30,xhci,tablet \
        -s 31,lpc -l com1,stdio \
        -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
        vm0

the previous error is gone,but ubuntu is not able to boot. It is freezed on a black screen,before to start loading the img file. I hear the noise that it tells that is trying to do something,but it can't for some reason.
 
In an old thread I've read that I should place a xorg.conf file like this inside the ubuntu VM. What do u think ? And maybe I should even install the nvidia drivers ?


Code: 
Section "ServerLayout"
    Identifier     "Layout0"
    Screen      0  "Screen0" 0 0
    InputDevice    "Keyboard0" "CoreKeyboard"
    InputDevice    "Mouse0" "CorePointer"
    Option         "Xinerama" "0"
EndSection

Section "Files"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Mouse0"
    Driver         "mouse"
    Option         "Protocol" "auto"
    Option         "Device" "/dev/psaux"
    Option         "Emulate3Buttons" "no"
    Option         "ZAxisMapping" "4 5"
EndSection

Section "InputDevice"

    # generated from default
    Identifier     "Keyboard0"
    Driver         "kbd"
EndSection

Section "Monitor"
    Identifier     "Monitor0"
    VendorName     "AOC"
    ModelName      "I2769VM"
    HorizSync       30.0 - 83.0
    VertRefresh     50.0 - 76.0
    Option         "DPMS"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "GeForce RTX 2080 Ti"
    Option         "NoLogo"
    BusID          "PCI:1:0:0"
EndSection

Section "Device"
    Identifier     "Device1"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "GeForce GTX 1060"
    Option         "NoLogo"
    BusID          "PCI:2:0:0"
EndSection

Section "Screen"
    Identifier     "Screen0"
    Device         "Device0"
    Monitor        "Monitor0"
    DefaultDepth    24
    Option         "Stereo" "0"
    Option         "nvidiaXineramaInfoOrder" "DFP-3"
    Option         "metamodes" "DP-2: nvidia-auto-select +1920+0, DP-4: nvidia-auto-select +0+0"
    Option         "SLI" "Off"
    Option         "MultiGPU" "Off"
    Option         "BaseMosaic" "off"
    SubSection     "Display"
        Depth       24
    EndSubSection
EndSection
 
When I was involved in hardware, there was a thing called "minimal config" - it means to remove every feature and try to get the thing running with the bare minimum of stuff - then add the features back one by one and see which one fails.

I would do the same here. I've not seen a linux in quite a time, but I think it should still be able to boot something like single-user maintenance mode. Try to get that far, and then from there advance further.
 
that's good. I will try with a minimal installation of ubuntu. the lastest released is the 18.04. Or even better,with another instance of freebsd as guest. I read that the bhyve pass-thru works for sure with freebsd as guest and there is less chance that it works with linux.
 
Here we go. I've installed FreeBSD instead of Linux with bhyve and I tried to pass thru the rtx 2080 ti using this command :

Code: 
bhyve   -S -c 4 -m 8G -w -H \
        -s 0,hostbridge \
        -s 1,passthru,1/0/0 \
        -s 1:1,passthru,1/0/2 \
        -s 1:2,passthru,1/0/3 \
        -s 2,virtio-blk,/mnt/da1p1/vms/os/freebsd-13R/freebsd-13R.img \
        -s 3,virtio-net,tap0 \
        -s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \
        -s 30,xhci,tablet \
        -s 31,lpc -l com1,stdio \
        -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
        vm0

as u can see :

1) I didn't pass thru the device 1/0/1 because it is the audio device integrated with the rtx 2080 ti
2) I have removed the emulated audio device
3) I have installed freebsd 13R instead of linux
4) without the passthru lines it works

but it didn't work. it freezes on the black screen before to itizialize the disk. On /boot/loader.conf I have added this :

Code: 
pptdevs="1/0/0 1/0/2 1/0/3"

Code: 
pciconf -v -l

ppt0@pci0:1:0:0:    class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e04 subvendor=0x19da subdevice=0x2503
    vendor     = 'NVIDIA Corporation'
    device     = 'TU102 [GeForce RTX 2080 Ti]'
    class      = display
    subclass   = VGA

hdac0@pci0:1:0:1:    class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x10f7 subvendor=0x19da subdevice=0x2503
    vendor     = 'NVIDIA Corporation'
    device     = 'TU102 High Definition Audio Controller'
    class      = multimedia
    subclass   = HDA

ppt1@pci0:1:0:2:    class=0x0c0330 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad6 subvendor=0x19da subdevice=0x2503
    vendor     = 'NVIDIA Corporation'
    device     = 'TU102 USB 3.1 Host Controller'
    class      = serial bus
    subclass   = USB

ppt2@pci0:1:0:3:    class=0x0c8000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad7 subvendor=0x19da subdevice=0x2503
    vendor     = 'NVIDIA Corporation'
    device     = 'TU102 USB Type-C UCSI Controller'
    class      = serial bus

Even this does not work :

-s 1,passthru,1/0/0 \
-s 1:1,passthru,1/0/2 \
-s 1:2,passthru,1/0/3 \
-s 1:3,passthru,2/0/0 \

or this :

-s 1,passthru,2/0/0 \
 
After a lot of tries (yes,Im stubborn),I have been able to avoid the black screen problem (and I found that even this guy has the same problem :
View: https://www.youtube.com/watch?v=Y1FuxpahDrM&t=5s&ab_channel=MarkoTasic.
Actually I have passed succesfully the RTX 2080 ti and the GTX 1060. Unfortunately,I don't see them inside the VM. I want to show you,adding a picture. Please don't complain....I think that Im close to the solution...

Screenshot_20210825_120924.png
 
Phishfry said:
Intel GPU on motherboard
Click to expand...
You mean Intel Integrated Graphics, right? Intel is trying to get into the discrete GPU game (nothing released yet), but the code you reference is for Integrated Graphics... which is known to work under FreeBSD.
 
shkhln said:
Try passing something simpler, say, a sound (motherboard audio should suffice as well) or network card.
Click to expand...

Already did that. I've passed the network card and it worked. And I passed some spare devices attached on my mobo and it worked.
 
astyle said:
You mean Intel Integrated Graphics, right? Intel is trying to get into the discrete GPU game (nothing released yet), but the code you reference is for Integrated Graphics... which is known to work under FreeBSD.
Click to expand...

No. The intel graphic chipset is not enough powerful. I'm not interested to invest my time trying to making the pass thru of it. I want to pass the RTX 2080 ti. it is powerful enough to be used inside the ubuntu VM for a lot of tasks. The nvidia driver refuses to install if it does not find an nvidia graphics card inside the vm. But I've thought to a solution. I could pass thru the card in linux with qemu-kvm from linux to the freebsd/ubuntu/bhyve/guest VM. When I have installed the driver,I will copy the img to freebsd and I will start bhyve.
 
ziomario said:
No. The intel graphic chipset is not enough powerful. I'm not interested to invest my time trying to making the pass thru of it. I want to pass the RTX 2080 ti. it is powerful enough to be used inside the ubuntu VM for a lot of tasks. The nvidia driver refuses to install if it does not find an nvidia graphics card inside the vm. But I've thought of a solution. I could pass thru the card in linux with qemu-kvm from linux to the freebsd/ubuntu/bhyve/guest os. When I have installed the driver,I will copy the img to freebsd and I will start bhyve.
Click to expand...
I was referencing Phishfry 's remarks. The gist was to start with something simple, and then to add on, until you get to the RTX 2080. Just a basic troubleshooting strategy by eliminating a clutter of things that may be in the way of finding a solution.
 
this :

Code: 
/boot/loader.conf

pptdevs="1/0/0 1/0/1 1/0/2 1/0/3"

bhyve -S -c 4 -m 11G -w -H \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/da1p1/vms/os/ubuntu-budgie-gpu/ubuntu-2104-gpu.img \
-s 2:0,passthru,1/0/0 \
-s 2:1,passthru,1/0/1 \
-s 2:2,passthru,1/0/2 \
-s 2:3,passthru,1/0/3 \
-s 6,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \
-s 30,xhci,tablet \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
vm0

produced this error :

root@marietto:/home/marietto/Desktop/Files/bhyve # ./os-uefi-hirsute.sh

VM:vm0 is not created.
fbuf frame buffer base: 0xb04600000 [sz 16777216]
vm exit[0]
reason VMX
rip 0x000000000000fff0
inst_length 3
status 0
exit_reason 48 (EPT violation)
qualification 0x0000000000000784
inst_type 0
inst_error 0
Segnale di annullamento

this is what says dmesg :

[drm] Unable to create a private tmpfs mount, hugepage support will be disabled(-19).
Failed to add WC MTRR for [0x40000000-0x5fffffff]: -22; performance may suffer
[drm] Got stolen memory base rx0, size 0x0
[drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[drm] Driver supports precise vblank timestamp query.
[drm] Connector eDP-1: get mode from tunables:
[drm] - kern.vt.fb.modes.eDP-1
[drm] - kern.vt.fb.default_mode
drmn1: successfully loaded firmware image 'i915/kbl_dmc_ver1_04.bin'
[drm] Finished loading DMC firmware i915/kbl_dmc_ver1_04.bin (v1.4)
[drm] failed to retrieve link info, disabling eDP
[drm] Connector HDMI-A-1: get mode from tunables:
[drm] - kern.vt.fb.modes.HDMI-A-1
[drm] - kern.vt.fb.default_mode
drmn1: Failed to program MOCS registers; expect performance issues.
sysctl_warn_reuse: can't re-use a leaf (hw.dri.debug)!
[drm] Initialized i915 1.6.0 20190822 for drmn1 on minor 0
WARNING: Device "fb" is Giant locked and may be deleted before FreeBSD 14.0.
VT: Replacing driver "efifb" with new "fb".
start FB_INFO:
type=11 height=1080 width=1920 depth=32
cmsize=16 size=8294400
pbase=0x40040000 vbase=0xfffff80040040000
name=drmn1 flags=0x0 stride=7680 bpp=32
cmap[0]=0 cmap[1]=7f0000 cmap[2]=7f00 cmap[3]=c4a000
end FB_INFO
drmn1: fb0: i915drmfb frame buffer device
WARNING: /mnt/da1p1 was not properly dismounted
tap0: link state changed to UP
pid 1841 (bhyve), jid 0, uid 0: exited on signal 6
tap0: link state changed to DOWN

I have a doubt : should I attach a monitor to an HDMI or Display port of the nvidia graphic card ? because I'm not doing it.
 
ziomario said:
The nvidia driver refuses to install if it does not find an nvidia graphics card inside the vm. But I've thought to a solution. I could pass thru the card in linux with qemu-kvm from linux to the freebsd/ubuntu/bhyve/guest VM. When I have installed the driver,I will copy the img to freebsd and I will start bhyve.
Click to expand...
Or you could simply install the driver from a package. Ubuntu definitely has it packaged.
 
ziomario said:
this :

Code: 
/boot/loader.conf

pptdevs="1/0/0 1/0/1 1/0/2 1/0/3"

bhyve -S -c 4 -m 11G -w -H \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/da1p1/vms/os/ubuntu-budgie-gpu/ubuntu-2104-gpu.img \
-s 2:0,passthru,1/0/0 \
-s 2:1,passthru,1/0/1 \
-s 2:2,passthru,1/0/2 \
-s 2:3,passthru,1/0/3 \
-s 6,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \
-s 30,xhci,tablet \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
vm0

I have a doubt : should I attach a monitor to an HDMI or Display port of the nvidia graphic card ? because I'm not doing it.
Click to expand...
Logically, you attach your monitor to whatever port is supported by the monitor. This MUST work for both text and graphical mode.
Just, as was mentioned here, why won't you try to passthrough just ONE graphics card to start with?
Second. Passing through hda sound you don't need device names like /dev/dsp8. This may not even work. Just use /dev/dsp (no number) and the system will create and use the RIGHT one.

Third. At this point you don't need yet nvidia-driver. Make sure things work at all, then you'll install it.
 
shkhln said:
Or you could simply install the driver from a package. Ubuntu definitely has it packaged.
Click to expand...

I'm not sure that the nvidia ubuntu packaged driver works. On the bug report that I've opened,Peter Grehan,asked :

Peter Grehan
freebsd_committer
2021-08-25 11:12:03 UTC

Is this using the proprietary Nvidia driver in the guest ?

So,I supposed that the VM needs the proprietary nvidia driver. Unfortunately he didn't reply anymore.
 
free-and-bsd said:
Logically, you attach your monitor to whatever port is supported by the monitor. This MUST work for both text and graphical mode.
Just, as was mentioned here, why won't you try to passthrough just ONE graphics card to start with?
Second. Passing through hda sound you don't need device names like /dev/dsp8. This may not even work. Just use /dev/dsp (no number) and the system will create and use the RIGHT one.

Third. At this point you don't need yet nvidia-driver. Make sure things work at all, then you'll install it.
Click to expand...

As I have already told you several times,I've passed only one graphic card. The PCI addresses that I've used and that you see BELOW are related to one only graphic card,the RTX 2080 ti.

-s 2:0,passthru,1/0/0 \
-s 2:1,passthru,1/0/1 \
-s 2:2,passthru,1/0/2 \
-s 2:3,passthru,1/0/3 \

2. I've removed the /dev/dsp8 parameter some time ago. Where do u see it ? I suspect that you are replying to my old posts and not to the newer ones.
 
ziomario said:
I'm not sure that the nvidia ubuntu packaged driver works. On the bug report that I've opened,Peter Grehan,asked :

Peter Grehan
freebsd_committer
2021-08-25 11:12:03 UTC

Is this using the proprietary Nvidia driver in the guest ?

So,I supposed that the VM needs the proprietary nvidia driver. Unfortunately he didn't reply anymore.
Click to expand...
You have no idea what you are doing.

Also,
shkhln said:
Please, show us. I'm specifically interested in lspci output.
Click to expand...
I read enough dumb things in this thread, I'm not going to simply believe a statement that it works.
 
shkhln said:
You have no idea what you are doing.

Also,

I read enough dumb things in this thread, I'm not going to simply believe a statement that it works.
Click to expand...

Excusme,but your answers sounds incomprehensible to me. I don't understand why u are telling that I don't know what I'm doing. I don't understand if Peter wanted me to install the nvidia proprietary driver or not. And I don't understand what u mean with "I read enough dumb things in this thread, I'm not going to simply believe a statement that it works". Anyway,it seems that your answers aren't kind. I'm trying to understand what to do,excuse me if I'm not so expert,but I'm trying to do my best to understand even better and to learn new things. I haven't any problem to try to install the ubuntu nvidia driver without to be able to pass thru the nvidia graphic card,anyway.
 
ok. I have installed the nvidia driver and then I did :

Code: 
bhyve -S -c 4 -m 11G -w -H \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/da1p1/vms/os/ubuntu-budgie-gpu/ubuntu-2104-gpu.img \
-s 2:0,passthru,1/0/0 \
-s 2:1,passthru,1/0/1 \
-s 2:2,passthru,1/0/2 \
-s 2:3,passthru,1/0/3 \
-s 6,virtio-net,tap0 \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
vm0

mount: /dev/da1p1: Device busy
VM:vm0 is not created.
fbuf frame buffer base: 0xb04600000 [sz 16777216]
Assertion failed: (error == 0), function modify_bar_registration, file /usr/src/usr.sbin/bhyve/pci_emul.c, line 501.
Segnale di annullamento

root@marietto:/home/marietto/Desktop/Files/bhyve # ./os-uefi-hirsute.sh
mount: /dev/da1p1: Device busy
vm_open: Invalid argument

root@marietto:/home/marietto/Desktop/Files/bhyve # ./os-uefi-hirsute.sh
mount: /dev/da1p1: Device busy
VM:vm0 is not created.
fbuf frame buffer base: 0xb04600000 [sz 16777216]

BLACK SCREEN and nothing happens.
 
You must log in or register to reply here.
Back
Top