bhyve Bhyve can't pass thru any of my NVIDIA graphic cards to an Ubuntu bhyved os : the vm freezes before recognizing the disk

OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 74
Messages: 763

Switch to a basic text terminal (Ctrl + Alt + F2, for example), log in and run nvidia-smi from there.

Ctrl + Alt + F2 works on FreeBSD host,not in Ubuntu VM in bhyve + TigerVNC. I don't know how to switch to a terminal with TigerVNC.
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 74
Messages: 763

From the viewer menu. Press F8 to see it.

I've been able to press CTRL + ALT + F1 and now I see the login prompt. But when I try to issue my login name,nothing happens. I don't see any letter printed on the screen,but I see the cursor moving a little bit each time I press a key.
 

astyle

Daemon

Reaction score: 764
Messages: 1,645

I've seen we
I've been able to press CTRL + ALT + F1 and now I see the login prompt. But when I try to issue my login name,nothing happens. I don't see any letter printed on the screen,but I see the cursor moving a little bit each time I press a key.
I've seen weird keyboards at work where you have to press the Fn key (or something similar, I can't recall off the top of my head) BEFORE typing in the login name. Only department staffers and IT people can really figure that out, and even then - only if they're not on autopilot from exhaustion. Maybe your keyboard has something similar? It's weird where paying attention can pay off, huh?
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 74
Messages: 763

After several reinstallations of all the various OSes,like Ubuntus and or Debians,I've been able to reach the result that I wanted. I've passed through all the sub devices of my graphic card,the RTX 2080 ti from FreeBSD 13r to Debian 11 using this command :

Code:
bhyve -S -c 8 -m 8G -w -H \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/da2s1/OS/bhyve/debian-11-gpu.img \
-s 2:0,passthru,1/0/0,rom=TU102.rom \
-s 2:1,passthru,1/0/1,rom=TU102.rom \
-s 2:2,passthru,1/0/2,rom=TU102.rom \
-s 2:3,passthru,1/0/3,rom=TU102.rom \
-s 6,virtio-net,tap0 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
vm0

Previously I had installed the xorg/nouveau driver on the Debian 11,but not the proprietary Nvidia driver. I will do this as next experiment. I have attached the secondary monitor on the HDMI port of the nvidia 2080 ti graphic card and I've seen xfce4 running on that monitor. So the RTX 2080 ti is working fine with the nouveau driver from within the VM. Anyway I don't know if actually it has the same features and performances of the nVidia proprietary graphic driver. Someone knows what could be missing ? Now the problem that I have is that,even if I have installed vnc server on the emulated debian VM,I'm not able to connect it to the IP address of Debian,let's say 192.168.1.3 ; it says connection refused. Someone knows why this happens ?
 

astyle

Daemon

Reaction score: 764
Messages: 1,645

After several reinstallations of all the various OSes,like Ubuntus and or Debians,I've been able to reach the result that I wanted. I've passed through all the sub devices of my graphic card,the RTX 2080 ti from FreeBSD 13r to Debian 11 using this command :

Code:
bhyve -S -c 8 -m 8G -w -H \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/da2s1/OS/bhyve/debian-11-gpu.img \
-s 2:0,passthru,1/0/0,rom=TU102.rom \
-s 2:1,passthru,1/0/1,rom=TU102.rom \
-s 2:2,passthru,1/0/2,rom=TU102.rom \
-s 2:3,passthru,1/0/3,rom=TU102.rom \
-s 6,virtio-net,tap0 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \
vm0

Previously I had installed the xorg/nouveau driver on the Debian 11,but not the proprietary Nvidia driver. I will do this as next experiment. I have attached the secondary monitor on the HDMI port of the nvidia 2080 ti graphic card and I've seen xfce4 running on that monitor. So the RTX 2080 ti is working fine with the nouveau driver from within the VM. Anyway I don't know if actually it has the same features and performances of the nVidia proprietary graphic driver. Someone knows what could be missing ? Now the problem that I have is that,even if I have installed vnc server on the emulated debian VM,I'm not able to connect it to the IP address of Debian,let's say 192.168.1.3 ; it says connection refused. Someone knows why this happens ?
If you know how to troubleshoot networking issues, then I'd suggest first checking the network setup of the VM itself (utilities, firewall, proper drivers for the emulated network card, etc), then the VNC server on that VM (Maybe vnc.conf file has some defaults that need to be changed). Sometimes getting a good night's sleep also helps, you wake up with a clear head, and realize the problem is actually easy to solve. :)
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 74
Messages: 763

Never worked the connection of VNC viewer to the IP number assigned to the VM. It always worked only using localhost. In your opinion I should open the port 5900 and 5901 on the router administration page ?
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 74
Messages: 763

This is the recurring error that prevents bhyve from running correctly the proprietary nvidia driver :

[ 2.148444] nvidia: module verification failed: signature and/or required key missing - tainting kernel
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 74
Messages: 763

and this ?

[ 20.467637] nvidia: loading out-of-tree module taints kernel.
[ 20.467645] nvidia: module license 'NVIDIA' taints kernel.
[ 20.467646] Disabling lock debugging due to kernel taint
[ 20.475527] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[ 20.486284] nvidia-nvlink: Nvlink Core is being initialized, major device number 246

[ 20.486915] nvidia 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:eek:wns=io+mem
[ 20.603744] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 470.63.01 Tue Aug 3 20:44:16 UTC 2021
[ 20.987010] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 470.63.01 Tue Aug 3 20:30:55 UTC 2021
[ 21.552490] [drm] [nvidia-drm] [GPU ID 0x00000002] Loading driver
[ 21.552492] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:00:02.0 on minor 0
[ 22.603846] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:02.1/sound/card0/input6
[ 22.604216] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:02.1/sound/card0/input7
[ 22.604269] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:02.1/sound/card0/input8
[ 22.604318] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:02.1/sound/card0/input9
[ 22.604368] input: HDA NVidia HDMI/DP,pcm=10 as /devices/pci0000:00/0000:00:02.1/sound/card0/input10
[ 22.604403] input: HDA NVidia HDMI/DP,pcm=11 as /devices/pci0000:00/0000:00:02.1/sound/card0/input11
[ 22.605064] input: HDA NVidia HDMI/DP,pcm=12 as /devices/pci0000:00/0000:00:02.1/sound/card0/input12
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 74
Messages: 763

I'm not interested in being helpful with who is not interested in being helpful.
 

shkhln

Daemon

Reaction score: 1,067
Messages: 2,398

You are actively trying to wiggle out from proving that passthrough works you for like a week already. I'm am being polite considering the circumstances. This is not easy.
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 74
Messages: 763

What kind of problems do u have ? Isn't this a place where to get support ? If u want to talk about education,don't look at me. I'm on the right place,doing the right things,trying,experimenting,asking,learning. In my opinion you should look inside you. I never been rude. You have been. In addition,the solutions found for the various problems faced can be useful for many people. Instead,you aren't helping anyone. Why are u here ? Coming back to the problem :

Code:
# nvidia-detect
Detected NVIDIA GPUs:
00:02.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] [10de:1e04] (rev a1)

Checking card:  NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] (rev a1)
Your card is supported by the default drivers.
Your card is also supported by the Tesla 460 drivers series.
Your card is also supported by the Tesla 450 drivers series.
Your card is also supported by the Tesla 418 drivers series.
It is recommended to install the
    nvidia-driver
package.

the passed through nvidia card is detected.
 

shkhln

Daemon

Reaction score: 1,067
Messages: 2,398

What kind of problems do u have ? Isn't this a place where to get support ?
No, this a place for collecting and organizing common knowledge. I only provide support in exchange for useful information. (Why, yes, nothing is ever free.) That might be as simple as set a set of steps for reproducing a bug, but it should be something. You, on the other hand, are just wasting my time for no reason, because you refuse to follow basic debugging procedures or downright ignore questions.
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 74
Messages: 763

I don't agree with you. You are only interested to have one only information from me. You are not interested to help. And you told this. When you have got the information u want,you will not give any help. So,this is not an equal agreement. You aren't helping me from the beginning. Don't joke with me. I never refused to give useful informations to the others guys. I did it once with you because I'm not going to be manipulated by you. And please don't confuse what I can't do because I don't know how to do it,from what I don't want to do. In addition,if I don't know how to do something and u don't want to explain it,it is normal that you will not have what you want and you will be a little responsible for this, because I am here to learn, but you are not here to explain. I believe that if you expect to find knowledgeable people like you in the forums, sometimes you will be disappointed. And no, I don't think that the motto "time is money" is universalizable. Sometimes it happened to find people who helped me a lot without making me weigh it. I have health problems, I can't have a good job and I can't earn a good amount of money : this is an hobby for me. Something that I can do because I must stay at home most of the time.
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 74
Messages: 763

In Linux,if I want to configure correctly the passthrough for my Nvidia card,I should create a file like this :

/etc/tmpfiles.d/nvidia_pm.conf

w /sys/bus/pci/devices/0000:01:00.0/power/control - - - - auto
w /sys/bus/pci/devices/0000:02:00.0/power/control - - - - auto

do u know how to do the same with FreeBSD ? I'm not able to find the same files and paths.
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 74
Messages: 763

I would like to specify that at the moment the passthru of the nvidia and amd graphic cards work only under Linux. Windows continues to does not work.
 
Top