How to setup and check if VirtualGL and TurboVNC are working correctly on Debian virtualized with bhyve

ziomario

Aspiring Daemon

Reaction score: 72
Messages: 746

Hello to everyone.

After several months of work I've been able to reach the goal that I was working on the last 4 months : to be able to use my RTX 2080 ti inside a Linux VM for rendering my Blender animations using cycles. To be honest,I haven't reached the goal perfectly as I wanted,because bhyve needs some patches to be able to run Xorg and the nvidia driver. Since its development is slow maybe it will not have those patches for a lot of time. BTW,maybe I found a workaround. But before to be sure that my workaround is good,I would like to know what you think. So,first of all,below you can see what I'm trying to explain :


Screenshot_20211106_215011.png



what you see is what I wanted to achieve : that Blender is able to detect my graphic card inside the linux VM,in this case I've used Debian. Since with ssh and X forwarding I can't execute programs that require 3D acceleration,I've installed VirtualGL and TurboVNC inside the VM and then I did :

Code:
marietto-FreeBSD:~# ssh -Y 192.168.1.3

marietto-BHYVE:~# /opt/TurboVNC/bin/vncserver -geometry 1440x900

Desktop 'TurboVNC: marietto-BHYVE:1' started on display marietto-BHYVE:1

Starting applications specified in /home/marietto/.vnc/xstartup.turbovnc
Log file is /home/marietto/.vnc/marietto-BHYVE:1.log

marietto-BHYVE:~# ps ax | grep vnc

1597 pts/0    Sl     0:00 /opt/TurboVNC/bin/Xvnc :1 -desktop TurboVNC: marietto-BHYVE:1 -httpd /opt/TurboVNC/b
in//../java -auth /home/marietto/.Xauthority -geometry 1440x900 -depth 24 -rfbwait 120000 -rfbauth /home/marietto/.vnc/passwd -x509cert /r
oot/.vnc/x509_cert.pem -x509key /home/marietto/.vnc/x509_private.pem -rfbport 5901 -fp /usr/share/fonts/X11/misc,/usr/share/fonts
/X11/75dpi,/usr/share/fonts/X11/100dpi,/usr/share/fonts/X11/Type1 -deferupdate 1 -dridir /usr/lib/x86_64-linux-gnu/dri -
registrydir /usr/lib/xorg

and then on FreeBSD I've launched TigerVNC viewer using the address : 192.168.1.2:1 and it worked.

BTW,I'm not sure that this solution is what I wanted. I'm not able to understand if I have the 3D acceleration,even if I can render my animations using the graphic card I want and even if the CUDA libraries are detected,so they work great. I would like to have your opinion if the whole configuration is great. I've added two log files. They contain the output of these commands :


Code:
xdpyinfo -display :0

attached log file : xdpyinfo.txt

Code:
/opt/VirtualGL/bin/glxinfo -display :0 -c

attached log file : glxinfo.txt


I've found these commands inside the tutorial that I've read :



To be honest,inside the tutorial there is another command to check the sanity for restricted access to 3D X Server,but it can't find it :


Code:
xauth merge /etc/opt/VirtualGL/vgl_auth_key

xauth: (argv):1:  merge:  unable to open file /etc/opt/VirtualGL/vgl_auth_key

marietto-BHYVE:/home/marietto/Scrivania# find / -name vgl_auth_key

nothing found.


Below you see something to fix (pulseaudio connection refused) :


Code:
marietto-FreeBSD:~# ssh -Y 192.168.1.2

marietto-BHYVE:~# startxfce4

/usr/bin/startxfce4: X server already running on display localhost:10.0
/usr/bin/iceauth:  creating new authority file /run/user/0/ICEauthority
gpg-agent: un gpg-agent è già in esecuzione - non iniziare uno nuovo
Another Window Manager (KWin) is already running on screen localhost:10.0
To replace the current window manager, try "--replace"

(xfwm4:1756): xfwm4-WARNING **: 16:34:47.185: Could not find a screen to manage, exiting

(xfce4-panel:1783): garcon-CRITICAL **: 16:34:56.514: garcon_gtk_menu_get_desktop_actions_menu: assertion 'actions != NULL' failed

(xfce4-panel:1783): garcon-CRITICAL **: 16:34:56.587: garcon_gtk_menu_get_desktop_actions_menu: assertion 'actions != NULL' failed

(xfce4-panel:1783): garcon-CRITICAL **: 16:34:56.630: garcon_gtk_menu_get_desktop_actions_menu: assertion 'actions != NULL' failed

(xfce4-panel:1783): garcon-CRITICAL **: 16:34:56.654: garcon_gtk_menu_get_desktop_actions_menu: assertion 'actions != NULL' failed

** (wrapper-2.0:1789): WARNING **: 16:34:56.785: No outputs have backlight property

(wrapper-2.0:1790): GLib-GIO-CRITICAL **: 16:34:56.882: g_file_new_for_path: assertion 'path != NULL' failed

(wrapper-2.0:1790): GLib-GIO-CRITICAL **: 16:34:56.882: g_file_monitor_file: assertion 'G_IS_FILE (file)' failed

(wrapper-2.0:1790): GLib-GObject-WARNING **: 16:34:56.882: invalid (NULL) pointer instance

(wrapper-2.0:1790): GLib-GObject-CRITICAL **: 16:34:56.882: g_signal_connect_data: assertion 'G_TYPE_CHECK_INSTANCE (instance)' failed

(wrapper-2.0:1790): Gtk-WARNING **: 16:34:56.882: Attempting to add a widget with type GtkToggleButton to a container of type XfcePanelPlugin, but the widget is already inside a container of type XfcePanelPlugin, please remove the widget from its existing container first.

** (wrapper-2.0:1788): WARNING **: 16:34:57.663: Binding 'XF86AudioLowerVolume' failed!

(wrapper-2.0:1788): pulseaudio-plugin-WARNING **: 16:34:57.663: Could not have grabbed volume control keys. Is another volume control application (xfce4-volumed) running?

(wrapper-2.0:1799): Gtk-WARNING **: 16:34:57.688: Negative content width -1 (allocation 1, extents 1x1) while allocating gadget (node button, owner XfceArrowButton)

(wrapper-2.0:1788): pulseaudio-plugin-WARNING **: 16:34:57.713: Disconected from the PulseAudio server. Attempting to reconnect in 5 seconds.

(wrapper-2.0:1788): pulseaudio-plugin-WARNING **: 16:34:57.713: pa_context_connect() failed: Accesso negato

(wrapper-2.0:1790): Gtk-WARNING **: 16:34:57.880: Negative content width -3 (allocation 1, extents 2x2) while allocating gadget (node button, owner GtkToggleButton)

(wrapper-2.0:1789): Gtk-WARNING **: 16:34:58.034: Negative content width -3 (allocation 1, extents 2x2) while allocating gadget (node button, owner PowerManagerButton)

** (light-locker:1860): ERROR **: 16:35:19.771: Environment variable XDG_SESSION_PATH not set. Is LightDM running?
Connessione non riuscita: Connessione rifiutata
pa_context_connect() non riuscita: Connessione rifiutata

** (xfce4-power-manager:1875): WARNING **: 16:35:20.160: Failed to get name owner: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Could not get owner of name 'org.freedesktop.PowerManagement': no such name


** (xfce4-power-manager:1875): WARNING **: 16:35:20.161: Failed to get name owner: GDBus.Error:org.freedesktop.DBus.Error.NameHasNoOwner: Could not get owner of name 'org.xfce.PowerManager': no such name


(xfce4-power-manager:1875): GLib-GObject-WARNING **: 16:35:20.383: ../../../gobject/gsignal.c:2614: signal 'Changed' is invalid for instance '0x556d9b1b1870' of type 'GDBusProxy'

(xfce4-power-manager:1875): xfce4-power-manager-WARNING **: 16:35:20.400: could not map keysym 1008ffa8 to keycode


** (xfce4-power-manager:1875): WARNING **: 16:35:20.403: No outputs have backlight property

(xfce4-power-manager:1875): xfce4-power-manager-WARNING **: 16:35:20.406: Failed to get keyboard max brightness level : GDBus.Error:org.freedesktop.DBus.Error.UnknownMethod: Interfaccia «org.freedesktop.UPower.KbdBacklight» inesistente sull'oggetto nel percorso /org/freedesktop/UPower/KbdBacklight

(wrapper-2.0:1788): pulseaudio-plugin-WARNING **: 16:35:20.411: Disconected from the PulseAudio server. Attempting to reconnect in 5 seconds.

(wrapper-2.0:1788): pulseaudio-plugin-WARNING **: 16:35:20.411: pa_context_connect() failed: Accesso negato

(wrapper-2.0:1788): Gtk-WARNING **: 16:35:20.439: Negative content width -3 (allocation 1, extents 2x2) while allocating gadget (node button, owner PulseaudioButton)

(wrapper-2.0:1788): pulseaudio-plugin-WARNING **: 16:35:25.348: Disconected from the PulseAudio server. Attempting to reconnect in 5 seconds.

(wrapper-2.0:1788): pulseaudio-plugin-WARNING **: 16:35:25.349: pa_context_connect() failed: Accesso negato

(wrapper-2.0:1788): pulseaudio-plugin-WARNING **: 16:35:30.355: Disconected from the PulseAudio server. Attempting to reconnect in 5 seconds.

(xfdesktop:1813): GLib-CRITICAL **: 16:36:22.983: g_variant_new_string: assertion 'string != NULL' failed
La cartella di lavoro "/root/[Invalid UTF-8]" non esiste. Non verrà utilizzata per deporre "xfce4-terminal".
(wrapper-2.0:1788): pulseaudio-plugin-WARNING **: 16:36:25.350: Disconected from the PulseAudio server. Attempting to reconnect in 5 seconds.

(wrapper-2.0:1788): pulseaudio-plugin-WARNING **: 16:36:25.350: pa_context_connect() failed: Accesso negato

(wrapper-2.0:1788): pulseaudio-plugin-WARNING **: 16:36:30.347: Disconected from the PulseAudio server. Attempting to reconnect in 5 seconds.

and how to fix this :

Code:
marietto@marietto-BHYVE:/$ vglrun glxgears
[VGL] ERROR: Could not open display :0.

marietto@marietto-BHYVE:/$ echo $DISPLAY
:1.0

marietto@marietto-BHYVE:/$ export DISPLAY=:1.0

marietto@marietto-BHYVE:/$ echo $DISPLAY
:1.0

marietto@marietto-BHYVE:/$ vglrun glxgears
[VGL] ERROR: Could not open display :0.
 

Attachments

  • Screenshot_20211106_190125.png
    Screenshot_20211106_190125.png
    684 KB · Views: 21
  • glxinfo.txt
    85.5 KB · Views: 17
  • xdpyinfo.txt
    94.9 KB · Views: 15
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 72
Messages: 746

Someone said to me : That VNC session is not accelerated. It is just a dumb frame buffer. VirtualGL uses VNC for forwarding the rendered graphics else where. You will need to install and configure VirtualGL and X on FreeBSD. To expand upon this, you are not looking to use the nVidia card from the VM via pass through etc. The graphics are being rendered on the FreeBSD box and would be being passed to the Linux VM via VNC for what you are looking to do. You would use VirtualGL only if you were not passing the PCI stuff on through. If you really want to use the PCI device from the VM, you'll need to debug what is going on in the VM as to why those libraries are not usable.

So,do you know if I can install and configure VirtualGL on FreeBSD and most of all,is it useful for what I want to do ? (rendering animations in blender inside the VM using the passed through nvidia graphic card) ? Can someone explain to me how it works ? What's the procedures that I should apply to achieve what I want. My ideas aren't clear. I don't understand why those libraries aren't usable.
 

kpedersen

Son of Beastie

Reaction score: 2,238
Messages: 3,064

VirtualGL is fairly messy in what it does (but in many ways it has to be because the idea is quite fiddly). Consider tigervnc Xvnc servers do not have direct access to the GPU (as you mentioned, they are simple unaccelerated framebuffers). The best they can get is LLVMpipe. What VirtualGL does is create a normal Xorg Xserver on the host and runs OpenGL applications on that, forwarding out the rendered visuals back to the unaccelerated framebuffers.

So unless you are able to run an accelerated framebuffer somehow on Bhyve, then this is still going to be a no-go.

However VirtualGL is possible to get working on FreeBSD. It isn't in ports and breaks fairly often IMO. However I did have it working about 5 years ago. Though for your use-case via Bhyve and GPU passthrough, I don't think it will be such a good option.

Would an easier solution for you not to just have an accelerated build server that when you commit your models into Git, it fetches them and renders them? You could perhaps script something fairly easily in the Python layer ontop of Blender.
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 72
Messages: 746

Do you mean rendering the projects inside the VM without to see at all what it is doing and then copy them from within the vm to outside ? could be useful,for this case,to use a dummy HDMI adapter ?
 

kpedersen

Son of Beastie

Reaction score: 2,238
Messages: 3,064

Really just any rendering inside the VM. It is a difficult problem (it seems). You have some solutions:
  • Stick to software rendering and just use a shedload of cores - More possible today than 10 years ago
  • Passthrough of entire PCI BUS - Most generic VM solution
  • Passthrough of GPU - Requires guest specific drivers, these won't be around forever.
  • Intelligent graphics protocol to render on host - Few have the facilities to maintain this custom approach (+ Wayland is breaking it)
So if you can avoid the above complexities entirely that will help. So yes, use Blender on FreeBSD and use the "basic" rendering just to ensure the camera / materials are in the right place. This should take no longer than 5 seconds. Then when you are ready for a more final bake, rsync it to a Linux (or probably better, Windows) host and do the main bake there. Then you get the full power of all the proprietary / unportable bits.
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 72
Messages: 746

  • Passthrough of GPU - Requires guest specific drivers, these won't be around forever.
which kind of drivers ? I'm using the cuda libraries inside the linux vm,what else kind of drivers I need ? Blender found everything it needs to start the rendering inside the vm.
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 72
Messages: 746

You forgot the most obvious. Just dual boot into Linux and directly use Blender like everyone else does.

nope. I'm not like everyone else. I don't like to "win" easily. You,all,should thanks me and the people like me,because we try to push the technology one step further to cover new uses and we are always not satisfied with what we have...
 

eternal_noob

Daemon

Reaction score: 892
Messages: 1,174

because we try to push the technology one step further to cover new uses and we are not satisfied with what we have...
Adding an additional, useless layer of complexity is not something desirable.
Keep it plain and simple, save resources.

I propose to either get the CUDA libraries working on plain FreeBSD or directly use Linux.
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 72
Messages: 746

---> Adding an additional, useless layer of complexity is not something desirable.

the desired goal is not always achieved using simple solutions and then different solutions are created, which are not simple. Or at least, one could say that they are elegant, which means that they are simple in their complexity. And so it becomes a matter of points of view. I would like to get CUDA on FreeBSD yes ! I agree that this solution is better than use them inside a Linux VM,but this is a very complicated task ! at least for me. Nvidia does not support FreeBSD for the CUDA libraries. It's just impossible to achieve this goal !
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 72
Messages: 746

For me it is,because I can't usa CUDA in FreeBSD,but I can use it inside the Linux VM. Blender recognizes my nvidia GPU and the CUDA libraries. So,this is a step forward. You could say that there are different problems to solve now. Yes,there are,but the situation is better than the previous one.

Code:
root@marietto-BHYVE:/usr/local/cuda-11.4/bin# ./nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Wed_Jul_14_19:41:19_PDT_2021
Cuda compilation tools, release 11.4, V11.4.100
Build cuda_11.4.r11.4/compiler.30188945_0
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 72
Messages: 746

I've bought a dummy HDMI adapter. It sounds like this :


Code:
marietto@marietto-BHYVE:~$ nvidia-xconfig --query-gpu-info


Number of GPUs: 1
GPU #0:
Name      : NVIDIA GeForce RTX 2080 Ti
UUID      : GPU-74c6f81e-9e6c-f279-2878-2d28f4333f6a
PCI BusID : PCI:0:3:0

Number of Display Devices: 1

Display Device 0 (TV-2):
    EDID Name             : XMD Mi TV
    Minimum HorizSync     : 15.000 kHz
    Maximum HorizSync     : 140.000 kHz
    Minimum VertRefresh   : 49 Hz
    Maximum VertRefresh   : 80 Hz
    Maximum PixelClock    : 600.000 MHz
    Maximum Width         : 4096 pixels
    Maximum Height        : 2160 pixels
    Preferred Width       : 1920 pixels
    Preferred Height      : 1080 pixels
    Preferred VertRefresh : 60 Hz
    Physical Width        : 600 mm
    Physical Height       : 340 mm


can this help me in some way ?
 

shkhln

Daemon

Reaction score: 1,065
Messages: 2,392

You,all,should thanks me and the people like me,because we try to push the technology one step further to cover new uses and we are not satisfied with what we have...
:what: Would you mind pushing punctuation a bit further? This a PITA to read and understand.
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 72
Messages: 746

:what: Would you mind pushing punctuation a bit further? This a PITA to read and understand.

Can you tell me how should I rephrase the sentence ? thanks. In the meantime,can you reply my question ? I mean this : I've bought a dummy HDMI adapter. It sounds like this....can this help me in some way ? if yes,how ? thanks.
 

shkhln

Daemon

Reaction score: 1,065
Messages: 2,392

I can't say anything about VirtualGL and CUDA/OpenCL compatibility without a dozen or so tests and I'm barely able to find motivation for things that I want to work on (this certainly isn't one).
 
OP
ziomario

ziomario

Aspiring Daemon

Reaction score: 72
Messages: 746

if you have some difficult to find motivations,I could help you because I'm a psychologist :). Yes,a psychologist who likes to make something for which he hasn't been trained. Isn't it cool ? You are the best in the area where I grope in the dark,but you need the help of someone like me to "heal" your mind. My medical prescription is : to increase your motivations you should grasp a good amount of money :p
 
Top