jails Is it possible to allow jail to use host GPU?

Here is my config. I have a simple 32bit jail on 64bit host. I have extracted FreeBSD 13-RELEASE base.txz in /usr/jail/wine32, also i enabled X11Forwarding and it works fine(but without 3D acceleration).

This works without a problem.

But when i try this:

I get this:
Error: couldn't get an RGB, Double-buffered visual

/etc/jail.conf:
wine32 {
host.hostname = www.wine32.com;
ip4 = inherit;
path = "/usr/jail/wine32";
mount.devfs;
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
allow.mount = 1;
enforce_statfs = 0;
allow.mount.devfs = 1;
allow.raw_sockets = 1;
allow.mount.procfs = 1;
devfs_ruleset=8;
}

and devfs rule number 8:
[devfsrules_unhide_xorg=8]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path agpgart unhide
add path console unhide
add path consolectl unhide
add path dri unhide
add path 'dri/*' unhide
add path io unhide
add path 'nvidia*' unhide
add path sysmouse unhide
add path mem unhide
add path pci unhide
add path tty unhide
add path ttyv0 unhide
add path ttyv1 unhide
add path ttyv8 unhide

As i can see this is not enough,
 
Do the files xterm and glxgears belong to different groups?

A better place to start is, does it work correctly if all devices are unhidden in the jail?

I wonder if it could have something to do with VAAPI, or another /dev device that has to do with additional video acceleration than dri. Do all dri devices in the host show up in the jail? Glxgears may need access to more devices than xterm.
 
Do the files xterm and glxgears belong to different groups?
This is what i got:
# ls -l /usr/local/bin | egrep 'xterm|glxgears'
-r-xr-xr-x 1 root wheel 16908 Oct 5 09:46 glxgears
-r-xr-xr-x 1 root wheel 24596 Oct 5 09:46 glxgears_fbconfig
-r-xr-xr-x 1 root wheel 14280 Oct 5 09:46 glxgears_pixmap
-r-xr-xr-x 1 root wheel 3772 Oct 5 07:22 koi8rxterm
-r-xr-xr-x 1 root wheel 4265 Oct 5 07:22 uxterm
-r-xr-xr-x 1 root wheel 660172 Oct 5 07:22 xterm
I wonder if it could have something to do with VAAPI, or another /dev device that has to do with additional video acceleration than dri.
On my host machine there is nothing /dev/dri* related. ls -l /dev | grep dri shown nothing.

Do all dri devices in the host show up in the jail?
Just this:
# ls -l /dev
total 2
crw------- 1 root wheel 0x6 Oct 15 13:49 console
crw------- 1 root wheel 0x20 Oct 15 11:43 consolectl
crw-rw-rw- 1 root wheel 0x58 Oct 15 11:43 crypto
dr-xr-xr-x 2 root wheel 512 Oct 15 13:48 fd
crw------- 1 root wheel 0x1f Oct 15 11:43 io
crw-r----- 1 root kmem 0x27 Oct 15 11:43 mem
crw-rw-rw- 1 root wheel 0xd Oct 15 14:00 null
crw-rw-rw- 1 root wheel 0x95 Oct 15 11:44 nvidia-modeset
crw-rw-rw- 1 root wheel 0x8d Oct 15 11:44 nvidia0
crw-rw-rw- 1 root wheel 0x94 Oct 15 11:44 nvidiactl
crw-r--r-- 1 root wheel 0xf Oct 15 11:43 pci
crw-rw-rw- 1 root wheel 0xb5 Oct 15 11:44 ptmx
dr-xr-xr-x 2 root wheel 512 Oct 15 13:48 pts
crw-r--r-- 1 root wheel 0x4 Oct 15 11:44 random
lrwxr-xr-x 1 root wheel 4 Oct 15 13:48 stderr -> fd/2
lrwxr-xr-x 1 root wheel 4 Oct 15 13:48 stdin -> fd/0
lrwxr-xr-x 1 root wheel 4 Oct 15 13:48 stdout -> fd/1
crw------- 1 root wheel 0x21 Oct 15 11:43 sysmouse
crw------- 1 root wheel 0x66 Oct 15 11:44 ttyv0
crw------- 1 root wheel 0x67 Oct 15 11:44 ttyv1
crw------- 1 root wheel 0x6e Oct 15 11:43 ttyv8
lrwxr-xr-x 1 root wheel 6 Oct 15 13:48 urandom -> random
crw-rw-rw- 1 root wheel 0xe Oct 15 11:43 zero

A better place to start is, does it work correctly if all devices are unhidden in the jail?
I will try to enable this right now.
 
On my host machine there is nothing /dev/dri* related. ls -l /dev | grep dri shown nothing.
This may be the problem. Un-hiding everything in the jail, can only match what's on the host, which doesn't have it. dri is lacking in the jail, also.

dri has to be set up on the host, with your user added to the video group, Thread how-to-light-desktop-setup-2018-2020s-update.64623/#post-376630. Then, the host will have a device that can be unhidden for the jail. If dri isn't showing on host, maybe the needed video kernel module for the card isn't installed. The Xorg driver is also needed, but that may not account for dri in the /dev/ directory
 
It should work (via both Jail and chroot). You may need to avoid SSH/X11 forwarding though (unless you are happy with software rendering via LLVMpipe). Some steps from memory.

  1. For now, just mount the entirety of /dev in your jail (we can cut that down later).
  2. ln your /tmp/.X11-unix/*** file into the jail under the same path. The catch is that it needs to be on the same filesystem (I think FreeBSD UFS in the installer does this by default these days).
  3. Either disable the security on Xorg (xhost +, xauth) or copy across your .Xauthority file from the host (home folder running X11) to the Jail (under the home folder of user running GUI program).
  4. Your jail will also need the dri / mesa related packages installed (for now just install xorg package just to check).
 
This may be the problem. Un-hiding everything in the jail, can only match what's on the host, which doesn't have it.

dri has to be set up on the host, with your user added to the video group, Thread how-to-light-desktop-setup-2018-2020s-update.64623/#post-376630. Then, the host will have a device that can be unhidden for the jail. If dri isn't showing on host, maybe the needed video kernel module for the card isn't installed. The Xorg driver is also needed, but that may not account for dri in the /dev/ directory
I added another devfs_ruleset that unhides everything and i have applied it to jail.
[devfsrules_unhide_all=9]
add unhide
Now i can see everything in jail. My user is already in video group. But glxgears still doesn't work.

Kernel modules loaded:
# kldstat
Id Refs Address Size Name
1 63 0xffffffff80200000 1f11f28 kernel
2 1 0xffffffff82112000 ae38 cryptodev.ko
3 1 0xffffffff8211d000 67feb0 zfs.ko
4 2 0xffffffff82a00000 1fa4bb0 nvidia.ko
5 2 0xffffffff82920000 388f8 linux.ko
6 4 0xffffffff82959000 db70 linux_common.ko
7 1 0xffffffff849a5000 107310 nvidia-modeset.ko
8 1 0xffffffff82967000 3250 ichsmb.ko
9 1 0xffffffff8296b000 2180 smbus.ko
10 1 0xffffffff8296e000 2340 uhid.ko
11 1 0xffffffff82971000 3380 usbhid.ko
12 1 0xffffffff82975000 31f8 hidbus.ko
13 1 0xffffffff82979000 3320 wmt.ko
14 1 0xffffffff8297d000 4350 ums.ko
15 1 0xffffffff82982000 30ac8 linux64.ko
16 1 0xffffffff829b3000 2260 pty.ko
17 1 0xffffffff829b6000 2a08 mac_ntpd.ko

Video drivers are working fine, i am using X11 without problems, also i play various video games and everything is fine. I am searching on the net about /dev/dri.
 
It should work (via both Jail and chroot). You may need to avoid SSH/X11 forwarding though (unless you are happy with software rendering via LLVMpipe). Some steps from memory.

  1. For now, just mount the entirety of /dev in your jail (we can cut that down later).
  2. ln your /tmp/.X11-unix/*** file into the jail under the same path. The catch is that it needs to be on the same filesystem (I think FreeBSD UFS in the installer does this by default these days).
  3. Either disable the security on Xorg (xhost +, xauth) or copy across your .Xauthority file from the host (home folder running X11) to the Jail (under the home folder of user running GUI program).
  4. Your jail will also need the dri / mesa related packages installed (for now just install xorg package just to check).

Thank you for reply. I have done this:
625 ln -s /tmp/.X11-unix/ /usr/jail/wine32/tmp/.X11-unix
627 ln -s /tmp/.X11-unix/X0 /usr/jail/wine32/tmp/.X11-unix/
629 cp /home/aca/.Xauthority /usr/jail/wine32/root/
630 cp /home/aca/.Xauthority /usr/jail/wine32/home/aca/

And i also installed mesa-dri and xorg packages in jail. But glxgears does not work again.
 
But glxgears does not work again.
Ah, remove both symlinks and make a "hard link" ( ln without -s)

Also, don't link the folder, just mkdir that folder in /tmp normally in the Jail.

After that are you just using the jail command to enter your session (as the correct user)? You may need to also set export DISPLAY=:0 environment variable.

What error are you getting? It might be easier to work from there. Can you at least run GUI programs across the local Jail boundary (not via X11/SSH).

Edit: Also you may need to make sure your Jailed user has an ID (or group) that matches that which is able to access the DRI/DRM devices in /dev. There are many fiddly things!
 
Ah, remove both symlinks and make a "hard link" ( ln without -s)
I am not able to do that because my .X11-unix/X0 is mounted under zroot/tmp and i need it to mount X0 under zroot/ROOT/default.. So this is happening, they cannot have same inode:
# cd /usr/jail/wine32/tmp/.X11-unix/
# ln /tmp/.X11-unix/X0
ln: ./X0: Cross-device link

EDIT:
I don't know really what i did but now jail doesn't accept any ssh connection.. Maybe it will be best for me to abandon this exercise. Start fresh jail and work my way slowly to achieve the wanted. But in the end i unhid every device available, i gave every authority i could find out and its not good enough.

I will let it die and start fresh. This got complicated enough.
 
I am not able to do that because my .X11-unix/X0 is mounted under zroot/tmp and i need it to mount X0 under zroot/ROOT/default.. So this is happening, they cannot have same inode:
Right. Yes, annoyingly that pipe needs to be on the same filesystem. You could mount one large one to /tmp and use that same FS to store your jails.

Zirias wrote a tool that allows for UNIX domain sockets to be sent through TCP. Perhaps this could be an option? I think the very latest SSH can forward these but can't convert them back to a domain socket on the other end.

I use GPU in Jails for a few projects at work but never found a solution that works with multiple filesystems (fairly important for jails IMO). So I have always been planning to give this tool a shot myself.
 
I noticed I was mentioned :cool:

Actually, this might work. Although, as far as I can see, this socket is owned by root. Is this necessary? I think it's better to run my tool as a non-privileged user (yep, I don't entirely trust my own code!) and right now, ownership of the (server) socket is "hard-wired" to the user/group remusockd is running as. OTOH, with a socket with mode 777, it probably shouldn't matter who owns it…

edit: I just noticed there might be a way around that: If remusockd isn't started as root, it never attempts to change its uid. But it still needs permissions in the target dir to set mode and ownership of the socket…
 
Yes, forward the correct devices in /dev (start with just mounting it all). Then you can either connect to the hosts running Xorg by forwarding the socket files in /tmp. Or you can run Xorg from within the guest jail.
 
Hmm, I'm struggling conceptually. Right now, my system is as follows: The host is my 'workstation' and my jail is a router. I assign the network devices to the router and create an Ethernet pair device with one end assigned to the workstation and the other the jail. The workstation still has the video device along with all USB devices. I log into the console, startx and have Xorg. I did this about a year or so ago to simplify management (and save a little bit on electricity :) ).

I access my jail using jexec as normal.

What confuses me is, if I instead make the router the host and the workstation the jail, wouldn't the video device be assigned over to the jail? Then, if that is the case, I would be logging into the workstation (jail) and for me to get to the host (router), I would need to ssh to it? Or, am I confused? Whenever I assign the network devices to the jail, the host can no longer see them. My concern is that once I assign the video device over to the jail, the host will no longer have video and that could make things very difficult very fast. What if the jail fails to start, I could be without a means to see what's going on unless I use an external device?

I am thinking about reversing my setup because I rarely have updates for the router jail as it has far fewer packages. On the other hand, my workstation has updates every few days. By reversing my setup, I won't need to reboot nearly as frequently and will further reduce downtime. I need to reboot in order to boot into a new Boot Environment.
 
Just to clarify, I'm not running Xorg on the host. The host is a barebones installation. It will have NTP so it can stay in sync, the kernel with all the necessary drivers to support the jails (support for pf altq). Where do I start? I am now planning to have 2 jails, 1 for acting like a router, and 1 for my workstation on demand. I am thinking I can avoid reboots by keeping the host as small as possible. I have to reboot presently in order to boot into a new Boot Environment. Rebooting a jail doesn't require me to bring the host down and that seems like a small win for me.
 
Back
Top