general/other Differences between running something through linuxlator vs chroot

[fahrenheit]

Hi all,

I have a question that I hope someone can help me (or at least point me to the right place to ask/report the issue).

Context:

My system: FreeBSD 15/stable (updated), drm-612-kmod, Radeon 780M, mesa-devel on freebsd (25.1.7 on rocky).
So I have /compat/linux with Alma Linux 10.2 installed (instead of rocky because I wanted to test the the i686 support of Alma) and setup my home there (nullfs mount point, same uids/gids), though what I'll describe would also happen with the linux-rl9 packages, I've installed Surviving Mars (a linux native 64bit game, installing thus to /home/<user>/GOG Games/Surviving Mars, available at the same place).

Problem:

If I try to start the game from FreeBSD (i.e. open a terminal or using the launcher) I get a segmentation fault in a few moments after loading (i.e. after the videos and some such) but if I chroot into /compat/linux (as my user, X11 properly setup) there is no segmentation fault and the game works fine.

Question:

So, disregarding that this is a game, my question is why is there a different behaviour? And can I do something to use the software as if running from the chroot but without actually doing the steps?

Reference:

Spoiler: FreeBSD LDD

ldd "/home/fahrenheit/GOG Games/Surviving Mars/game/MarsGOG"
/home/fahrenheit/GOG Games/Surviving Mars/game/MarsGOG:
libpthread.so.0 => not found (0)
libdl.so.2 => not found (0)
libcurl-gnutls.so.4 => not found (0)
libopenal.so.1 => /usr/local/lib/libopenal.so.1 (0x670c556a000)
libssl.so.1.0.0 => /usr/local/lib/libssl.so.1.0.0 (0x670c6ef0000)
libcrypto.so.1.0.0 => /usr/local/lib/libcrypto.so.1.0.0 (0x670c7561000)
libX11.so.6 => /usr/local/lib/libX11.so.6 (0x670c5e8a000)
libXext.so.6 => /usr/local/lib/libXext.so.6 (0x670c875b000)
libXrender.so.1 => /usr/local/lib/libXrender.so.1 (0x670ca169000)
libXrandr.so.2 => /usr/local/lib/libXrandr.so.2 (0x670ca81c000)
libXt.so.6 => /usr/local/lib/libXt.so.6 (0x670cb3b6000)
libSDL2-2.0.so.0 => /usr/local/lib/libSDL2-2.0.so.0 (0x670c9d50000)
libpops_api.so => not found (0)
libstdc++.so.6 => /usr/local/lib/gcc13/libstdc++.so.6 (0x670cd000000)
libm.so.6 => not found (0)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x670c9783000)
libc.so.6 => /usr/local/lib/compat/libc.so.6 (0x670cb594000)
libportaudio.so.2 => /usr/local/lib/libportaudio.so.2 (0x670cb921000)
libjack.so.0 => /usr/local/lib/libjack.so.0 (0x670cc51f000)
libsndio.so.7.3 => /usr/local/lib/libsndio.so.7.3 (0x670ce08b000)
libpulse.so.0 => /usr/local/lib/libpulse.so.0 (0x670ce595000)
libpipewire-0.3.so.0 => /usr/local/lib/libpipewire-0.3.so.0 (0x670ce70b000)
libdbus-1.so.3 => /usr/local/lib/libdbus-1.so.3 (0x670cf727000)
libm.so.5 => /lib/libm.so.5 (0x670d0556000)
libc++.so.1 => /lib/libc++.so.1 (0x670d06da000)
libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x670d146c000)
libthr.so.3 => /lib/libthr.so.3 (0x670d1dec000)
libc.so.7 => /lib/libc.so.7 (0x670d224d000)
libxcb.so.1 => /usr/local/lib/libxcb.so.1 (0x670d3529000)
libSM.so.6 => /usr/local/lib/libSM.so.6 (0x670d3a4a000)
libICE.so.6 => /usr/local/lib/libICE.so.6 (0x670d3e58000)
libXcursor.so.1 => /usr/local/lib/libXcursor.so.1 (0x670d4ce2000)
libXi.so.6 => /usr/local/lib/libXi.so.6 (0x670d5425000)
libXfixes.so.3 => /usr/local/lib/libXfixes.so.3 (0x670d5cf0000)
libXss.so.1 => /usr/local/lib/libXss.so.1 (0x670d6173000)
libdrm.so.2 => /usr/local/lib/libdrm.so.2 (0x670d67e0000)
libgbm.so.1 => /usr/local/lib/libgbm-devel.so.1 (0x670d6881000)
libEGL.so.1 => /usr/local/lib/libEGL.so.1 (0x670d6e1a000)
libwayland-egl.so.1 => /usr/local/lib/libwayland-egl.so.1 (0x670d7a6b000)
libwayland-cursor.so.0 => /usr/local/lib/libwayland-cursor.so.0 (0x670d7e0f000)
libwayland-client.so.0 => /usr/local/lib/libwayland-client.so.0 (0x670d7ff1000)
libxkbcommon.so.0 => /usr/local/lib/libxkbcommon.so.0 (0x670d88fc000)
libusb.so.3 => /usr/lib/libusb.so.3 (0x670d8aeb000)
libusbhid.so.4 => /usr/lib/libusbhid.so.4 (0x670d8dbc000)
librt.so.1 => /lib/librt.so.1 (0x670d92d1000)
libopus.so.0 => /usr/local/lib/libopus.so.0 (0x670dac00000)
libsamplerate.so.0 => /usr/local/lib/libsamplerate.so.0 (0x670d96b1000)
libpulsecommon-17.0.so => /usr/local/lib/pulseaudio/libpulsecommon-17.0.so (0x670d9dec000)
libdl.so.1 => /usr/lib/libdl.so.1 (0x670db6b1000)
libintl.so.8 => /usr/local/lib/libintl.so.8 (0x670dc50e000)
libsys.so.7 => /lib/libsys.so.7 (0x670dcbb3000)
libXau.so.6 => /usr/local/lib/libXau.so.6 (0x670dcc3a000)
libXdmcp.so.6 => /usr/local/lib/libXdmcp.so.6 (0x670dd54e000)
libGLdispatch.so.0 => /usr/local/lib/libGLdispatch.so.0 (0x670de10f000)
libepoll-shim.so.0 => /usr/local/lib/libepoll-shim.so.0 (0x670de362000)
libffi.so.8 => /usr/local/lib/libffi.so.8 (0x670ded9f000)
libsndfile.so.1 => /usr/local/lib/libsndfile.so.1 (0x670df664000)
libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x670dfe4d000)
libvorbisenc.so.2 => /usr/local/lib/libvorbisenc.so.2 (0x670e009f000)
libFLAC.so.14 => /usr/local/lib/libFLAC.so.14 (0x670e10b5000)
libmpg123.so.0 => /usr/local/lib/libmpg123.so.0 (0x670e1c60000)
libmp3lame.so.0 => /usr/local/lib/libmp3lame.so.0 (0x670e21a8000)
libvorbis.so.0 => /usr/local/lib/libvorbis.so.0 (0x670e2c08000)
libogg.so.0 => /usr/local/lib/libogg.so.0 (0x670e2ce8000)
libelf.so.2 => /lib/libelf.so.2 (0x670e2ec5000)
[vdso] (0x670c4618000)

 

[NapoleonWils0n]

Hi Mate

I have done a lot of work on running Linux applications with the Linuxulator, Jails and Podman

And you are right there is a difference between launching application from the Freebsd side
vs from inside the Linux side

You always want to launch applications from inside the Linux side as it were
in you case using a chroot

The reason is because then the application can see all the libraries needed to run the linux application
whereas if you launch them from the Freebsd side it cant

You can actually launch applications from the Freebsd side by using wrapper scripts
by having a script on the Freebsd side that runs a script on the linux side and launchers the application

Its then possible to create a desktop entry that you use with wrapper scripts
that allows you to open linux applications with your application launcher on Freebsd

I would actually recommend using rocky linux for the linuxulator
with rocky linux podman containers instead of the linuxulator on its own

What you can then do is mount the nvidia driver and libraries from freebsd into the container
and have nvidia and cuda working with wayland or x11 and audio with a pulseaudio socket

having done a lot of work with the linuxulator, jails and podman container
podman is the best way to run linux applications on Freebsd

if you have a look at my freebsd-cuda project you will see how i use wrapper scripts
and a desktop entry to launch linux applications

freebsd cuda project using rocky linux podman containers

GitHub - NapoleonWils0n/freebsd-cuda: Run Linux command line and gui applications on FreeBSD with full Nvidia Cuda hardware acceleration.

Run Linux command line and gui applications on FreeBSD with full Nvidia Cuda hardware acceleration. - NapoleonWils0n/freebsd-cuda

github.com

Davinci resolve in a rocky linux jail

GitHub - NapoleonWils0n/davinci-resolve-freebsd-jail-rocky: Daviinci Resolve in a Rocky Linux Freebsd jail

Daviinci Resolve in a Rocky Linux Freebsd jail. Contribute to NapoleonWils0n/davinci-resolve-freebsd-jail-rocky development by creating an account on GitHub.

github.com

 

[fahrenheit]

Hi NapoleonWils0n I know of your repos and work, impressive work, I've took some stuff from there plus from shkhln's work on steam.

What I have working on the chroot is:

• Vulkan (CUDA doesn't matter to me because I don't have an Nvidia gpu)
• X11 (.xauthority + socket)
• pulse (through the socket, tough you can just map the XDG_RUNTIME_DIR to the same effect)
• docker-cli (through liberal use of the freebsd podman socket)

While I understand why you went with podman, I prefer to just update the chroot with dnf as needed, no need to add another layer on top of an already hard issue (plus I just need to manage 2 oses ).

Regarding your suggestion of wrapper scripts, if I understood correctly you are using jexec in your wrapper scripts, which doesn't apply to /compat/linux (as far as I know, that is), so chroot -n -u <username> /compat/linux <command> (assuming security.bsd.unprivileged_chroot is set to 1 to avoid using sudo/doas/mdo) is your suggestion? If so, that works and answers the second part of my question (how to make stuff work).

Do you know if there's some documentation or something on why running outside of the chroot fails?

Thanks

 

[NapoleonWils0n]

hi mate

for the wrapper scripts for jails and podman im using jexec
podman containers are actually jails on freebsd thats why it works

i guess you could use this instead of jexec in a wrapper script
but i havent tried

chroot -n -u <username> /compat/linux <command>

I dont know of any documentation on why trying to running outside the chroot/jail fails
but after many years of playing around with the linuxulator/jails/podman thats definitely the case

the reason is because freebsd is looking for the libraries paths on the freebsd side which dont exist
but when you execute the command from inside chroot/jail the library path resolve correctly if that makes sense

the reason i use podman is because in the compose.yaml file you can specify

# platform linux
platform: linux/amd64
# os linux
os: linux

which tricks applications to think they are really running on linux

thats need for python applications so they pull down the linux wheels
whereas when you use a jail it can "see" that its actually on freebsd and then tries to download freebsd wheel and fails

 

[fahrenheit]

hi mate

for the wrapper scripts for jails and podman im using jexec
podman containers are actually jails on freebsd thats why it works

i guess you could use this instead of jexec in a wrapper script
but i havent tried

chroot -n -u <username> /compat/linux <command>

I dont know of any documentation on why trying to running outside the chroot/jail fails
but after many years of playing around with the linuxulator/jails/podman thats definitely the case

the reason is because freebsd is looking for the libraries paths on the freebsd side which dont exist
but when you execute the command from inside chroot/jail the library path resolve correctly if that makes sense

the reason i use podman is because in the compose.yaml file you can specify

# platform linux
platform: linux/amd64
# os linux
os: linux

which tricks applications to think they are really running on linux

thats need for python applications so they pull down the linux wheels
whereas when you use a jail it can "see" that its actually on freebsd and then tries to download freebsd wheel and fails

Yeah, I have to do Java development from inside the chroot because while FreeBSD has a build for OpenJDK and I do use Eclipse as an IDE (native) most libs don't ship proper detection of FreeBSD and/or native libraries.
My podman (host) <-> docker (chroot) socket works because I've forced the platform globally on the chroot. The downside is that it breaks the host side (I could probably work around it but it's too much work), and as there are some issues running C# containers and others (e.g. postgresql without the ocijail flag) I try to stick to native (either on FreeBSD or on Alma).

Anyways thanks for your time, your feedback aligns with what I'm experiencing.