Solved SOLVED: changed the absolute symlink of ld-linux-x86-64.so.2 w/ relative: Any way to execute Linux ELF binaries without chrooting into the Linux jail

I

IThoughtTooLongForThis

I was wondering if it's possible to execute Linux ELF binaries without having to chroot into my Ubuntu jail.
If I chroot into the jail, I can do pretty much anything with Linux apps (even with X).
If I try to run the same binary without being in my chroot, I get the following error:
Code: 
ELF interpreter /lib64/ld-linux-x86-64.so.2 not found, error 2
Abort
Now, I know what this means and it's because it's trying to find the library associated with the Linux install directly on FreeBSD's root instead of the Ubuntu jail's root. Is there a way to redirect the library loading or is that just something I have to deal with?
 
zirias@ said:
That's what /compat/linux is for,
Click to expand...
Yes. Nowadays you can actually change this and have it point to /compat/ubuntu for example, the sysctl compat.linux.emul_path can be set to change this path. The /compat/linux directory is what emulators/linux_base-c7 uses and was, for a long time, pretty much a hardcoded. path.

The idea here is to install a small linux distribution, some libraries, executables, etc. in order to satisfy the Linux binary requirements. Then using brandelf(8), mark the binary to execute a certain ABI (in this case, Linux). And you should be able to directly execute it.
 
SirDice said:
Then using brandelf(8), mark the binary to execute a certain ABI (in this case, Linux). And you should be able to directly execute it.
Click to expand...

Is this something that needs to be done to every Linux ELF? I was thinking that was already the case - the Linux ELF already being "branded" as such.

Also, I just checked my sysctl (and sysctl.conf file) output and I do have it set to /compat/ubuntu with the same issue.
compat.linux.emul_path: /compat/ubuntu

I would prefer my own Linux jail over linux_base-c7 as it's a bit more up to date, afik.
 
As far as I know only the ones that are executed "from" FreeBSD. It just tells the system it shouldn't treat it as a FreeBSD executable. It may already be branded correctly, but that's easy enough to check.
 
Wait a moment, I guess I should explain a bit more here. /compat/linux is not just a minimal GNU/Linux userland (any distribution), but should be tuned to integrate well with the FreeBSD userland. This especially means it should not "shadow" (as it's used like a filesystem overlay for Linux binaries) files and paths you'll always want to use from your FreeBSD system. Typical examples are your home directory, some configurations in /etc, and so on. It also shouldn't contain any "low-level" Linux tools that wouldn't work on FreeBSD anyways, and so on. As a side note, I don't remember the actual path was ever hardcoded, but maybe it was a long time ago ... not too relevant anyways.

So sure, you could just use some Ubuntu base install for that, but you won't get the "perfect" result. You could also try and adapt your Ubuntu userland (as roughly described above) yourself for better integration results.

And also sure, the linux-c7 userland is very dated now, which is already a problem trying to get some Linux software to run. Replacing it isn't as simple as just re-package some other Linux distribution 1:1 though. That's also why I'm experimenting with a different approach, building a Linuxulator userland entirely from source instead. Still not sure it will lead anwhere...
 
BobSlacker said:
Why not Gentoo?
Click to expand...
I don't see what should be the point. If you meant using Gentoo's "portage" to build packages, this would be just a cumbersome thing to do ... we have our own ports framework, using that to build Linux packages from source is much more straight-forward. If you meant using Gentoo binary packages, AFAIR they're still a "second class citizen" in Gentoo world, so apart from being conceptually the same as repackaging any other distribution, it would most likely be more cumbersome as well.
 
SirDice said:
As far as I know only the ones that are executed "from" FreeBSD. It just tells the system it shouldn't treat it as a FreeBSD executable. It may already be branded correctly, but that's easy enough to check.
Click to expand...
I just ran "brandelf -t Linux *linux-bin*" on a few Linux binaries. Running "brandelf *linux-bin*" does show that it's now labeled as a Linux binary instead of a SysV binary.
I'm still receiving the same error.
 
zirias@ said:
Wait a moment, I guess I should explain a bit more here. /compat/linux is not just a minimal GNU/Linux userland (any distribution), but should be tuned to integrate well with the FreeBSD userland. This especially means it should not "shadow" (as it's used like a filesystem overlay for Linux binaries) files and paths you'll always want to use from your FreeBSD system. Typical examples are your home directory, some configurations in /etc, and so on. It also shouldn't contain any "low-level" Linux tools that wouldn't work on FreeBSD anyways, and so on. As a side note, I don't remember the actual path was ever hardcoded, but maybe it was a long time ago ... not too relevant anyways.

So sure, you could just use some Ubuntu base install for that, but you won't get the "perfect" result. You could also try and adapt your Ubuntu userland (as roughly described above) yourself for better integration results.

And also sure, the linux-c7 userland is very dated now, which is already a problem trying to get some Linux software to run. Replacing it isn't as simple as just re-package some other Linux distribution 1:1 though. That's also why I'm experimenting with a different approach, building a Linuxulator userland entirely from source instead. Still not sure it will lead anwhere...
Click to expand...

I just reread this to attempt to understand it more clearly.
When you say your approach is building Linuxulator userland from source, are you talking about the actual syscall translation system for FreeBSD or an actual Linux userland?
 
IThoughtTooLongForThis said:
I'm still receiving the same error.
Click to expand...
You will get the error from your OP when there's no /compat/linux/lib64/ld-linux-x86-64.so.2 on your system. Or, if you changed the compat.linux.emul_path sysctl, the same file in a different path. Or when Linuxulator isn't loaded at all.

Any dynamically linked Linux binary will need at least the program interpreter (ld-linux-x86-64.so.2 on Linux/x86_64) plus whatever shared libraries it links in order to run.

IThoughtTooLongForThis said:
When you say your approach is building Linuxulator userland from source, are you talking about the actual syscall translation system for FreeBSD or an actual Linux userland?
Click to expand...
There's no need to translate any syscalls, the Linuxulator (in the FreeBSD kernel, loaded as a module) offers all the Linux syscalls needed. The branding of the ELF binary just tells the kernel which set of syscalls to offer, the native ones or the Linux ones. So yes, this is just about building the actual userland. As I said, I'm still not sure whether it would work better than repackaging some existing Linux distribution. I just tried to quickly explain why using a Linux distribution "as is" won't cut it, it must be adapted to work well as a userland for Linuxulator.
 
zirias@ said:
You will get the error from your OP when there's no /compat/linux/lib64/ld-linux-x86-64.so.2 on your system. Or, if you changed the compat.linux.emul_path sysctl, the same file in a different path. Or when Linuxulator isn't loaded at all.

Any dynamically linked Linux binary will need at least the program interpreter (ld-linux-x86-64.so.2 on Linux/x86_64) plus whatever shared libraries it links in order to run.


There's no need to translate any syscalls, the Linuxulator (in the FreeBSD kernel, loaded as a module) offers all the Linux syscalls needed. The branding of the ELF binary just tells the kernel which set of syscalls to offer, the native ones or the Linux ones. So yes, this is just about building the actual userland. As I said, I'm still not sure whether it would work better than repackaging some existing Linux distribution. I just tried to quickly explain why using a Linux distribution "as is" won't cut it, it must be adapted to work well as a userland for Linuxulator.
Click to expand...
I figured out my issue and it directly relates to your last sentence.
Turns out that I had to replace the absolute symlink for ld-linux-x86-64.so.2 with a relative one and now I can run Linux binaries without the need to go into the chroot first!

I originally setup my Linux jail with this guide: https://wiki.freebsd.org/LinuxJails
Which doesn't say anything about symlinks needing to be relative instead of absolute. I got help from the #FreeBSD Libera Chat IRC channel.
 
IThoughtTooLongForThis said:
I originally setup my Linux jail with this guide: https://wiki.freebsd.org/LinuxJails
Which doesn't say anything about symlinks needing to be relative instead of absolute.
Click to expand...
Well, that makes sense, because for a Linux jail, such things won't matter as you'll be chroot'ed inside the jail.

Interesting discovery though, I wasn't aware that the target of a symlink isn't resolved again through the overlay. But indeed yet another example of things to consider for the actual Linuxulator userland in /compat/linux. And as I said, there are quite some other things, like e.g. not shadowing configuration files that would be valid for Linux binaries as well.

To me, this issue is yet another hint about the need to finally replace our linux-c7 userland. Users shouldn't have to install their own /compat/linux and fiddle with it for all the integration issues...
 
zirias@ said:
Well, that makes sense, because for a Linux jail, such things won't matter as you'll be chroot'ed inside the jail.

Interesting discovery though, I wasn't aware that the target of a symlink isn't resolved again through the overlay. But indeed yet another example of things to consider for the actual Linuxulator userland in /compat/linux. And as I said, there are quite some other things, like e.g. not shadowing configuration files that would be valid for Linux binaries as well.

To me, this issue is yet another hint about the need to finally replace our linux-c7 userland. Users shouldn't have to install their own /compat/linux and fiddle with it for all the integration issues...
Click to expand...
I would love do to whatever I can to “modernize” the Linux userland as I’m going to try gaming on FreeBSD. No, I’m not a masochist, either.

Any good resources for me to look into that’s more in depth than just setting up a Linux jail? Heads up, I have, for all intents and purposes, no programming experience other than writing “hello would” and basic cli apps in Python, Rust, and C.
 
IThoughtTooLongForThis said:
I would love do to whatever I can to “modernize” the Linux userland as I’m going to try gaming on FreeBSD. No, I’m not a masochist, either.

Any good resources for me to look into that’s more in depth than just setting up a Linux jail? Heads up, I have, for all intents and purposes, no programming experience other than writing “hello would” and basic cli apps in Python, Rust, and C.
Click to expand...
Well, it's not a simple thing :(

First things first, the (IMHO?) only sane way to create a Linuxulator userland is by using ports for that. So the first thing to read and understand would probably be Porter's Handbook.

Unfortunately, a Linux port works quite differently from regular ports. For the classic approach of "re-packaging" from an existing Linux distribution, the only "documentation" I'm aware of is some pretty old blog writing here: https://www.leidinger.net/blog/2011/08/29/howto-create-a-new-linux_base-port/ – because of its age, some technical infos are outdated, so, careful.

Once you have some understanding how FreeBSD ports (including the "framework" in /usr/ports/Mk) work, it might also help to just have a look at the existing linux-c7 ports.

As I said, I'm trying a completely different approach myself, which is building the Linux ports/packages from source (and I still have to repeat I can't decide yet whether that's a good idea in practice). If you would be interested to help with that, have a look at my work in progress here: https://github.com/Zirias/freebsd-ports-overlay-linuxsrc ... it's a ports overlay that's e.g. easy to use with poudriere.
 
zirias@ said:
Well, it's not a simple thing :(

First things first, the (IMHO?) only sane way to create a Linuxulator userland is by using ports for that. So the first thing to read and understand would probably be Porter's Handbook.

Unfortunately, a Linux port works quite differently from regular ports. For the classic approach of "re-packaging" from an existing Linux distribution, the only "documentation" I'm aware of is some pretty old blog writing here: https://www.leidinger.net/blog/2011/08/29/howto-create-a-new-linux_base-port/ – because of its age, some technical infos are outdated, so, careful.

Once you have some understanding how FreeBSD ports (including the "framework" in /usr/ports/Mk) work, it might also help to just have a look at the existing linux-c7 ports.

As I said, I'm trying a completely different approach myself, which is building the Linux ports/packages from source (and I still have to repeat I can't decide yet whether that's a good idea in practice). If you would be interested to help with that, have a look at my work in progress here: https://github.com/Zirias/freebsd-ports-overlay-linuxsrc ... it's a ports overlay that's e.g. easy to use with poudriere.
Click to expand...
I’ve looked into wanting to become a port maintainer for a few things that I’ve compiled by hand but only from a high level and haven’t given it much thought after that.

I suppose my main concern is I use pkg for all my packages and not ports. I’m assuming this would be fine within a jail system so I can keep my binaries separate?

So your project is strictly just the user land, not based on any specific distro? I suppose that makes sense with what you’ve been saying about compiling from source, the Linux user land; almost LFS like?

Now that FreeBSD is all that I use, I imagine I can spend some more time into some projects like this.
 
IThoughtTooLongForThis said:
So your project is strictly just the user land, not based on any specific distro?
Click to expand...
Yes. I just added a separate thread trying to explain it a bit, because it's off-topic in here I guess: https://forums.freebsd.org/threads/a-linuxulator-userland-built-from-source.90649/
IThoughtTooLongForThis said:
almost LFS like?
Click to expand...
I certainly consulted LFS to get around some issues especially with the cross toolchain that's needed first and the base toolchain ports 😉
 
You must log in or register to reply here.
Back
Top