Legend or truth, concerning the libraries's loader?

[devEric69]

Hello,

(!!!) nb: Everything I say below , must be contextualized in a desktop development, with a graphic desktop, and an installation in the /home/.myAppli of a user.
The paradigm of a server (for example, a web server, with a only semi-graphic desktop like "Midnight commander", and very serious packages like Apache, Php + a framework, etc., are by the nature of their installation under the root "/', off topic with my question below.


In many forums on the net, I find that FreeBSD can easily load a *.so placed in the same directory of an application, that asks to load it (like Windows), which would be convenient.

But according to https://www.freebsd.org/cgi/man.cgi?query=ld.so , FreeBSD seems to use exactly the same ELF loader ( ld-linux.so.X) as Linux. So, it means that FreeBSD would not load - without further intervention, or program compilation directive - a library, looking first in same the application directory, like Windows which allows to keep control over the versioning of libraries used, and allows you to deliver everything in the application's directory like /home/.myAppli ( i. e. leaving all the binaries grouped together); it's this behavior that made it possible to move back what was called "DLL-hell" for a while in Windows: look the requested library(ies) would not be there, before all other directories, by a "happy chance" .

It's not a troll: several (not many, but at least 3) FreeBSD users told me so. Is that a false statement ?

Best regards,
Eric.

 

[shkhln]

But according to https://www.freebsd.org/cgi/man.cgi?query=ld.so , FreeBSD seems to use exactly the same ELF loader ( ld-linux.so.X) as Linux.

They are largely compatible, but FreeBSD and GNU (glibc) dynamic linkers are different programs written by different people.

So, it means that FreeBSD would not load - without further intervention, or program compilation directive - a library, looking first in same the application directory

By default. You can adjust that behavior with -rpath compiler flag or with LD_LIBRARY_PATH environment variable.

 

[zirias@]

it's this behavior that made it possible to move back what was called "DLL-hell" for a while in Windows: [...]

I don't think that having the application's directory by default at the beginning of the shared library search path is a good approach to solve the "DLL hell" issue. Sure it solves it, but this also means there's no way to fix e.g. a security issue in a library by just applying a patch to it system-wide.

The dynamic loader in FreeBSD (as well as in GNU) is much closer to a good solution -- using semantic versioning and symlinks, allowing to install different versions (with slightly incompatible API) of the same library side by side system-wide. This is rock solid as long as the libraries are versioned correctly, and for special extra requirements, you can have versioned symbols as well.

Yet it might suit an open source system a bit better than a closed source one, as even without changing the API, a new version of a library *might* introduce subtle bugs -- I guess that's the reason why Microsoft chose a simple approach that's close to abandoning shared libraries altogether (cause most applications have their private copies anyways).

 

[devEric69]

All right.

shkhln: clearly, I have read false statements on the net (I am saddened. LD_LIBRARY_PATH is not practical, as it must be written in a *.sh script executed each time a user /home is started, otherwise it is deleted at the next session). Alone, $ORIGIN is what seems the most flexible because it can resonate relatively to the location of the executable - which can therefore be moved with its libraries - but I have the impression that it is only valid with C compilers, like gcc. In any case, it is unknown to other compilers like FreePascal that I use).

Thank you for the information, shkhln.

 

[devEric69]

@Zirias: Overall, Microsoft has abandoned shared dlls, because everyone put their protected copies of Microsoft versioned Dlls in the middle of these last ones, so that their program would work, even if it meant that the other existing applications would be broken. Hence, Windows library's loader searches automatically for a library ***first*** in the same directory as the launched application.

Like this, with Windows, the management of the versioning of the Dlls is in fact - without subsequent complications - the responsibility of the application, which is therefore not at the mercy of an "apt-get update & upgrade" (I specify again, that I am talking about an application on desktop, not to use \ develop for and on a centralized OS server (Web, databases, etc.)).

Thank you for your answer, Zirias.

 

[xtremae]

This would certainly mitigate the dll hell issue, but it has the potential to raise a much bigger problem by opening a portal to security hell.

 

[devEric69]

In the paradigm of an application for an OS-desktop version, it's the programmer's role to check if the *.so loaded from the same directory as the binary requesting with a "LoadLibrary('abc.so.1')" or a "LoadLibrary('abc.so')", is indeed a dependent version compatible with the version of "its beloved software" requiring the *.so. It allows him to manage *easily* its library version dependencies, and not to be at the mercy of an update (apt-get update, upgrade, ...) of *.so files updating the shell, and destroying the compatibility of the library versions on which his ELF-binary counted.
I'm starting to repeat myself, but under Windows, there are programs that are not that simple, that you can change directories or rename, without recompiling them, and they continue to work; what concerns me is why you can't have the same thing under POSIX?
Because of security?
Regarding security, I understand that with a server version (a website is exposed front-end to the whole world), it is better to trust only proven packages from official repositories (I would add that the 100 to 500 most serious projects are well packaged, but that we should not be under any illusions about the following: there are installation bugs or security biases, because they are misused in a way that was not imagined before; even Microsoft knows that there is no perfect security: they require a role elevation to install an application for the first time, to make the installer more responsible for the consequences of his installation).
But I repeat myself again, the desktop world is a different world. The desktop application should be able to install everything - as much as possible in any case - in its directory as often as possible:
- like this, the desktop program keeps better control over the management of the libraries's versions, it uses (already said).
- it does not need to modify the loader's registers (one less security problem).
- a desktop program is hidden: behind a firewall from its access provider, behind a firewall from the Box, and often behind a router if there is a local network, and behind a minimalist firewall with rules meaning that the desktop should not be queried in the row of IANA server ports, or even behind a HIPS firewall asking if a particular application can go outside (asking "all the time?", or "for the session only?"). The way to use a server OS version and a desktop OS version of a distribution is not the same. For example, I don't like the idea of installing a real Php server on a desktop. On the other hand, if I find a little LAMP application that installs everything I need in my /home/.myLAMP directory, then yes, I agree to install this mini-server "emulator", to create \ test a model using loopback 127.0.0.1, before doing the same thing on an OS-server.

 

[kpedersen]

Yeah as shkhln mentioned; you can execute something like this:

$ LD_LIBRARY_PATH=.    ./myprog

 

[devEric69]

Yeah, as I replied: it will no longer be valid for the next session (see my post above).

Okay: I thought the FreeBSD's library loader was more convenient than other loaders in the POSIX family (with Windows in mind, comparatively).
That is not the case, and that is unfortunate .
➔ I really think that an evolution of this loader - just for OS-desktop versions, at least - would allow it, to become as conciliatory and practical with the use of DLLs, as Microsoft's loader is. That is all I have to say.

Thank you for all your answers.

 

[kpedersen]

Hmm thats weird, it is still valid for the next session for me. At the end of the day you are just telling the loader where to look for libraries. You can also put it in .profile for a more permanant solution (though that is a security risk).

what concerns me is why you can't have the same thing under POSIX?

Actually, when I was a UNIX beginner I also thought the same. I even used to think that binaries must have the library path hard coded in (this behaviour is old fashioned now and very rare). Binaries look for libraries in windows here:

.
%PATH%
system32

and on POSIX; here:

. (with tweak in .profile)
$LD_LIBRARY_PATH

You basically just need to make sure your program is not using a windows-centric layout. For example Windows programs would not find their .dll if you put them in a lib folder would it?

On POSIX, programs are not self contained, they are scattered over the filesystem in bin, etc, lib, share, etc. There are positives and negatives to both approaches (I actually prefer the Windows approach here and if the rest of the OS and Microsoft wasn't utterly broken, I would probably be using Windows).

 

[George]

Considering ports or packages, you are not supposed to put the executable in the same directory as the library. This is different from Windows.
You have, like, /usr/bin, and /usr/lib.

 

[shkhln]

I really think that an evolution of this loader - just for OS-desktop versions, at least - would allow it, to become as conciliatory and practical with the use of DLLs, as Microsoft's loader is. That is all I have to say.

Even completely statically built applications still depend on at least kernel APIs. Then there is D-Bus, various configuration files, etc. Windows provides more stable desktop environment solely because it provides stable APIs for desktop needs like drawing things on screen, manipulating windows, playing sounds and so on. MS also typically keeps around legacy APIs as long as possible. The dynamic library placement is a very minor factor there.

 

[devEric69]

>
> Binaries look for libraries in windows here:
> .
> %PATH%
>system32
>
> and on POSIX; here:
> . (with tweak in .profile)
> $LD_LIBRARY_PATH
>

Very well summarized: everything is said.
I would add that the priority search for ".", is becoming topical with solutions like Snap, or https://github.com/AppImage/appImageKit , which is strangely similar to Windows installations (it allows to create self-extracting packages with everything you need inside, which can be launched from a USB key, under a "lamda" user account: the user does not write a single command. Well, it's not yet a WYSIWYG installator, but it goes in that direction).

>
> Windows provides more stable desktop environment solely because it provides
> stable APIs for desktop needs like drawing things on screen, manipulating windows,
> playing sounds and so on.
>

Indeed, Windows can do everything very well on average.

According to Linus Torvalds, it is especially the oversized *nix ecosystem - including the "office war" between the Linux distributions - that makes that the crowd find the *nix desktop being too over-complicated. Now, the disappearance of Unity, which had moved far, if not too far away from Gnome tronk, makes one factor of extreme divergence less. However, it is a multitude of details that make the "user friendly" difference, not a single point, in my opinion.

As far as graphical interfaces are concerned, MS is still one step ahead - MS has expertise in graphical interface *sophistication* - but, it's much less obvious than it was 15 years ago. My workstation under Ubuntu has dark or light or ... themes, a Nemo file manager,... Everything falls quickly "under the mouse".

➲ Returning to the original theme (allowing a coded executable, and its coded DLLs and some DLLs like embedded Firebird or SQLite or..., to easily be loaded together from the same directory: yeah, I'm a little lazy), here is what I found on the internet. Those who use gcc have compilation options that make their lives easier (I'm thinking of the option '$ORIGIN'). Unfortunately, I use Lazarus (which still allows cross-compiling) which does not have this option.

Nevertheless, I am gluing a "backup" theoretical solution here:

It goes like this, when compiling with gcc:
gcc ... myAppli ... -Wl,-rpath=$ORIGIN/../lib64
(the rpath is stored in the elf executable, in the dynamic section. it can be a relative path)

readelf -d myAppli
.../...
0x0000000f (RPATH) Library rpath: [$ORIGIN/../lib64]
.../...

$ORIGIN is a special variable that means ‘this executable’, and it means the actual executable filename, as readlink would see it, so symlinks are followed.
➔In other words, $ORIGIN is special, and resolves to wherever the binary is at runtime.

There is a tool named chrpath. But above all, there is a more universal tool than chrpath called patchelf. It was originally created for use in making packages for Nix and NixOS (packaging system and a GNU/Linux distribution).

● In case there is no rpath in a binary (here called myAppli), chrpath fails:
chrpath -r '$ORIGIN/../lib64' myAppli
myAppli: no rpath or runpath tag found.

● On the other hand...,
patchelf --set-rpath '$ORIGIN/../lib64' myAppli
...succeeds just fine (that's the theory: I still have to test the practice).

Regards.

 

[_martin]

With ELF binary format (and dynamic executable) FreeBSD loads libraries in a way specified in a dynamic section of a binary. You can use readelf(1) to read these sections. -d for dynamic section. With standard libraries standard (predefined) paths are used to search for a library.
You can overwrite this behavior to an extent with environment variables (LD_*). Certain security rules apply then when doing so (e.g. setuid binaries..).

You can influence all this during compilation/linking phase. As already above suggested rpath and its expansion with $ORIGIN. You can go even as far as setting it so that library will be loaded from current directory first (-Wl,-rpath,.).

 

[unitrunker]

It's hard to compare Windows vs. FreeBSD since Windows supports side-by-side libraries where the app uses a manifest to select the best match.

FreeBSD at least has one ace card to isolate binaries: Put the troublemakers in a jail.

 

[balanga]

By default. You can adjust that behavior with -rpath compiler flag or with LD_LIBRARY_PATH environment variable.

I'm damned if I can get this set!

set LD_LIBRARY_PATH=/mylibs appears to work but it does not show up when I type env. If I try export I get command not found.

How do I set it?

 

[_martin]

balanga export works with sh/bash, set as you showed does work with csh. I can list those variables with env too.
What shell are you using ?

 

[shkhln]

set LD_LIBRARY_PATH=/mylibs appears to work but it does not show up when I type env. If I try export I get command not found.

export works for sh and bash, setenv works for csh. If you are trying to permanently add a directory to the default search path, you should consider ldconfig_paths setting in rc.conf.

 

[balanga]

balanga export works with sh/bash, set as you showed does work with csh. I can list those variables with env too.
What shell are you using ?

shell is /bin/csh

I can set LD_LIBRARY_PATH=/mylibs and echo $LD_LIBRARY_PATH shows /mylibs but the variable does not show up under env

I'm PXEBOOTING and running diskless if that makes any difference.

I'm having similar problems appending a directory to the PATH.

Just noticed that under sh() I can use export .....

 

[balanga]

export works for sh and bash, setenv works for csh. If you are trying to permanently add a directory to the default search path, you should consider ldconfig_paths setting in rc.conf.

Unfortunately /etc is readonly in the environment I'm testing.

I'm basically trying to run programs which are not installed but are accessible via a mounted path.

 

[_martin]

balanga Where are you setting this variable ? You can always set custom variables in ~/.cshrc (in your home profile) when using csh. Use setenv in profile to do so. Frankly I'm always confusing set and setenv, mostly because I don't use csh that much.

 

[balanga]

It's a readonly environment.

Eventually I gave up but figured out it was easiest to mount a filesystem and chroot() into it

 

[tingo]

It would be interesting if you could explain how you tought making persistent changes to a readonly environment would work?