How do you get Linux 64-bit emulation to work?

Jay F. Shachter

Jay F. Shachter

(Posting to the "General" forum because I could not find another forum that was better suited to the topic; feel free to relocate the discussion if it belongs elsewhere)

I am developing an application that will run on a server which will be exposed to the Internet; I have persuaded my employer to run it on *BSD rather than on Linux, for security reasons. The application will include some proprietary libraries for which we have paid, and which were provided to us only in binary format. The proprietary libraries are essential to the application. These proprietary, black-box libraries are provided only for Microsoft Windows and Linux, and only 64-bit versions are provided for Linux. Therefore, I either have to have 64-bit Linux emulation on FreeBSD, or split the application into two servers -- one exposed to the Internet, running FreeBSD, and one hidden from the Internet, running Linux -- or keep it one application, and run it on a Linux server. The last option is undesirable for reasons of security, and the 2nd option is undesirable for reasons of complexity.

The server on which we are currently developing the application (an Amazon cloud machine) is running 10.2-RELEASE. I have seen evidence of work that was done, I believe initially by Alan Jude, that implements 64-bit Linux emulation on FreeBSD. Is it possible to get 64-bit Linux emulation on 10.2-RELEASE? I saw some discussion of this question -- not very recent -- which implied a need to do it before installing any ports, but that is, of course, impossible, we have installed plenty of ports on our system, a FreeBSD system is very close to worthless if you have not installed any ports on it.

If it is not possible to get 64-bit Linux emulation on 10.2-RELEASE, I assume that it is possible in 11.0-CURRENT. Creating a new cloud machine running 11.0-CURRENT (because I don't believe you can do freebsd-update from 10.2-RELEASE to 11.0-CURRENT, please correct me if I am wrong) is very far from my first preference, for reasons stated above, newly-installed FreeBSD systems are close to worthless, and it takes days of work to make them usable. If, however, that is the only option, then that is what we shall do, and my question then becomes, How do you obtain 64-bit Linux emulation on 11.0-CURRENT? As always, thank you in advance for any and all replies.
 
As far as I know it's not available on 10.2-RELEASE. You'd need a recent 10-STABLE. It should become available in the upcoming 10.3-RELEASE.
 
I installed 11.0-CURRENT and found that I was able to do kldload linux64. When I do, I am able to execute a 64-bit Linux binary without getting the error message that ELF binary type "3" is not known.

However, I did not get very far. Because the program uses dynamic libraries (and there is no way to build it so that it does not, because the proprietary libraries we will be using are provided to us only as shared objects), I instantly received the error message that the ELF interpreter /lib64/ld-linux-x86-64 is not found. Do I have to copy that file manually from a 64-bit Linux system, and place it under the /compat/linux directory (parenthetically, is it /compat/linux for 64-bit Linux emulation, or is it, perhaps, a different directory?)? I notice that the /compat directory does not exist on my system. I am reluctant to create it manually, because I have never had to create that directory manually before on any other FreeBSD system. I suspect that there is some port, the installation of which will create that directory properly, which I might not do if I create it myself.

So, then, in a small number of words: 11.0-CURRENT seems to support 64-bit Linux emulation; how do I get it to work? As always, thank you in advance for any and all replies.
 
Did you install emulators/linux_base-c6? To achieve Linux emulation, you need to have a Linux userland as well as a kernel, or in the case of FreeBSD, a binary interface. The emulators/linux_base-c6 is the base system of CentOS 6 and contains all the binaries and libraries needed to run Linux programs in FreeBSD. However, the port installs a 32-bit userland, which cannot be used for a 64-bit application. Fortunately, according to this commit, this can be resolved by adding a few lines to /etc/make.conf:
Code: 
OVERRIDE_LINUX_BASE_PORT=c6_64
OVERRIDE_LINUX_NONBASE_PORTS=c6_64
I would advise you to read the commit to help you fully understand what you are doing.
 
Also note that 11-CURRENT can be quite unstable, it's a development version after all. The Linux 64 bit emulation should be MFC'ed now so running 10-STABLE may be a better choice.
 
Jay F. Shachter said:
(Posting to the "General" forum because I could not find another forum that was better suited to the topic; feel free to relocate the discussion if it belongs elsewhere)

I am developing an application that will run on a server which will be exposed to the Internet; I have persuaded my employer to run it on *BSD rather than on Linux, for security reasons. The application will include some proprietary libraries for which we have paid, and which were provided to us only in binary format.
Click to expand...
I would remove my post as you are leaving the trace that you gave a really bad advise to your employer. If the proprietary libraries require operating system XXX than that is what you need to use.

IMHO FreeBSD should remove Linux binary compatibility layer as it serves no serious purpose other than fooling people that they can run Linux code (serious code like MATLAB or OracleDB which people truly care about can't be run on the emulation layer). Even worse emulation has nothing to do with drivers (it is application layer emulation).
 
You really need to know yourself which dynamic libraries your application needs if you're not using a port that installs them for you, with this new 64-bit emulation there are no pre-made ports for 64-bit runtime libraries so you're very much on your own to get them in place. Same pretty much goes for using FreeBSD CURRENT in general, don't expect support for problems that are trivialities for professionals.
 
Oko said:
IMHO FreeBSD should remove Linux binary compatibility layer as it serves no serious purpose other than fooling people that they can run Linux code (serious code like MATLAB or OracleDB which people truly care about can't be run on the emulation layer). Even worse emulation has nothing to do with drivers (it is application layer emulation).
Click to expand...
Just because Linux emulation doesn't run some software doesn't mean it should be ripped out completely. I for one use it to run servers for games such as Garry's Mod and Counter Strike, where it works very well and I can leverage the benefits of FreeBSD while running binaries compiled for Linux. If I didn't have this emulation available to me I would have to resort to using Linux, which in my opinion, is inferior to FreeBSD. I don't think that all of the developers and committers throughout the world would appreciate the fact that years of hard work would be destroyed so carelessly. Instead, it is a much better use of time and effort to improve Linux emulation so that one day it could have a chance of running much more complicated software such as MATLAB or OracleDB. The WINE developers didn't give up because they saw that they couldn't run advanced Windows software. They just kept working and eventually they were able to make their previous observation untrue. If people want to blindly install FreeBSD on every Linux machine they see and think that they can get commercial software working perfectly, it is their fault, not the Linux emulation developers' fault.

Jay F. Shachter you should not abandon a working system simply because you see an alternative as more secure. The security of a system depends on many factors, of which the operating system is only one of many. One great example I can think of is Thread Hacked server (question about sendmail configuration).54694. quamenzullo didn't take proper security measures with the CMS he was using and was hacked as a result. If this had happened on a Linux machine, he still would have been hacked. You shouldn't choose FreeBSD simply because it offers what you believe to be better security.
 
Last edited:
atomicbeef said:
Just because Linux emulation doesn't run some software doesn't mean it should be ripped out completely.
Click to expand...
Yes it does because it greatly increase complexity of the code for no added benefit. Complexity in turn introduces bugs and insecurities. Unlike proprietary systems BSDs should have no second thoughts to prune any legacy code which turned out to be a bad idea. That is the beauty of BSDs comparing to proprietary systems like Linux for example. Linux emulation layer is a bad idea just like Wine. If you need an application A which run only on operating system B than you should run operating system B unless you can use alternative application. As simple as that. I am not a teenager any more. I run computers in part to make my living. If MATLAB requires RHEL to run that is what I run. When we need to test the things on the Windows that is what we run.
 
Oko said:
Yes it does because it greatly increase complexity of the code for no added benefit.
Click to expand...
The thing is, I see the Linux emulation as very beneficial, while you do not. This is simply a matter of different use cases. Like you, I strive for as much simplicity as possible. In fact, simplicity was one of the reasons I moved from Linux to FreeBSD. However, I find that not being able to use FreeBSD for my game servers (which is the OS I use for everything else) is much more complex, especially considering the fact that the Linux landscape changes so frequently. Due to the Linux emulation layer, I don't have to maintain two separate OSes, each with their own set of quirks and features. Of course, this is only more simple from a user point of view, while you were talking about simplicity from a programming point of view. In this regard, you are right. The Linux emulation layer does add complexity to the codebase, but I think that this complexity is worth the functionality. From a security standpoint, I don't think the Linux emulation layer introduces too many vulnerabilities, as it has only had 3 vulnerabilities in history according to the security advisories. There is the potential for unknown bugs to crop up 15 years later as an exploit, but overall, I think that Linux emulation is pretty secure. If you are really opposed to Linux emulation, you can compile a custom kernel without it, but that's not really necessary because it exists as a completely optional kernel module by default.

As for testing software, I would absolutely use it on the native OS it was built for. Not doing so is a great way to not get support from the developer because you aren't using a supported OS.

This whole conversation is starting to get off-topic though and would be better discussed in a new thread.
 
  • Thanks
Reactions: Oko
Sorry I've been away, I had to spend 10 hours today installing the ports that would render my newly-installed FreeBSD system minimally functional. If I may now be permitted to return to the original topic, for which I am still requesting some assistance:

I installed emulators/linux_base-c6, as I was advised to do. When I tried to run my application, I still received error messages due to the nonexistence of certain shared libraries which the abovementioned port did not provide: libgdamm-5.0.so.13, libglibmm-2.4.so.1, libboost_date_time.so.1.54, libsigc-2.0.so.0, libgda-5.0.so.4, libxml2.so.2, libsecret-1.so.0, liblzma.so.5, libgcrypt.so.11, and libgpg-error.so.2. I copied all of these files from the Linux system on which they reside to the appropriate subdirectories of /compat/linux on my 11.0-CURRENT system, reducing the unresolved dependencies one by one. Now when I invoke my application, I see this (the name of the application has been redacted, so as to protect my employer's intellectual property):

Code: 
/home/jay/XXX: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /home/jay/XXX)
/home/jay/XXX: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.14' not found (required by /home/jay/XXX)
/home/jay/XXX: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /home/jay/XXX)
/home/jay/XXX: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/lib64/libglibmm-2.4.so.1)
/home/jay/XXX: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /usr/lib64/libglibmm-2.4.so.1)
/home/jay/XXX: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /usr/lib64/libboost_date_time.so.1.54.0)
/home/jay/XXX: /usr/lib64/libstdc++.so.6: version `GLIBCXX_3.4.15' not found (required by /usr/lib64/libsigc-2.0.so.0)
/home/jay/XXX: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/lib64/libgda-5.0.so.4)
/home/jay/XXX: /lib64/libz.so.1: version `ZLIB_1.2.3.3' not found (required by /usr/lib64/libxml2.so.2)
/home/jay/XXX: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/lib64/libxml2.so.2)
/home/jay/XXX: /lib64/libc.so.6: version `GLIBC_2.15' not found (required by /usr/lib64/libxml2.so.2)
/home/jay/XXX: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /usr/lib64/libsecret-1.so.0)
/home/jay/XXX: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /lib64/liblzma.so.5)
/home/jay/XXX: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /lib64/libgcrypt.so.11)
/home/jay/XXX: /lib64/libc.so.6: version `GLIBC_2.15' not found (required by /lib64/libgcrypt.so.11)
/home/jay/XXX: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /lib64/libgpg-error.so.0)

In other words, I am now running into incompatibilities between the libraries that were installed by emulators/linux_base-c6 and the corresponding libraries on the system where the application was built.

Is this problem solely due to the fact that I developed the application on an Ubuntu system rather than on CentOS 6? If that is the problem, I am prepared to install a CentOS 6 system on the development machine and develop the application on CentOS 6 rather than on Ubuntu 14.04. But I would like to hear from the possessors of specialized knowledge who read this forum, before making that investment. Would CentOS 7 be satisfactory (I prefer to work on the most recent supported version of an operating system, when possible)? If it must be CentOS 6, is there a particular point release of CentOS 6 that I must install? It will potentially save me -- and my employer -- quite a bit of time if someone with knowledge and experience in this matter (perhaps purchased with the same time that I am now trying to save myself) can provide these details. And, finally, if recreating my binary on a CentOS system is not the solution to my problem, then what is? As always, thank you in advance for any and all replies.
 
The library incompatibilities are caused because the libraries themselves were linked to different libraries when compiled. In order to get the system working properly, you would have to install a CentOS 6.6 system and compile your program there. Then you need to copy over whatever libraries you need from the CentOS 6.6 machine to the FreeBSD machine.
 
Linux tends to use symbol versioning on the shared libraries to protect against linking with wrong libraries and the errors such as version `GLIBCXX_3.4.15' not found (required by /usr/lib64/libglibmm-2.4.so.1) is an indication that your application was compiled and linked against a different set of libraries that you have installed.
 
Based on the above, then, the development platform will be CentOS, and the deployment platform, for security reasons, will be a version of FreeBSD (presently 11.0-CURRENT, perhaps 10.3-RELEASE when it becomes available) that supports 64-bit Linux emulation. I do not want to expose a Linux system to the Internet when alternatives are available (although Linux is certainly safer than Microsoft Windows, there are other operating systems that are much more secure), but feel free to continue to debate the advisability or the need to use FreeBSD at all; I enjoyed reading your postings on that subject, and I may even learn something from them. In the meantime, though, I have two final questions, before I install another OS on the computer:

1. Does the development platform have to be CentOS 6? CentOS 7 is the most recent version.
2. Does it have to be CentOS 6.6 and not 6.7? CentOS 6.6 is not supported (I have not checked whether Springdale, or Scientific Linux, or Oracle Linux -- which are all equivalent to CentOS -- support version 6.6, but CentOS does not); 6.7 is supported, but not 6.6.
 
You are required to use CentOS 6.6 and CentOS 6.6 only. Any other version will most likely result in incompatible libraries.

On another note, you really should install 10-STABLE instead of 11-CURRENT as it will be easier to update the OS without introducing major changes to the environment which could possibly break things.
 
If the binaries are supposed to be run on Linux, run it on Linux, otherwise your supplier will refuse any responsibility for any possible issues - there was no QA and testing for FreeBSD. Regarding the Linux security - I don't think that properly managed FreeBSD is any safer than properly managed RHEL/CentOS with SELinux configured, surely not CURRENT. The reasons why I would choose FreeBSD are different, but not in this case.
 
Oko said:
IMHO FreeBSD should remove Linux binary compatibility layer as it serves no serious purpose [...] like MATLAB or OracleDB which people truly care about
Click to expand...
There are other serious things besides MATLAB or OracleDB, for example android studio. If you tell me how to setup this without linux emulation I will be grateful. Especially now that the new versions of the android build and platform tools are all 64 bit linux binaries.
 
laufdi said:
There are other serious things besides MATLAB or OracleDB, for example android studio. If you tell me how to setup this without linux emulation I will be grateful. Especially now that the new versions of the android build and platform tools are all 64 bit linux binaries.
Click to expand...
I agree. Personally I just care for MATLAB and Mathematica and lot of people complained about OracleDB. However I am not retracting the statement about Linux emulation. If Android upstream truly care about BSDs you guys should be able to build its tools natively. They will even benefit as BSD build will probably expose many bugs. If they don't you are doing a wrong thing by developing Android application on FreeBSD.
 
As I get the error /lib64/libc.so.6: version `GLIBC_2.14' not found, is there any way to update the libc.so.6?
I have linux-c6... installed, is there a linux port with newer libc?
 
laufdi said:
As I get the error /lib64/libc.so.6: version `GLIBC_2.14' not found, is there any way to update the libc.so.6?
I have linux-c6... installed, is there a linux port with newer libc?
Click to expand...
I am not sure if there are Cent OS 7 libraries in ports as FreeBSD Linux emulation is usually quite a bit behind commonly used Linux-es. Cent OS 6 uses glibc 2.12 and it is not possible to update. Cent 7 is 2.17 glibc
 
laufdi said:
I don't see anything like c7 or centos in ports
Click to expand...
Dude you need Linux. Using Linux compatibility is beating a dead horse. If you opt for Red Hat based distro you need 7 the one with systemd:D if you want libc 2.14 or higer.
 
You must log in or register to reply here.
Back
Top