Editorial
It has been two times now that relying on a Jail managed to get my system and myself out of a heap of trouble. The first time was when I was unable to build a certain port but the error messages made me suspicious; I didn't believe that this could have been a screw up from the ports maintainer. Then it hit me: my jail! I rebuild the port within the (clean!) environment of my jail (using the same configuration, obviously) and it build cleanly. Examining the logs helped me to find the cause of the problems (a dumb experiment a few months prior) and fix it.
A continuance of this is my base system itself. Yesterday I tried to rebuild my kernel only to be greeted with an error message. As it turned out this was caused by a (self inflicted!) problem with Clang. And once more it was my jail which helped me to fix it by building the entire base (and kernel) within the confines of said jail.
Time for me to give a little love back and share this with you guys. I'm well aware that most (?) of you will probably be well familiar with the whole concept already, but these writeups serve more purposes
And there's always a chance that some people might actually learn something new here.
This will not be your standard "how to install a Jail" guide Sorta...
What is a Jail?
A Jail is basically a virtual FreeBSD environment running "inside" your main environment. Somewhat comparable to what you can achieve with chroot, and since I like to go all out in my guides lets start by taking a closer look at this mysterious chroot command.
Change the root ("chroot")
As you probably know the heart of a Unix system (other than the kernel and/or init) is the root. It all starts at / which is the 'beginning' of your system; the whole directory hierarchy starts here. Unix doesn't know about drive letters and all that nonsense * but instead places everything within "the one" directory structure. But this does pose a risk. For example: /dev/mem basically provides access to your system's memory. Or /etc/fstab which could help an attacker to learn more about your setup.
But there are also less paranoid reasons: if a user has all their data in their home directory, then why bother them with stuff they can't use in the first place? Why not confine them in a place of their own? This is what chroot does. Once again: the root is the heart of your Unix(-like) system, and this allows you to "shift" that perception onto another location within your Unix environment.
However, it is not a command you can "just" use. It needs some preparations:
Pay close attention to that last error. As you can see the error seems bogus because when I use
That's what's triggered our error message. But no problem: we'll just copy the executable and be done with it. Well... maybe not:
Have you ever wondered what the big deal with /rescue is? If so: here you go. See: executables within FreeBSD rely on libraries to function. Sometimes quite a few of them. If you want to learn more about that then ldd can be a very useful tool:
So in order to make this work we'd have to copy all those libraries within our new root environment before csh can run. ...or. Did you know that all the files within the /rescue location have been "statically compiled"? What that means? This:
Notice something? /rescue/csh is a lot bigger than the other executable. The reason for that is that /rescue/csh does not rely on any other external libraries. They're combined into one package. In official terms: "/rescue/csh is not a dynamic ELF executable". Instead it's self-supporting. As such:
Please don't tell me that you're not getting a little bit excited here?
So what is happening? I'm logged in as root and the default shell for that user is /bin/csh. I'm also trying to change my root ("chroot") to /home/peter/temp. By doing so I'm basically telling the system "please execute /bin/csh and confine it to /home/peter/temp". The result can be seen above. Since I never copied /bin/ls to /home/peter/temp/bin the command cannot be started. Same for /usr/local.
I basically told the system to exchange / for /home/peter/temp. So what was once / has now physically become /home/peter/temp. Pretty cool, right?
So what about Jails?
A jail takes the above concept one step further. Instead of merely locking a process into a specific file system location it will actually "fork" a full blown kernel process and start that within the confines of a specific location. In other words: it will start a full blown - yet virtual - FreeBSD environment and limit its access not only to that specific filesystem location. No, this goes much deeper than that: because this is a separate kernel process it will also generate a whole new "userland environment". In other words: it separates kernel processes. If you want to know what processes are running then you can use ps for that info. ps will then ask the kernel and relay that info back to you. But by starting a new kernel process ps would ask the kernel about this while that kernel process in itself has absolutely no knowledge about anything happening outside its own perimeter. As a result anything inside the jail would be oblivious to everything else.
Talk is cheap so lets put this to the test:
Note: my server restricts process listings to current users only. Even so, now lets try the same thing but this time from within a jail:
That's quite the difference, wouldn't you agree? All the system noticed was the same process we just started.
This is what sets jails apart from a chroot'ed environment: full separation. As mentioned earlier it's basically a virtual system running on top of your current one.
How to set it up?
There are tons of ways to do this and there's honestly no "best" here. Never forget: what works for me doesn't have to work for you. But having said that I do think I have some useful tips to share...
Now, before I start I need to mention sysutils/ezjail. Note that I have 0 experience with this port but I know plenty of people use this to manage their ports so it seems only fair to mention it. If you want easy then this could be for you.
Thing is: I don't want easy because 'easy' always comes with a price attached. Even here. For starters: uniformity. If you use this port then I'm pretty sure you'll confine yourself to its set out standards. Standards which are out in the open for everyone to learn. So once someone finds out their inside a jail and they're familiar with ezjail it just might give them an edge if they're also gained access to your host. Unlikely, for sure, but still something to consider. Especially when security is an issue.
Another aspect is that setting up a jail really isn't all that hard.
Userland
As mentioned above a jail is basically a separate kernel process, and what does a kernel need? A userland! So basically an environment to use and allow usage from users. It is for that reason why I prefer to set up a jail environment with a copy of the general base system. In other words:
My server runs FreeBSD 11.1 and what you see above are the base packages for that OS. My psi jail is basically nothing more but an extracted copy of base.txz (and a bit of manual tweaking).
This is the first step in setting up a jail: the userland.
I prefer to set up a directory (actually: a virtual ZFS filesystem) and then extract the base.txz in there so that I have a pristine FreeBSD setup. Next stop: configure the jail, that can be done using /etc/jail.conf:
This is pretty much all you need. A hostname, valid IP address and the path to your set up userspace followed by some optional preferences. For this jail I took things easy and set it up so that the IP address would become an alias on em0 which is my main network interface. But you can easily change this if you want. My setup gives the jail pretty much unlimited network access. A good way to prevent this is to utilize lo0 and then set up a NAT connection between your jail and the main interface.
Fire it up!
When everything has been set up all you have to do is actually start, or create, your jail. To be honest I think this is a poor choice of wording but at the same time I cannot come up with anything else. I mean... To me the process of creation is done when you set up the base system (for example by extracting base.txz or by installing the userland manually using
The cool part here is that because your jail is a basically a virtual environment you can even associate processes to be used within the jail. In normal English: set up /etc/rc.conf in order to automatically start certain processes. So that whenever you start the jail you'll also (automatically) start those processes. Just like you'd do on a "normal" FreeBSD system.
Which brings me to my last item... How do we start all this?
Well, using /etc/rc.conf:
This code snippet ensures that whenever my server boots it'll also fire up my Psi jail.
Why would you want a jail?
There are many purposes. Security (basically: an extra security layer) is one of those. Install any public services into a jail and if one gets run over due to some freak backdoor then the attacker will only be confined within your jail, so mostly ruling out any risk for the base system. Reset the jail, and you're good to go again.
But there's more. Are you building your own ports? I do too. The problem though is that a lot of this depends on your environment. So what if you have some kind of problem with said environment (maybe a library which you forgot to rebuild against other (upgraded) libraries)? That's when a jail can become quite useful. Because it's separated from your main host it wouldn't be "contaminated" in the same way, thus providing the perfect backup as building environment.
Of course you should always be careful here, because although you may have ruled out the dependency on certain libraries during the build process, what about the actual usage process?
What to think about DMZ? Aka: de-militirazed zones. No, not to host public services, but what about running your next browsing session from within the confines of a jail? Downloaded any malware? Who cares if it was specifically targeted for FreeBSD: you're inside a jail so worst case scenario is that your jail gets messed up. A quick reset and you're good to go again.
And there you have it....
A different take on Jails. I'm well aware that most I shared is also documented elsewhere, such as the awesome FreeBSD handbook, but I still hope that this could be useful for some of you. Jails are amazing. Or at least they can be
It has been two times now that relying on a Jail managed to get my system and myself out of a heap of trouble. The first time was when I was unable to build a certain port but the error messages made me suspicious; I didn't believe that this could have been a screw up from the ports maintainer. Then it hit me: my jail! I rebuild the port within the (clean!) environment of my jail (using the same configuration, obviously) and it build cleanly. Examining the logs helped me to find the cause of the problems (a dumb experiment a few months prior) and fix it.
A continuance of this is my base system itself. Yesterday I tried to rebuild my kernel only to be greeted with an error message. As it turned out this was caused by a (self inflicted!) problem with Clang. And once more it was my jail which helped me to fix it by building the entire base (and kernel) within the confines of said jail.
Time for me to give a little love back and share this with you guys. I'm well aware that most (?) of you will probably be well familiar with the whole concept already, but these writeups serve more purposes
And there's always a chance that some people might actually learn something new here.This will not be your standard "how to install a Jail" guide
Sorta...What is a Jail?
A Jail is basically a virtual FreeBSD environment running "inside" your main environment. Somewhat comparable to what you can achieve with chroot, and since I like to go all out in my guides lets start by taking a closer look at this mysterious chroot command.
Change the root ("chroot")
As you probably know the heart of a Unix system (other than the kernel and/or init) is the root. It all starts at / which is the 'beginning' of your system; the whole directory hierarchy starts here. Unix doesn't know about drive letters and all that nonsense * but instead places everything within "the one" directory structure. But this does pose a risk. For example: /dev/mem basically provides access to your system's memory. Or /etc/fstab which could help an attacker to learn more about your setup.
But there are also less paranoid reasons: if a user has all their data in their home directory, then why bother them with stuff they can't use in the first place? Why not confine them in a place of their own? This is what chroot does. Once again: the root is the heart of your Unix(-like) system, and this allows you to "shift" that perception onto another location within your Unix environment.
However, it is not a command you can "just" use. It needs some preparations:
Code:
peter@unicron:/home/peter $ ls
Desktop/ devel/ mc/
Mail/ downloads/ passwd
NetBeansProjects/ etc/ pictures/
VisualParadigm/ infocom/ spigot@
backups/ kamps/ temp/
bin/ mail/ transfer/
bugzilla.png man/
peter@unicron:/home/peter $ chroot temp
chroot: temp: Operation not permitted
peter@unicron:/home/peter $ su
Password:
root@unicron:/home/peter # chroot temp
chroot: /bin/csh: No such file or directory
root@unicron:/home/peter # ls /bin/csh
/bin/csh* ls /bin/csh it shows up easily. So what's this about "No such file"? Well... this is what chroot is all about. /bin/csh exists. But the chroot command changes the root to a new location, in my attempt above I tried to change this to /home/peter/temp. So we need to exchange / for /home/peter/temp and guess what?
Code:
root@unicron:/home/peter # file /home/peter/temp/bin/csh
/home/peter/temp/bin/csh: cannot open `/home/peter/temp/bin/csh' (No such file or directory)
Code:
root@unicron:/home/peter # mkdir temp/bin
root@unicron:/home/peter # cp /bin/csh temp/bin
root@unicron:/home/peter # chroot temp
ELF interpreter /libexec/ld-elf.so.1 not found, error 2
Abort
Code:
root@unicron:/home/peter # ldd `which csh`
/bin/csh:
libncursesw.so.8 => /lib/libncursesw.so.8 (0x800885000)
libcrypt.so.5 => /lib/libcrypt.so.5 (0x800ae3000)
libc.so.7 => /lib/libc.so.7 (0x800d02000)
Code:
root@unicron:/home/peter # ls -l /bin/csh /rescue/csh
-r-xr-xr-x 2 root wheel 432272 Mar 25 11:26 /bin/csh*
-r-xr-xr-x 139 root wheel 10901936 Mar 25 11:27 /rescue/csh*
Code:
root@unicron:/home/peter # cp /rescue/csh temp/bin
root@unicron:/home/peter # chroot temp
# ls /
ls: Command not found.
# cd /usr/local
/usr/local: No such file or directory.So what is happening? I'm logged in as root and the default shell for that user is /bin/csh. I'm also trying to change my root ("chroot") to /home/peter/temp. By doing so I'm basically telling the system "please execute /bin/csh and confine it to /home/peter/temp". The result can be seen above. Since I never copied /bin/ls to /home/peter/temp/bin the command cannot be started. Same for /usr/local.
I basically told the system to exchange / for /home/peter/temp. So what was once / has now physically become /home/peter/temp. Pretty cool, right?
So what about Jails?
A jail takes the above concept one step further. Instead of merely locking a process into a specific file system location it will actually "fork" a full blown kernel process and start that within the confines of a specific location. In other words: it will start a full blown - yet virtual - FreeBSD environment and limit its access not only to that specific filesystem location. No, this goes much deeper than that: because this is a separate kernel process it will also generate a whole new "userland environment". In other words: it separates kernel processes. If you want to know what processes are running then you can use ps for that info. ps will then ask the kernel and relay that info back to you. But by starting a new kernel process ps would ask the kernel about this while that kernel process in itself has absolutely no knowledge about anything happening outside its own perimeter. As a result anything inside the jail would be oblivious to everything else.
Talk is cheap so lets put this to the test:
Code:
root@unicron:/home/peter # mount -t devfs devfs temp/dev
root@unicron:/home/peter # chroot temp
# ps
PID TT STAT TIME COMMAND
1145 v0 IWs 0:00.00 login [pam] (login)
61750 v0 S 5:30.32 /usr/local/bin/X :0 -auth /home/peter/.serverauth.61736 ( 1146 v1 Is+ 0:00.01 /usr/libexec/getty Pc ttyv1 1147 v2 Is+ 0:00.01 /usr/libexec/getty Pc ttyv2 1148 v3 Is+ 0:00.00 /usr/libexec/getty Pc ttyv3 1149 v4 Is+ 0:00.01 /usr/libexec/getty Pc ttyv4 1150 v5 Is+ 0:00.01 /usr/libexec/getty Pc ttyv5
1151 v6 Is+ 0:00.01 /usr/libexec/getty Pc ttyv6
2778 4 IW 0:00.00 su
2779 4 IW 0:00.00 _su (csh)
2782 4 I+ 0:01.48 mc
2783 5 Is+ 0:00.38 /bin/csh
64628 7 IW 0:00.00 su
64629 7 S 0:00.09 _su (csh)
64886 7 S 0:00.01 /bin/csh -i
64889 7 R+ 0:00.00 ps
Code:
root@unicron:/home/peter # jexec psi ps
PID TT STAT TIME COMMAND
64955 7 R+J 0:00.02 psThis is what sets jails apart from a chroot'ed environment: full separation. As mentioned earlier it's basically a virtual system running on top of your current one.
How to set it up?
There are tons of ways to do this and there's honestly no "best" here. Never forget: what works for me doesn't have to work for you. But having said that I do think I have some useful tips to share...
Now, before I start I need to mention sysutils/ezjail. Note that I have 0 experience with this port but I know plenty of people use this to manage their ports so it seems only fair to mention it. If you want easy then this could be for you.
Thing is: I don't want easy because 'easy' always comes with a price attached. Even here. For starters: uniformity. If you use this port then I'm pretty sure you'll confine yourself to its set out standards. Standards which are out in the open for everyone to learn. So once someone finds out their inside a jail and they're familiar with ezjail it just might give them an edge if they're also gained access to your host. Unlikely, for sure, but still something to consider. Especially when security is an issue.
Another aspect is that setting up a jail really isn't all that hard.
Userland
As mentioned above a jail is basically a separate kernel process, and what does a kernel need? A userland! So basically an environment to use and allow usage from users. It is for that reason why I prefer to set up a jail environment with a copy of the general base system. In other words:
Code:
peter@unicron:/home/peter $ zfs list | grep jails
zroot/opt/jails 6.20G 97.0M 153M /opt/jails
zroot/opt/jails/psi 6.05G 49.4G 6.05G /opt/jails/psi
peter@unicron:/home/peter $ ls /opt/jails
base.txz kernel.txz lib32.txz psi/This is the first step in setting up a jail: the userland.
I prefer to set up a directory (actually: a virtual ZFS filesystem) and then extract the base.txz in there so that I have a pristine FreeBSD setup. Next stop: configure the jail, that can be done using /etc/jail.conf:
Code:
peter@unicron:/home/peter $ cat /etc/jail.conf
## Jail(s) config
psi {
host.hostname = "psi.intranet.lan";
ip4.addr = 10.0.1.6;
path = /opt/jails/psi;
mount.devfs;
mount.fstab = /etc/fstab.psi;
persist;
interface = em0;
# devfs_ruleset = 2;
# enforce_statfs = 1;
allow.raw_sockets;
allow.chflags;
allow.mount.procfs;
allow.mount.devfs;
}Fire it up!
When everything has been set up all you have to do is actually start, or create, your jail. To be honest I think this is a poor choice of wording but at the same time I cannot come up with anything else. I mean... To me the process of creation is done when you set up the base system (for example by extracting base.txz or by installing the userland manually using
# make DESTDIR=/path/to/jail/root installworld). But in all honesty: this only becomes a real jail once you "forked" the kernel process: # jail -c psi.The cool part here is that because your jail is a basically a virtual environment you can even associate processes to be used within the jail. In normal English: set up /etc/rc.conf in order to automatically start certain processes. So that whenever you start the jail you'll also (automatically) start those processes. Just like you'd do on a "normal" FreeBSD system.
Which brings me to my last item... How do we start all this?
Well, using /etc/rc.conf:
Code:
jail_enable="YES"
jail_list="psi"Why would you want a jail?
There are many purposes. Security (basically: an extra security layer) is one of those. Install any public services into a jail and if one gets run over due to some freak backdoor then the attacker will only be confined within your jail, so mostly ruling out any risk for the base system. Reset the jail, and you're good to go again.
But there's more. Are you building your own ports? I do too. The problem though is that a lot of this depends on your environment. So what if you have some kind of problem with said environment (maybe a library which you forgot to rebuild against other (upgraded) libraries)? That's when a jail can become quite useful. Because it's separated from your main host it wouldn't be "contaminated" in the same way, thus providing the perfect backup as building environment.
Of course you should always be careful here, because although you may have ruled out the dependency on certain libraries during the build process, what about the actual usage process?
What to think about DMZ? Aka: de-militirazed zones. No, not to host public services, but what about running your next browsing session from within the confines of a jail? Downloaded any malware? Who cares if it was specifically targeted for FreeBSD: you're inside a jail so worst case scenario is that your jail gets messed up. A quick reset and you're good to go again.
And there you have it....
A different take on Jails. I'm well aware that most I shared is also documented elsewhere, such as the awesome FreeBSD handbook, but I still hope that this could be useful for some of you. Jails are amazing. Or at least they can be
Ok, a small disclaimer.. I don't really think drive letters are stupid, and I can even understand the motivation behind them. Let's not forget the target audience here: DOS / Windows tries to cater to a larger user base, including people who don't share our expertise. So trying to keep a separation between the system (C:) and an optional data volume (D:)? I honestly think it's actually a good idea!
But... This is FreeBSD we're talking, we're amongst greybeards here. And within the Unix mindset a drive letter is plain out nonsense; a mountpoint is all we need. As such the comment above.
But... This is FreeBSD we're talking, we're amongst greybeards here. And within the Unix mindset a drive letter is plain out nonsense; a mountpoint is all we need. As such the comment above.
Last edited:
