Strategy for jails

[Peacekeeper2000]

:q I am looking for a guide or recommendation what kind of services you should keep in separate jails and what kind of services it is easier to keep them combined. E.g. for a web server using also a database (and the database server is also used by other services), would you recommend to combine them or keep them in separate jails?

I know it all depends, and I am sure I am not the first to raise this (but a short search has not given me good sources back) therefore maybe there is a guide out here that you could recommend? Of course I read the handbook already, but there is not really a deep dive about the strategy how to split your services).

I want to use the server like a home brewed NAS for music, video, picture sharing, web server, FTP server, file shares (AFP/SMB), Git repository, so generally a store for everything at home and hobby development. I have one big pool zRAID with 4 drives as a foundation for all file storage and currently the first services are running without jails. Maybe to put everything in only one jail could be also a strategy?

Any feedback is appreciated - and sorry when I overlooked an "everybody know this guide " type of document - newbee :r

Thanks and cheers.

 

[SirDice]

There's no real answer for this, so there won't be a guide. It will be perfectly fine if everything runs on the host. Using jails just adds another layer of complexity. And it's mostly up to you if the added security of running separate jails outweighs the added work you're going to have to do to keep it running and up to date.

 

[wblock@]

What is the goal of putting services in jails? Security? Some people put services in jails just to isolate them and keep them from interfering with other services.

@SirDice is right, it adds complexity. Each jail is another system to be kept updated.

FTP has been trying to crawl off and quietly die for over a decade. Please let it go. Consider sftp(1) for a less-terrible replacement.

 

[max21]

Reading this thread, it sounds like jails have been down-graded. Not once word of encouragement other than what the OP already know.... or do I need to provide quotes with any standard reader comments who are not afraid to be hated.

Has jail been hacked by the new hackers of today, or is it dying beginning since 10.x or 9x because of complications with new features added () and to be added in the future that will makes the entire system, including jails, less secure?

The OP said “picture sharing, web server, FTP server, ” .... There was no need to say sftp or Apache web server. Even PcBSD say connecting with the host system is as weak as Linux and Windows when it comes to security. The whole world of FreeBSD experts say “don’t do it”. You guys know the new truth. Please share more of the facts before it’s too late. It’s already too late for me maybe. I use 10.0 and don’t want to turn back. Thank you.

No disrespect but if you was a searcher or a common member what would these replies it sound like to you? (a snow-job)

 

[Peacekeeper2000]

There's no real answer for this, so there won't be a guide. It will be perfectly fine if everything runs on the host. Using jails just adds another layer of complexity. And it's mostly up to you if the added security of running separate jails outweighs the added work you're going to have to do to keep it running and up to date.

Thanks for your post. I thought - beside the complexity - I also get a better security in case one of the service has a leakage. I am aware that it will be an additional effort and that is why I am struggling at the moment to understand how can I balance the maintenance effort vs the improved security. I also though that a tool like ezJail would make the jail-babysitting a bit easier ?.

Are there some Standard packages that you normally put into a jail ? E.g. mail as single jail, while all file server services (NFS,AFP,SMB etc) could be grouped together in one jail ?

BTW: What is the effort after jails are in place ? A portsnap fetch update && postmaster -a for each jail ? The freebsd-update would anyhow only run on the host, nor ?

 

[Peacekeeper2000]

What is the goal of putting services in jails? Security? Some people put services in jails just to isolate them and keep them from interfering with other services.

@SirDice is right, it adds complexity. Each jail is another system to be kept updated.

FTP has been trying to crawl off and quietly die for over a decade. Please let it go. Consider sftp(1) for a less-terrible replacement.

Thanks for your post. Your are right , I missed to provide the underlaying reason. I assumed everybody is using jails only for security reasons. And I might also use jail to simulate another AFP Server ( good idea) - so hat could also be a jail strategy :beer

In preparation of the jail effort , I already found out it is an effort to align all network ports to the right interfaces. Alone this exercise was helpful to prevent the server to listen to everywhere and every port.

As ftpd I am using ftp/pure-ftpd which works also nice with my standard shell shells/zsh and of course the sftp .

 

[max21]

BTW: What is the effort after jails are in place ? A portsnap fetch update && postmaster -a for each jail ? The freebsd-update would anyhow only run on the host, nor ?

I hope I got that right too. I once thought that the jails has it own service such as apache, but I recently read that jails use the hosting system applications by way of nullfs. I also gather that you can use one of your jails to compile ports if you wish (most people thinks that’s all it good for it seems), but it application would still be written to the host system when all is said and done. I’m a newbee too when it comes to getting serious about the advantages of using FreeBSD as a production server as well as a desktop (where we might have to use the full-hosting work-station for that ... but I don’t know for sure yet) to connect to that remote server, without fear, and I do understand, or only understand at lease 90% of what I read, finally. Up until this year it was only a thrill just to see FreeBSD work and run stuff as a desktop so I really can 't be too much help. I'll be searching and asking the same within a few weeks, again. Good Luck.

 

[scottro]

The nullfs thing is one method of using jails. I believe both ezjail and qjail use that, but it's not a requirement--that is, you can run jails without making use of nullfs.

Jails can be handy to run a few different things on one server. While you could put, for example, your web server and mail server on one machine, by using jails, with or without nullfs to make two separate machines, it can make things a bit simpler and a bit less likely for a change in one to break the other.

It can add a layer of security as well. There are various use cases. At work, we frequently use it to divide jobs between different jails, such as the one I just mentioned, a single jail host running one web server and one mail server.

 

[wblock@]

Reading this thread, it sounds like jails have been down-graded. Not once word of encouragement other than what the OP already know.... or do I need to provide quotes with any standard reader comments who are not afraid to be hated.
...
No disrespect but if you was a searcher or a common member what would these replies it sound like to you? (a snow-job)

Jails are a tool. They may or may not be the right tool for the job, but to answer that question correctly requires knowing details. I use a few jails, and I'm pretty sure @SirDice uses them a lot more. We both know they have features and drawbacks. So we don't just say "yes, jails are the solution to all problems!"

We also comment on other problems when we see them.

 

[SirDice]

I use a few jails, and I'm pretty sure @SirDice uses them a lot more. We both know they have features and drawbacks. So we don't just say "yes, jails are the solution to all problems!"

Yep, I use them quite a lot. I don't really use them for the added security (although that is a bonus) but to separate different configurations. I can have one jail with a certain Apache configuration and another with a completely different (and incompatible with the first one) configuration. My jail usage is more like using virtual machines to test things without having to touch the host system.

I also make heavy use of nullfs(5) but it's not a requirement for jails, it just makes it a lot easier to 'share' data between jails. I definitely recommend using nullfs(5) to share the host's /usr/ports. That way you only have one ports tree to update. I've also used it to share /usr/home across the jails.

I'd recommend ZFS in combination with jails too. It's quite easy to create a basic jail, snapshot it and copy it multiple times. Really quick way of getting multiple jails up and running. Tools like ezjail and qjail also make use of this functionality.

 

[Peacekeeper2000]

The nullfs thing is one method of using jails. I believe both ezjail and qjail use that, but it's not a requirement--that is, you can run jails without making use of nullfs.

Ahh, the nullfs(5) I need to have a look at it as I want to share some zfs() shares across multiple jails. And also a good tip to get the /usr/ports also shared across the jails.

My biggest concern is at the moment to go public with a web server and so open the server also for my internal used services like SSH or other attacks. I also don't want to share music,video and pictures outside of my home network. To balance the effort and to still have an effective security layer, I wonder if a jail just for the web server is good enough or if I also need jail(s) for every other service to be secure.

Also can I put a firewall in a jail together with the web server or would you recommend a firewall on host level? (I have a firewall currently only on the router, but worked very well until today).

 

[Peacekeeper2000]

Yep, I use them quite a lot. I don't really use them for the added security (although that is a bonus) but to separate different configurations. I can have one jail with a certain Apache configuration and another with a completely different (and incompatible with the first one) configuration. My jail usage is more like using virtual machines to test things without having to touch the host system.

Yeah, I also though about something like a lab-jail where I can install and try out new software without impacting the host - ok sound like jails would be fine for that. Thanks!

I'd recommend ZFS in combination with jails too. It's quite easy to create a basic jail, snapshot it and copy it multiple times. Really quick way of getting multiple jails up and running. Tools like ezjail and qjail also make use of this functionality.

Are you using a separate zpool() or just zfs() all on the same pool? As mentioned I have a zRAID pool with 12 TB also for data protection. So I would not be able to build a separate pool for each jail.

One question to backups: do you do that for each pool or how do you secure/backup the different jail content?

 

[wblock@]

I'm interested in learning what you found to be a drawback using jails.

Jails are pretty much like using multiple computers. Each is another copy of settings and applications to keep updated. Beyond that, hardware access and networking in a jail are not quite the same as on a full system.

That said, nullfs(5) can reduce the number of things to update. sysutils/ezjail lets multiple jails share the same userland code and ports tree. Multiple jail filesystems can all be seen from a single host, which is easier than having multiple physical machines.

 

[SirDice]

Are you using a separate zpool() or just zfs() all on the same pool? As mentioned I have a zRAID pool with 12 TB also for data protection. So I would not be able to build a separate pool for each jail.

I do have two pools, one I use for my ports tree, source tree and a couple of jails, it's basically a scratch pool. The other has all my movies, pictures and whatever. I like to keep them separate but it's not required.

Short process I use:

zfs create -o mountpoint=/jails/clean_jail fbsd0/jails/clean_jail
# create jail as in jail(8) and use DESTDIR=/jails/clean_jail
# set root password:
chroot /jails/clean_jail /bin/passwd
# make snapshot:
zfs snapshot fbsd0/jails/clean_jail@clean
# create a 'new' jail
zfs clone fbsd0/jails/clean_jail@clean fbsd0/jails/myjail
# change a few settings in /jails/myjail/etc for example
# add jail to jail.conf

Because you can zfs clone the snapshot multiple times (and works almost instantly) you can quickly create multiple jails. You just have to change a few settings and you're off.

 

[kpa]

The downside of using ZFS snapshots and clones is that the clones (that are read/write copy-on-write copies of the original read-only snapshot) will have a life of their own independent of the original file system the snapshot was made from. To update all the jails based on the same snapshot you'll have to re-create them from scratch. That's not however much work since you can again take a snapshot of the master jail and do the cloning as you did before.

 

[AzaShog]

With regards to using jails for security, please correct me if I'm wrong, still learning this, but the way I see it, the security benefit of jails is is not that much greater than running unprivileged on the host, except to "virtualize the root". You can run processes requiring root privilege inside jails and those roots would be limited only to the jail's process space and filesystem root mount, etc... It's "just" awesome operating-system-level virtualization.

But, jails are running off of the same kernel, and their security is only as tight as the kernel's. In other words, a jail process can exploit the kernel (if a vulnerability exists, of course) and gain access to the privileged host or other jails' processes and filesystem. It's a convoluted set of circumstances, but it happens all the time at least in the Linux land - insecure (web) apps allowing random binaries that exploit the kernel. I know people running unpatched kernels because vulnerabilities found so far were "local only, required a local, unprivileged user", and yet they at the same time run (PHP) apps with bad track record in allowing upload of random code - gateways for remote attackers to easily become "unprivileged, local users".

 

[Peacekeeper2000]

I do have two pools, one I use for my ports tree, source tree and a couple of jails, it's basically a scratch pool. The other has all my movies, pictures and whatever. I like to keep them separate but it's not required.

Short process I use:

... Thanks ! that is an easy start :beer

Because you can zfs clone the snapshot multiple times (and works almost instantly) you can quickly create multiple jails. You just have to change a few settings and you're off.

Yeah, a pretty cool way to build a lot of jails in a sec. I will remember that. For deployment of different services , do you also have a special trick or do you just do the normal build of the service in each jail ?
BTW: Any suggestions on the backup topic ?

 

[Peacekeeper2000]

The downside of using ZFS snapshots and clones is that the clones (that are read/write copy-on-write copies of the original read-only snapshot) will have a life of their own independent of the original file system the snapshot was made from. To update all the jails based on the same snapshot you'll have to re-create them from scratch. That's not however much work since you can again take a snapshot of the master jail and do the cloning as you did before.

You are right, after the first creation , you have to maintain each of the jails separate. That is exactly why I think about a strategy to minimize the no of jails , but also keep the security higher than a standard host.

 

[Peacekeeper2000]

But, jails are running off of the same kernel, and their security is only as tight as the kernel's. In other words, a jail process can exploit the kernel (if a vulnerability exists, of course) and gain access to the privileged host or other jails' processes and filesystem. It's a convoluted set of circumstances, but it happens all the time at least in the Linux land - insecure (web) apps allowing random binaries that exploit the kernel. I know people running unpatched kernels because vulnerabilities found so far were "local only, required a local, unprivileged user", and yet they at the same time run (PHP) apps with bad track record in allowing upload of random code - gateways for remote attackers to easily become "unprivileged, local users".

I agree, that jails are a compromise. My CPU is not a very strong one, otherwise maybe a full virtualization would be possible. Also my Webserver will be a simple one. BTW: this is one of my reasons why I build a system from scratch ( and not buying a NAS or installing FreeNAS) . I want to understand what sw package is running on my server and why.
I think Security is an ongoing activity you need to do

 

[wblock@]

A jail’s reduced /dev is considered as a feature, not as a drawback.

Actually, the contents of /dev in a jail can be controlled with /etc/devfs.rules. But some hardware or kernel access is simply not allowed. For example, changing the host clock is not allowed. ntpd(), a network service that would otherwise be a prime candidate for a jail, can't update the host clock from there.

 

[Peacekeeper2000]

A jail’s reduced /dev is considered as a feature, not as a drawback.

Actually, the contents of /dev in a jail can be controlled with /etc/devfs.rules. But some hardware or kernel access is simply not allowed. For example, changing the host clock is not allowed. ntpd(), a network service that would otherwise be a prime candidate for a jail, can't update the host clock from there.

:q Oh, that is interesting ! I have moved to net/openntpd as I thought it would be able to update the system time ( like ntpd(), but just bound to a single IP Adress ). So do I talk about the same clock ?
( Of course I am not expecting ntpd() to update the BIOS clock )

 

[Peacekeeper2000]

... A bit off topic ...

... As it has been pointed here already to the shared kernel, the quality of the kernel is crucial. Luckily the FreeBSD kernel has still a good reputation, ...

Yeah, even when you go into a full virtual machine ( where you probably have the same type of kernel in each VM ) the kernel is a security possibility. I compare that with a strong castle : The kernel would be the Donjon/keep while you still have strong walls, yards , gatekeeper etc around. I have had MS HomeServer, Debian and now switched to FreeBSD 10.0p7 (since Version 8 ) as it is simpler, very good structured, I can hack the ports if needed and very reliable - the system is just running without reboot every time - just awesome ! (reminds me on my good old HP-UX days on Apollo )

PS: Security is considered as a drawback in a wellness context.

I agree it is - on the other side: Easy and comfortable usage always goes along with more complex systems. At the end the more complex a system is, the more difficult to secure. And today there are still castles completely available as they have been total uninteresting to "hack". :stud

 

[jb_fvwm2]

New port sysutils/rcadm which has its manpage downloadable-only still AFAIK, may be of interest to those running many jails. Or not. Just toyed with it a few minutes so far.

 

[Peacekeeper2000]

Looks like we got some fallout from the drawback branch.
Some more time related jail hints:
...
and run tzsetup in the jail.

Thanks for that hint !

Coming back to the strategy topic, I learned already lot from all posters. I realy have to re-think what and why I am "jailing" . At the moment I think I will

• put my UPS and ntpd and others more host related services in one jail ( not sure as UPS sysutils/nut needs to connect to USB, and Ntpd to the clock - so maybe they stay on the host level ) ,
• I will simulate different files sharing servers with separate simple jails and
• one jail for Webservices ( include also the database and the web server activities ) and
• another jail for my development activities.

Also jails could be quickly used to build "test servers" for a weekend hack and for learning more about jails

 

[Peacekeeper2000]

New port sysutils/rcadm which has its manpage downloadable-only still AFAIK, may be of interest to those running many jails. Or not. Just toyed with it a few minutes so far.

Thanks for the hint. My problem with such a pkg: it is exactly one tool that I would like to keep out of my pkg collection. To be honest: the content of the shell script that this port would install is a perfect excercise for shells/zsh or shells/bash-completion. From a security point of view I would like to do a bit of a lazy check of new versions. So when postmaster -a would update such a pkg, it could simple re-scope the purpose of the shell script and could inject whatever trojan horse. When I balance the risk against the benefit of such a pkg, I would mark it as to high risk compared to the benefit it provides ( as I cold solve the problem with something I already have in place)