Thin jail woes

F

fengshaun

I've been using full jails until now, but updating them has become too cumbersome. So, I decided to try thin jails as instructed here: <https://clinta.github.io/freebsd-jails-the-hard-way/>.

I have created a template with bsdinstall jail, modified it with a few packages, and made the fstab and everything like so:

Code: 
# /jails/fstab/thinjail.fstab
/jails/template /jails/thinjail nullfs ro 0 0
/jails/thinjails/mail /jails/thinjail unionfs rw,noatime 0 0

Code: 
# /etc/jail.conf
exec.start = "/bin/sh /etc/rc";
exec.stop  = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

interface = "igb0";
host.hostname = "$name.local";
path = "/jails/$name";
ip4.addr = 10.0.0.$ip;

thickjail {
  $ip = 25;
}

thinjail {
  $ip = 26;
  mount.fstab = "/jails/fstab/$name.fstab";
}

When I try to create the jail with jail -c thinjail I get this output (notice "ps: empty file: Invalid argument"):

Code: 
$ jail -c thinjail
thinjail: created
ps: empty file: Invalid argument
Setting hostname: thinjail.
Creating and/or trimming log files.
Starting syslogd.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
32-bit compatibility ldconfig path: /usr/lib32
Clearing /tmp (X related).
Updating motd:.
Starting sendmail_submit.
Starting sendmail_msp_queue.
Starting cron.
ps: empty file: Invalid argument

Also when I try to run portmaster(8), I get the following at the beginning of the output:

Code: 
ps: empty file: Invalid argument
[: -eq: unexpected operator
.
.
.
ps: /boot/kernel/kernel: No such file or directory
ps: /boot/kernel/kernel: No such file or directory
ps: /boot/kernel/kernel: No such file or directory
[: : bad number

And when I try to install Perl, I get the following error:
Code: 
===>  Configuring for perl5-5.20.3_8
***
*** I'm sorry, but /dev/null appears to be a file rather than a device.
*** Please consult your operating sytem's notes for making a device
*** in /dev.
*** Cannot continue, aborting.
***
===>  Script "Configure" failed unexpectedly.

I noticed that even though mount reports that devfs(8) is mounted under jail/dev, /dev under the jail is empty with the exception of a null file which is not even a character device! My template doesn't have a /dev and my thinjail doesn't either.

I have no idea what is going on at this point. Any help is appreciated.
 
The link to the howto is broken.

First mount_unionfs(8):
THIS FILE SYSTEM TYPE IS NOT YET FULLY SUPPORTED (READ: IT DOESN'T WORK) AND USING IT MAY, IN FACT, DESTROY DATA ON YOUR SYSTEM. USE AT YOUR OWN RISK.
Click to expand...
I had several problems with unionfs in the past and would recommend another approach to jail templates / basejails (see below)

I also got an empty /dev folder for all my jails but I can't test if it's required for sitting at a Linux laptop..


1) create the template as a normal jail with whatever methode you want
2) plan what you want to do in those jails (like having the same ports everywhere or truely independet jails) and be aware which paths you'll need to be writable to accomplish this
3) either create a single mountpoint (like described there: https://www.freebsd.org/doc/handbook/jails-application.html ) OR just mount the template to a different folder and make a nullfs mount over the template stuff (to get a fully working jail you only need /tmp, /var and /usr/local). I'd also recommend to just create soft links for config files in /etc to your jail folder instead of managing a whole /etc folder for every jail. In this way I set up servers with over a hunded jails without any trouble what so ever with /dev (and thes also where just about 1M disk space each until we deployed the actual data). And also you exactly know which files are updated when updating the template and which files you need to care on a per jail basis. I find Unionfs rather unpredictable in that respect.
(If you have a lot of jails with the same software requirements you can do awesome stuff like having a template for all those jails and not mounting /usr/local but /usr/local/etc or even just link a single configuration file to another config folder.)
 
The thinest jail I ever came up with was:
Code: 
mkdir base test test-ro
tar xzvf base.txz -C ./base
mount -t nullfs -o ro ./base ./test-ro
mount -t unionfs -o noatime -o below ./test-ro ./test
test will contain all the writes performed, like a diff from base.

As krawall states, unionfs are dangerous. They will panic your kernel, at some point, guaranteed. I would stay waaay clear if I were you.

If you don't understand, read jail(8) and if you don't understand that, as SirDice states, use a framework from ports.
 
I'm also experiencing the same issue as the original poster, but nobody has provided sensible information yet -- unionfs would appear to be fine, as it is in fact heavily used in both sysutils/ezjail and sysutils/iocage, and iocage(8) is just a shell script wrapper around jail(8).

For those keeping score, the updated URL for the original tutorial is

https://clinta.github.io/freebsd-jails-the-hard-way/

EDIT: it turns out there's some sort of issue with jail wherein it does not properly mount devfs(8) sometime after the freebsdFreeBSD 9.x series. There's a bug that's almost two years old filed to this regard:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=186360

There appear to be several attempted workarounds; I will try to update this thread if I can get the solution in http://srobb.net/nullfsjail.html working.

EDIT 2: ok, got it working.

It appears that dev is only properly mounted if it is touched (e.g., ls dev) before the jail is actually created. There may be some kind of ordering or optimization that was applied, but jail wasn't properly updated to handle it.

The workaround is to include a line in your jail.conf which runs ls on dev after the devfs(8) is mounted but before the jail is created. Note the exec.prestart line, which accomplishes this, in my jail.conf:

Code: 
interface = "lo1";
host.hostname = "$name.domain.local";
path = "/usr/local/jails/$name";
ip4.addr = 10.0.0.$ip;
mount.devfs;
mount.fstab = "/usr/local/jails/$name.fstab";
exec.prestart = "ls /usr/local/jails/$name/dev";

exec.clean;
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";

# The jail definition for zjail1
zjail1 {
    $ip = 16;
    mount.fstab = "";
}

# The jail definition for thinjail1
thinjail1 {
    $ip = 17;
}
 
Just to reinforce the point, quoting unionfs(8)
BUGS
THIS FILE SYSTEM TYPE IS NOT YET FULLY SUPPORTED (READ: IT DOESN'T WORK)
AND USING IT MAY, IN FACT, DESTROY DATA ON YOUR SYSTEM. USE AT YOUR OWN
RISK. BEWARE OF DOG. SLIPPERY WHEN WET. BATTERIES NOT INCLUDED.
Click to expand...
and going from personal experience, you will see a panic at some point.
It's a shame, because it's very useful when it works, but I can't afford the fsck times restarting a server.
 
All of the major jail tools appear to use unionfs to implement thin jails, so hopefully that disclaimer, and your experience, are not triggered in this specific use case.
 
Felix Gallo said:
All of the major jail tools appear to use unionfs to implement thin jails, so hopefully that disclaimer, and your experience, are not triggered in this specific use case.
Click to expand...

No, they are not, sysutils/ezjail using skeleton strategy by implementing hard links, sysutils/iocage using ZFS clone strategy...

mount_unionfs(8) it's just a layer for file system - if it worked well - then most of the jails have to be done by it - idea of this file system simply rocks, but don't working well at the moment. I heard that mount_unionfs(8) will be redone from the start... so we have to wait a little.. think it worth waiting.

But anyways, both ezjail(7) and iocage(8) are very nicely done and they do their job and actually they are solid solution to implement JAILS for end users without a pain and a great thanks and great respect going to their makers.
 
I believe you are mistaken about sysutils/ezjail. I see no mention of unionfs in the code, man, nor website. It does use mount_nullfs(8) extensively, which is very solid.

Edit: Looks like the unionfs was added 3 months ago to iocage for the sole purpose of recording changes to create a "jail package", not for general runtime use.
 
You must log in or register to reply here.
Back
Top