Jail stopped working

bsduni · Nov 11, 2013

Hi!

I have a FreeBSD 9.1 server with a jail named 'www' to host web pages. Everything was working fine, and all of a sudden the server hung without any known reason, i.e. while I have not accessed the machine for a few days (and, I'm the only admin) and no changes have been made.

Now, after I rebooted the server, the jail is not coming up. When I issue service jail start the output is:

Code:

Configuring jails:.
Starting jails: cannot start jail "www":
jail: getpwnam root: No such file or directory
jail: /bin/sh /etc/rc: failed
.

The host machine's /etc/rc.conf is as follows:

Code:

# Set-up jails
jail_enable="YES"
jail_list="www"       # Names of the jails separated by space
jail_interface="bge0"
jail_set_hostname_allow="NO"
jail_socket_unixiproute_only="YES"
jail_sysvipc_allow="NO"
jail_exec_start="/bin/sh /etc/rc"
jail_exec_stop="/bin/sh /etc/rc.shutdown"
jail_devfs_enable="YES"
jail_fdescfs_enable="NO"
jail_procfs_enable="YES"
jail_mount_enable="NO"            
jail_fstab=""
jail_flags="-l -U root"

# Create aliases
ifconfig_bge0_alias0="inet 192.168.1.1 netmask 255.255.255.0"   # www jail

# Settings of jail1
jail_www_rootdir="/jail/www"                 # Jail's root directory
jail_www_hostname="www.example.com"  # Jail's hostname
jail_www_ip="192.168.1.1"                           # Jail's IP address
...

And, with this configuration the jail was working for more than a month without any issue.

One thing I noted is that the /etc folder within the jail is no longer accessible. That is, it is shown with ls -al, but, can not access the directory with cd.

Code:

# ls -al /jail/www/
...
lrwxr-xr-x   1 root  wheel    5 Oct  2 15:33 etc -> s/etc
...
# cd /jail/www/etc/
/jail/www/etc/: No such file or directory.

Any clues/hints?

Thank you in advance.

ShelLuser · Nov 12, 2013

What do you have in your /etc/jail.conf?

bsduni · Nov 12, 2013

There is no /etc/jail.conf in my system (and it was not there either, when it was functioning).

Also tried solution given in a previous thread:

Code:

# /usr/sbin/pwd_mkdb -d /usr/jail/www/etc -p /usr/jail/www/etc/master.passwd
pwd_mkdb: /usr/jail/www/etc/master.passwd: No such file or directory

The /etc/fstab:

Code:

# Jails
/jail/mroot       /jail/www        nullfs      ro     0      0
/jailspace/www    /jail/www/s      nullfs      rw     0      0
/home             /jail/www/s      nullfs      ro     0      0

and all the mountings are shown with df as expected.

Though the /jailspace/www is mounted on /jail/www/s as 'rw', non of the subdirectories of /jailspace/www are listed with ls /jail/www/s and the /jail/www/s remains READ-ONLY (despite 'rw' mount).

The reason I am trying to modify /jail/www/s is that within the 'www' jail, etc is 'ln'ed to s/etc.

Hope this additional info is also helpful for some expert to understand the exact scenario.

nk · Nov 12, 2013

bsduni said:
The /etc/fstab:

Code:

# Jails /jail/mroot /jail/www nullfs ro 0 0 /jailspace/www /jail/www/s nullfs rw 0 0 /home /jail/www/s nullfs ro 0 0

and all the mountings are shown with df as expected.

Though the /jailspace/www is mounted on /jail/www/s as 'rw', non of the subdirectories of /jailspace/www are listed with ls /jail/www/s and the /jail/www/s remains READ-ONLY (despite 'rw' mount).

The reason I am trying to modify /jail/www/s is that within the 'www' jail, etc is 'ln'ed to s/etc.

jail_www_rootdir in rc.conf differs from your fstab entries. The problem is there's no /etc in jail_www_rootdir.

kisscool-fr · Nov 12, 2013

To be sure, can you show us the output of mount? From what you showed, you are mounting /home over /jailspace/www in /jails/www/s. Unmount it once and try to start the jail.

# umount /jail/www/s
# service jail start www

ShelLuser · Nov 12, 2013

Although this setup works I'd still advice to utilize jail.conf as well. Because that gives you direct access to the "jail controlling programs" such as jexec and jail. The advantage should be obvious: you can control a jail without the extra "rc.d layer" on top.

Solely based on the output I see when booting my FreeBSD server I'm convinced that using the jail command to start (or stop) your jail can provide much more detailed information as to what is going wrong than the rc.d script.

bsduni · Nov 13, 2013

@nk:
It was a typing mistake. Both have got /jail/www.

@kisscool-fr:
Thanks a lot. Your tip helped to start the jail again. I looks like addition of a user within the jail (with the useradd command) triggered the issue.
Because the jail was there with the same configuration files during several reboots in the past, and was working fine. The only change I did lately before the problematic reboot was an addition of a new user (possibly contradicting user name with the one in the host list).
Now I can troubleshoot it.

@ShelLuser:
Thank you for that tip. I am a newbie on FreeBSD with some Linux background, and have got a lot to learn on FreeBSD stuffs.
I am definitely going to try the jail.conf suggested.

Thank you every one.

kisscool-fr · Nov 13, 2013

bsduni said:
@kisscool-fr:
Thanks a lot. Your tip helped to start the jail again. I looks like addition of a user within the jail (with the useradd command) triggered the issue.
Because the jail was there with the same configuration files during several reboots in the past, and was working fine. The only change I did lately before the problematic reboot was an addition of a new user (possibly contradicting user name with the one in the host list).
Now I can troubleshoot it.

I don't understand how it could work before, it is not logical.

Can you tell what is the purpose of mounting /home to /jail/www/s? This mountpoint, I mean /jail/www/s, is the place where your changes to your jail are kept. So having it mounted in read write mode here is ok.

But mounting your /home directory and in read mode only over /jail/www/s has no sense.