jails with vnet in rc.conf

kisscool-fr · Jun 22, 2011

Hi all,

I'm trying to make work a vnet jail in a FreeBSD 8.2 host on vmware. I searched here and Googled a lot. A lot of references I found referred to those pages:
http://wiki.polymorf.fr/index.php/Howto:FreeBSD_jail_vnet
http://lifanov.com/doc/vimage.html

Following the first one, I compiled the kernel with VIMAGE and without SCTP and patched /etc/rc.d/jail.

Code:

# uname -a
FreeBSD jsrv.host.lan 8.2-RELEASE-p2 FreeBSD 8.2-RELEASE-p2 #2: Tue Jun 14 16:39:14 CEST 2011     root@jsrv.host.lan:/usr/obj/usr/src/sys/JKERN  i386

Code:

# egrep -v '^#|^$' /usr/src/sys/i386/conf/JKERN
include GENERIC
ident   JKERN
nooptions       SCTP
options         VIMAGE

Im the host rc.conf file, I have for jails:

Code:

jail_enable="YES"
jail_v2_enable="YES"
jail_list=""
jail_set_hostname_allow="NO"

In the jail configuration file /etc/jails/jsmb1.conf:

Code:

jail_list="$jail_list jsmb1"
jail_jsmb1_name="jsmb1"
jail_jsmb1_hostname="jsmb1.host.lan"
jail_jsmb1_devfs_enable="YES"
jail_jsmb1_rootdir="/jails2/dir/jsmb1"
#jail_jsmb1_devfs_ruleset="devfsrules_jail"
#jail_jsmb1_exec_start="/bin/sh /etc/rc"
jail_jsmb1_vnet_enable="YES"

jail_jsmb1_exec_prestart0="ifconfig epair0 create"
jail_jsmb1_exec_prestart1="ifconfig bridge0 addm epair0a"
jail_jsmb1_exec_prestart2="ifconfig epair0a up"
jail_jsmb1_earlypoststart0="ifconfig epair0b vnet jsmb1"
#jail_jsmb1_poststart0="ifconfig epair0b vnet jsmb1"
#jail_jsmb1_afterstart0="ifconfig lo0 127.0.0.1"
#jail_jsmb1_afterstart1="ifconfig epair0b 192.168.3.89 netmask 255.255.255.0"
#jail_jsmb1_afterstart2="ifconfig add default 192.168.3.251"
#jail_jsmb1_afterstart3="/bin/sh /etc/rc"
jail_jsmb1_exec_poststop0="ifconfig bridge0 deletem epair0a"
jail_jsmb1_exec_poststop1="ifconfig epair0a destroy"

The virtual network stack is active in jail but I have a problem with the network configuration. The prestart and poststop commands work correctly. The earlypoststart and afterstart commands don't. If I execute the commands manually, they work.

I don't know if something changed between the moment the article was created and the 8-2 release(-p2).

Did anybody already have this problem? And how did you solve it (if you solved it)? Any advice is welcome.

Thanks.

usdmatt · Jun 22, 2011

I am also running 8.2-RELEASE-p2 and looking in the jail rc script can not see any reference to earlypoststart so I'm not sure this is a valid rc variable name.

I have the following (in this order):

Code:

jail_name_exec_prestartX
jail_name_exec_afterstartX
jail_name_exec_poststartX

jail_name_exec_prestopX
jail_name_exec_poststopX

Also note that they all contain _exec_ whereas you don't appear to have this in your afterstart and poststart entries.

Also curious is that X appears to start at 1 for the _exec_afterstart variables. Maybe I'm just reading the script wrong but the loop variable appears to be initialised to 1 when it's set to 0 for all the rest. Is anyone else able to explain why this was done or whether it's a bug in the jail rc script?

kisscool-fr · Jun 22, 2011

usdmatt said:
I am also running 8.2-RELEASE-p2 and looking in the jail rc script can not see any reference to earlypoststart so I'm not sure this is a valid rc variable name.

earlypoststart exists once /etc/rc.d/jail is patched

Code:

# grep -i -C 4 early /etc/rc.d/jail
        done

        i=0
        while : ; do
                eval _exec_earlypoststart${i}=\"\${jail_${_j}_exec_earlypoststart${i}:-\${jail_exec_earlypoststart${i}}}\"
                [ -z "$(eval echo \"\$_exec_earlypoststart${i}\")" ] && break
                i=$((i + 1))
        done

        i=0
--
                        fi

                        i=0
                        while : ; do
                                eval out=\"\${_exec_earlypoststart${i}:-''}\"
                                [ -z "$out" ] && break
                                ${out}
                                i=$((i + 1))
                        done

usdmatt said:
Also note that they all contain _exec_ whereas you don't appear to have this in your afterstart and poststart entries.

That's it, shame on me :r
I didn't seen something big like this, I must be too tired.

usdmatt said:
Also curious is that X appears to start at 1 for the _exec_afterstart variables. Maybe I'm just reading the script wrong but the loop variable appears to be initialised to 1 when it's set to 0 for all the rest. Is anyone else able to explain why this was done or whether it's a bug in the jail rc script?

Don't know what you are talking about but there is another mistake (ifconfig whereas it should be route) for afterstart2.

The main problem was the _exec_ part.

Thanks

kisscool-fr · Jun 23, 2011

Following the wiki guide to create a vnet jail in rc.conf, there still isn't possible to use netstat command in jail.

Code:

# jexec jsmb1 netstat -rn
netstat: kvm not available: /dev/mem: No such file or directory
Routing tables
rt_tables: symbol not in namelist

To make it work, it is necessary to add this to /etc/devfs.rules file on the host system.

Code:

# cat /etc/devfs.rules
[devfsrules_unhide_mem=5]
add path mem unhide
add path kmem unhide

[devfsrules_unhide_bpf=6]
add path bpf unhide

# Devices usually found in a jail.
#
[devfsrules_jail2=7]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add include $devfsrules_unhide_mem
add include $devfsrules_unhide_bpf

and to modify the jail configuration file by replacing the line

Code:

jail_jsmb1_devfs_ruleset="devfsrules_jail"

by

Code:

jail_jsmb1_devfs_ruleset="devfsrules_jail2"

After restarting the jail, netstat works well.

Code:

# jexec jsmb1 netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.3.251      UGS         1       25 epair0
127.0.0.1          link#1             UH          0        0    lo0
192.168.3.0/24     link#2             U           0        0 epair0
192.168.3.89       link#2             UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::1                               ::1                           UH          lo0
fe80::%lo0/64                     link#1                        U           lo0
fe80::1%lo0                       link#1                        UHS         lo0
fe80::%epair0b/64                 link#2                        U       epair0b
fe80::c0:44ff:fe00:60b%epair0b    link#2                        UHS         lo0
ff01:1::/32                       ::1                           U           lo0
ff01:2::/32                       fe80::c0:44ff:fe00:60b%epair0b U       epair0b
ff02::%lo0/32                     ::1                           U           lo0
ff02::%epair0b/32                 fe80::c0:44ff:fe00:60b%epair0b U       epair0b

I added bpf too for tcpdump testing purposes but it isn't necessary for netstat.

There still exist a little annoying thing when jail starts. It is blocking for about 1 minute at motd step.

Code:

Clearing /tmp (X related).
Updating motd:.

If anyone has an explication and a solution, you're welcome

usdmatt · Jun 23, 2011

Code:

update_motd="NO"

in the jails /etc/rc.conf?

It appears to just be updating the FreeBSD version info displayed at the top of the MOTD banner. You probably don't really need it, although it's possible that something running after this is causing the hang...

In fact, sorry, you can probably ignore this post - the '.' after 'Updating motd:' seems to suggest it's already completed this step so it must be something else and I don't know enough about jails to really know what it could be.

kisscool-fr · Jun 23, 2011

Thanks for advice.

After investigating, I found that I had a nameserver in the jailed resolv.conf which was not reachable at this moment.

With no name servers or with a valid name server, the start process is just normal

kisscool-fr · Aug 12, 2011

Hi,

I'm back with an another little question about vnet jails.

When working with standard jails (or v1 jails), we can see with the jls command, the adresse assigned to the jail.

For example (this is from the handbook):

Code:

# jls
   JID  IP Address      Hostname                      Path
     3  [B]192.168.3.17[/B]    ns.example.org                /home/j/ns
     2  [B]192.168.3.18[/B]    mail.example.org              /home/j/mail
     1  [B]62.123.43.14[/B]    www.example.org               /home/j/www

And when working with vnet jails (or v2 jails), this information is missing.

Code:

# jls
   JID  IP Address      Hostname                      Path
     2  [B]-[/B]               ldap1.hostname.lan               /jails/ldap1
     3  [B]-[/B]               ldap2.hostname.lan               /jails/ldap2
     4  [B]-[/B]               jail1.hostname.lan               /jails/jail1

Does anyone know if it is possible to view with jls the adress assigned to vnet jails ?
I tried looking in the patch to apply to have v2 jails, but I can't find anything that could help me with this.

Thanks for any help you could provide me.

jails with vnet in rc.conf

kisscool-fr

usdmatt

kisscool-fr

kisscool-fr

usdmatt

kisscool-fr

kisscool-fr