Solved [SOLVED] FreeBSD 10 - ZFS - Jail- jail.conf - Don't work.

Hi,

Please, look at my steps and tell me what am i doing wrong. I have no strength and I do not understand why it is not working.

fstab
Code:
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/ada0s1a            /               ufs             rw      1       1
/dev/ada0s1b            none            swap            sw      0       0
/dev/ada0s1d            /root           ufs             rw      2       2
/dev/ada0s1e            /var            ufs             rw      2       2
/dev/ada0s1f            /var/tmp                ufs             rw      2       2
/dev/ada0s1g            /usr            ufs             rw      2       2
/dev/ada0s1h            /usr/home               ufs             rw      2       2
#/dev/ada0s2a           /jails          ufs             rw      2       2
proc                    /proc           procfs  rw              0       0
Using zpool and zfs:
Code:
root@ks3360102:/ # umount /jails
root@ks3360102:/ # zpool create jail ada0s2a
root@ks3360102:/ # zfs create -o mountpoint=/jail -o compress=lz4 jail/jail
root@ks3360102:/ # zfs create jail/Oksymoron
root@ks3360102:/ # zfs create jail/Oksymoron/usr
root@ks3360102:/ # zfs create jail/Oksymoron/usr/home
root@ks3360102:/ # zfs create jail/Oksymoron/var
root@ks3360102:/ # zfs create jail/Oksymoron/tmp
root@ks3360102:/usr/src # make installworld DESTDIR=/jail/Oksymoron
root@ks3360102:/usr/src # make distribution DESTDIR=/jail/Oksymoron
root@ks3360102:/usr/src # cat /etc/jail.conf
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";

Oksymoron {
        path = /jail/Oksymoron;
        mount.devfs;
        allow.mount;
        mount;
        host.hostname = Oksymoron.edu.pl;
        ip4.addr = 192.168.0.2;
        interface = lo1;

};

root@ks3360102:/usr/src # jail -c Oksymoron
Oksymoron: created
/etc/rc: WARNING: $hostname is not set -- see rc.conf(5).
Generating host.conf.
Creating and/or trimming log files.
ln: /dev/log: Operation not permitted
Starting syslogd.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
Clearing /tmp (X related).
Updating motd:.
Starting cron.
Jail "works":
Code:
root@ks3360102:/usr/src # jls
   JID  IP Address      Hostname                      Path
     1  192.168.0.2     Oksymoron.edu.pl              /jail/Oksymoron

Reboot and:
Code:
root@ks3360102:~ # ls /jail/
root@ks3360102:~ #
My jail disappeared. Why ?
Check zfs and zpool, look good.
Code:
root@ks3360102:~ # zfs list
NAME                      USED  AVAIL  REFER  MOUNTPOINT
jail                      244M   356G   144K  /jail
jail/Oksymoron            243M   356G  17.3M  /jail/Oksymoron
jail/Oksymoron/tmp        176K   356G   176K  /jail/Oksymoron/tmp
jail/Oksymoron/usr        225M   356G   224M  /jail/Oksymoron/usr
jail/Oksymoron/usr/home   144K   356G   144K  /jail/Oksymoron/usr/home
jail/Oksymoron/var        564K   356G   564K  /jail/Oksymoron/var
jail/jail                 152K   356G   152K  /jail
root@ks3360102:~ # zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
jail   362G   244M   362G     0%  1.00x  ONLINE  -
root@ks3360102:~ #
root@ks3360102:~ #

Please help me, why isn't working ?
 
Re: FreeBSD 10 - ZFS - Jail - Don't work.

I'm guessing you've already put the following in /etc/rc.conf on the host. I've not used jails for a while, and never with the jail.conf style, but I'll be surprised if they run by default without being enabled in rc.conf.

Code:
jail_enable="YES"

Also, you have made 2 ZFS filesystems with the same mountpoint. When creating the pool jail, a 'root' dataset called jail is created, and mounted on /jail. You can see this at the top of your zfs list output. You've then created a jail/jail dataset, and mounted it in the same place. It would of made more sense to just use the root dataset:

Code:
# zpool create -O compress=lz4 jail device (will mount root dataset on /jail with compression on)
# zfs create jail/Oksymoron

To add to this, you've enabled compression on the jail/jail dataset, then made your first jail under jail/Oksymoron, so I suspect if you view the compression ratio you'll find it isn't compressed. (Unless you enabled compression on the root separately and haven't shown the command in your post)

Code:
# zfs get ratio jail/Oksymoron
 
Re: FreeBSD 10 - ZFS - Jail - Don't work.

I've added in rc.conf.
Code:
# Jail
jail_enable="YES"
Ok i removed zpool and zfs:
Code:
root@ks3360102:~ # zfs destroy -r jail
root@ks3360102:~ # zpool destroy jail
root@ks3360102:~ # zpool list
no pools available
root@ks3360102:~ # zfs list
no datasets available
root@ks3360102:~ #
Than:
Code:
root@ks3360102:~ # zpool create -O compress=lz4 jail ada0s2a
root@ks3360102:~ # zpool list
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
jail   362G   732K   362G     0%  1.00x  ONLINE  -
root@ks3360102:~ # zfs create jail/Oksymoron
root@ks3360102:~ # zfs create jail/Oksymoron/usr
root@ks3360102:~ # zfs create jail/Oksymoron/usr/home
root@ks3360102:~ # zfs create jail/Oksymoron/var
root@ks3360102:~ # zfs create jail/Oksymoron/tmp
Installing jail:
Code:
root@ks3360102:/usr/src # make installworld DESTDIR=/jail/Oksymoron
root@ks3360102:/usr/src # make distribution DESTDIR=/jail/Oksymoron

And reboot:
Code:
root@ks3360102:~ # jls
   JID  IP Address      Hostname                      Path
root@ks3360102:~ # ls /jail/
root@ks3360102:~ # zfs list
NAME                      USED  AVAIL  REFER  MOUNTPOINT
jail                      121M   356G   152K  /jail
jail/Oksymoron            120M   356G  10.0M  /jail/Oksymoron
jail/Oksymoron/tmp        144K   356G   144K  /jail/Oksymoron/tmp
jail/Oksymoron/usr        109M   356G   109M  /jail/Oksymoron/usr
jail/Oksymoron/usr/home   144K   356G   144K  /jail/Oksymoron/usr/home
jail/Oksymoron/var        492K   356G   492K  /jail/Oksymoron/var
root@ks3360102:~ # ls /jail/
root@ks3360102:~ #
And everything in /jail has disappeared. Is this some magic ? Becouse i had never any problem with earlier versions. So what's wrong ? Why this all the time disappears ?

#edit
Ratio:
Code:
root@ks3360102:/jail # zfs get ratio jail/Oksymoron
NAME            PROPERTY       VALUE  SOURCE
jail/Oksymoron  compressratio  2.15x  -
 
Re: FreeBSD 10 - ZFS - Jail - Don't work.

Do you also have the following in /etc/rc.conf to make your system mount ZFS on boot?

Code:
zfs_enable="YES"
 
Re: FreeBSD 10 - ZFS - Jail - Don't work.

Simple, you have created an overlap. If you look carefully at the output of the zfs list you'll notice the following:

Code:
jail                      244M   356G   144K  /jail
jail/jail                 152K   356G   152K  /jail
You need to keep in mind that if you create a ZFS pool then you're actually doing 2 things: you're creating the pool itself, but also the main filesystem for that pool.

So when you created the pool 'jail' you actually created a filesystem which got automatically mounted on /jail. After that you created the filesystem jail/jail but you set the mountpoint to /jail as well. And as such created the overlap I mentioned earlier.

My suggestion would be to skip the creation of the new jail/jail filesystem but instead use the 'main' filesystem of the ZFS pool.

(Edit: I noticed that my message overlapped with another one, but since mine also added some background on the reason this happened I figured I'd leave it).
 
Re: FreeBSD 10 - ZFS - Jail - Don't work.

Another aspect to keep in mind is the 'jailed' property for ZFS filesystems. If that property is set then the host won't be able to access those filesystems any longer as they are now only accessible from inside the jail. This is done to prevent impossible mountpoints to interfere with the host while they may be perfectly acceptable within the confines of said jail.

Try using zfs get jailed /jail/Oksymoron.
 
Re: FreeBSD 10 - ZFS - Jail - Don't work.

usdmatt said:
Do you also have the following in /etc/rc.conf to make your system mount ZFS on boot?

Code:
zfs_enable="YES"

Thanks. I think, you're right. I had but with "#" :(

But I still can't run the jail:
Code:
root@ks3360102:/jail/Oksymoron # jail -c Oksymoron
ifconfig: ioctl (SIOCAIFADDR): File exists
jail: Oksymoron: /sbin/ifconfig em0 inet 192.168.0.2 netmask 255.255.255.255 alias: failed
root@ks3360102:/jail/Oksymoron #
root@ks3360102:/jail/Oksymoron # cat /etc/jail.conf
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";

Oksymoron {
        path = /jail/Oksymoron;
        mount.devfs;
        allow.mount;
        mount;
        host.hostname = Oksymoron.edu.pl;
        ip4.addr = 192.168.0.2;
        interface = em0;

}
Ifconfig:
Code:
root@ks3360102:/jail/Oksymoron # ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:22:4d:a9:c3:ae
        inet 37.*.*.* netmask 0xffffff00 broadcast 37.187.97.255
        inet6 fe80::222:4dff:fea9:c3ae%em0 prefixlen 64 scopeid 0x1
        inet6 2001:41d0:a:2197::1 prefixlen 128
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 192.168.0.1 netmask 0xfffffff8
        inet 192.168.0.2 netmask 0xffffffff
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
Cloned interfaces
Code:
# pf
pf_enable="YES"
cloned_interfaces="lo1"
ipv4_addrs_lo1="192.168.0.1-2/29"

ShelLuser said:
Another aspect to keep in mind is the 'jailed' property for ZFS filesystems. If that property is set then the host won't be able to access those filesystems any longer as they are now only accessible from inside the jail. This is done to prevent impossible mountpoints to interfere with the host while they may be perfectly acceptable within the confines of said jail.

Try using zfs get jailed /jail/Oksymoron.

Code:
root@ks3360102:/jail/Oksymoron # zfs get jailed /jail/Oksymoron
NAME            PROPERTY  VALUE   SOURCE
jail/Oksymoron  jailed    off     default
 
Re: FreeBSD 10 - ZFS - Jail - Don't work.

You appear to be getting more into jail problems now which really isn't my area.

The error you get when starting the jail looks like it's complaining because the address you have given the jail, 192.168.0.2, is already assigned to the lo1 interface. As far as I'm aware, you shouldn't be configuring your system with the addresses given to the jail. They will configure that themselves when they start.

Edit: I also find your configuration a bit confusing though. You have a 37.x address assigned to em0, which appears to be a 'real' public address (purely assumed because it's not in a private range). However, you've configured your jail to use the same interface, with a private address?
 
Re: FreeBSD 10 - ZFS - Jail - Don't work.

Code:
root@ks3360102:~ # jail -c Oksymoron
Oksymoron: created
/etc/rc: WARNING: $hostname is not set -- see rc.conf(5).
Generating host.conf.
Creating and/or trimming log files.
ln: /dev/log: Operation not permitted
Starting syslogd.
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
Clearing /tmp (X related).
Updating motd:.
/etc/rc: WARNING: run_rc_command: cannot run /usr/sbin/sendmail
/etc/rc: WARNING: run_rc_command: cannot run /usr/sbin/sendmail
Starting cron.

Mon Jan 27 14:20:37 UTC 2014
root@ks3360102:~ # w
 3:21PM  up  1:05, 1 user, load averages: 0.11, 0.10, 0.08
USER       TTY      FROM                      LOGIN@  IDLE WHAT
root       pts/0    89-67-168-84.dynamic.che  3:17PM     - w
root@ks3360102:~ # jls
   JID  IP Address      Hostname                      Path
     1  192.168.0.2     Oksymoron.edu.pl              /jail/Oksymoron

After reboot works very well. I fixed /etc/jail.conf
Code:
root@ks3360102:~ # cat /etc/jail.conf
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";

Oksymoron {
        path = /jail/Oksymoron;
        mount.devfs;
        allow.mount;
        mount;
        host.hostname = Oksymoron.edu.pl;
        ip4.addr = 192.168.0.2;
        interface = lo1;

}
I've copied /etc/resolv.conf to /jail/Oksymoron/etc but i can get access to the internet.
/etc/pf.conf
Code:
root@ks3360102:~ # cat /etc/pf.conf
IP_PUB="37.*.*.*
IP_JAIL="192.168.0.2"
NET_JAIL="192.168.0.0/24"
PORT_JAIL="{80,443,31337}"
scrub in all
nat pass on em0 from $NET_JAIL to any -> $IP_PUB
rdr pass on em0 proto tcp from any to $IP_PUB port $PORT_JAIL -> $IP_JAIL
Ifconfig
Code:
root@ks3360102:~ # ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:22:4d:a9:c3:ae
        inet 37.*.*.* netmask 0xffffff00 broadcast 37.187.97.255
        inet6 fe80::222:4dff:fea9:c3ae%em0 prefixlen 64 scopeid 0x1
        inet6 2001:41d0:a:2197::1 prefixlen 128
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 192.168.0.1 netmask 0xfffffff8
        inet 192.168.0.2 netmask 0xffffffff
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
I'm wondering what's wrong. Everything looks good i think soo but can't conect with internet :(. On host works perfectly, problem is with jail.
 
Re: FreeBSD 10 - ZFS - Jail - Don't work.

Why use NAT / redirection anyway?

When I need a jail to have access to the network itself I simply set it up with a 'real' IP address and thus allow it to directly access the network.

For example, on my backup server I use a jail as some sort of 'interface' as contact for the rest of my servers. In other words; my servers never contact my backup server directly, but only the jail which runs on it. To that end I simply don't assign any IP address to the network device I'm using (vtnet1) but instead have the address assigned to vtnet1 by creating the jail.

(edit)

For example (from my rc.conf):

Code:
# Jail
jail_enable="YES"
jail_list="dogma"
jail_sysvipc_allow="YES"

jail_dogma_rootdir="/usr/jails/dogma"
jail_dogma_hostname="central.dogma"
jail_dogma_ip="inet xx.xx.xx.xx netmask 255.255.255.248"
jail_dogma_interface="vtnet1"
jail_dogma_devfs_enable="YES"
jail_dogma_fstab="/etc/fstab.dogma"
This sets up the jail, makes sure that vtnet1 is assigned with the new IP address after which all other hosts in the network can now directly access the services running inside this jail. In my opinion this is much easier than messing with NAT and redirects.
 
Re: FreeBSD 10 - ZFS - Jail - Don't work.

So you have 192.168.0.1 as the gateway on the jail?
You may also need to enable forwarding on the host otherwise it won't foward packets from lo1 to em0

/etc/rc.conf
Code:
gateway_enable="YES"
 
Re: FreeBSD 10 - ZFS - Jail - Don't work.

ShelLuser said:
Why use NAT / redirection anyway?

When I need a jail to have access to the network itself I simply set it up with a 'real' IP address and thus allow it to directly access the network.

For example, on my backup server I use a jail as some sort of 'interface' as contact for the rest of my servers. In other words; my servers never contact my backup server directly, but only the jail which runs on it. To that end I simply don't assign any IP address to the network device I'm using (vtnet1) but instead have the address assigned to vtnet1 by creating the jail.

(edit)

For example (from my rc.conf):

Code:
# Jail
jail_enable="YES"
jail_list="dogma"
jail_sysvipc_allow="YES"

jail_dogma_rootdir="/usr/jails/dogma"
jail_dogma_hostname="central.dogma"
jail_dogma_ip="inet xx.xx.xx.xx netmask 255.255.255.248"
jail_dogma_interface="vtnet1"
jail_dogma_devfs_enable="YES"
jail_dogma_fstab="/etc/fstab.dogma"
This sets up the jail, makes sure that vtnet1 is assigned with the new IP address after which all other hosts in the network can now directly access the services running inside this jail. In my opinion this is much easier than messing with NAT and redirects.

Everything works very good on all kind of version FreeBSD. When i installed FreeBSD 10.0, and configuration is in /etc/jail.conf works nothing. I have only one IP on server, and i want to give access to shell for friends. I had redirection on port ex: 31337 for ssh, cloned interface, jail was only on the same ip as now (192.168.0.2) andy everything worked very good. In pf.conf i add ports which i want to be redirection to jail in/out. Now works nothing.
rc.conf
Code:
# pf
pf_enable="YES"
cloned_interfaces="lo1"
ipv4_addrs_lo1="192.168.0.1-2/29"
pf.conf
Code:
root@ks3360102:~ # cat /etc/pf.conf
IP_PUB="37.*.*.*"
IP_JAIL="192.168.0.2"
NET_JAIL="192.168.0.0/24"
PORT_JAIL="{80,443,31337}"
scrub in all
nat pass on em0 from $NET_JAIL to any -> $IP_PUB
rdr pass on em0 proto tcp from any to $IP_PUB port $PORT_JAIL -> $IP_JAIL
 
Re: [SOLVED] FreeBSD 10 - ZFS - Jail- jail.conf - Don't work

@ShelLuser, @usdmatt

Thank you for your support. Everything works perfectly. Problem with net was in resolv.conf. Creator blocked nameserver using "#" in resolv.conf.

Greetz,
 
Back
Top