networking with jails with a host beeing a VM?

lalebarde

lalebarde

Hi all,

My desktop is under Gentoo Linux, and I am setting up servers under FreeBSD. To learn and to perform trials, I use Qemu. My FreeBSD Qemu client is a 9.1 full zfs install. I can SSH to it and access the web from inside. Now, I have to set up jails for future servers. I use ezjail. My first jail is created and I can access to its console, but then I don't manage to set up networking, and I need some help please. At the end of the story, I want a full IPv4/IPv6 setting, but here, let's focus on IPv4 only.

Here are my settings:

Briefly speaking, the FreeBSD VM and its jails are all on the same subnet 192.168.99.0. The host is on 192.168.99.66, the first jail on 192.168.99.2. The router is on 192.168.99.1 which is a bridge with a tap interface in the Qemu host. NAT is used thanks to iptables to route the traffic through the Qemu host which acts like a gateway. But into the FreeBSD VM, as everything is on the same subnet, I have not activated pf (that would be for later). With such a setup, I assume being in a VM does not change anything, though I would appreciate a confirmation or rebuttal by an expert. Anyway, I provide here the VM host setup.

Gentoo Linux Desktop / qemu host:
Code: 
# ifconfig 
br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.99.1  netmask 255.255.255.0  broadcast 192.168.99.255
        inet6 fe80::2095:a4ff:fe97:bedb  prefixlen 64  scopeid 0x20<link>
        ether 8a:0b:c2:53:d3:e2  txqueuelen 0  (Ethernet)
        RX packets 802908  bytes 112142378 (106.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 905639  bytes 678999653 (647.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.0.10  netmask 255.255.255.0  broadcast 192.168.0.255
        inet6 fe80::862b:2bff:fe99:f465  prefixlen 64  scopeid 0x20<link>
        inet6 xxxx:xxxx:xxxx:xxxx:862b:2bff:fe99:f465  prefixlen 64  scopeid 0x0<global>
        inet6 xxxx:xxxx:xxxx:xxxx::10  prefixlen 64  scopeid 0x0<global>
        ether 84:2b:2b:99:f4:65  txqueuelen 1000  (Ethernet)
        RX packets 1232833  bytes 1051661312 (1002.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1140081  bytes 153213168 (146.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17  

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 16436
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 0  (Boucle locale)
        RX packets 1034406  bytes 61739784 (58.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1034406  bytes 61739784 (58.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

qtap0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        inet6 fe80::880b:c2ff:fe53:d3e2  prefixlen 64  scopeid 0x20<link>
        ether 8a:0b:c2:53:d3:e2  txqueuelen 500  (Ethernet)
        RX packets 802970  bytes 123388670 (117.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 915392  bytes 679643447 (648.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
Code: 
# iptables -L -n
Chain INPUT (policy DROP)
target     prot opt source               destination         
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0            reject-with tcp-reset
REJECT     udp  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0           
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0            icmptype 0
ACCEPT     udp  --  192.168.0.0/24       0.0.0.0/0            udp dpt:631
ACCEPT     tcp  --  192.168.0.0/24       0.0.0.0/0            tcp dpt:631

Chain FORWARD (policy DROP)
target     prot opt source               destination         
ACCEPT     all  --  192.168.99.0/24     !192.168.0.0/24      
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
Code: 
# iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  0.0.0.0/0            0.0.0.0/0

Here is how I launch my FreeBSD VM:
Code: 
$ qemu-kvm -hda freebsd-9.1-amd64-0.img -hdb freebsd-9.1-amd64-1.img -hdc ufs_scripts_et_tarballs.img -boot c -m 1500 -k fr -smp 3 -cpu Nehalem -net nic,macaddr=02:5a:4b:3c:2d:1e -net tap,ifname=qtap0,script=no,downscript=no

FreeBSD VM / qemu client:
Code: 
root@host:/root # ifconfig 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 02:5a:4b:3c:2d:1e
	inet 192.168.99.66 netmask 0xffffff00 broadcast 192.168.99.255
	inet6 fe80::5a:4bff:fe3c:2d1e%em0 prefixlen 64 scopeid 0x1 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
Code: 
root@host:/root # cat /etc/rc.conf | sed -e 's/#.*//;/^\s*$/d'
zfs_enable=YES
hostname="host.mydomain.com"
ifconfig_em0="inet 192.168.99.66 netmask 0xffffff00"
defaultrouter="192.168.99.1"
sshd_enable="YES"
moused_enable="YES"
dumpdev="AUTO"
font8x8="iso-8x8"
font8x14="iso-8x14"
font8x16="iso-8x16"
scrnmap="iso-8859-1_to_cp437"
keymap="fr.iso.acc"
ipv6_activate_all_interfaces="YES"
ipv6_ifconfig_em0_alias0="inet6 xxxx:xxxx:xxxx:xxxx::99:66 prefixlen 64"
ipv6_ifconfig_em0_alias1="inet6 fe80::99:66 prefixlen 64"
ipv6_defaultrouter="fe80:2095:a4ff:fe97:bedb"
ezjail_enable="YES" 
openntpd_enable="YES" 
inetd_enable="NO" 
rpc_bind_enable="NO" 
sendmail_enable="NO" 
syslogd_enable="YES" 
syslogd_flags="-s -b 127.0.0.1" 
gateway_enable="YES"

FreeBSD jail n°1:
Code: 
root@el001:/root # ifconfig 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 02:5a:4b:3c:2d:1e
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
Code: 
root@el001:/root # cat /etc/rc.conf | sed -e 's/#.*//;/^\s*$/d'
hostname="el001.mydomain.com"
keymap="fr.iso.acc.kbd"
ifconfig_em0="inet 192.168.99.2 netmask 0xffffff00"
defaultrouter="192.168.99.1"
network_interfaces=""
rpcbind_enable="NO"
cron_flags="$cron_flags -J 15"
syslogd_flags="-ss"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
 
As far as I have understood, jails parameters may be defined in rc.conf, jail.conf, or if ezjail is used: /usr/local/etc/ezjail.conf, /usr/local/etc/ezjail/myjail. So since I use ezjail, I checked both last ones.

/usr/local/etc/ezjail.conf
Code: 
ezjail_jaildir=/usr/jails
ezjail_jailtemplate=${ezjail_jaildir}/newjail
ezjail_jailbase=${ezjail_jaildir}/basejail
ezjail_sourcetree=/usr/src
ezjail_mount_enable="YES"
ezjail_devfs_enable="YES"
ezjail_procfs_enable="YES"
ezjail_fdescfs_enable="YES"
ezjail_use_zfs="YES"
ezjail_use_zfs_for_jails="YES"
ezjail_jailzfs="sys/jails"

/usr/local/etc/ezjail/el001.mydomain.com
Code: 
export jail_el001_dataensafe_com_hostname="el001.mydomain.com"
export jail_el001_dataensafe_com_ip="192.168.99.2"
export jail_el001_dataensafe_com_rootdir="/usr/jails/el001.mydomain.com"
export jail_el001_dataensafe_com_exec_start="/bin/sh /etc/rc"
export jail_el001_dataensafe_com_exec_stop=""
export jail_el001_dataensafe_com_mount_enable="YES"
export jail_el001_dataensafe_com_devfs_enable="YES"
export jail_el001_dataensafe_com_devfs_ruleset="devfsrules_jail"
export jail_el001_dataensafe_com_procfs_enable="YES"
export jail_el001_dataensafe_com_fdescfs_enable="YES"
export jail_el001_dataensafe_com_image=""
export jail_el001_dataensafe_com_imagetype="zfs"
export jail_el001_dataensafe_com_attachparams=""
export jail_el001_dataensafe_com_attachblocking=""
export jail_el001_dataensafe_com_forceblocking=""
export jail_el001_dataensafe_com_zfs_datasets=""
export jail_el001_dataensafe_com_cpuset=""
export jail_el001_dataensafe_com_fib=""

If I am not mistaken, it looks correct, but I am really not sure at this time, and I still not understand why the interfaces lo and em0 are not configured when the jail starts (of course, I have restarted it). Help please.
 
From http://forums.freebsd.org/showpost.php?p=201904&postcount=2, I added the following line into /usr/local/etc/ezjail/el001_mydomain_com:
Code: 
export jail_el001_dataensafe_com_interface="em0"
I can ping now localhost and the jail IP:
Code: 
root@el001:/root # ping localhost
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.079 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.068 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.102 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.066 ms
^C
--- localhost ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.066/0.079/0.102/0.014 ms
root@el001:/root # ping 192.168.99.2
PING 192.168.99.2 (192.168.99.2): 56 data bytes
64 bytes from 192.168.99.2: icmp_seq=0 ttl=64 time=0.073 ms
64 bytes from 192.168.99.2: icmp_seq=1 ttl=64 time=0.069 ms
64 bytes from 192.168.99.2: icmp_seq=2 ttl=64 time=0.062 ms
^C
--- 192.168.99.2 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.062/0.068/0.073/0.005 ms
Code: 
root@el001:/root # ifconfig 
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
	ether 02:5a:4b:3c:2d:1e
	inet 192.168.99.2 netmask 0xffffffff broadcast 192.168.99.2
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>

I can communicate with the outside:
Code: 
root@el001:/root # host google.fr
google.fr has address 173.194.45.63
google.fr has address 173.194.45.56
google.fr has address 173.194.45.55
google.fr has IPv6 address 2a00:1450:4007:803::1018
.....

Still, I wonder why
Code: 
inet 127.0.0.1 netmask 0xff000000
do not appear.
 
This will not appear in the jails.

Code: 
inet 127.0.0.1 netmask 0xff000000

It will only show up the host. Pinging localhost in the jail gets translated to pinging the address of that jail.
 
You must log in or register to reply here.
Back
Top