Problem reaching physical hosts from jails

Please do not feel discouraged to read given what might look like a "complex" scheme. I'm pretty sure the solution is a simple as fixing a routing problem. Summary of the problem is the fact that I am unable to establish a connection between the gateway and my jails, although I am able to connect between the jails and each other.

I have a box with 5 physical interfaces, I am only working with three of them at the moment.

What I would like to ultimately achieve is the following scenario (gotta love the ascii):

Code:
				      ____________________
				     |      Cipher BOX    |
				     |      _________     |
                                     |     |         |    0a
notebook <---> {red network} <-----> em0<->x  jail1  s<   |
10.0.0.3                             |     |         | \  1a
                                     |     |_________|  > |
				     |                   msk0 <-----> {black network} <-----> same as left side}
				     |      _________   > |
                                     |     |         | /  |
notebook <---> {red network} <-----> em1<->j  jail2  i<   |
172.16.0.3                           |     |         |  bridge0
                                     |     |_________|    |
				     |____________________|

Physical Interfaces
	em0 = IP Address is 10.0.0.5
	em1 = IP address is 172.16.0.5
	msk0 = IP address is 192.168.1.5

Virtual Interfaces
	x = IP address is 10.0.0.2
	s = epair0b = IP address is 192.168.1.2
	j = IP address is 172.16.0.2
	i = epair1b = IP address is 192.168.1.11
	0a = epair0a = IP address is 192.168.1.10
	1a = epair1a = IP address is 192.168.1.20
	bridge0 = IP address is 192.168.1.1 (has members epair0a and epair1a)

Let's focus on the Cipher box, I am currently in the process of implementing this. I am using jails to create separate network stacks for each virtual interface inside the jails.
I currently have the two jails running, each jail having two interfaces only, the loopback interface and the s and i virtual interfaces. I have basically implemented everything in the right side of the IP cipher box.

With my current settings, I am able to establish a connection (using PING) between the jails using the s and i interfaces. However, although the bridge0 is supposed to be there to faciliate this connection, I am not able to ping to it, which makes me think that the echo replies I am getting are not from the correct jails.

In command line words, both of these commands:

# jexec 1 ping 192.168.1.11
# jexec 2 ping 192.168.1.2
would return echo replies

However, none of these commands would return anything:
# jexec 1 ping 192.168.1.5
# jexec 1 ping 192.168.1.10
# jexec 1 ping 192.168.1.20
# jexec 1 ping 192.168.1.1

There is no echo reply of any sort, not even host unreachable, It would just stay there until I CTRL+C when it would finally display 100.0% packet loss.


I am leaning to believe this is merely a routing problem rather than an implementation problem. I'm kind of a newb when it comes to routing, I'd appreciate any guidance.


UPDATE
after a little bit of tampering, running
# jexec 1 ping 192.168.1.5
returns
Code:
ping:sendto:Host is down.
ifconfig msk0 shows that msk0 is active. Pinging the other interfaces from jail1 (192.168.1.10, 192.168.1.20, 192.168.1.1) still do not return anything.

Also, performing ping from the main box to both jails give
Code:
ping:sendto:Host is down.


Edit
Here are my ifconfigs and netstats of the main host and the jails.

# ifconfig
Code:
em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
	ether 00:15:17:96:0d:08
	media: Ethernet autoselect
	status: no carrier
em1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
	ether 00:15:17:96:0d:09
	media: Ethernet autoselect
	status: no carrier
em2: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
	ether 00:15:17:96:0d:0a
	media: Ethernet autoselect
	status: no carrier
em3: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
	ether 00:15:17:96:0d:0b
	media: Ethernet autoselect
	status: no carrier
msk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=11a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4>
	ether 00:1e:90:9d:ee:4e
	inet 192.168.1.5 netmask 0xffffff00 broadcast 192.168.1.255
	media: Ethernet autoselect (1000baseT <full-duplex,flag0,flag1,flag2>)
	status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7 
	inet6 ::1 prefixlen 128 
	inet 127.0.0.1 netmask 0xff000000 
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 3a:83:7f:af:12:a9
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 11 priority 128 path cost 14183
	member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 10 priority 128 path cost 14183
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33200
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c0:24:00:0a:0a
	inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c0:24:00:0b:0a
	inet 192.168.1.20 netmask 0xffffff00 broadcast 192.168.1.255

# jexec 1 ifconfig
Code:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c0:24:00:0b:0b
	inet6 fe80::c0:24ff:fe00:b0b%epair0b prefixlen 64 scopeid 0x2 
	inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255

# jexec 2 ifconfig
Code:
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
epair1b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c0:24:00:0c:0b
	inet6 fe80::c0:24ff:fe00:c0b%epair1b prefixlen 64 scopeid 0x2 
	inet 192.168.1.11 netmask 0xffffff00 broadcast 192.168.1.255

# netstat -rn
Code:
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
127.0.0.1          link#7             UH          0        0    lo0
192.168.1.0/24     link#5             U           0       46   msk0
192.168.1.1        link#8             UHS         0        0    lo0
192.168.1.5        link#5             UHS         0        0    lo0
192.168.1.10       link#10            UHS         0        0    lo0
192.168.1.20       link#11            UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::1                               ::1                           UH          lo0
fe80::%lo0/64                     link#7                        U           lo0
fe80::1%lo0                       link#7                        UHS         lo0
ff01:7::/32                       fe80::1%lo0                   U           lo0
ff02::%lo0/32                     fe80::1%lo0                   U           lo0

Executing netstat -rn in the jails returned the following error message:
Code:
netstat: kvm not available: /dev/mem: No such file or directory
Routing tables
rt_tables: symbol not in namelist
 
Please post (or edit the existing post) to list the netmask for each of the IP interfaces. Can you also list the route table (use netstat -rn) for each jail as well as the physical host?
 
@DutchDaemon
executing sysctl security.jail.allow_raw_sockets=1 did not solve my problem. I was still unable to ping between the main host and the jails, although pinging between jails is functional.

@godon
I just edited the main post to include ifconfigs and netstat -rn's.
Here is also the content of my /etc/rc.conf to see how I configured everything

Code:
# -- sysinstall generated deltas -- # Thu Apr 29 16:31:48 2010
# Created: Thu Apr 29 16:31:48 2010
# Enable network daemons for user convenience.
# Please make all changes to this file, not to /etc/defaults/rc.conf.
# This file now contains just the overrides from /etc/defaults/rc.conf.
keymap="us.iso"

# -- sysinstall generated deltas -- # Thu Apr 29 16:44:09 2010
#ifconfig_msk0="DHCP"
rpcbind_enable="YES"
amd_enable="NO"
hostname="VPN.SSG-550"
hald_enable="YES"
dbus_enable="YES"

# enable ssh
sshd_enable="YES"

ifconfig_msk0="inet 192.168.1.5 netmask 255.255.255.0"
inetd_enable="YES"

# -- sysinstall generated deltas -- # Mon May  3 13:11:00 2010
font8x8="cp437-8x8"
font8x14="cp437-8x14"
font8x16="cp437-8x16"

#racoon_enable="yes"
#ipsec_enable="YES"
#ipsec_file="/usr/local/etc/racoon/setkey.conf"

# Interface Settings
cloned_interfaces="bridge0"
ifconfig_bridge0="inet 192.168.1.1 netmask 255.255.255.0 up"

# JAIL Settings
# ezjail_enable="YES"
jail_enable="YES"
jail_v2_enable="YES"
jail_list="jail1 jail2"
jail_set_hostname_allow="YES"
jail_exec_start="/bin/sh /etc/rc"
jail_exec_stop="/bin/sh /etc/rc.shutdown"

# jail1 Settings
jail_jail1_name="jail1"
jail_jail1_hostname="jail1"
jail_jail1_rootdir="/jails/jail1"
jail_jail1_vnet_enable="YES"

# jail2 Settings
jail_jail2_name="jail2"
jail_jail2_hostname="jail2"
jail_jail2_rootdir="/jails/jail2"
jail_jail2_vnet_enable="YES"

# network settings
# Create epairs
jail_jail1_exec_prestart0="ifconfig epair0 create"
jail_jail2_exec_prestart0="ifconfig epair1 create"

# Give IP addresses to epairs
jail_jail1_exec_prestart1="ifconfig bridge0 addm epair0a"
jail_jail1_exec_prestart2="ifconfig epair0a 192.168.1.10 up"
jail_jail2_exec_prestart1="ifconfig bridge0 addm epair1a"
jail_jail2_exec_prestart2="ifconfig epair1a 192.168.1.20 up"

# Put epairs in jails
jail_jail1_exec_earlypoststart0="ifconfig epair0b vnet jail1"
jail_jail2_exec_earlypoststart0="ifconfig epair1b vnet jail2"

# Give IP addresses to epairs in jails
jail_jail1_exec_afterstart0="ifconfig lo0 127.0.0.1"
jail_jail1_exec_afterstart1="ifconfig epair0b 192.168.1.2 netmask 255.255.255.0 up"
jail_jail2_exec_afterstart0="ifconfig lo0 127.0.0.1"
jail_jail2_exec_afterstart1="ifconfig epair1b 192.168.1.11 netmask 255.255.255.0 up"

jail_jail1_exec_afterstart2="route add default 192.168.1.1"
jail_jail1_exec_afterstart3="/bin/sh /etc/rc"
jail_jail2_exec_afterstart2="route add default 192.168.1.1"
jail_jail2_exec_afterstart3="/bin/sh /etc/rc"


jail_jail1_exec_poststop0="ifconfig bridge0 deletem epair0a"
jail_jail1_exec_poststop1="ifconfig bridge0 deletem epair1a"
# Destroy epairs on shutdown
jail_jail1_exec_poststop2="ifconfig epair0a destroy"
jail_jail2_exec_poststop1="ifconfig epair1a destroy"
gateway_enable="YES"

# PF
#pf_enable="YES"
#pf_rules="/etc/pf.conf"
#pflog_enable="YES"


On a final note, the computer I am using is not connected to the Internet. Actually, my entire scheme is not connected to the Internet. Just a local LAN with physical wiring.
 
You have 4 interfaces trying to live on the same subnet. It's surprising to me that things are working as well as they are. You need to remove the IP addresses on epair0a and epair1a. They aren't needed since the bridge0 interface has an IP address. You need to change the msk0 interface to use some other subnet or remove the IP from msk0 and add it as a member of bridge0 (depending on whether you want to do some sort of NAT or bridge your jails).

Give that a try and see if it works for you.
 
Okay. I removed the IP addresses on epair0a and epair1a. But I kept them as members of bridge0. I also gave msk0 a different IP address.

Code:
ifconfig_msk0="inet 1.1.1.1 netmask 255.255.0.0"
...snpped...
jail_jail1_exec_prestart1="ifconfig bridge0 addm epair0a"
#jail_jail1_exec_prestart2="ifconfig epair0a 192.168.1.10 up"
jail_jail2_exec_prestart1="ifconfig bridge0 addm epair1a"
#jail_jail2_exec_prestart2="ifconfig epair1a 192.168.1.20 up"

ping between epair0b and epair1b which used to work did not work anymore. ping between the bridge and the jails also did not work.

I then removed msk0's IP addres, and added it as a member of bridge0. Ping still did not work.

I finally removed epair0a and epair1a from the bridge0 member list, and removed msk0 from the brdige0 member list. and it doesn't work either...


I need epair0b, epair1b and msk0 to be on the same network. I only added bridge0 for routing, I don't really want it.
 
OK so I played still around with it. This time, I added back the epair0a and epair1a with their IP addresses, and added them as members to the bridge0 interface. I only changed the IP address of msk0 to be 1.1.1.1 with netmask 255.0.0.0 just to see if it works. IT DOES!

Now ping works between the jails and themselves, between the jails and the bridge, and between the jails and msk0. However, I have no idea what is going on and how it is solved, or why it was not working before. You can go ahead and mark this thread as solved, but if anyone understands networking, could you please explain to me?

Here are my final settings:
Code:
ifconfig main host
msk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=11a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4>
	ether 00:1e:90:9d:ee:4e
	inet 1.1.1.1 netmask 0xffffff00 broadcast 192.255.255.255
	media: Ethernet autoselect (none)
	status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7 
	inet6 ::1 prefixlen 128 
	inet 127.0.0.1 netmask 0xff000000 
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 3a:83:7f:af:12:a9
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 11 priority 128 path cost 14183
	member: msk0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 5 priority 128 path cost 55
	member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 10 priority 128 path cost 14183
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33200
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c0:24:00:0a:0a
	inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c0:24:00:0b:0a
	inet 192.168.1.20 netmask 0xffffff00 broadcast 192.168.1.255

ifconfig of jail1 and jail2 are the same as first post.

netstat -rn on main host returns:
Code:
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
1.0.0.0/8	   link#5             U           0        0   msk0
1.1.1.1		   link#5             UHS         0        0    lo0
127.0.0.1          link#7             UH          0        1    lo0
192.168.1.0/24     link#8             U           0       69 bridge
192.168.1.1        link#8             UHS         0        1    lo0
192.168.1.10       link#9             UHS         0        0    lo0
192.168.1.20       link#10            UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::1                               ::1                           UH          lo0
fe80::%lo0/64                     link#7                        U           lo0
fe80::1%lo0                       link#7                        UHS         lo0
ff01:7::/32                       fe80::1%lo0                   U           lo0
ff02::%lo0/32                     fe80::1%lo0                   U           lo0
 
Back
Top