Same settings, one works and one doesnt!!

beaute · Jul 26, 2010

Okay so this is really weird because it used to work. It's probably something as dumb as turning forwarding on/off but I just can't figure it out. I have the same settings on two machines, I basically copied the files from one machine to the other, and changed what's necessary. It worked for some time, but while I was playing with the firewall, I must have done something but I can't seem to reverse it because I don't know what I turned off/on ><

I have disabled the firewall (no lines to cause the firewall to start on boot in rc.conf)

I have a host machine with three physical interfaces, namely 1.1.1.1, 2.2.2.2 and 3.3.3.2. This machine has two jails, jail1 and jail2. Each jail has two virtual interfaces, so for jail1 I have 192.168.1.2 and 10.0.0.1. For jail2 I have 192.168.1.4 and 172.16.0.1.

Now the problem is that I can ping from jail1 to all three physical interfaces just fine. I just can't do so from jail2.
# jexec 2 ping 1.1.1.1

Code:

ping: send to: No route to host

Here is my ifconfig of the host machine (I am aware of the no-carrier portion of the interfaces, I'm not connecting them to anything at the moment except msk0, I'm just trying to reach them from within the jails):

Code:

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
	ether 00:15:17:96:0d:08
	inet 2.2.2.2 netmask 0xff000000 broadcast 2.255.255.255
	media: Ethernet autoselect
	status: no carrier
em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4>
	ether 00:15:17:96:0d:09
	inet 3.3.3.2 netmask 0xff000000 broadcast 3.255.255.255
	media: Ethernet autoselect
	status: no carrier
msk0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=18<VLAN_MTU,VLAN_HWTAGGING>
	ether 00:1e:90:9d:ee:4e
	inet 1.1.1.1 netmask 0xff000000 broadcast 1.255.255.255
	media: Ethernet autoselect (1000baseT <full-duplex,flag0,flag1,flag2>)
	status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7 
	inet6 ::1 prefixlen 128 
	inet 127.0.0.1 netmask 0xff000000 
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether c2:88:df:e6:cd:8d
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
	maxage 20 holdcnt 6 proto rstp maxaddr 100 timeout 1200
	root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
	member: epair1a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 11 priority 128 path cost 14183
	member: msk0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 5 priority 128 path cost 55
	member: epair0a flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
	        ifmaxaddr 0 port 10 priority 128 path cost 14183
gif0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
	tunnel inet 1.1.1.1 --> 1.1.1.2
	inet 101.0.0.1 --> 101.0.0.2 netmask 0xffffff00 
	options=1<ACCEPT_REV_ETHIP_VER>
epair0a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c0:24:00:0a:0a
	inet 192.168.1.4 netmask 0xffffff00 broadcast 192.168.1.255
epair2a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c0:24:00:0c:0a
	inet 10.0.0.2 netmask 0xff000000 broadcast 10.255.255.255
epair1a: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c0:24:00:0b:0a
	inet 192.168.1.5 netmask 0xffffff00 broadcast 192.168.1.255
epair3a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c0:24:00:0e:0a
	inet 172.16.0.2 netmask 0xffff0000 broadcast 172.16.255.255

And here is my ifconfig of jail1

Code:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c0:24:00:0b:0b
	inet6 fe80::c0:24ff:fe00:b0b%epair0b prefixlen 64 scopeid 0x2 
	inet 192.168.1.2 netmask 0xffffff00 broadcast 192.168.1.255
epair2b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c0:24:00:0d:0b
	inet6 fe80::c0:24ff:fe00:d0b%epair2b prefixlen 64 scopeid 0x3 
	inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255

and my ifconfig of jail2

Code:

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet 127.0.0.1 netmask 0xff000000 
	inet6 ::1 prefixlen 128 
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 
epair1b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c0:24:00:0d:0b
	inet6 fe80::c0:24ff:fe00:d0b%epair1b prefixlen 64 scopeid 0x2 
	inet 192.168.1.3 netmask 0xffffff00 broadcast 192.168.1.255
epair3b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	ether 02:c0:24:00:0f:0b
	inet6 fe80::c0:24ff:fe00:f0b%epair3b prefixlen 64 scopeid 0x3 
	inet 172.16.0.1 netmask 0xffffff00 broadcast 172.16.0.255

and finally my netstat

Code:

Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
1.0.0.0/8          link#5             U           0        0   msk0
1.1.1.1            link#5             UHS         0        0    lo0
2.0.0.0/8          link#1             U           0        0    em0
2.2.2.2            link#1             UHS         0        0    lo0
3.0.0.0/8          link#2             U           0        0    em1
3.3.3.2            link#2             UHS         0        0    lo0
10.0.0.0/8         link#12            U           0        0 epair2
10.0.0.2           link#12            UHS         0        0    lo0
101.0.0.1          link#9             UHS         0        0    lo0
101.0.0.2          link#9             UH          0        0   gif0
127.0.0.1          link#7             UH          0        0    lo0
172.16.0.0/16      link#14            U           0        3 epair3
172.16.0.2         link#14            UHS         0        0    lo0
192.168.1.0/24     link#8             U           0       33 bridge
192.168.1.1        link#8             UHS         0        0    lo0
192.168.1.4        link#10            UHS         0        0    lo0
192.168.1.5        link#11            UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::1                               ::1                           UH          lo0
fe80::%lo0/64                     link#7                        U           lo0
fe80::1%lo0                       link#7                        UHS         lo0
ff01:7::/32                       fe80::1%lo0                   U           lo0
ff02::%lo0/32                     fe80::1%lo0                   U           lo0

I have the same settings on a different machine and everything works fine. It just doesn't work on this one.

SirDice · Jul 26, 2010

Can you post the rc.conf file for both hosts (the one that works and the one that doesn't)?

beaute · Jul 27, 2010

This is the rc.conf of the host that does not work:

Code:

# -- sysinstall generated deltas -- # Thu Apr 29 16:31:48 2010
# Created: Thu Apr 29 16:31:48 2010
# Enable network daemons for user convenience.
# Please make all changes to this file, not to /etc/defaults/rc.conf.
# This file now contains just the overrides from /etc/defaults/rc.conf.
keymap="us.iso"

# -- sysinstall generated deltas -- # Thu Apr 29 16:44:09 2010
#ifconfig_msk0="DHCP"
rpcbind_enable="YES"
amd_enable="NO"
hostname="VPN.SSG-550"
hald_enable="YES"
dbus_enable="YES"

# -- sysinstall generated deltas -- # Mon May  3 13:11:00 2010
font8x8="cp437-8x8"
font8x14="cp437-8x14"
font8x16="cp437-8x16"

# SSHD ENABLE
sshd_enable="YES"

# Interface Settings
cloned_interfaces="bridge0 gif0"
gif_interfaces="gif0"

ifconfig_msk0="inet 1.1.1.1 netmask 255.0.0.0"
ifconfig_em0="inet 2.2.2.2 netmask 255.0.0.0"
ifconfig_em1="inet 3.3.3.2 netmask 255.0.0.0"
ifconfig_bridge0="inet 192.168.1.1 netmask 255.255.255.0 up"
gifconfig_gif0="1.1.1.1 1.1.1.2"
ifconfig_gif0="inet 101.0.0.1 101.0.0.2 netmask 255.255.255.0 mtu 1500"

gateway_enable="YES"
inetd_enable="YES"
default_router="192.168.1.1"

#ipsec_enable="YES"
#ipsec_file"/usr/local/etc/racoon/setkey.conf"
#racoon_enable="yes"
#ipsec_enable="YES"
#ipsec_file="/usr/local/etc/racoon/setkey.conf"

# JAIL Settings
jail_enable="YES"
jail_v2_enable="YES"
jail_list="jail1 jail2"
jail_set_hostname_allow="YES"

# jail1 Settings
jail_jail1_name="jail1"
jail_jail1_hostname="jail1"
jail_jail1_devfs_enable="YES"
jail_jail1_rootdir="/usr/jails/jail1"
jail_jail1_vnet_enable="YES"

# jail2 Settings
jail_jail2_name="jail2"
jail_jail2_hostname="jail2"
jail_jail2_devfs_enable="YES"
jail_jail2_rootdir="/usr/jails/jail2"
jail_jail2_vnet_enable="YES"

# Network Settings
# Create epairs
jail_jail1_exec_prestart0="ifconfig epair0 create"
jail_jail2_exec_prestart0="ifconfig epair1 create"
jail_jail1_exec_prestart1="ifconfig epair2 create"
jail_jail2_exec_prestart1="ifconfig epair3 create"

# Give IP addresses to epairs
jail_jail1_exec_prestart2="ifconfig bridge0 addm epair0a"
jail_jail1_exec_prestart3="ifconfig epair0a 192.168.1.4 up"
jail_jail2_exec_prestart2="ifconfig bridge0 addm epair1a"
jail_jail2_exec_prestart3="ifconfig epair1a 192.168.1.5 up"
jail_jail1_exec_prestart4="ifconfig bridge0 addm msk0"
jail_jail1_exec_prestart5="ifconfig epair2a 10.0.0.2 up"
jail_jail2_exec_prestart4="ifconfig epair3a 172.16.0.2 up"

# Put epairs in jails
jail_jail1_exec_earlypoststart0="ifconfig epair0b vnet jail1"
jail_jail2_exec_earlypoststart0="ifconfig epair1b vnet jail2"
jail_jail1_exec_earlypoststart1="ifconfig epair2b vnet jail1"
jail_jail2_exec_earlypoststart1="ifconfig epair3b vnet jail2"

# Give IP addresses to epairs in jails
jail_jail1_exec_afterstart0="ifconfig lo0 127.0.0.1"
jail_jail1_exec_afterstart1="ifconfig epair0b 192.168.1.2 netmask 255.255.255.0 up"
jail_jail1_exec_afterstart2="ifconfig epair2b 10.0.0.1 netmask 255.255.255.0 up"
jail_jail2_exec_afterstart0="ifconfig lo0 127.0.0.1"
jail_jail2_exec_afterstart1="ifconfig epair1b 192.168.1.3 netmask 255.255.255.0 up"
jail_jail2_exec_afterstart2="ifconfig epair3b 172.16.0.1 netmask 255.255.255.0 up"

# Routing Information and starting the jails
jail_jail1_exec_afterstart3="route add default 192.168.1.1"
jail_jail1_exec_afterstart4="/bin/sh /etc/rc"
jail_jail2_exec_afterstarr3="route add default 192.168.1.1"
jail_jail2_exec_afterstart4="/bin/sh /etc/rc"

jail_jail1_exec_poststop0="ifconfig bridge0 deletem epair0a"
jail_jail1_exec_poststop1="ifconfig bridge0 deletem epair1a"
jail_jail1_exec_poststop2="ifconfig bridge0 deletem msk0"

# Destroy epairs on shutdown
jail_jail1_exec_poststop3="ifconfig epair0a destroy"
jail_jail1_exec_poststop4="ifconfig epair2a destroy"
jail_jail2_exec_poststop0="ifconfig epair1a destroy"
jail_jail2_exec_poststop1="ifconfig epair3a destroy"

# PF
#pf_enable="YES"
#pf_rules="/etc/pf.conf"
#pflog_enable="YES"

and this is the rc.conf of the host the does work.

Code:

# -- sysinstall generated deltas -- # Thu Apr 29 16:31:48 2010
# Created: Thu Apr 29 16:31:48 2010
# Enable network daemons for user convenience.
# Please make all changes to this file, not to /etc/defaults/rc.conf.
# This file now contains just the overrides from /etc/defaults/rc.conf.
keymap="us.iso"

# -- sysinstall generated deltas -- # Thu Apr 29 16:44:09 2010
#ifconfig_msk0="DHCP"
rpcbind_enable="YES"
amd_enable="NO"
hostname="VPN.SSG-550"
hald_enable="YES"
dbus_enable="YES"

# -- sysinstall generated deltas -- # Mon May  3 13:11:00 2010
font8x8="cp437-8x8"
font8x14="cp437-8x14"
font8x16="cp437-8x16"

# SSHD ENABLE
sshd_enable="YES"

# Interface Settings
cloned_interfaces="bridge0 gif0"
gif_interfaces="gif0"

ifconfig_msk0="inet 1.1.1.2 netmask 255.0.0.0"
ifconfig_em0="inet 4.4.4.2 netmask 255.0.0.0"
ifconfig_em1="inet 5.5.5.2 netmask 255.0.0.0"
ifconfig_bridge0="inet 206.165.1.1 netmask 255.255.255.0 up"
gifconfig_gif0="1.1.1.2 1.1.1.1"
ifconfig_gif0="inet 101.0.0.2 101.0.0.1 netmask 255.255.255.0 mtu 1500"

gateway_enable="YES"

inetd_enable="YES"
default_router="206.165.1.1"

#racoon_enable="yes"
#ipsec_enable="YES"
#ipsec_file="/usr/local/etc/racoon/setkey.conf"

# JAIL Settings
jail_enable="YES"
jail_v2_enable="YES"
jail_list="jail3 jail4"
jail_set_hostname_allow="YES"

# jail3 Settings
jail_jail3_name="jail3"
jail_jail3_hostname="jail3"
jail_jail3_devfs_enable="YES"
jail_jail3_rootdir="/usr/jails/jail3"
jail_jail3_vnet_enable="YES"

# jail4 Settings
jail_jail4_name="jail4"
jail_jail4_hostname="jail4"
jail_jail4_devfs_enable="YES"
jail_jail4_rootdir="/usr/jails/jail4"
jail_jail4_vnet_enable="YES"

# Network Settings
# Create epairs
jail_jail3_exec_prestart0="ifconfig epair0 create"
jail_jail4_exec_prestart0="ifconfig epair1 create"
jail_jail3_exec_prestart1="ifconfig epair2 create"
jail_jail4_exec_prestart1="ifconfig epair3 create"

# Give IP addresses to epairs
jail_jail3_exec_prestart2="ifconfig bridge0 addm epair0a"
jail_jail3_exec_prestart3="ifconfig epair0a 206.165.1.4 up"
jail_jail4_exec_prestart2="ifconfig bridge0 addm epair1a"
jail_jail4_exec_prestart3="ifconfig epair1a 206.165.1.5 up"
jail_jail3_exec_prestart4="ifconfig bridge0 addm msk0"
jail_jail3_exec_prestart5="ifconfig epair2a 10.0.0.2 up"
jail_jail4_exec_prestart4="ifconfig epair3a 172.16.0.2 up"

# Put epairs in jails
jail_jail3_exec_earlypoststart0="ifconfig epair0b vnet jail3"
jail_jail4_exec_earlypoststart0="ifconfig epair1b vnet jail4"
jail_jail3_exec_earlypoststart1="ifconfig epair2b vnet jail3"
jail_jail4_exec_earlypoststart1="ifconfig epair3b vnet jail4"

# Give IP addresses to epairs in jails
jail_jail3_exec_afterstart0="ifconfig lo0 127.0.0.1"
jail_jail3_exec_afterstart1="ifconfig epair0b 206.165.1.2 netmask 255.255.255.0 up"
jail_jail3_exec_afterstart2="ifconfig epair2b 10.0.0.1 netmask 255.255.255.0 up"
jail_jail4_exec_afterstart0="ifconfig lo0 127.0.0.1"
jail_jail4_exec_afterstart1="ifconfig epair1b 206.165.1.3 netmask 255.255.255.0 up"
jail_jail4_exec_afterstart2="ifconfig epair3b 172.16.0.1 netmask 255.255.255.0 up"

# Routing Information and starting the jails
jail_jail3_exec_afterstart3="route add default 206.165.1.1"
jail_jail3_exec_afterstart4="/bin/sh /etc/rc"
jail_jail4_exec_afterstart3="route add default 206.165.1.1"
jail_jail4_exec_afterstart4="/bin/sh /etc/rc"

jail_jail3_exec_poststop0="ifconfig bridge0 deletem epair0a"
jail_jail3_exec_poststop1="ifconfig bridge0 deletem epair1a"
jail_jail3_exec_poststop2="ifconfig bridge0 deletem msk0"

# Destroy epairs on shutdown
jail_jail3_exec_poststop3="ifconfig epair0a destroy"
jail_jail3_exec_poststop4="ifconfig epair2a destroy"
jail_jail4_exec_poststop0="ifconfig epair1a destroy"
jail_jail4_exec_poststop1="ifconfig epair3a destroy"

# PF Enable
#pf_enable="YES"
#pf_rules="/etc/pf.conf"
#pflog_enable="YES"
#pflog_logfile="/var/log/pflog"

beaute · Jul 28, 2010

OK, I played around with my settings and it appears that I had mis-spelled afterstart (wrote it afterstarr) for the route add for the second jail. I fixed it, but it still doesn't work.

If I add the route manually it works. In command-line language:
# jexec 2 route add default 192.168.1.1

will allow me to reach the hosts 1.1.1.1, 2.2.2.2 and 3.3.3.2. So it appears the line that adds the route is not executed. I fixed the typo but it still doesn't run. Can anyone tell me why?

Code:

jail_jail2_exec_afterstart3="route add default 192.168.1.1"

It might be helpful to note that the rc.conf of the jail has three lines only:

Code:

hostname="jail2"
default_router="192.168.1.1"
sshd_enable="NO"

beaute · Jul 28, 2010

nvm I solved it.

sixtydoses · Jul 28, 2010

beaute said:
nvm I solved it.

Share your solution please.

beaute · Jul 28, 2010

# mv /etc/rc.conf /home/user/rc.conf.backup
# scp [email]root@1.1.1.2:/etc/rc.conf[/email] /etc/rc.conf
# vi /etc/rc.conf

in vi

Code:

:s/jail3/jail1/g
:s/jail4/jail2/g
:s/206.165/192.168

there was a typo I couldn't locate..........

Same settings, one works and one doesnt!!

beaute

SirDice

Administrator

beaute

beaute

beaute

sixtydoses

beaute