can't ping inside of jail

kr651129 · #1 November 26th, 2012, 16:49

I think I'm missing something in my jail. The setup went fine but I don't have inet access inside my jail. I can ping the jail from the host system.

Code:

# jls
JID   IP Address    Hostname    Path
 2    10.10.10.101  webserver   /usr/jails/webserver

# ifconfig
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
        ether bc:5f:f4:59:97:a9
        inet6 fe80::be5f:f4ff:fe59:97a9%re0 prefixlen 64 scopeid 0x7
        inet 192.168.15.26 netmask 0xffffff00 broadcast 192.168.15.255
        inet 10.10.10.101 netmask 0xffffffff broadcast 10.10.10.101
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

# jexec 2 csh

webserver# ifconfig
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
        ether bc:5f:f4:59:97:a9
        inet 10.10.10.101 netmask 0xffffffff broadcast 10.10.10.101
ifconfig: socket(AF_INET6, SOCK_DGRAM): Protocol not supported
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
ifconfig: socket(AF_INET6, SOCK_DGRAM): Protocol not supported

webserver# ping www.google.com
ping: cannot resolve www.google.com: Host name lookup failure

Is there something I've missed?

mamalos · #2 November 26th, 2012, 16:58

Most probably you're not NAT-ing on your external interface, so the jail tries to ping using IPv4 address 10.10.10.101, the packet leaves your host's external interface and cannot be routed back, since nobody knows how to reply to 10.10.10.101 (except if your host machine is the gateway for the machine you're trying to ping, or the machine knows a route to 10.10.10.101/32, which I doubt). In short: Do you NAT on your external interface? If not, you may find difficulties pinging from inside the jail.

kr651129 · #3 November 26th, 2012, 17:04

I'm going to assume I don't NAT on the external interface since sysinstall setup the network for me when I first installed the system.

chatwizrd · #4 November 26th, 2012, 19:00

If you cant resolve the hostname then do you have your nameservers in /etc/resolv.conf

kr651129 · #5 November 26th, 2012, 19:18

I have nameservers in /etc/resolv.conf on the host, but they are the defaults that were setup at the first install.

jaxxed · #6 November 26th, 2012, 19:23

This is a common networking problem in jails.

From inside the host, what do you get from this:
$/> netstan -rn

You should get a list of all the routing rules for your system, including the IP on the jail.is there a routing rule that passes the jail net traffice to your network?

A common implementation is to NAT your jails to you external IP, or to pick jail ips that make more sense on your host systen network. Many of us are using the loopback approach, where we creat a loopbakc device lo1 just for jails, put all the jails the itheir own subnet(s) on the loopback intercae and then NAT them to the external. Additionally you can alias any jail on your external interface. This approach gives lt's of firewalling/routing options inside the jail setup, jail to host and jail to network. It also works on laptops, with changing ips.

kr651129 · #7 November 26th, 2012, 19:35

jaxxed,
Thanks for the reply. Here's my output

Code:

> netstat -rn
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.15.1       UGS         0   219743    re0
10.10.10.101       link#7             UHS         0      254    lo0 =>
10.10.10.101/32    link#7             U           0        0    re0
127.0.0.1          link#10            UH          0       75    lo0
192.168.15.0/24    link#7             U           0    67276    re0
192.168.15.26      link#7             UHS         0        0    lo0

Internet6:
Destination                       Gateway                       Flags      Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               ::1                           UH          lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%re0/64                     link#7                        U           re0
fe80::be5f:f4ff:fe59:97a9%re0     link#7                        UHS         lo0
fe80::%lo0/64                     link#10                       U           lo0
fe80::1%lo0                       link#10                       UHS         lo0
ff01::%re0/32                     fe80::be5f:f4ff:fe59:97a9%re0 U           re0
ff01::%lo0/32                     ::1                           U           lo0
ff02::/16                         ::1                           UGRS        lo0
ff02::%re0/32                     fe80::be5f:f4ff:fe59:97a9%re0 U           re0
ff02::%lo0/32                     ::1                           U           lo0

Do you have any good documentation/howto's that you can link me to for any of implementations you suggested?

jaxxed · #8 November 27th, 2012, 09:09

I guess it depends on what jail system you're using. I'm using the PCBSD warden system, but you're likely using the EZJail system.

Check this link:
http://kbeezie.com/freebsd-jail-single-ip/

The nice thing about this approach is that once you have it set up, it works for all jails automatically, even with network changes etc. It becomes low hassle when adding/removing jails.

I treat all of my jails as a part of a host only internal network on lo1, and I nat that traffic out to the internet through my main interface. If I want a jail to have an external interface, then I manually pick an external interface IP and modify the jail to add the interface:
$/> jail -m jid=# ip4.addr="{internal_ip}" ip4.addr="{new_external_ip}
(I actually have modified the warden scripts for this)

One note is that I don't think that you need to manually specifiy your external IP in the pf.conf, as you can specify the interface and use ${ext_if}:network instead. This allows me to move my machine to new networks without worrying about having to adapt my firewall/router everytime.

The current weakness of my approach are:
zeroconf, which doesn't behave well. Firstly it binds to the most recent IP/alias of an interface, so adding IPs throws everything off, and secondly only one instance of avahi-daemon can bind to an interface, so either the host, or one jail can do it externally, and internally.
Hmmm - there was one other that I can't remember now.

I use this scenario on a dell inspiron notebook, that I use for web developemnt. I have individual jails set up for client web server development environments. I have additionally one jail as a mysql server, that the others can see. I manually bind a client jail to my external interface if I want to share the server with other developers.
I map my user home folders over (one set of source on all machines) to the jails, an then symlink source folders into /usr/local/www ... folders, but could use homedir stuff. Because of this I maintain a user uid map between the jails (which warden does automatically.)

In a less personal space I would consider using the same approach, perhaps without the portjail concept, but sticking to internal network for internal space (connecting web server to db server) and then either RDR external server traffic, or map external IPS. I like the network separation, and control. It maps really well will network control and monitoring.

mamalos · #9 November 27th, 2012, 09:46

Guys, I really don't understand why you are stretching this thread...

@OP: NAT!!!! Keep the rest of your configuration as is and add the following to the configuration files mentioned:

In your /etc/rc.conf:

Code:

pf_enable="YES"
pflog_enable="YES"

Create /etc/pf.conf with your "favorite editor", set its permissions 0700 and write these lines inside it:

Code:

ext_if="re0"
ext_ip="192.168.15.26"
jail_ip="10.10.10.101"

set skip on lo0

nat pass on $ext_if -> $ext_ip

pass out quick on $ext_if inet from { $ext_ip, $jail_ip } to any keep state

And either restart your machine or give:

# /etc/rc.d/pf start && /etc/rc.d/pflog start

on your command line. This way you will enable a firewall that allows ONLY all traffic originating from either one of your local IPv4 addresses (your host's and your jail's IPv4 addresses) that is destined to any machine outside of your external interface, statefully. For more information on how to build a firewall with PF read this and consult this forum's firewall section.

As mentioned previously, make sure that your jail's /etc/resolv.conf contains your reachable namservers. If the above contents don't work (I haven't tested them...), replace the last line with this one:

Code:

pass all

CAUTION: this will make your firewall to accept all traffic! Of course, if you haven't setup a firewall on your box, then this is already your case

.

gqgunhed · #10 November 27th, 2012, 12:22

some more things to consider:

1.) Inside the jail, check the contents of your /etc/resolv.conf. Please add any nameservers here if needed, see your /etc/resolv.conf in the host-system (outside of the jail). Caution: These may change if set by your ISP via DHCP.

2.) Ping is not allowed AFAIK because raw sockets are disabled for security reason by default. Check this via:

# sysctl security.jail.allow_raw_sockets

If you want to test connectivity of the jail by pinging, you must allow_raw_sockets. But try dig or fetch(1) to test connectivity (by IP and DNS-names) and isolate the cause.

kr651129 · #11 December 3rd, 2012, 00:11

Quote:

Originally Posted by jaxxed

I guess it depends on what jail system you're using. I'm using the PCBSD warden system, but you're likely using the EZJail system.

Check this link:
http://kbeezie.com/freebsd-jail-single-ip/

The nice thing about this approach is that once you have it set up, it works for all jails automatically, even with network changes etc. It becomes low hassle when adding/removing jails.

I treat all of my jails as a part of a host only internal network on lo1, and I nat that traffic out to the internet through my main interface. If I want a jail to have an external interface, then I manually pick an external interface IP and modify the jail to add the interface:
$/> jail -m jid=# ip4.addr="{internal_ip}" ip4.addr="{new_external_ip}
(I actually have modified the warden scripts for this)

One note is that I don't think that you need to manually specifiy your external IP in the pf.conf, as you can specify the interface and use ${ext_if}:network instead. This allows me to move my machine to new networks without worrying about having to adapt my firewall/router everytime.

The current weakness of my approach are:
zeroconf, which doesn't behave well. Firstly it binds to the most recent IP/alias of an interface, so adding IPs throws everything off, and secondly only one instance of avahi-daemon can bind to an interface, so either the host, or one jail can do it externally, and internally.
Hmmm - there was one other that I can't remember now.

I use this scenario on a dell inspiron notebook, that I use for web developemnt. I have individual jails set up for client web server development environments. I have additionally one jail as a mysql server, that the others can see. I manually bind a client jail to my external interface if I want to share the server with other developers.
I map my user home folders over (one set of source on all machines) to the jails, an then symlink source folders into /usr/local/www ... folders, but could use homedir stuff. Because of this I maintain a user uid map between the jails (which warden does automatically.)

In a less personal space I would consider using the same approach, perhaps without the portjail concept, but sticking to internal network for internal space (connecting web server to db server) and then either RDR external server traffic, or map external IPS. I like the network separation, and control. It maps really well will network control and monitoring.

Thanks! I'd like to take this approach and I followed the instructions. I can ping the jail now (192.168.0.1) but I still don't have any external access.

Code:

# jls
   JID  IP Address      Hostname                      Path
     1  192.168.0.1     apache                        /usr/jails/apache

/etc/resolv.conf (from inside the jail)

Code:

nameserver 8.8.8.8
nameserver 8.8.4.4

/etc/pf.conf (from the host)

Code:

IP_PUB="192.168.15.26"
IP_JAIL="192.168.0.2"
NET_JAIL="192.168.0.0/24"
PORT_JAIL="{80,443,2020}"
scrub in all
nat pass on re1 from $NET_JAIL to any -> $IP_PUB
rdr pass on re1 proto tcp from any to $IP_PUB port $PORT_JAIL -> $IP_JAIL

/etc/rc.conf (host)

Code:

# -- sysinstall generated deltas -- # Sat Dec  1 10:36:25 2012
ifconfig_re0="DHCP"
ipv6_activate_all_interfaces="YES"
hostname="host"

sshd_enable="YES"
ezjail_enable="YES"

cloned_interfaces="lo1"
ipv4_addrs_lo1="192.168.0.1-9/29"

ifconfig (from jail)

Code:

apache# ifconfig
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
        ether bc:5f:f4:59:97:a9
ifconfig: socket(AF_INET6, SOCK_DGRAM): Protocol not supported
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
ifconfig: socket(AF_INET6, SOCK_DGRAM): Protocol not supported
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=3<RXCSUM,TXCSUM>
        inet 192.168.0.2 netmask 0xffffffff
ifconfig: socket(AF_INET6, SOCK_DGRAM): Protocol not supported

Edit:
I can ping any networked device except for the router 192.168.15.1 which is what the host's default name server is.

**SirDice** · #12 December 3rd, 2012, 16:27

Quote:

Originally Posted by kr651129

/etc/pf.conf (from the host)

Code:

nat pass on re1 from $NET_JAIL to any -> $IP_PUB
rdr pass on re1 proto tcp from any to $IP_PUB port $PORT_JAIL -> $IP_JAIL

You're NAT'ing on re1, which doesn't exist on your system. You do have an re0.

mamalos · #13 December 4th, 2012, 10:14

Bravo and thanx SirDice,

I've been trying to tell this to the OP from day 1 and included the code to do this on day 2...then I stopped believing there is a point in that

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
[Solved] Ping from inside ezjail failed	Werner	Networking	9	January 6th, 2013 06:22
VirtualBox inside a jail	lockdoc	General	1	February 23rd, 2012 21:23
mfs (tmp) inside a jail?	mamalos	General	7	March 17th, 2010 22:22
DNS inside jail	gpatrick	Web & Network Services	0	November 1st, 2009 04:43
ezjail and zfs from inside the jail	wonslung	General	9	June 2nd, 2009 00:05

The Following User Says Thank You to SirDice For This Useful Post:
mamalos (December 4th, 2012)