Why does my jail not resolve hostnames?

[osx-addict]

Ok.. I've got my first real jail setup but find it can only ping the host interface -- other numbered IP addresses fail, name lookups fail,etc... Any ideas? I do not have any firewall running (that I'm aware of).. Do I need something to route these requests properly or is it something else?

Below are the relevant config settings:

Host "rc.conf":

# -- sysinstall generated deltas -- # Sun Mar 1 08:55:26 2009
# Created: Sun Mar 1 08:55:26 2009
# Enable network daemons for user convenience.
# Please make all changes to this file, not to /etc/defaults/rc.conf.
# This file now contains just the overrides from /etc/defaults/rc.conf.
defaultrouter="10.0.1.254"
hostname="srv1.mydomain.com"
ifconfig_fxp0="inet 10.0.1.198 netmask 255.255.254.0"
linux_enable="NO"
sshd_enable="NO"
rpcbind_enable="NO"
tcp_extensions="YES"
clear_tmp_enable="YES"

# -- sysinstall generated deltas -- # Sun Mar 1 17:35:09 2009
sendmail_enable="NO"
cyrus_pwcheck_enable="NO"
cyrus_imapd_enable="NO"
saslauthd_enable="NO"
spamass_milter_enable="NO"
spamd_enable="NO"
spamd_flags="-c -u nobody"
clamav_clamd_enable="NO"
clamav_milter_enable="NO"
clamav_freshclam_enable="NO"

ntpdate_flags="north-america.pool.ntp.org"
ntpdate_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.0.1.198"
syslogd_flags="-a 10.0.1.198"

# Jail items...
jail_enable="YES"
jail_set_hostname_allow="NO"
jail_list="jailbase"
jail_socket_unixproute_only="YES"
jail_interface="fxp0"
jail_procfs_enable="YES"

# Jailbase
jail_jailbase_rootdir="/usr/jails/jailbase"
jail_jailbase_hostname="jailbase.mydomain.com"
jail_jailbase_ip="192.168.10.30"
jail_jailbase_devfs_enable="YES"
jail_jailbase_exec_start="/bin/sh /etc/rc"

ifconfig_vr0_alias0="inet 192.168.10.30 netmask 255.255.255.0"


Here's the jailbase version of rc.conf:

# -- sysinstall generated deltas -- # Sun Mar 1 08:55:26 2009
# Created: Sun Mar 1 08:55:26 2009
# Enable network daemons for user convenience.
# Please make all changes to this file, not to /etc/defaults/rc.conf.
# This file now contains just the overrides from /etc/defaults/rc.conf.

ifconfig_fxp0="inet 192.168.10.30 netmask 255.255.255.0"
defaultrouter="10.0.1.254"

sshd_enable="NO"
#network_interfaces=""
sendmail_enable="NO"

rpcbind_enable="NO"
clear_tmp_enable="YES"

syslogd_enable="YES"
syslogd_flags="-ss"


Here's my jailbase resolv.conf:

nameserver 208.67.222.222
nameserver 208.67.220.220

[plamaiziere]

Ok.. I've got my first real jail setup but find it can only ping the host interface -- other numbered IP addresses fail, name lookups fail,etc... Any ideas? I do not have any firewall running (that I'm aware of).. Do I need something to route these requests properly or is it something else?


I think you should add gateway_enable="YES" in rc.conf.
or by command line : sysctl net.inet.ip.forwarding=1

[osx-addict]

I tried doing the sysctl line from above on the host side and it didn't seem to make any difference.. Any other ideas?

[osx-addict]

If this helps...(this is done within the jail)

jailbase# netstat -rn
netstat: kvm not available: /dev/mem: No such file or directory
Routing tables
rt_tables: symbol not in namelist

[SirDice]

ifconfig_fxp0="inet 192.168.10.30 netmask 255.255.255.0"
defaultrouter="10.0.1.254"
The default gateway is outside of the jail's subnet.

[osx-addict]

Ok.. I updated the IP for the jail to be 10.0.1.200 with the host interface having an address of 10.0.1.198.. I'm able to ping anything on my local network now but nothing outside yet. If I try to ping my external DNS servers I get nothing...


jailbase# ping 200.67.222.222
PING 200.67.222.222 (200.67.222.222): 56 data bytes
^C
--- 200.67.222.222 ping statistics ---
6 packets transmitted, 0 packets received, 100.0% packet loss
jailbase#


Here's the jail's rc.conf :


ifconfig_fxp0="inet 10.0.1.200 netmask 255.255.254.0"
defaultrouter="10.0.1.254"

sshd_enable="NO"
#network_interfaces=""
sendmail_enable="NO"

rpcbind_enable="NO"
clear_tmp_enable="YES"

syslogd_enable="YES"
syslogd_flags="-ss"


Any other ideas...?

[SirDice]

Make sure the traffic from the jail is NATted on your gateway.

[anomie]

Now that osx-addict's host and jail are on the same subnet, NAT is not required.

In your host's /etc/rc.conf, you should change the alias to:
ifconfig_vr0_alias0="inet 10.0.1.200 netmask 255.255.255.255"

Finally, in the jail's /etc/rc.conf there is no reason to have an ifconfig directive.

[SirDice]

Now that osx-addict's host and jail are on the same subnet, NAT is not required.
You will if you want to talk to the outside (internet). AFAIK osx-addict's DNS servers are on the Internet. The NAT is done on the gateway of course.

Finally, in the jail's /etc/rc.conf there is no reason to have an ifconfig directive.
I have an empty interfaces directive in my jail's rc.conf to stop the warnings.

interfaces=""

[anomie]

I see what you mean. Yes, his gateway will have to provide NAT for the 10.0.1/23 network. (But his host system should not be acting as the jail's gateway, as was implied early on in the thread.)

[osx-addict]

Wow.. Thanks for the help guys! Unfortunately I can't try any of this out as I'm now at work.. I'll have to wait until this evening.

In the meantime, I can fill in a few holes in case it changes any answers.


My NAT'ing is being done by my Cisco router.. You're correct that it's wanting to see things in the 10.0.1.x address range. I stupidly re-used the 192.168.x.x values thinking it wouldn't matter.. Oh well.
I did change the inet alias in the hosts' rc.conf to update the jail's IP to 10.0.1.200 -- I forgot to mention that. Now, do I need to reboot for that to take effect? I did an "ifconfig -a" on the host side and didn't see any alias listed anymore (as I recall) -- but it was there last night after I put it in by hand -- perhaps the jail start does something with it behind my back?
On the topic of these "N" aliases (e.g. ifconfig_vr0_alias0) I usually see in tutorials.. How do they get allocated? Is the one labeled "0" provided to the first jail started,etc? Is there any rhyme or reason to the naming? Should the above "vr0" be "fxp0" to match my hosts' network port name?
So -- I can remove the ifconfig directive from the host jail -- should I replace it with network_interfaces="" instead?


Sorry.. I feel a bit like a fish out of water on some of this.. Thanks!

[SirDice]

2&3: The jail_*_ip directive in rc.conf takes care of that. The alias will be added/removed when the jail is started/stopped.

4: Yes and yes. That last one will stop warnings from appearing when you start the jail.

You can start/stop jails using /etc/rc.d/jail start and /etc/rc.d/jail stop. You can add the jail's name (jailbase in your case) at the end to stop/start just one.

[osx-addict]

You can start/stop jails using /etc/rc.d/jail start and /etc/rc.d/jail stop. You can add the jail's name (jailbase in your case) at the end to stop/start just one.

Thanks.. That's actually the way I've been starting/stopping them -- works like a charm.

I'll update things tonight (unless I go home at lunch) and see how it goes and report back..

[anomie]

To add to the previous comments:

I did change the inet alias in the hosts' rc.conf to update the jail's IP to 10.0.1.200 -- I forgot to mention that.

You should also change the netmask for the alias to 255.255.255.255. From the ifconfig(8) manpages:
alias

Establish an additional network address for this interface. This
is sometimes useful when changing network numbers, and one wishes
to accept packets addressed to the old interface. If the address
is on the same subnet as the first network address for this
interface, a non-conflicting netmask must be given. Usually
0xffffffff is most appropriate.

Now, do I need to reboot for that to take effect? I did an "ifconfig -a" on the host side and didn't see any alias listed anymore

AFAIK, /etc/rc.d/netif restart should do it.

On the topic of these "N" aliases (e.g. ifconfig_vr0_alias0) I usually see in tutorials.. How do they get allocated? Is the one labeled "0" provided to the first jail started,etc? Is there any rhyme or reason to the naming? Should the above "vr0" be "fxp0" to match my hosts' network port name?

Yes, the alias directive should reflect the name of the network device (fxp0 in your case) that you're creating the alias on.

The first alias should use 0, the second 1, the third 2, etc. Example:
ifconfig_xl0_alias0="inet 10.1.1.51 netmask 255.255.255.255"
ifconfig_xl0_alias1="inet 10.1.1.55 netmask 255.255.255.255"
ifconfig_xl0_alias2="inet 10.1.1.59 netmask 255.255.255.255"

[osx-addict]

Yes, the alias directive should reflect the name of the network device (fxp0 in your case) that you're creating the alias on.

The first alias should use 0, the second 1, the third 2, etc. Example:
ifconfig_xl0_alias0="inet 10.1.1.51 netmask 255.255.255.255"
ifconfig_xl0_alias1="inet 10.1.1.55 netmask 255.255.255.255"
ifconfig_xl0_alias2="inet 10.1.1.59 netmask 255.255.255.255"

Thanks.. I thinking about it more, I guess it doesn't really matter what order the aliases are in -- as long as they exist and each jail (which needs one) gets the respective alias applied to the host's ethernet interface which in turn will do the mapping and allow access out.

Just to refresh my brain.. When I get home I think I want to do the following tasks based on the help gathered from above :


Change the netmasks to 255.255.255.255 (e.g. 0xffffffff) for the host-side aliases defined in /etc/rc.conf
Remove 'ifconfig' references in any jail' rc.conf file.
Add 'network_interfaces=""' in any jail' rc.conf file. (to quiet up startup complaints)
Do I still need the "defaultrouter=x.y.z.a" line in the jailed rc.conf file?


I think that's about it..

[SirDice]

The order of aliases and jails doesn't really matter indeed (alias0 can be tied to jail id 6 i.e.).

1) yes
2) yes
3) yes
4) no, only one on the host.

addendum 1, if you use the jail_*_ip variable you don't need the ifconfig_*_alias[012] settings. The alias will be set on the fly when the jail starts.

If you do use the ifconfig_*_alias[0123] you will need to use the /etc/rc.d/netif script to make the aliases appear.

addendum 4, IIRC each jail can now have it's own routing table. This wasn't the case when I started playing with jails. So in theory you can have the jail in a 192.168.1/24 network with a 192.168.1/24 gateway. Mental note: must play with the new jail network toys :e

[osx-addict]

Ok.. We're making progress.. I'm now able to ping the outside world from within the jail.. However, name lookups are still horked up.. See below..


jailbase# cd etc
jailbase# more resolve.conf
nameserver 208.67.222.222
nameserver 208.67.220.220

jailbase# ping 208.67.222.222
PING 208.67.222.222 (208.67.222.222): 56 data bytes
64 bytes from 208.67.222.222: icmp_seq=1 ttl=51 time=61.621 ms
64 bytes from 208.67.222.222: icmp_seq=2 ttl=51 time=63.812 ms
64 bytes from 208.67.222.222: icmp_seq=3 ttl=51 time=60.774 ms
64 bytes from 208.67.222.222: icmp_seq=4 ttl=51 time=64.109 ms
^C
--- 208.67.222.222 ping statistics ---
5 packets transmitted, 4 packets received, 20.0% packet loss
round-trip min/avg/max/stddev = 60.774/62.579/64.109/1.417 ms
jailbase# ping yahoo.com
ping: cannot resolve yahoo.com: Host name lookup failure
jailbase# nslookup
> google.com
;; connection timed out; no servers could be reached


Here are my relevant config files :

first is the jailed rc.conf:

network_interfaces=""

sshd_enable="NO"
sendmail_enable="NO"

rpcbind_enable="NO"
clear_tmp_enable="YES"

syslogd_enable="YES"
syslogd_flags="-ss"


The next is the host version of the rc.conf file :


# -- sysinstall generated deltas -- # Sun Mar 1 08:55:26 2009
# Created: Sun Mar 1 08:55:26 2009
# Enable network daemons for user convenience.
# Please make all changes to this file, not to /etc/defaults/rc.conf.
# This file now contains just the overrides from /etc/defaults/rc.conf.
defaultrouter="10.0.1.254"
hostname="srv1.mydomain.com"
ifconfig_fxp0="inet 10.0.1.198 netmask 255.255.254.0"
linux_enable="NO"
sshd_enable="NO"
rpcbind_enable="NO"
tcp_extensions="YES"
clear_tmp_enable="YES"

# -- sysinstall generated deltas -- # Sun Mar 1 17:35:09 2009
sendmail_enable="YES"
cyrus_pwcheck_enable="NO"
cyrus_imapd_enable="NO"
saslauthd_enable="YES"
spamass_milter_enable="YES"
spamd_enable="YES"
spamd_flags="-c -u nobody"
clamav_clamd_enable="YES"
clamav_milter_enable="YES"
clamav_freshclam_enable="YES"

ntpdate_flags="north-america.pool.ntp.org"
ntpdate_enable="YES"
inetd_enable="YES"
inetd_flags="-wW -a 10.0.1.198"
syslogd_flags="-a 10.0.1.198"

# Jail items...
jail_enable="YES"
jail_set_hostname_allow="NO"
jail_list="jailbase"
#jail_socket_unixiproute_only="YES"
jail_interface="fxp0"
jail_procfs_enable="YES"

# Jailbase
jail_jailbase_rootdir="/usr/jails/jailbase"
jail_jailbase_hostname="jailbase.mydomain.com"
jail_jailbase_ip="10.0.1.200"
jail_jailbase_devfs_enable="YES"
jail_jailbase_exec_start="/bin/sh /etc/rc"


Any ideas?

[osx-addict]

Mental note.. Never copy files and specify their names by hand..

It turns out that my /etc/resolv.conf was actually named /etc/resolve.conf.

I found the following when using truss to diagnose the ping failures and this made me know what the problem was:


jailbase# truss ping yahoo.com > & t.t
jailbase# grep open t.t
open("/etc/libmap.conf",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/var/run/ld-elf.so.hints",O_RDONLY,00) = 3 (0x3)
open("/lib/libm.so.5",O_RDONLY,00) = 3 (0x3)
open("/lib/libipsec.so.3",O_RDONLY,027757765354) = 3 (0x3)
open("/lib/libc.so.7",O_RDONLY,027757765354) = 3 (0x3)
open("/etc/resolv.conf",O_RDONLY,0666) ERR#2 'No such file or directory'
open("/etc/nsswitch.conf",O_RDONLY,0666) = 4 (0x4)
open("/etc/hosts",O_RDONLY,0666) = 4 (0x4)


The fix is below.. How stupid of me!!!!

jailbase# cd /etc
jailbase# mv resolve.conf resolv.conf
jailbase# ping yahoo.com
;; res_setoptions(" debug
", "conf")..
;; debug
;; res_init()... default dnsrch list:
;; ca-flower.com
;; ..END..
;; res_nquerydomain(yahoo.com, <Nil>, 1, 1)
;; res_query(yahoo.com, 1, 1)
;; res_nmkquery(QUERY, yahoo.com, IN, A)
;; res_send()
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58491
;; flags: rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; yahoo.com, type = A, class = IN
;; Querying server (# 1) address = 66.75.164.90
;; new DG socket
;; got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58491
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 7, ADDITIONAL: 7
;; yahoo.com, type = A, class = IN
yahoo.com. 4h15m23s IN A 206.190.60.37
yahoo.com. 4h15m23s IN A 68.180.206.184
yahoo.com. 1d23h59m38s IN NS ns6.yahoo.com.
yahoo.com. 1d23h59m38s IN NS ns3.yahoo.com.
yahoo.com. 1d23h59m38s IN NS ns2.yahoo.com.
yahoo.com. 1d23h59m38s IN NS ns1.yahoo.com.
yahoo.com. 1d23h59m38s IN NS ns8.yahoo.com.
yahoo.com. 1d23h59m38s IN NS ns5.yahoo.com.
yahoo.com. 1d23h59m38s IN NS ns4.yahoo.com.
ns2.yahoo.com. 1d23h57s IN A 68.142.255.16
ns6.yahoo.com. 1d23h20m5s IN A 202.43.223.170
ns5.yahoo.com. 1d22h25m28s IN A 119.160.247.124
ns1.yahoo.com. 1d23h1m27s IN A 68.180.131.16
ns3.yahoo.com. 1d23h40s IN A 217.12.4.104
ns4.yahoo.com. 1d23h46m28s IN A 68.142.196.63
ns8.yahoo.com. 1d22h36m40s IN A 202.165.104.22
PING yahoo.com (206.190.60.37): 56 data bytes
64 bytes from 206.190.60.37: icmp_seq=0 ttl=51 time=93.212 ms
64 bytes from 206.190.60.37: icmp_seq=1 ttl=51 time=90.645 ms
64 bytes from 206.190.60.37: icmp_seq=2 ttl=51 time=94.420 ms
^C
--- yahoo.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 90.645/92.759/94.420/1.574 ms


I've got debugging enabled in the /etc/resolv.conf which is why the above is verbose..

Thanks everyone for the help & persistance in solving this silly problem.. I did learn a lot about jails though -- a good thing!