Internet access from jails

[rinalds]

I'm using jails to isolate Apache virtual hosts, which works great. The only problem is that internet access from the jails does not work well. For example:

jail# telnet freebsd.org 80
...spends about 10 seconds to resolve the name...
Trying 69.147.83.40...
...spends about a minute here...
telnet: connect to address 69.147.83.40: Operation timed out

However, the strange thing is that sometimes it manages to connect after some 5-10 seconds. Connection to server's public IP address works without problems.

If I try to run this through truss, I get this:

....
__sysctl(0x7fffffffe7c0,0x4,0x80152e0f0,0x7fffffffe868,0x0,0x0) = 0 (0x0)
socket(PF_INET6,SOCK_DGRAM,17) ERR#43 'Protocol not supported'
socket(PF_INET,SOCK_DGRAM,17) = 3 (0x3)
connect(3,{ AF_INET 69.147.83.40:1 },16) = 0 (0x0)
getsockname(3,{ AF_INET 10.0.0.5:63056 },0x7fffffffe868) = 0 (0x0)
close(3) = 0 (0x0)
fstat(1,{ mode=crw--w---- ,inode=101,size=0,blksize=4096 }) = 0 (0x0)
ioctl(1,TIOCGETA,0xffffdf40) = 0 (0x0)
Trying 69.147.83.40...
write(1,"Trying 69.147.83.40...\n",23) = 23 (0x17)
socket(PF_INET,SOCK_STREAM,6) = 3 (0x3)
getuid(0x2,0x1,0x6,0x80139baac,0xffffffff80ac3f40,0x7fffffffe8a8) = 0 (0x0)
setuid(0x0,0x1,0x6,0x8013263bc,0xffffffff80ac4040,0x7fffffffe8a8) = 0 (0x0)
setsockopt(0x3,0x0,0x3,0x5199e0,0x4,0x7fffffffe8a8) = 0 (0x0)
>> stucks here >> connect(3,{ AF_INET 69.147.83.40:80 },16) ERR#60 'Operation timed out'
telnet: write(2,"telnet: ",8) = 8 (0x8)
connect to address 69.147.83.40write(2,"connect to address 69.147.83.40",31) = 31 (0x1f)
: write(2,": ",2) = 2 (0x2)

Jail's /etc/rc.conf:

hostname="support.livecart.com"
network_interfaces=""

Host's /etc/rc.conf:

defaultrouter="67.220.195.137"
hostname="localhost.localdomain"
ifconfig_fxp0="inet 67.220.195.138 netmask 255.255.255.248"
ifconfig_fxp0_alias0="inet 67.220.195.139 netmask 255.255.255.255"

cloned_interfaces="lo1"
ifconfig_lo1="inet 10.0.0.254 netmask 255.255.255.0"
ifconfig_lo1_alias0="inet 10.0.0.5 netmask 255.255.255.255"

gateway_enable="YES"
pf_enable="YES"
pf_rules="/etc/pf.conf"

# Jails
jail_set_hostname_allow="YES"
jail_enable="YES"
jail_devfs_enable="YES"
jail_procfs_enable="YES"
jail_sysvipc_allow="YES"
jail_socket_unixiproute_only="YES"

ezjail_enable="YES"

/etc/pf.conf:

nat on fxp0 from lo1:network to any -> (fxp0)

/etc/resolv.conf (same for jail and host):

nameserver 206.251.73.9
nameserver 4.2.2.1

sysctl -a | grep jail

security.jail.jailed: 0
security.jail.mount_allowed: 0
security.jail.chflags_allowed: 0
security.jail.allow_raw_sockets: 1
security.jail.enforce_statfs: 2
security.jail.sysvipc_allowed: 1
security.jail.socket_unixiproute_only: 1
security.jail.set_hostname_allowed: 1

Any suggestions would be appreciated.

 

[anomie]

Any suggestions would be appreciated.

Run tcpdump(8) in a separate terminal (from inside the jail) to try to determine where it is lagging. Is it taking a long time to send the SYN packet? Is it sending the SYN right away but waiting a long time to receive the SYN-ACK? Or something else?

# tcpdump port 80

... and then:

# nc -zvw 1 [url]http://www.freebsd.org[/url] 80

Post the results here in code tags, and try to explain at which stages in the output the lagging occurred.

(P.S. telnet is just fine, but I strongly prefer nc or nmap for this sort of testing, which I why I put it in my example.)

 

[SirDice]

Can you explain why you're using lo1 and NAT? Why not bind the jail directly to the external IP?

 

[rinalds]

Thanks for your help Strangely, I'm getting mixed results.

Attempt 1:

jail# nc -zvw 1 http://www.freebsd.org 80
nc: connect to http://www.freebsd.org port 80 (tcp) failed: Operation timed out

host# tcpdump port 80 | grep freebsd
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on fxp0, link-type EN10MB (Ethernet), capture size 96 bytes
14:50:32.192577 IP 192.168.0.1.58470 > http://www.freebsd.org.http: S 2214706377:2214706377(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 85318051 0>
14:50:35.235645 IP 192.168.0.1.58470 > http://www.freebsd.org.http: S 2214706377:2214706377(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 85321051 0>
14:50:38.486750 IP 192.168.0.1.58470 > http://www.freebsd.org.http: S 2214706377:2214706377(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 85324251 0>
14:50:41.753038 IP 192.168.0.1.58470 > http://www.freebsd.org.http: S 2214706377:2214706377(0) win 65535 <mss 1460,sackOK,eol>
14:50:45.006141 IP 192.168.0.1.58470 > http://www.freebsd.org.http: S 2214706377:2214706377(0) win 65535 <mss 1460,sackOK,eol>
14:50:48.238330 IP 192.168.0.1.58470 > http://www.freebsd.org.http: S 2214706377:2214706377(0) win 65535 <mss 1460,sackOK,eol>
14:50:54.524275 IP 192.168.0.1.58470 > http://www.freebsd.org.http: S 2214706377:2214706377(0) win 65535 <mss 1460,sackOK,eol>
14:51:06.867269 IP 192.168.0.1.58470 > http://www.freebsd.org.http: S 2214706377:2214706377(0) win 65535 <mss 1460,sackOK,eol>
14:51:31.434749 IP 192.168.0.1.58470 > http://www.freebsd.org.http: S 2214706377:2214706377(0) win 65535 <mss 1460,sackOK,eol>

Attempt 2:

jail# nc -zvw 1 http://www.freebsd.org 80
Connection to http://www.freebsd.org 80 port [tcp/http] succeeded!

host# tcpdump port 80 | grep freebsd
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on fxp0, link-type EN10MB (Ethernet), capture size 96 bytes
15:22:29.129832 IP 67-220-195-139.hosted.static.webnx.com.61577 > http://www.freebsd.org.http: S 1423380358:1423380358(0) win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 87208838 0>
15:22:29.139458 IP http://www.freebsd.org.http > 67-220-195-139.hosted.static.webnx.com.61577: S 2103184992:2103184992(0) ack 1423380359 win 65535 <mss 1460,nop,wscale 3,sackOK,timestamp 4251315625 87208838>
15:22:29.139480 IP 67-220-195-139.hosted.static.webnx.com.61577 > http://www.freebsd.org.http: . ack 1 win 8326 <nop,nop,timestamp 87208848 4251315625>
15:22:29.139623 IP 67-220-195-139.hosted.static.webnx.com.61577 > http://www.freebsd.org.http: F 1:1(0) ack 1 win 8326 <nop,nop,timestamp 87208848 4251315625>
15:22:29.149400 IP http://www.freebsd.org.http > 67-220-195-139.hosted.static.webnx.com.61577: . ack 2 win 8326 <nop,nop,timestamp 4251315635 87208848>
15:22:29.149412 IP http://www.freebsd.org.http > 67-220-195-139.hosted.static.webnx.com.61577: F 1:1(0) ack 2 win 8326 <nop,nop,timestamp 4251315635 87208848>
15:22:29.149423 IP 67-220-195-139.hosted.static.webnx.com.61577 > http://www.freebsd.org.http: . ack 2 win 8325 <nop,nop,timestamp 87208858 4251315635>

Sometimes it connects, but usually it doesn't. I noticed that the successful connect logged address resolved from the public IP (67-220-195-139.hosted.static.webnx.com), while the failed attempt showed 192.168.0.1. No configuration was changed in between the attempts. I tried adding 192.168.0.1 to /etc/hosts, but it didn't help.

Interestingly, when I try to ping the resolved address, I get this:

host# ping 67-220-195-139.hosted.static.webnx.com
ping: cannot resolve 67-220-195-139.hosted.static.webnx.com: Unknown host

Can you explain why you're using lo1 and NAT? Why not bind the jail directly to the external IP?

There are a lot more virtual hosts than I have public IPs available. I'm using nginx to proxy the http requests to Apache instances on local IPs.

 

[SirDice]

There are a lot more virtual hosts than I have public IPs available. I'm using nginx to proxy the http requests to Apache instances on local IPs.

Sounds reasonable :e

I have been looking at your configuration and I can't find anything relating to 192.168.0.1. It's not set in rc.conf?!?!?

What does ifconfig say inside the running jail?

Try adding jail_*_ip and jail_*_interface to /etc/rc.conf to make sure the jail's bound to the correct interface and ip address.

 

[rinalds]

Solved!

Ahh! 192.168.0.1 was indeed added to rc.conf as another alias for fxp0... I left it out from my initial post stupidly thinking it was not relevant. Removed it and everything seems to work finally :h

Many thanks again for helping to figure this out!