jail with loopback IP, IPFW, and natd - Outbound connections fail

ChrisStone

ChrisStone

I've got a FreeBSD 9.0 server. It has several jails, but they all have this same issue. They can not initiate connections to the outside world. They communicate with each other and the host fine.

Relevant rc.conf settings:

Code: 
firewall_enable="YES"                   # IPFW
firewall_type="/etc/ipfw.rules"         # Rule script for IPFW

natd_enable="YES"                       # NAT for Internet Routing
natd_interface="wan0"                   # NAT Card
natd_flags="-f /etc/natd.conf -dynamic" # NAT Conf

ifconfig_lo1_name="jail1"
ifconfig_jail1="inet 192.168.1.101/32"

jail_asdf_rootdir="/jails/asdf"
jail_asdf_hostname="asdf.example.net"
jail_asdf_ip="192.168.1.101"
jail_asdf_devfs_enable="YES"

From sysctl.conf:

Code: 
security.jail.allow_raw_sockets=1

From ipfw.rules:

Code: 
# XXX 00050 divert natd ip4 from any to any via wan0
add 00060 check-state

# Allow me out
add 00135 allow ip from me to any keep-state
add 00136 allow ip6 from me6 to any keep-state

# HTTP
add 11010 allow tcp from any to me http setup keep-state
add 11011 allow tcp from any to me6 http setup keep-state
add 11012 allow tcp from any to me https setup keep-state
add 11013 allow tcp from any to me6 https setup keep-state
.... lots more rules like the above ....

# General Network - ICMP
add 61001 allow icmp from any to any

# XXX last rule is deny everything

From natd.conf:

Code: 
redirect_port tcp 192.168.1.101:80 80
redirect_port tcp 192.168.1.101:443 443

This works great for incoming connections. The I have tested from multiple computers, I can get to the website just fine.

When I [cmd=]jexec 1 csh[/cmd] to get a shell in the Jail I can't create an outgoing connection. The Jail's resolv.conf points to the host server, and it performs name resolution fine. With ICMP still being passed without exception I can ping from the Jail.

I can do a [cmd=]tcpdump -i wan0 host 1.2.3.4[/cmd] on the host and watch the traffic as it passes. I see the SYN go out, and a SYN ACK come back. Then a few seconds later the same again as the Jail retries.

I believe I understand the problem. The outgoing packet starts through the firewall rules, gets NAT translated, allowed out and recorded as the external IP making an outgoing connection (ie $wan0_ip -> 1.2.3.4). When the return packet comes back it goes through translation, but now doesn't match the check-state rule as the packet has the internal IP (ie 1.2.3.4 -> 192.168.1.101).

How can I allow outgoing connections from my jails?
 
Sorry guys, don't be mad: The above two posts were unhelpful, with the exception of the example configuration.

The key to the problem here, as noted in the original post, is that address translation was always occurring before the dynamic state rules were checked. Thus there were different apparently local addresses depending on the direction of the traffic. The answer should have been obvious, that translation needs to be split depending on direction.

A corrected example version of the ipfw.rules

Code: 
add 00050 divert natd ip4 from any to any via wan0 in
add 00060 check-state

# Talking to myself
add 00200 allow ip from me to me keep-state

# Let me out
add 00210 skipto 63000 tcp from any to any via wan0 out setup keep-state
add 00211 skipto 63000 ip  from any to any via wan0 out keep-state

# HTTP
add 11010 skipto 63000 tcp from any to me http,https setup keep-state
add 11011 skipto 63000 tcp from any to me6 http,https setup keep-state

# General Network - ICMP
add 61001 allow icmp from any to any

# Last rule of "normal" traffic
add 62000 deny ip from any to any

# Only for my outbound and specifically allowed incoming
add 63000 divert natd ip from any to any via wan0 out
add 63001 allow ip from any to any

# XXX default to deny everything

I'm sure this could be improved upon, but connections work as expected.
 
You must log in or register to reply here.
Back
Top