IPFW Jails with Proxy

P

perkypork

I have been trying to get this setup to work for a few days.

I have a single WAN interface (bce0) that has a couple of public IPs attached to it. I would like to use one of these IPs for the host (wanip1) and jails and another for a jail (wanip2) that requires a public IP. I have a cloned lo0 interface, lo1 with a subnet of 10.1.0.0/24 which all the jails reside on. I would like to forward ports 80,443 coming in on wanip1 to the jail proxy, which then forwards the traffic off to the correct jail. I would like the jails to be able to have their own firewall setups, which I currently do by including a firewall script writable by each jail. I would also like each jail to be able to connect to the internet.

I have been trying to follow this post here - https://forums.freebsd.org/threads/ipfw-nat-setting.46929/#post-262399 but this post is to do NAT on the interface not the IP. I cannot use the interface as there is another public IP attached to it. I realize that I should be able to make some small changes to the example in this post and get it to work but that doesn't seem to be the case. I got it to work when I followed exactly but it stopped working when I change the references from <wan interface> to <wan ip> (where I could).
 
Below is the firewall rules I think should work to achieve what I want above but this does not work. I would really appreciate it if someone gave me some pointers as this is the first time I have used IPFW NAT. I have not found a good explanation of how to use IPFW with in kernel NAT.

Code: 
#!/bin/sh

wif="bce0" # wan interface
jif="lo1" # jails interface
wip1="119.111.111.111" # wan ip 1
wip2="119.111.111.222" # wan ip 2

cmd="ipfw -q add"

ipfw -q -f flush

$cmd nat 1 config ip $wip same_ports reset \
redirect_port tcp 10.1.0.1:80 80 \
redirect_port tcp 10.1.0.1:443 443

# Allow anything within the LAN -- the interface with heaviest traffic shall come first
$cmd 10 allow ip from any to any via lo0
$cmd 11 allow ip from any to any via $jif

# NAT rule for incomming packets
$cmd 100 nat 1 ip from any to any via $wif in
$cmd 101 check-state

# Allow access to NAT redirected services listening on a LAN interface behind the NAT
$cmd 201 skipto 10000 tcp from any to any 80 via $wif in setup keep-state
$cmd 202 skipto 10000 tcp from any to any 443 via $wif in keep-state

# Allow all other outgoing connections
$cmd 2000 skipto 10000 tcp from any to any via $wif out setup keep-state
$cmd 2010 skipto 10000 udp from any to any via $wif out keep-state

# Jails firewalls are included here (these firewalls only restrict traffic on the jails private IP)

# NAT rule for outgoing packets
$cmd 10000 nat 1 ip from any to any via $wif out

# Deny Remainder and Log
$cmd deny log all from any to any
 
I have done some further research on this as I am under the pump to get it working correctly.

I found a couple of interesting things on Google:
- https://lists.freebsd.org/pipermail/freebsd-questions/2011-June/231271.html
The person replying to the initial question has gone to quite some trouble to explain how to construct an IPFW ruleset using kernel NAT, though I still do not quite understand. What I like most about this conversation is the poster explaining that its better to use xmit/recv than via on rule sets. This is the only example I have been able to find where xmit/recv is used instead of via.
- https://forums.freebsd.org/threads/ipfw-nat-setting.46929/#post-262399
This one is also quite good and shows what should be a working config, though it does not work for me.

I have found the handbook on IPFW and ipfw(8) to be particularly useless in getting a working config in place.

I have yet to find a IPFW using kernal NAT example where NAT is being done on the IP not the interface. I am not sure why the examples wouldn't be doing NAT on the IP as this would be more flexible - as it allows more than one public IP on the interface (should that occur).

I will continue to post my findings, until (hopefully) I get a working config - which might save someone else a few days.
 
Just jotting down a couple more things I have discovered in my searching:

1. I cloned the lo0 interface (lo1) for the local Jails interface. Traffic on cloned interfaces does not go over cloned interface, it goes over lo0. You cannot use tcpdump(1) on cloned interfaces, it will see no traffic.

2. If using IPFW and kernel NAT you should disable TSO (check with ifconfig bce0, look for TSO4 or TSO8) on the interface you are natting (https://www.freebsd.org/cgi/man.cgi?query=ipfw(8)#end). You can do this by setting -tso in /etc/rc.conf where you configure your interface - ifconfig_bce0="inet 119.111.222.333/27 -tso"

3. You must set net.inet.ip.fw.one_pass=0 in /etc/sysctl.conf
 
Well.....

After about 30 hours trying to get this to work, I gave up and moved over to pf(4). It took me about 2 hours to get a working pf firewall up and running (having never used it before) with about 100 or so rules. I was surprised at how easy it was to get NAT running in PF in comparison to IPFW. I would still really like to use IPFW as my firewall so I am still hoping that someone on the IPFW mailing list or these forums can help me out.
 
So I am still working on this. It took a couple hours to get my PF firewall script working perfectly, except for one problem, which took forever to solve. I couldn't work out why a Jail couldn't proxy pass (Nginx) to another jail. Each of my Jails can see each other but they don't talk to each other direct, they talk to a load balancer, which then sends requests off to the proxy servers, which then send requests off to the jails. If a jail is proxy passing something to another connection, then the jail needs to initiate a connection out the WAN IP, connect to the load balancer, which forwards the request off to a proxy, which then sends the request to a jail. Sometimes the Nginx and the proxy would show a private IP (from the jail subnet) in their logs, when it should have been a public IP. Took me a few hours of reading but pf(4) does not support NAT redirection (or hairpin NAT or loopback NAT). So I am now back to getting this working with IPFW with kernel NAT as it is supposedly supported. I guess I will find out soon enough.
 
Hi perky,

I am too working on to make the in kernel nat work with IPFW.
My setup is simple - jails on a cloned lo1 interface having private IPs and I am just trying to make the jails get access to internet.

Allowing access to jails from outside will be next.

I would suggest you to look at the way cbsd is doing nat with IPFW.
I intend to do this next.
 
You must log in or register to reply here.
Back
Top