FreeBSD firewall: how to get "nat loopback" functionality?

tingo · #1 January 30th, 2012, 20:13

Hello,

I'm currently using ipfw+natd for my firewall. I have some web servers on my local network. They use DynDNS for the web site name(s), using the public IP address I get from my ISP and a dyndns client. This makes it vital to be able to access these web sites from my internal network using the public IP address. This is sometimes referred to as "nat loopback", (here is a quite good explanation of the issue).

I want to avoid using a split DNS, both to keep the admin / management down (a fancy way of saying "less work"), and also because it is easier to test if a web site "works" if I am using the public IP address (I can see at once if I have messed up a firewall rule, for instance).

How do I get a firewall built on FreeBSD to have "NAT loopback" functionality, so that I can access the web sites on my local network using the public IP address? Preferably with ipfw + natd, if possible.

Note: the the web sites work correctly if accessed from another place on the internet. I've just switched to a new ISP; my previous ISP had a router / modem box (ADSL) with this functionality. My new ISP (cable) only provides me with a modem (better in many ways, I don't have to do double NAT anymore).

rolfheinrich · #2 January 31st, 2012, 01:05

I am almost sure that ipfw+natd should give you "nat loopback" out of the box. However, I cannot prove this for now, because some time ago I did the switch from natd to in-Kernel NAT.

I read also your other "natd"-thread, and I understand that you want to stay with natd, and I won't argue against this.

My 2 cents are, that it is not that much effort to do the switch from natd to in-Kernel-NAT, and everything is working well, including "nat loopback". I just checked it on my installation, which seems to be quite similar to what you are aiming for.

tingo · #3 February 6th, 2012, 22:28

Quote:

Originally Posted by rolfheinrich

I am almost sure that ipfw+natd should give you "nat loopback" out of the box. However, I cannot prove this for now, because some time ago I did the switch from natd to in-Kernel NAT.

Yes, I thought so too. (Sorry for the late response, I've been busy with other things.)
I tried messing with my firewall rules today, but that didn't help.
I wish I had a tool that could be like traceroute, but for http traffic. That would make it easier to figure out where things are stopping.

Quote:

Originally Posted by rolfheinrich

My 2 cents are, that it is not that much effort to do the switch from natd to in-Kernel-NAT, and everything is working well, including "nat loopback". I just checked it on my installation, which seems to be quite similar to what you are aiming for.

Well, if I can't get ipfw+natd to work, I shall certainly try out this solution.

**DutchDaemon** · #4 February 6th, 2012, 23:59

Quote:

Originally Posted by tingo

traceroute, but for http traffic.

net/tcptraceroute
tcptraceroute somehost 80

rolfheinrich · #5 February 7th, 2012, 00:01

Quote:

Originally Posted by tingo

I wish I had a tool that could be like traceroute, but for http traffic.

Up to now, I understood, that you are talking about domains of virtual hosts, that are served by the same httpd server at the nat machine. In this case traceroute from a client machine in your local network should be sufficient for troubleshooting, and it would show you only 1 hop, for example something like the following:

Code:

$ traceroute mydyndom.ath.cx
traceroute to mydyndom.ath.cx (111.222.333.444), 64 hops max, 52 byte packets
 1  dynip.frommyisp.com (111.222.333.444)  1.027 ms  0.415 ms  0.395 ms

This is all, that you can expect from nat-loopback. If you get a similar result in your LAN, then nat-loopback is basically working. If not, then you might want to check your DNS resolver on the respective client, that means whether really all your dynamic domains resolve to your public IP.

If said websites are hosted by different machines in your local network, then it might be helpful to learn a little bit more about your setup, in this case, quite possibly you cannot achieve your objective with NAT, but you need to install a web proxy on the gateway machine.

tingo · #6 February 10th, 2012, 11:29

Quote:

Originally Posted by rolfheinrich

Up to now, I understood, that you are talking about domains of virtual hosts, that are served by the same httpd server at the nat machine. In this case traceroute from a client machine in your local network should be sufficient for troubleshooting, and it would show you only 1 hop, for example something like the following:

Except for one small detail: the natd (firewall) machine doesn't have a web server; the web server is "behind" the nat machine.

Quote:

Originally Posted by rolfheinrich

This is all, that you can expect from nat-loopback. If you get a similar result in your LAN, then nat-loopback is basically working.

I guess I was hoping for more detailed information. I get this result:

Code:

tingo@kg-v2$ tcptraceroute tingox.blogdns.org 80
Selected device nfe0, address 10.1.150.17, port 27396 for outgoing packets
Tracing the path to tingox.blogdns.org (84.215.134.159) on TCP port 80, 30 hops max
 1  cm-84.215.134.159.getinternet.no (84.215.134.159) [closed]  9.620 ms  9.933 ms  9.980 ms

It is interesting that it says that the port is closed.
If I try to check my mailserver:

Code:

tingo@kg-v2$ tcptraceroute tingox.blogdns.org 25
Selected device nfe0, address 10.1.150.17, port 62129 for outgoing packets
Tracing the path to tingox.blogdns.org (84.215.134.159) on TCP port 25, 30 hops max
 1  cm-84.215.134.159.getinternet.no (84.215.134.159) [open]  9.871 ms  10.029 ms  9.939 ms

Quote:

Originally Posted by rolfheinrich

If said websites are hosted by different machines in your local network, then it might be helpful to learn a little bit more about your setup, in this case, quite possibly you cannot achieve your objective with NAT, but you need to install a web proxy on the gateway machine.

The webserver in question is also a web proxy for other web servers on my local net, but that isn't relevant here. For this particular case, I'm only concerned with the http traffic to the webserver.

Perhaps it will be useful to show a "before" and "after" scenario.
With my old ISP, the setup looked like this:
Internet - ISP router (nat to FW) - firewall (natd) machine - internal net (web servers + clients)

With my new ISP, it looks like this:
Internet - ISP modem - firewall (natd) machine - internal net (web servers + clients)

The main difference is that my firewall is now the "outside router" and has a public IP address on the outside interface.

rolfheinrich · #7 February 10th, 2012, 15:06

Quote:

Originally Posted by tingo

Except for one small detail: the natd (firewall) machine doesn't have a web server; the web server is "behind" the nat machine.

I guess I was hoping for more detailed information. I get this result:

Code:

tingo@kg-v2$ tcptraceroute tingox.blogdns.org 80
Selected device nfe0, address 10.1.150.17, port 27396 for outgoing packets
Tracing the path to tingox.blogdns.org (84.215.134.159) on TCP port 80, 30 hops max
 1  cm-84.215.134.159.getinternet.no (84.215.134.159) [closed]  9.620 ms  9.933 ms  9.980 ms

It is interesting that it says that the port is closed.

Since you said, that there is no web server at the nat machine, why would you expect something to responding on port 80? Did you adjust in the natd configuration port redirection to your web proxy in your LAN? If not, how shall the packets to port 80 know, that they have to go to that other machine?

Quote:

Originally Posted by tingo

If I try to check my mailserver:

Code:

tingo@kg-v2$ tcptraceroute tingox.blogdns.org 25
Selected device nfe0, address 10.1.150.17, port 62129 for outgoing packets
Tracing the path to tingox.blogdns.org (84.215.134.159) on TCP port 25, 30 hops max
 1  cm-84.215.134.159.getinternet.no (84.215.134.159) [open]  9.871 ms  10.029 ms  9.939 ms

Because there is a smtp server listening on port 25 on the nat machine. Check the listening ports with the following command:

sockstat -4

Quote:

Originally Posted by tingo

The webserver in question is also a web proxy for other web servers on my local net, but that isn't relevant here. For this particular case, I'm only concerned with the http traffic to the webserver.

As said already, if the web-proxy is not running on the nat machine, then you need to configure nat-port-redirection. Otherwise, external traffic on port 80 is stuck at the nat machine, because nothing is listening on port 80, and nothing is telling the packets where to go either.

tingo · #8 February 19th, 2012, 00:48

Some findings:
net.inet.ip.fw.one_pass was 1, now changed to 0. Unfortunately, it hasn't changed the behavior of my firewall at all. I'm a bit surprised by this; is it because I am using natd that this works even with net.inet.ip.fw.one_pass=1?

net.inet.ip.forwarding=1 is ok (no surprise, I have gateway_enable="YES" in /etc/rc.conf on the firewall machine).

Currently, I don't use IPv6, so net.inet6.ip6.forwarding=0 doesn't matter to me.

One thing I'm wondering about: why do you suggest to set net.inet.tcp.tso=0? AFAICT, this setting only applies to fxp(4), and the firewall machine doesn't have any fxp interfaces.

I have analyzed my firewall rules again (with logging on), but haven't found anything of interest. To me, it looks like the firewall rules are working ok, and passing necessary traffic in and out.

I think I will have to try a different angle to solve this problem when I have time.

rolfheinrich · #9 February 19th, 2012, 16:09

Quote:

Originally Posted by tingo

Some findings:
net.inet.ip.fw.one_pass was 1, now changed to 0. Unfortunately, it hasn't changed the behavior of my firewall at all. I'm a bit surprised by this; is it because I am using natd that this works even with net.inet.ip.fw.one_pass=1?

Yes, maybe - I am using in kernel nat, and I just checked it again, without net.inet.ip.fw.one_pass=0, nat port forwarding does not work in my firewall setup.

Quote:

Originally Posted by tingo

One thing I'm wondering about: why do you suggest to set net.inet.tcp.tso=0? AFAICT, this setting only applies to fxp(4), and the firewall machine doesn't have any fxp interfaces.

Disabling TSO on the NIC is recommended at the end of ipfw(8). I preferred to disable TSO for all my NICs at once by setting net.inet.tcp.tso=0.

My machine got a RealTek NIC, managed by the re(4) driver, and this supports TSO as well.

tingo · #10 February 22nd, 2012, 21:16

This is my first try with in-kernel NAT, based on rolfheinrich's setup. Here is the script' I'm using:

Code:

root@kg-omni1# ipfw show
00010     36     4414 allow ip from any to any via lo0
00020      0        0 deny ip from any to 127.0.0.0/8
00030      0        0 deny ip from 127.0.0.0/8 to any
00040      0        0 deny ip from any to ::1
00050      0        0 deny ip from ::1 to any
00080 140116 21406815 allow ip from any to any via rl0
00090      0        0 deny ip from any to any not antispoof in
00100  66120  8101485 nat 1 ip from any to any via xl0 in
00101      0        0 check-state
00200    345   194901 skipto 10000 tcp from any to any dst-port 22,80 via xl0 in setup keep-state
00202 102485 15419899 skipto 10000 tcp from any to any dst-port 4661,4662,4242,5231 via xl0 in setup keep-state
00203   8106   502774 skipto 10000 udp from any to any dst-port 4665,4672 via xl0 in keep-state
01000      0        0 deny ip from not me to any dst-port 137,138,139 via xl0 out
02000   8409  3805091 skipto 10000 tcp from any to any via xl0 out setup keep-state
02010   3184   366877 skipto 10000 udp from any to any via xl0 out keep-state
05000      0        0 allow tcp from any to any dst-port 22,80 via xl0 in setup limit src-addr 50
05010      0        0 allow tcp from any to any dst-port 4661,4662,4242,5231 via xl0 in setup
05020      0        0 allow udp from any to any dst-port 4665,4672 via xl0 in
09998  22149  1290595 deny tcp from any to any via xl0
09999    257    79584 deny udp from any to any via xl0
10000  62106 12568598 nat 1 ip from any to any via xl0 out
65534 122589 20294725 allow ip from any to any
65535   1188   211088 deny ip from any to any

and this is the NAT config:

Code:

root@kg-omni1# ipfw nat 1 show config
ipfw nat 1 config if xl0 reset
 redirect_port tcp 10.1.150.8:5231 5231
 redirect_port tcp 10.1.150.8:4242 4242
 redirect_port udp 10.1.150.8:4672 4672
 redirect_port udp 10.1.150.8:4665 4665
 redirect_port tcp 10.1.150.8:4662 4662
 redirect_port tcp 10.1.150.8:4661 4661
 redirect_port tcp 10.1.150.8:22 22
 redirect_port tcp 10.1.10.100:80 80

(wrapped to improve readability)
I'm probably doing something wrong, because I still can't access my internal web server(s) by using my public ip address when I'm "behind" the NAT firewall.
I haven't tested from another network yet. Update: now tested from another network. As before, access to the web servers behind my firewall works.

tingo · #11 February 23rd, 2012, 22:47

This evening I tried running a second NAT instance (reverse NAT), on the internal interface, by adding these rules:
(iif="rl0")

Code:

${fwcmd} nat 2 config if ${iif} log reverse redirect_port tcp 10.1.10.100:80 80

$add 70 skipto 20000 log tcp from any to me 80 via ${iif} keep-state

$add 20000 nat 2 ip from any to any via ${iif}

But that didn't work either. I can see traffic going through these rules:

Code:

root@kg-omni1# ipfw show
00010    200    10848 allow ip from any to any via lo0
00020      0        0 deny ip from any to 127.0.0.0/8
00030      0        0 deny ip from 127.0.0.0/8 to any
00040      0        0 deny ip from any to ::1
00050      0        0 deny ip from ::1 to any
00070     90     4752 skipto 20000 log tcp from any to me dst-port 80 via rl0 keep-state
00080  10190   792298 allow ip from any to any via rl0
00090      0        0 deny log ip from any to any not antispoof in
00100  49081  6824483 nat 1 ip from any to any via xl0 in
00101      0        0 check-state
00200  10254  6692420 skipto 10000 tcp from any to any dst-port 22,80 via xl0 in setup keep-state
00202 125849  5681118 skipto 10000 tcp from any to any dst-port 4661,4662,4242,5231 via xl0 in setup keep-state
00203   8460   636458 skipto 10000 udp from any to any dst-port 4665,4672 via xl0 in keep-state
01000      0        0 deny log ip from not me to any dst-port 137,138,139 via xl0 out
02000  35172 12062860 skipto 10000 tcp from any to any via xl0 out setup keep-state
02010   1134   115349 skipto 10000 udp from any to any via xl0 out keep-state
05000      0        0 allow tcp from any to any dst-port 22,80 via xl0 in setup limit src-addr 50
05010      0        0 allow tcp from any to any dst-port 4661,4662,4242,5231 via xl0 in setup
05020      0        0 allow udp from any to any dst-port 4665,4672 via xl0 in
09998   2864   193743 deny log tcp from any to any via xl0
09999    645   196121 deny log udp from any to any via xl0
10000  48212  6331138 nat 1 ip from any to any via xl0 out
20000  89245 12501428 nat 2 ip from any to any via rl0
65534 185188 25429833 allow ip from any to any
65535   1472   250035 deny ip from any to any

but connecting to the web server doesn't work, it times out after a while.
As before, connecting from another network still works.

tingo · #12 February 25th, 2012, 00:26

And now (with my "second try .." setup) I get errors from ipfw:

Code:

Feb 25 00:13:14 kg-omni1 kernel: ipfw: ipfw_install_state: Too many dynamic rules
Feb 25 00:13:45 kg-omni1 last message repeated 27 times
Feb 25 00:15:46 kg-omni1 last message repeated 110 times

Not good.

RusDyr · #13 February 28th, 2012, 13:38

Increase maximum number of dynamic rules from default 4096 to, for example, 65536:# sysctl net.inet.ip.fw.dyn_max=65536
and see actual number after:
# sysctl net.inet.ip.fw.dyn_count.
Don't forget to save sysctl value in /etc/sysctl.conf.

And, would you be so kind to leave only nat-related rules and check it after?

tingo · #14 March 1st, 2012, 20:17

Quote:

Originally Posted by RusDyr

Increase maximum number of dynamic rules from default 4096 to, for example, 65536:# sysctl net.inet.ip.fw.dyn_max=65536
and see actual number after:
# sysctl net.inet.ip.fw.dyn_count.
Don't forget to save sysctl value in /etc/sysctl.conf.

These are good to know about - thanks!

Quote:

Originally Posted by RusDyr

And, would you be so kind to leave only nat-related rules and check it after?

I'm not sure I follow? Are you saying that removing all rules except NAT rules will help me to figure out how to get "NAT loopback" working?

tingo · #15 March 1st, 2012, 22:50

Update: For now, I am using a workaround: I installed nginx as a reverse proxy on my firewall.

Thread Tools
Show Printable Version Email this Page
Display Modes
Switch to Linear Mode Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Nat Proxy Firewall recomendations	breese	Firewalls	0	August 19th, 2011 23:23
[Solved] loopback no IP address	murias	Networking	9	November 30th, 2010 08:04
ZFS functionality Question	Sylgeist	General	7	August 2nd, 2010 17:06
[Solved] curious about cloned loopback and routing	wonslung	Networking	2	November 19th, 2009 18:58
Firewall + DNS + NAT + DHCP	gpatrick	Firewalls	2	November 17th, 2009 08:56

The Following User Says Thank You to DutchDaemon For This Useful Post:
tingo (February 10th, 2012)

The Following User Says Thank You to RusDyr For This Useful Post:
tingo (March 1st, 2012)