FreeBSD firewall: how to get "nat loopback" functionality?

T

tingo

Hello,

I'm currently using ipfw+natd for my firewall. I have some web servers on my local network. They use DynDNS for the web site name(s), using the public IP address I get from my ISP and a dyndns client. This makes it vital to be able to access these web sites from my internal network using the public IP address. This is sometimes referred to as "nat loopback", (here is a quite good explanation of the issue).

I want to avoid using a split DNS, both to keep the admin / management down (a fancy way of saying "less work"), and also because it is easier to test if a web site "works" if I am using the public IP address (I can see at once if I have messed up a firewall rule, for instance).

How do I get a firewall built on FreeBSD to have "NAT loopback" functionality, so that I can access the web sites on my local network using the public IP address? Preferably with ipfw + natd, if possible.

Note: the the web sites work correctly if accessed from another place on the internet. I've just switched to a new ISP; my previous ISP had a router / modem box (ADSL) with this functionality. My new ISP (cable) only provides me with a modem (better in many ways, I don't have to do double NAT anymore).
 
I am almost sure that ipfw+natd should give you "nat loopback" out of the box. However, I cannot prove this for now, because some time ago I did the switch from natd to in-Kernel NAT.

I read also your other "natd"-thread, and I understand that you want to stay with natd, and I won't argue against this.

My 2 cents are, that it is not that much effort to do the switch from natd to in-Kernel-NAT, and everything is working well, including "nat loopback". I just checked it on my installation, which seems to be quite similar to what you are aiming for.
 
rolfheinrich said:
I am almost sure that ipfw+natd should give you "nat loopback" out of the box. However, I cannot prove this for now, because some time ago I did the switch from natd to in-Kernel NAT.
Click to expand...

Yes, I thought so too. (Sorry for the late response, I've been busy with other things.)
I tried messing with my firewall rules today, but that didn't help.
I wish I had a tool that could be like traceroute, but for http traffic. That would make it easier to figure out where things are stopping.

rolfheinrich said:
My 2 cents are, that it is not that much effort to do the switch from natd to in-Kernel-NAT, and everything is working well, including "nat loopback". I just checked it on my installation, which seems to be quite similar to what you are aiming for.
Click to expand...

Well, if I can't get ipfw+natd to work, I shall certainly try out this solution.
 
tingo said:
I wish I had a tool that could be like traceroute, but for http traffic.
Click to expand...

Up to now, I understood, that you are talking about domains of virtual hosts, that are served by the same httpd server at the nat machine. In this case traceroute from a client machine in your local network should be sufficient for troubleshooting, and it would show you only 1 hop, for example something like the following:

Code: 
$ traceroute mydyndom.ath.cx
traceroute to mydyndom.ath.cx (111.222.333.444), 64 hops max, 52 byte packets
 1  dynip.frommyisp.com (111.222.333.444)  1.027 ms  0.415 ms  0.395 ms

This is all, that you can expect from nat-loopback. If you get a similar result in your LAN, then nat-loopback is basically working. If not, then you might want to check your DNS resolver on the respective client, that means whether really all your dynamic domains resolve to your public IP.

If said websites are hosted by different machines in your local network, then it might be helpful to learn a little bit more about your setup, in this case, quite possibly you cannot achieve your objective with NAT, but you need to install a web proxy on the gateway machine.
 
rolfheinrich said:
Up to now, I understood, that you are talking about domains of virtual hosts, that are served by the same httpd server at the nat machine. In this case traceroute from a client machine in your local network should be sufficient for troubleshooting, and it would show you only 1 hop, for example something like the following:
Click to expand...
Except for one small detail: the natd (firewall) machine doesn't have a web server; the web server is "behind" the nat machine.

rolfheinrich said:
This is all, that you can expect from nat-loopback. If you get a similar result in your LAN, then nat-loopback is basically working.
Click to expand...
I guess I was hoping for more detailed information. I get this result:
Code: 
tingo@kg-v2$ tcptraceroute tingox.blogdns.org 80
Selected device nfe0, address 10.1.150.17, port 27396 for outgoing packets
Tracing the path to tingox.blogdns.org (84.215.134.159) on TCP port 80, 30 hops max
 1  cm-84.215.134.159.getinternet.no (84.215.134.159) [closed]  9.620 ms  9.933 ms  9.980 ms
It is interesting that it says that the port is closed.
If I try to check my mailserver:
Code: 
tingo@kg-v2$ tcptraceroute tingox.blogdns.org 25
Selected device nfe0, address 10.1.150.17, port 62129 for outgoing packets
Tracing the path to tingox.blogdns.org (84.215.134.159) on TCP port 25, 30 hops max
 1  cm-84.215.134.159.getinternet.no (84.215.134.159) [open]  9.871 ms  10.029 ms  9.939 ms
rolfheinrich said:
If said websites are hosted by different machines in your local network, then it might be helpful to learn a little bit more about your setup, in this case, quite possibly you cannot achieve your objective with NAT, but you need to install a web proxy on the gateway machine.
Click to expand...

The webserver in question is also a web proxy for other web servers on my local net, but that isn't relevant here. For this particular case, I'm only concerned with the http traffic to the webserver.

Perhaps it will be useful to show a "before" and "after" scenario.
With my old ISP, the setup looked like this:
Internet - ISP router (nat to FW) - firewall (natd) machine - internal net (web servers + clients)

With my new ISP, it looks like this:
Internet - ISP modem - firewall (natd) machine - internal net (web servers + clients)

The main difference is that my firewall is now the "outside router" and has a public IP address on the outside interface.
 
tingo said:
Except for one small detail: the natd (firewall) machine doesn't have a web server; the web server is "behind" the nat machine.

I guess I was hoping for more detailed information. I get this result:
Code: 
tingo@kg-v2$ tcptraceroute tingox.blogdns.org 80
Selected device nfe0, address 10.1.150.17, port 27396 for outgoing packets
Tracing the path to tingox.blogdns.org (84.215.134.159) on TCP port 80, 30 hops max
 1  cm-84.215.134.159.getinternet.no (84.215.134.159) [closed]  9.620 ms  9.933 ms  9.980 ms
It is interesting that it says that the port is closed.
Click to expand...

Since you said, that there is no web server at the nat machine, why would you expect something to responding on port 80? Did you adjust in the natd configuration port redirection to your web proxy in your LAN? If not, how shall the packets to port 80 know, that they have to go to that other machine?


tingo said:
If I try to check my mailserver:
Code: 
tingo@kg-v2$ tcptraceroute tingox.blogdns.org 25
Selected device nfe0, address 10.1.150.17, port 62129 for outgoing packets
Tracing the path to tingox.blogdns.org (84.215.134.159) on TCP port 25, 30 hops max
 1  cm-84.215.134.159.getinternet.no (84.215.134.159) [open]  9.871 ms  10.029 ms  9.939 ms
Click to expand...

Because there is a smtp server listening on port 25 on the nat machine. Check the listening ports with the following command:

sockstat -4


tingo said:
The webserver in question is also a web proxy for other web servers on my local net, but that isn't relevant here. For this particular case, I'm only concerned with the http traffic to the webserver.
Click to expand...

As said already, if the web-proxy is not running on the nat machine, then you need to configure nat-port-redirection. Otherwise, external traffic on port 80 is stuck at the nat machine, because nothing is listening on port 80, and nothing is telling the packets where to go either.
 
rolfheinrich said:
Since you said, that there is no web server at the nat machine, why would you expect something to responding on port 80? Did you adjust in the natd configuration port redirection to your web proxy in your LAN?
Click to expand...
I'm sorry if I haven't explained myself good enough.
Yes, natd redirects everything on port 80 to my webserver. From /etc/natd.conf:
Code: 
  redirect_port tcp 10.1.10.100:80 80
This has been in place the whole time.


rolfhheinric said:
Because there is a smtp server listening on port 25 on the nat machine.
Click to expand...
Yes, I already know that. :)
And because of the natd redirection, I was expecting the packets for port 80 to be redirected to the webserver. If it worked, which it currently does not.

rolfhheinric said:
As said already, if the web-proxy is not running on the nat machine, then you need to configure nat-port-redirection. Otherwise, external traffic on port 80 is stuck at the nat machine, because nothing is listening on port 80, and nothing is telling the packets where to go either.
Click to expand...

As explained above, nat port redirection is in place. Sorry if I haven't been clear on that.
 
tingo said:
I'm sorry if I haven't explained myself good enough.
Yes, natd redirects everything on port 80 to my webserver. From /etc/natd.conf:
Code: 
  redirect_port tcp 10.1.10.100:80 80
This has been in place the whole time.

rolfheinrich said:
Because there is a smtp server listening on port 25 on the nat machine. Check the listening ports with the following command:
sockstat -4
Click to expand...

Yes, I already know that. :)
And because of the natd redirection, I was expecting the packets for port 80 to be redirected to the webserver. If it worked, which it currently does not.

As explained above, nat port redirection is in place. Sorry if I haven't been clear on that.
Click to expand...

For me the resume of the opera is, that "nat loopback" is working on your system, while "nat port redirection" does not work as expected.

I think it's time now that you analyze your firewall rules.

In addition, for nat port redirection to work, the packets must be passing at least two times the whole fw rule set, so, don't forget to set net.inet.ip.fw.one_pass to 0 in /etc/sysctl.conf. And once you are there, please check that the other necessary sysctl settings as shown below are in place.

Code: 
net.inet.ip.fw.one_pass=0
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
net.inet.tcp.tso=0

Best regards

Rolf
 
Some findings:
net.inet.ip.fw.one_pass was 1, now changed to 0. Unfortunately, it hasn't changed the behavior of my firewall at all. I'm a bit surprised by this; is it because I am using natd that this works even with net.inet.ip.fw.one_pass=1?

net.inet.ip.forwarding=1 is ok (no surprise, I have gateway_enable="YES" in /etc/rc.conf on the firewall machine).

Currently, I don't use IPv6, so net.inet6.ip6.forwarding=0 doesn't matter to me.

One thing I'm wondering about: why do you suggest to set net.inet.tcp.tso=0? AFAICT, this setting only applies to fxp(4), and the firewall machine doesn't have any fxp interfaces.

I have analyzed my firewall rules again (with logging on), but haven't found anything of interest. To me, it looks like the firewall rules are working ok, and passing necessary traffic in and out.

I think I will have to try a different angle to solve this problem when I have time.
 
tingo said:
Some findings:
net.inet.ip.fw.one_pass was 1, now changed to 0. Unfortunately, it hasn't changed the behavior of my firewall at all. I'm a bit surprised by this; is it because I am using natd that this works even with net.inet.ip.fw.one_pass=1?
Click to expand...

Yes, maybe - I am using in kernel nat, and I just checked it again, without net.inet.ip.fw.one_pass=0, nat port forwarding does not work in my firewall setup.

tingo said:
One thing I'm wondering about: why do you suggest to set net.inet.tcp.tso=0? AFAICT, this setting only applies to fxp(4), and the firewall machine doesn't have any fxp interfaces.
Click to expand...

Disabling TSO on the NIC is recommended at the end of ipfw(8). I preferred to disable TSO for all my NICs at once by setting net.inet.tcp.tso=0.

My machine got a RealTek NIC, managed by the re(4)() driver, and this supports TSO as well.
 
First try with in-kernel NAT

This is my first try with in-kernel NAT, based on rolfheinrich's setup. Here is the script' I'm using:
Code: 
root@kg-omni1# ipfw show
00010     36     4414 allow ip from any to any via lo0
00020      0        0 deny ip from any to 127.0.0.0/8
00030      0        0 deny ip from 127.0.0.0/8 to any
00040      0        0 deny ip from any to ::1
00050      0        0 deny ip from ::1 to any
00080 140116 21406815 allow ip from any to any via rl0
00090      0        0 deny ip from any to any not antispoof in
00100  66120  8101485 nat 1 ip from any to any via xl0 in
00101      0        0 check-state
00200    345   194901 skipto 10000 tcp from any to any dst-port 22,80 via xl0 in setup keep-state
00202 102485 15419899 skipto 10000 tcp from any to any dst-port 4661,4662,4242,5231 via xl0 in setup keep-state
00203   8106   502774 skipto 10000 udp from any to any dst-port 4665,4672 via xl0 in keep-state
01000      0        0 deny ip from not me to any dst-port 137,138,139 via xl0 out
02000   8409  3805091 skipto 10000 tcp from any to any via xl0 out setup keep-state
02010   3184   366877 skipto 10000 udp from any to any via xl0 out keep-state
05000      0        0 allow tcp from any to any dst-port 22,80 via xl0 in setup limit src-addr 50
05010      0        0 allow tcp from any to any dst-port 4661,4662,4242,5231 via xl0 in setup
05020      0        0 allow udp from any to any dst-port 4665,4672 via xl0 in
09998  22149  1290595 deny tcp from any to any via xl0
09999    257    79584 deny udp from any to any via xl0
10000  62106 12568598 nat 1 ip from any to any via xl0 out
65534 122589 20294725 allow ip from any to any
65535   1188   211088 deny ip from any to any
and this is the NAT config:
Code: 
root@kg-omni1# ipfw nat 1 show config
ipfw nat 1 config if xl0 reset
 redirect_port tcp 10.1.150.8:5231 5231
 redirect_port tcp 10.1.150.8:4242 4242
 redirect_port udp 10.1.150.8:4672 4672
 redirect_port udp 10.1.150.8:4665 4665
 redirect_port tcp 10.1.150.8:4662 4662
 redirect_port tcp 10.1.150.8:4661 4661
 redirect_port tcp 10.1.150.8:22 22
 redirect_port tcp 10.1.10.100:80 80
(wrapped to improve readability)
I'm probably doing something wrong, because I still can't access my internal web server(s) by using my public ip address when I'm "behind" the NAT firewall.
I haven't tested from another network yet. Update: now tested from another network. As before, access to the web servers behind my firewall works.
 
Second try with in-kernel NAT

This evening I tried running a second NAT instance (reverse NAT), on the internal interface, by adding these rules:
(iif="rl0")
Code: 
${fwcmd} nat 2 config if ${iif} log reverse redirect_port tcp 10.1.10.100:80 80

$add 70 skipto 20000 log tcp from any to me 80 via ${iif} keep-state

$add 20000 nat 2 ip from any to any via ${iif}
But that didn't work either. I can see traffic going through these rules:
Code: 
root@kg-omni1# ipfw show
00010    200    10848 allow ip from any to any via lo0
00020      0        0 deny ip from any to 127.0.0.0/8
00030      0        0 deny ip from 127.0.0.0/8 to any
00040      0        0 deny ip from any to ::1
00050      0        0 deny ip from ::1 to any
00070     90     4752 skipto 20000 log tcp from any to me dst-port 80 via rl0 keep-state
00080  10190   792298 allow ip from any to any via rl0
00090      0        0 deny log ip from any to any not antispoof in
00100  49081  6824483 nat 1 ip from any to any via xl0 in
00101      0        0 check-state
00200  10254  6692420 skipto 10000 tcp from any to any dst-port 22,80 via xl0 in setup keep-state
00202 125849  5681118 skipto 10000 tcp from any to any dst-port 4661,4662,4242,5231 via xl0 in setup keep-state
00203   8460   636458 skipto 10000 udp from any to any dst-port 4665,4672 via xl0 in keep-state
01000      0        0 deny log ip from not me to any dst-port 137,138,139 via xl0 out
02000  35172 12062860 skipto 10000 tcp from any to any via xl0 out setup keep-state
02010   1134   115349 skipto 10000 udp from any to any via xl0 out keep-state
05000      0        0 allow tcp from any to any dst-port 22,80 via xl0 in setup limit src-addr 50
05010      0        0 allow tcp from any to any dst-port 4661,4662,4242,5231 via xl0 in setup
05020      0        0 allow udp from any to any dst-port 4665,4672 via xl0 in
09998   2864   193743 deny log tcp from any to any via xl0
09999    645   196121 deny log udp from any to any via xl0
10000  48212  6331138 nat 1 ip from any to any via xl0 out
20000  89245 12501428 nat 2 ip from any to any via rl0
65534 185188 25429833 allow ip from any to any
65535   1472   250035 deny ip from any to any
but connecting to the web server doesn't work, it times out after a while.
As before, connecting from another network still works.
 
And now (with my "second try .." setup) I get errors from ipfw:
Code: 
Feb 25 00:13:14 kg-omni1 kernel: ipfw: ipfw_install_state: Too many dynamic rules
Feb 25 00:13:45 kg-omni1 last message repeated 27 times
Feb 25 00:15:46 kg-omni1 last message repeated 110 times
Not good.
 
Increase maximum number of dynamic rules from default 4096 to, for example, 65536:# sysctl net.inet.ip.fw.dyn_max=65536
and see actual number after:
# sysctl net.inet.ip.fw.dyn_count.
Don't forget to save sysctl value in /etc/sysctl.conf.

And, would you be so kind to leave only nat-related rules and check it after?
 
RusDyr said:
Increase maximum number of dynamic rules from default 4096 to, for example, 65536:# sysctl net.inet.ip.fw.dyn_max=65536
and see actual number after:
# sysctl net.inet.ip.fw.dyn_count.
Don't forget to save sysctl value in /etc/sysctl.conf.
Click to expand...
These are good to know about - thanks!

RusDyr said:
And, would you be so kind to leave only nat-related rules and check it after?
Click to expand...
I'm not sure I follow? Are you saying that removing all rules except NAT rules will help me to figure out how to get "NAT loopback" working?
 
You must log in or register to reply here.
Back
Top