IPFW Cannot Get IPFW NAT to work

[JKallen]

I have spent days trying to get what I thought should be a simple set of ipfw nat rules set up. With less than zero success. I have read the documentation and scoured the web, and I assume I am just missing something.

Scenario: I have one NIC card with four public IPs. I am running a bunch of jails that each handle specific tasks. (i.e. dns, mysql, apache, etc.) I do not want my jails to have public IPs for a bunch of reasons. So I am trying to set up the jails on clones of lo0, and then use ipfw nat rules to handle port redirection. Seems pretty straight forward but I am screwing something up here. The net effect of everything below is that ipfw show shows EVERYTHING going through the nat rule. For instance, i run SSH on A.B.C.D on port 22 (not A.B.C.G which is what I am trying to NAT) and all traffic is going thrugh the NAT 1. What am I missing here?!?!

High Level System Info:
FreeBSD 11.1-RELEASE

/etc/rc.conf

hostname="XYZ"
dumpdev="AUTO"

# NETWORK INITIALIZATION
ifconfig_em1="inet A.B.C.D netmask 255.255.255.0"
defaultrouter="A.B.C.1"
gateway_enable="YES"

# IPV6 INITIALIZATION
ifconfig_em1_ipv6="inet6 1.2.3.4.5.6.7.8:0 prefixlen 64"
ipv6_defaultrouter="1.2.3.4::a"
ipv6_gateway_enable="YES"

firewall_enable="YES"
firewall_nat_enable="YES"
firewall_script="/etc/firewall.rules"
firewall_logging="YES"
ipv6_firewall_logging="YES"

sendmail_enable="NO"

syslogd_enable="YES"
syslogd_flags="-b A.B.C.D -4"


ifconfig_em1_alias0="inet A.B.C.E netmask 255.255.255.255 broadcast A.B.C.E"
ifconfig_em1_alias1="inet A.B.C.F netmask 255.255.255.255 broadcast A.B.C.F"
ifconfig_em1_alias2="inet A.B.C.G netmask 255.255.255.255 broadcast A.B.C.G"


ifconfig_em1_alias3="inet6 1.2.3.4.5.6.7.9 prefixlen 64"

# ^^^^A BUNCH MORE IPV6 ALIASES ARE HERE - OMITTED FOR BREVITY

ifconfig_em1_alias20="inet6 1.2.3.4.5.6.7.8.15 prefixlen 64"


# LOCAL IPV4 ADDRESSING FOR JAILS

cloned_interfaces="lo1 lo2 lo3 lo4"

ipv4_addrs_lo1="10.0.0.1/8"
ipv4_addrs_lo2="10.0.0.1/16"
ipv4_addrs_lo3="10.0.0.1/24"
ipv4_addrs_lo4="10.0.1.1/24"


# THIS WILL SERVE AS THE NET WORK OF DEV SERVERS
ifconfig_lo3_alias0="inet 10.0.0.1 netmask 255.255.255.255 broadcast 10.0.0.1"
ifconfig_lo3_alias1="inet 10.0.0.2 netmask 255.255.255.255 broadcast 10.0.0.2"
ifconfig_lo3_alias2="inet 10.0.0.3 netmask 255.255.255.255 broadcast 10.0.0.3"
ifconfig_lo3_alias3="inet 10.0.0.4 netmask 255.255.255.255 broadcast 10.0.0.4"
ifconfig_lo3_alias4="inet 10.0.0.5 netmask 255.255.255.255 broadcast 10.0.0.5"
ifconfig_lo3_alias5="inet 10.0.0.6 netmask 255.255.255.255 broadcast 10.0.0.6"
ifconfig_lo3_alias6="inet 10.0.0.7 netmask 255.255.255.255 broadcast 10.0.0.7"

#THIS WILL SERVE AS THE NETWORK OF PROD SERVERS
ifconfig_lo4_alias0="inet 10.0.1.1 netmask 255.255.255.255 broadcast 10.0.1.1"
ifconfig_lo4_alias1="inet 10.0.1.2 netmask 255.255.255.255 broadcast 10.0.1.2"
ifconfig_lo4_alias2="inet 10.0.1.3 netmask 255.255.255.255 broadcast 10.0.1.3"
ifconfig_lo4_alias3="inet 10.0.1.4 netmask 255.255.255.255 broadcast 10.0.1.4"
ifconfig_lo4_alias4="inet 10.0.1.5 netmask 255.255.255.255 broadcast 10.0.1.5"
ifconfig_lo4_alias5="inet 10.0.1.6 netmask 255.255.255.255 broadcast 10.0.1.6"
ifconfig_lo4_alias6="inet 10.0.1.7 netmask 255.255.255.255 broadcast 10.0.1.7"

jail_enable="YES"

/etc/sysctl.conf

security.bsd.see_other_uids=0
net.inet.ip.fw.one_pass=0
net.inet.ip.fw.verbose=1
net.inet.ip.fw.verbose_limit=5
net.inet.ip6.fw.verbose=1
net.inet.ip6.fw.verbose_limit=5

/etc/firewall.rules

# NOTE THAT A.B.C.G below corresponds to fourth public ip address on external facing NIC

ipfw nat 1 config ip A.B.C.G same_ports unreg_only redirect_port tcp 10.0.0.5:80 80

#################################################################
# No restrictions on Loopback Interface
#################################################################
$cmd 00030 allow all from any to any via lo0

$cmd 00050 check-state

$cmd 00051 nat 1 all from any to any;

# A WHOLE BUNCH OF RULES HAPPEN HEAR

In short, all traffic seems to be going through 00051 even if it was originally destined for one of the other three public IP addresses.

 

[BSDAppentic3]

Since I crashed my system trying to mount a jail, I never try to do it again. I'll do it, in a VM.
But why you use ipv6?

 

[SirDice]

This is bad:

ipv4_addrs_lo1="10.0.0.1/8"
ipv4_addrs_lo2="10.0.0.1/16"
ipv4_addrs_lo3="10.0.0.1/24"

Not only does each interface have the same IP address (IP conflicts) you also have different subnet masks. That's never going to work correctly.

 

[JKallen]

This is bad:

ipv4_addrs_lo1="10.0.0.1/8"
ipv4_addrs_lo2="10.0.0.1/16"
ipv4_addrs_lo3="10.0.0.1/24"

Not only does each interface have the same IP address (IP conflicts) you also have different subnet masks. That's never going to work correctly.

I was wondering about that. I have removed that. The idea behind that was I was trying to configure a sub net for 10, and subnet for 10.0, and then two subnets one for 10.0.0 and 10.0.1.

With respect to the main issue which is ipfw and kernel nat, it seems part of the issue is that TSO4 and TXCSUM are set. Once I have everything figured out I am going to post the response, because every "definitive" solution I have found on the internet on ipfw w kernel nat seem to contradict each other. Some say to have the nat config'd with recv, others say to use via,....there just seems to be a lot of confusion around this subject area. The answer I like least is "just use pf"....I don't want to

 

[BSDAppentic3]

This is bad:

ipv4_addrs_lo1="10.0.0.1/8"
ipv4_addrs_lo2="10.0.0.1/16"
ipv4_addrs_lo3="10.0.0.1/24"

Not only does each interface have the same IP address (IP conflicts) you also have different subnet masks. That's never going to work correctly.

Why? They should be different IP adress?

 

[JKallen]

Since I crashed my system trying to mount a jail, I never try to do it again. I'll do it, in a VM.
But why you use ipv6?

I have no trouble with jails. I think they are very useful, easy to set up, configure and manage. I havent worked much with VM. With respect to ipv6....the answer is because, thats the direction the internet is going. I find it easier to work with ipv6 than ipv4 with CIDR and NATing. Its much more difficult to get hacked indiscriminately because there are 65,000+ addresses per square meter on this planet. So finding an IP Address and a Port, and then finding a security hole for that IP and Port with ipv6 is more difficult just because of the orders of magnitude more possible combinations.

 

[BSDAppentic3]

JKallen You should download and install some emulator, then run FreeBSD there. Just in case that some went wrong.
I use emulators/virtualbox-ose which is relatively easy to install and use.
At least...you want to do this in your host system

 

[BSDAppentic3]

JKallen I think that there shouldn't be any problem...but you want to use jails...

 

[BSDAppentic3]

I have no trouble with jails. I think they are very useful, easy to set up, configure and manage. I havent worked much with VM. With respect to ipv6....the answer is because, thats the direction the internet is going. I find it easier to work with ipv6 than ipv4 with CIDR and NATing. Its much more difficult to get hacked indiscriminately because there are 65,000+ addresses per square meter on this planet. So finding an IP Address and a Port, and then finding a security hole for that IP and Port with ipv6 is more difficult just because of the orders of magnitude more possible combinations.

Yes, I read that the new protocol rises up the x.xxx.xxx number of IP to xxx.xxx.xxx.xxx. It say: if a hacker wants to attack to one IP, he/she should be search around 10000 IP...sorry, I remember it vaguely: correct me if I am wrong, but I think we're talking about the same.
But my question was because I have troubles setting IPv6 on this system. But if you hadn't troubles with that...

 

[JKallen]

Yes, I read that the new protocol rises up the x.xxx.xxx number of IP to xxx.xxx.xxx.xxx. It say: if a hacker wants to attack to one IP, he/she should be search around 10000 IP...sorry, I remember it vaguely: correct me if I am wrong, but I think we're talking about the same.
But my question was because I have troubles setting IPv6 on this system. But if you hadn't troubles with that...

I had troubles initially because i had a firewall and ICMP is necessary for IPv6 so it took hours with tcpdump to work out the bare minimum icmp access to make IPv6 work. After that though, its been a piece of cake for me. I can attach two separate IPv6 addresses to a jail, one for admining (like ssh) and another for a service port like 80 or 443, and never have to worry about running out of ip addresses.