IPFW IPFW and kernel NAT redirect_port not working.

u_gene · Jan 7, 2022

Hello!
I need help to understand how redirect_port working in kernel NAT on FreeBSD 12.3.
I have ipfw rules like this:

Code:

#!/bin/sh -
fwcmd="/sbin/ipfw"
lanout="ng0"
netin="192.168.1.0"
netmask="24"

${fwcmd} -f flush

${fwcmd} nat 100 config log ip ${ipout} reset same_ports \
redirect_port udp 192.168.1.57:16451 16451 \
redirect_port tcp 192.168.1.57:16451 16451 \

${fwcmd} add deny icmp from any to any frag

${fwcmd} add pass all from any to any via lo0
${fwcmd} add pass all from any to any via br0

${fwcmd} add skipto 10000 all from any to any in recv ${lanout}
${fwcmd} add skipto 20000 all from any to any out xmit ${lanout}

${fwcmd} add deny log logamount 100 ip from any to any

############################################## any to any in recv lanout
${fwcmd} add 10000 count all from any to any

${fwcmd} add deny ip from any to 10.0.0.0/8
${fwcmd} add deny ip from any to 172.16.0.0/12
${fwcmd} add deny ip from any to 192.168.0.0/16
${fwcmd} add deny ip from any to 0.0.0.0/8
${fwcmd} add deny ip from any to 169.254.0.0/16
${fwcmd} add deny ip from any to 240.0.0.0/4


${fwcmd} add count tcp from any to me 16451
${fwcmd} add count udp from any to me 16451

${fwcmd} add nat 100 all from any to any in

${fwcmd} add count tcp from any to me 16451
${fwcmd} add count udp from any to me 16451

${fwcmd} add allow ip from any to any established

${fwcmd} add allow ip from any to ${netin}/${netmask}

${fwcmd} add deny log logamount 10000 all from any to any

############################################## any to any out xmit {lanout}
${fwcmd} add 20000 count all from any to any

${fwcmd} add nat 100 all from ${netin}/${netmask} to any out

${fwcmd} add allow tcp from me to any
${fwcmd} add allow udp from me to any
${fwcmd} add allow icmp from me to any icmptypes 0,3,4,8,11,12

${fwcmd} add deny log logamount 10000 ip from any to any

While NAT working, some TCP and UDP packets still unredirected.

Code:

>ipfw show
00100       0          0 deny icmp from any to any frag offset
00200       0          0 check-state :default
00300    8214    3071966 allow ip from any to any via lo0
00800 2050165 1573806088 allow ip from any to any via br0
00900       0          0 deny ip from any to 127.0.0.0/8
01000       0          0 deny ip from 127.0.0.0/8 to any
01100 1295067 1266330049 skipto 10000 ip from any to any in recv ng0
01200  738901  292538839 skipto 20000 ip from any to any out xmit ng0
01300       3        120 deny log logamount 100 ip from any to any
10000 1295062 1266323749 count ip from any to any
10100       0          0 deny ip from any to 10.0.0.0/8
10200       0          0 deny ip from any to 172.16.0.0/12
10300       0          0 deny ip from any to 192.168.0.0/16
10400       0          0 deny ip from any to 0.0.0.0/8
10500       0          0 deny ip from any to 169.254.0.0/16
10600       0          0 deny ip from any to 240.0.0.0/4
10700  200233   12651885 count tcp from any to me 16451
10800   11824    1391077 count udp from any to me 16451
10900 1295057 1266323325 nat 100 ip from any to any in
11000       1         40 count tcp from any to me 16451
11100      17       2137 count udp from any to me 16451
11200  374093  167231079 allow ip from any to any established
11300  920792 1099074330 allow ip from any to 192.168.1.0/24
11400     167      11616 deny log logamount 10000 ip from any to any
20000  738876  292532631 count ip from any to any
20100  736146  292124427 nat 100 ip from 192.168.1.0/24 to any out
20200  590155  275644023 allow tcp from me to any
20300  148719   16888348 allow udp from me to any
20400       2        260 allow icmp from me to any icmptypes 0,3,4,8,11,12
20500       0          0 deny log logamount 10000 ip from any to any

Counters on rules 11000 and 11100 must be 0, but it not.
Why?
How it works??

obsigna · Jan 7, 2022

u_gene said:

Hello!
I need help to understand how redirect_port working in kernel NAT on FreeBSD 12.3.
I have ipfw rules like this:

Code:

#!/bin/sh -
fwcmd="/sbin/ipfw"
lanout="ng0"
netin="192.168.1.0"
netmask="24"

${fwcmd} -f flush

${fwcmd} nat 100 config log ip ${ipout} reset same_ports \
redirect_port udp 192.168.1.57:16451 16451 \
redirect_port tcp 192.168.1.57:16451 16451 \

${fwcmd} add deny icmp from any to any frag

${fwcmd} add pass all from any to any via lo0
${fwcmd} add pass all from any to any via br0

${fwcmd} add skipto 10000 all from any to any in recv ${lanout}
${fwcmd} add skipto 20000 all from any to any out xmit ${lanout}

${fwcmd} add deny log logamount 100 ip from any to any

############################################## any to any in recv lanout
${fwcmd} add 10000 count all from any to any

${fwcmd} add deny ip from any to 10.0.0.0/8
${fwcmd} add deny ip from any to 172.16.0.0/12
${fwcmd} add deny ip from any to 192.168.0.0/16
${fwcmd} add deny ip from any to 0.0.0.0/8
${fwcmd} add deny ip from any to 169.254.0.0/16
${fwcmd} add deny ip from any to 240.0.0.0/4


${fwcmd} add count tcp from any to me 16451
${fwcmd} add count udp from any to me 16451

${fwcmd} add nat 100 all from any to any in

${fwcmd} add count tcp from any to me 16451
${fwcmd} add count udp from any to me 16451

${fwcmd} add allow ip from any to any established

${fwcmd} add allow ip from any to ${netin}/${netmask}

${fwcmd} add deny log logamount 10000 all from any to any

############################################## any to any out xmit {lanout}
${fwcmd} add 20000 count all from any to any

${fwcmd} add nat 100 all from ${netin}/${netmask} to any out

${fwcmd} add allow tcp from me to any
${fwcmd} add allow udp from me to any
${fwcmd} add allow icmp from me to any icmptypes 0,3,4,8,11,12

${fwcmd} add deny log logamount 10000 ip from any to any

While NAT working, some TCP and UDP packets still unredirected.

Code:

>ipfw show
00100       0          0 deny icmp from any to any frag offset
00200       0          0 check-state :default
00300    8214    3071966 allow ip from any to any via lo0
00800 2050165 1573806088 allow ip from any to any via br0
00900       0          0 deny ip from any to 127.0.0.0/8
01000       0          0 deny ip from 127.0.0.0/8 to any
01100 1295067 1266330049 skipto 10000 ip from any to any in recv ng0
01200  738901  292538839 skipto 20000 ip from any to any out xmit ng0
01300       3        120 deny log logamount 100 ip from any to any
10000 1295062 1266323749 count ip from any to any
10100       0          0 deny ip from any to 10.0.0.0/8
10200       0          0 deny ip from any to 172.16.0.0/12
10300       0          0 deny ip from any to 192.168.0.0/16
10400       0          0 deny ip from any to 0.0.0.0/8
10500       0          0 deny ip from any to 169.254.0.0/16
10600       0          0 deny ip from any to 240.0.0.0/4
10700  200233   12651885 count tcp from any to me 16451
10800   11824    1391077 count udp from any to me 16451
10900 1295057 1266323325 nat 100 ip from any to any in
11000       1         40 count tcp from any to me 16451
11100      17       2137 count udp from any to me 16451
11200  374093  167231079 allow ip from any to any established
11300  920792 1099074330 allow ip from any to 192.168.1.0/24
11400     167      11616 deny log logamount 10000 ip from any to any
20000  738876  292532631 count ip from any to any
20100  736146  292124427 nat 100 ip from 192.168.1.0/24 to any out
20200  590155  275644023 allow tcp from me to any
20300  148719   16888348 allow udp from me to any
20400       2        260 allow icmp from me to any icmptypes 0,3,4,8,11,12
20500       0          0 deny log logamount 10000 ip from any to any

Counters on rules 11000 and 11100 must be 0, but it not.
Why?
How it works??

Replace the counter rules 11000 and 11100 by the following allow rules:

Code:

...
${fwcmd} add allow tcp from any to 192.168.1.57 16451
${fwcmd} add allow udp from any to 192.168.1.57 16451
...

You need also to add to /etc/sysctl.conf the following:

Code:

net.inet.ip.fw.one_pass=0

u_gene · Jan 7, 2022

Code:
Code:

>sysctl -a | grep one_pass net.inet.ip.fw.one_pass: 0

Replace the counter rules 11000 and 11100 by the following allow rules:

Code:

${fwcmd} add allow tcp from any to 192.168.1.57 16451
${fwcmd} add allow udp from any to 192.168.1.57 16451

...

...same action in rule 11300

Rules 11000 & 11100 just for check. After NAT with redirect_port on 10900 - all incoming TCP and UDP packets on port 16451 must be with destination addr 192.168.1.57:16541. But counters shows that 200232 TCP packets was redirected (1 left with unchanged dest addr) and 11807 UDP packets was redirected (17 left with unchanged dest addr).
Why??

covacat · Jan 7, 2022

just log the packets that 'escape', you may get more info

facedebouc · Jan 8, 2022

You can try with this setting sysctl net.inet.ip.fw.dyn_keepalive=0

D-FENS · Jan 8, 2022

u_gene said:
...same action in rule 11300

Rules 11000 & 11100 just for check. After NAT with redirect_port on 10900 - all incoming TCP and UDP packets on port 16451 must be with destination addr 192.168.1.57:16541. But counters shows that 200232 TCP packets was redirected (1 left with unchanged dest addr) and 11807 UDP packets was redirected (17 left with unchanged dest addr).
Why??

What is the IP address of the host with this firewall? 192.168.1.57 is a different host, I assume. Right?

You probably assume the rules 11000 and 11100 are logging packets that are supposed to be already NAT-ed to 192.168.1.57:16541, right? So they should never get logged by those rules.

What you are missing though is that the NAT is applied only on packets received at ${ipout}. What about the packets received on other interfaces (they could have the port number 16541, there is no rule blocking them from coming).
To summarize, you are apparently logging packets that did not come via ${ipout} and coincidentally had the same port number 16541.

For example, let's assume the following:
* ${ipout} is 192.168.1.2
* You have an additional network interface (possibly, an Epair interface to connect to a jail on the same host, a TUN interface for a VPN, a TAP for a virtual machine, or a second network interface connecting to another LAN) with the IP address 192.168.2.2.
* You make a network connection to another host on this second interface, for example open a web page on the jail: https://192.168.2.3.
* Your browser randomly uses port 16541 on the client socket.
* When the response comes back from your jail's web server - boom, you get an incoming packet to "me" with port number 16541 but a NAT was not applied because it is not received on ${ipout}.

This would explain the counts you see.
If you want more details on the packets, simply add a log keyword to the count and then see what were the exact packets.

u_gene · Jan 9, 2022

roccobaroccoSC said:
What is the IP address of the host with this firewall? 192.168.1.57 is a different host, I assume. Right?

Yes. It's a jail on different host with running transmissionbt daemon.

roccobaroccoSC said:
What you are missing though is that the NAT is applied only on packets received at ${ipout}. What about the packets received on other interfaces (they could have the port number 16541, there is no rule blocking them from coming).

Blocking rule for other interfaces is 01300.

roccobaroccoSC said:
To summarize, you are apparently logging packets that did not come via ${ipout} and coincidentally had the same port number 16541.

I've changed port number to 16542 for testing.

roccobaroccoSC said:
This would explain the counts you see.
If you want more details on the packets, simply add a log keyword to the count and then see what were the exact packets.

Code:

ipfw rules:
11000      0         0 count log logamount 1000 tcp from any to me 16452
11100      2       180 count log logamount 1000 udp from any to me 16452

Log file:
Jan  9 14:22:52 hs kernel: ipfw: 11100 Count UDP 95.84.240.97:15830 213.155.204.84:16452 in via ng0
Jan  9 14:22:52 hs kernel: ipfw: 11400 Deny UDP 95.84.240.97:15830 213.155.204.84:16452 in via ng0

I've tested this ipfw script on FreeBSD 11.4. Only fragmentation UDP packets counted and denyed by rules 11000-11100.
Something changed in NAT in FreeBSD 12???

mer · Jan 9, 2022

OP: this block. did you really mean to have that trailing backslash on the second redirect_port line?


${fwcmd} nat 100 config log ip ${ipout} reset same_ports \
redirect_port udp 192.168.1.57:16451 16451 \
redirect_port tcp 192.168.1.57:16451 16451 \

obsigna · Jan 9, 2022

ng0 is a netgraph interface, so I assume that your system is connected by PPP. I never used that, and therefore the question. Is ipfw setup before or after the connection via ng0 becomes established? In the latter case there might be already some traffic in the queue which may stray into the nascending rules of ipfw. For one-time testing it might be worth to start the PPP connection manually after ipfw has been setup completely. Then compare the ipfw logs with the case when PPP was setup automatically.

u_gene · Jan 9, 2022

I've spent about 5 day trying to solve this issue. Nothing work.
Now, i'm downgraded FreeBSD to 12.2 and everything works perfect with the same IPFW script!
What a strange thing?
Something changed in NAT??
Thank's everyone to help.

covacat · Jan 9, 2022

if you use mpd5 (ng0) you can try it's internal nat support

D-FENS · Jan 10, 2022

u_gene said:
Blocking rule for other interfaces is 01300.

Indeed. Maybe you have multiple IP addresses on the same interface? Rule 01100 would allow the packets (because the interface is the same) but they would not be NAT-ed (because the IP address is different)?
This could happen if you use jails with a different IP address on your ng0 interface (which is the standard case if you are not using VNET).
Do you have jails/VMs on the host?

u_gene said:
I've changed port number to 16542 for testing.

u_gene said:
I've tested this ipfw script on FreeBSD 11.4. Only fragmentation UDP packets counted and denyed by rules 11000-11100.
Something changed in NAT in FreeBSD 12???

What I am guessing here is that the packets are randomly coinciding with the same port number. If my hypothesis is correct you cannot rely on testing on another machine, because it might still behave incorrectly it could not be seen due to random chance.

D-FENS · Jan 10, 2022

covacat said:
if you use mpd5 (ng0) you can try it's internal nat support

I believe it is a configuration issue, not a bug in the kernel NAT. If the configuration is done correctly it should work as expected.

u_gene · Jan 10, 2022

roccobaroccoSC said:
Do you have jails/VMs on the host?

No. No jail/VM on this machine. No multiple IP's. It's just a router with bridge (br0) and ng interfaces.

I believe it is a configuration issue, not a bug in the kernel NAT. If the configuration is done correctly it should work as expected.

Yes. I think the same. But, my opinion is - configuration issue in FreeBSD 12.3 (kernel, drivers or something). The same problem persisted when I changed kernel NAT with NATD daemon (with divert call).
But, in FreeBSD 12.2 on same machine with GENERIC kernel both NAT work perfect.

D-FENS · Jan 11, 2022

u_gene said:
No. No jail/VM on this machine. No multiple IP's. It's just a router with bridge (br0) and ng interfaces.

Yes. I think the same. But, my opinion is - configuration issue in FreeBSD 12.3 (kernel, drivers or something). The same problem persisted when I changed kernel NAT with NATD daemon (with divert call).
But, in FreeBSD 12.2 on same machine with GENERIC kernel both NAT work perfect.

I have an idea! Maybe some packets come through while the firewall brings up the rules (right at start)?
You could try and reset the log counters, then make a measurement over a longer period and see if something comes up.

Also, try to activate log and see which packets exactly come through as suggested above.

obsigna · Jan 11, 2022

https://www.freebsd.org/security/advisories/FreeBSD-EN-22:06.libalias.asc

This securitiy advisory, which was published today, seems to be relevant. It would explain, why 12.2 does work but 12.3 not. Perhaps, you want to upgrade to 12.3-RELEASE-p1

IPFW IPFW and kernel NAT redirect_port not working.

u_gene

obsigna

Profile disabled

u_gene

covacat

facedebouc

D-FENS

u_gene

mer

obsigna

Profile disabled

u_gene

covacat

D-FENS

D-FENS

u_gene

D-FENS

obsigna

Profile disabled