IPFW ipfw in kernel NAT not works

Hi, guys

After upgrade to 13.0, I found a nat rule not works.
rule 2004 should modify the source ip and then 2005 should not match again. (I make a mistake, rule 2005 match in packet from tun1, not out packet)
but both counter and ipfw log prove nat not works.

one more thing, although I could capture packet match rule 2005, but I can't capture packet from all other interface, just seems like this packet disappear.
if I disable rule 2004, I can capture packet on genet0.

The same config works fine on 12.0

The packet path:
ng0 (pppoe, source ip 192.168.192.100) -> tun0 (ipfw fwd) -> tun1 (userspace nat, source ip nat to 192.168.194.2) -> genet0(in kernel nat, source ip should nat to 192.168.2.119)

I think the packet disappear after match rule 2004, does it relate with checksum? I have set net.inet.tcp.tso="0" and disable one_pass.
How could I check state of libalias?

Code:
root@rpi4:/var/log # ipfw show
00100   276    28952 allow ip from any to any via lo0
00200     0        0 deny ip from any to 127.0.0.0/8
00300     0        0 deny ip from 127.0.0.0/8 to any
00400     0        0 deny ip from any to ::1
00500     0        0 deny ip from ::1 to any
00600     0        0 allow ipv6-icmp from :: to ff02::/16
00700     0        0 allow ipv6-icmp from fe80::/10 to fe80::/10
00800     0        0 allow ipv6-icmp from fe80::/10 to ff02::/16
00900     0        0 allow ipv6-icmp from any to any icmp6types 1
01000     0        0 allow ipv6-icmp from any to any icmp6types 2,135,136
01001     0        0 setfib 1 ip from 192.168.2.205 to any in via genet0
01002 10450  8024205 nat 11 ip from any to any in via tun0
01003     0        0 nat 11 ip from 192.168.2.205 to any out via tun0
01004  1168   100446 setfib 1 ip from 192.168.2.11 to any in via genet0
01006   989    82436 nat 11 ip from 192.168.2.11 to any out via tun0
01050     0        0 deny ip from any to 192.168.0.0/16 in via genet0.100
01051     0        0 setfib 1 ip from 192.168.100.0/24 to any in via genet0.100
01052     0        0 nat 11 ip from 192.168.100.0/24 to any out via tun0
01053   587    37337 nat 10 ip from 192.168.192.0/24 to 192.168.2.0/24 out via genet0
02000   587    37337 skipto 65000 ip from 192.168.192.0/24 to 192.168.2.0/24
02002 53925 16308466 nat 10 ip from any to any in via genet0
02003 22506  3467602 fwd 192.168.192.11 ip from 192.168.192.0/24{100-120} to not 192.168.192.0/24{90-120}
02004   560    49418 nat 10 ip from 192.168.194.0/24 to any out via genet0
02005   174    22412 allow log ip from 192.168.194.0/24 to any
65000 71728 26613162 allow ip from any to any
65535     0        0 deny ip from any to any
root@rpi4:/var/log # ipfw nat show config
ipfw nat 10 config ip 192.168.2.119
ipfw nat 11 config if tun0
 
To make a minimal test, I install wireguard-go, and make it bind with tun0, but it still not works, any issue in ipfw config?
Code:
[root@rpi4 /home/freebsd/wireguard]# wg
interface: tun0
  public key: 2zstlhI0j9UoEBMPln6YBSJUIANlMV0lrXsmmZXmXGo=
  private key: (hidden)
  listening port: 11819

peer: TaAAEdIDsa8rJrg+RTh1Zx/f+D+47CGamKh6Jk6WXz0=
  endpoint: 192.168.2.119:45525
  allowed ips: 10.0.0.2/32
  latest handshake: 1 minute, 9 seconds ago
  transfer: 16.80 KiB received, 1.45 KiB sent

[root@rpi4 /home/freebsd/wireguard]# ipfw show
00100   204   19224 allow ip from any to any via lo0
00200     0       0 deny ip from any to 127.0.0.0/8
00300     0       0 deny ip from 127.0.0.0/8 to any
00400     0       0 deny ip from any to ::1
00500     0       0 deny ip from ::1 to any
00600     0       0 allow ipv6-icmp from :: to ff02::/16
00700     0       0 allow ipv6-icmp from fe80::/10 to fe80::/10
00800     0       0 allow ipv6-icmp from fe80::/10 to ff02::/16
00900     0       0 allow ipv6-icmp from any to any icmp6types 1
01000     0       0 allow ipv6-icmp from any to any icmp6types 2,135,136
02000   108    6480 nat 10 ip from 10.0.0.0/24 to any out xmit genet0
02001   661   60993 nat 10 ip from any to any in via genet0
65000 11003 3993502 allow ip from any to any
65535     0       0 deny ip from any to any
[root@rpi4 /home/freebsd/wireguard]# ipfw nat show config
ipfw nat 10 config if genet0
 
One more test, try log the packet after NAT, I can see the log of rule 2002, but can't capture packet on genet0.

Code:
root@rpi4:/home/freebsd # ipfw list
00100 allow ip from any to any via lo0
00200 deny ip from any to 127.0.0.0/8
00300 deny ip from 127.0.0.0/8 to any
00400 deny ip from any to ::1
00500 deny ip from ::1 to any
00600 allow ipv6-icmp from :: to ff02::/16
00700 allow ipv6-icmp from fe80::/10 to fe80::/10
00800 allow ipv6-icmp from fe80::/10 to ff02::/16
00900 allow ipv6-icmp from any to any icmp6types 1
01000 allow ipv6-icmp from any to any icmp6types 2,135,136
02000 nat 10 log tag 10 ip from 10.0.0.0/24 to any out via genet0
02001 nat 10 ip from any to any in via genet0
02002 allow log tagged 10
65000 allow ip from any to any
65535 deny ip from any to any

root@rpi4:/home/freebsd/wireguard # tail -f /var/log/security
Apr  9 21:30:34 rpi4 kernel: ipfw: 2000 Nat ICMP:8.0 10.0.0.2 114.114.114.114 out via genet0
Apr  9 21:30:34 rpi4 kernel: ipfw: 2002 Accept ICMP:8.0 192.168.2.19 114.114.114.114 out via genet0
Apr  9 21:30:36 rpi4 kernel: ipfw: 2000 Nat ICMP:8.0 10.0.0.2 114.114.114.114 out via genet0
Apr  9 21:30:36 rpi4 kernel: ipfw: 2002 Accept ICMP:8.0 192.168.2.19 114.114.114.114 out via genet0             <=== nat works and source ip have changed.

...

root@rpi4:/home/freebsd # tcpdump -i genet0 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on genet0, link-type EN10MB (Ethernet), capture size 262144 bytes
(No packet could be captured)
 
So it's not relate with NAT, it cause by packet drop before send out to genet0 on rpi4.
After disable txchecksum on genet0, I can see packet on genet0, but I can't see packet on router, maybe genet0 drop packet?
Even forward without nat, genet drop packet.
ifconfig genet0 -txcsum
 
Back
Top