IPFW Trouble with filtering local traffic with port number using ipfw

I am using jails for an emulation task. Each jail has its own IP, and each node hosting jails has its own IP which is not used by a jail. Sometimes I want to redirect traffic intended for a jail to the host node where jail is running. I use ipfw_nat for that. The IPFW ruleset roughly looks like:

1. Allow all traffic to a jail on a port belonging to something being emulated
2. Direct all other traffic to a NAT that translates to the host node address

For some reason this allows traffic to jails running on other nodes, but traffic intended for jails running on the same physical interface end up not matching rule 1, and get NAT'd. I notice if I have a big catchall rule before the NAT redirect like:

Code:
ipfw add allow all from me to me keep-state

Then things work. Why is that? How can I get such filtering on specific ports? Is there another approach you could suggest?

Here is a specific example ruleset, pruned a bit for relevancy:
Code:
00300 allow ip from any to 169.254.228.82 dst-port 65456 keep-state
00400 allow ip from any to 169.254.228.86 dst-port 65456 keep-state
00500 nat 1001 ip from any to 169.254.228.82 keep-state
00600 nat 3001 ip from any to 169.254.228.86 keep-state
65535 allow ip from any to any

The NATs are:
Code:
ipfw nat 3001 config ip 1.111.228.82 same_ports
ipfw nat 1001 config ip 1.111.228.82 same_ports

The interface corresponds to an Infiniband card, though I'm not sure if that is relevant:

Code:
ib1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 2044
  options=80018<VLAN_MTU,VLAN_HWTAGGING,LINKSTATE>
  lladdr 0.0.4.6.fe.80.0.0.0.0.0.0.0.15.1b.0.10.81.c.df
  inet 1.111.228.82 netmask 0xffff0000 broadcast 1.111.255.255 zone 1
  inet 169.254.228.82 netmask 0xffff0000 broadcast 169.254.255.255 zone 1
  inet 169.254.228.86 netmask 0xffff0000 broadcast 169.254.255.255 zone 1
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Infiniband autoselect
  status: active
 
Perhaps the problem is caused by the Link-local addresses that you utilize in your setup. Those addresses are special in many respects, e.g. routers are not supposed to forward packets to addresses in the space 169.254.0.0/16. Perhaps it might be a good idea to change the link local addresses to addresses from one of the 3 private network address spaces.

The keep-state directives in your rules do not make so much sense, because you got no check-state rule in your rule set. If you really want to setup a stateful nat'ting firewall, then you need to take much more care about the position of the rules. To begin with, in a stateful ipfw(8) rule set you need to define 2 rules for each NAT instance, one for outgoing and one for incoming traffic. And all rules that may pass the respective NAT instance must be embraced by said 2 NAT rules each. In addition, in the stateful case, you need to switch off the sysctl(8) variable net.inet.ip.fw.one_pass, i.e. set it to 0 in file /etc/sysctl.conf.
 
Thanks a ton for the reply.

Perhaps the problem is caused by the Link-local addresses that you utilize in your setup. Those addresses are special in many respects, e.g. routers are not supposed to forward packets to addresses in the space 169.254.0.0/16. Perhaps it might be a good idea to change the link local addresses to addresses from one of the 3 private network address spaces.

I've changed this to 192.168.0.0/16 to no avail. But worth a try.

The keep-state directives in your rules do not make so much sense, because you got no check-state rule in your rule set.

Can you update my knowledge on this? My reading of the man page and elsewhere is that check-state is just an optimization. keep-state implies a check against the dynamic rules, whereas check-state means 'check against the dynamic rules now', to avoid checking against potentially many other rules. In this case there aren't a ton of rules, so I don't see the need for that optimization. Does check-state do something else as well? I tried throwing a rule for this at the top of my set and it didn't seem to fix the problem. Again..worth a try.

If you really want to setup a stateful nat'ting firewall, then you need to take much more care about the position of the rules. To begin with, in a stateful ipfw(8) rule set you need to define 2 rules for each NAT instance, one for outgoing and one for incoming traffic. And all rules that may pass the respective NAT instance must be embraced by said 2 NAT rules each. In addition, in the stateful case, you need to switch off the sysctl(8) variable net.inet.ip.fw.one_pass, i.e. set it to 0 in file /etc/sysctl.conf.

Thanks for the input here; I flipped off the one_pass sysctl and saw no effect, but that does seem like the right thing to do. Would it be asking too much to ask for an example of how to wrap NAT differently in this case? I've tried all the combinations of in/out dst/src ports etc. and it doesn't seem to solve this particular problem of communicating within the same node. I can still communicate out to other nodes and they can communicate back using the rules that bypass the NAT, so I suspect this problem does not relate to the NAT per se, but to the rules that help traffic avoid it. I could be wrong.
 
No, keep-state is what tells IPFW to create a so called "dynamic rule" on the first match of the rule with the keep-state keyword, essentially a state. The dynamic rules are not checked though until you use check-state in your ruleset.
 
mbryan, reading the description of the check-state action rule in the manual ipfw(8), I realize, that you are correct, the first keep-state directive in a rule set adopts the role of check-state in case it is not present.
ipfw 8 said:
check-state
Checks the packet against the dynamic ruleset. If a match is
found, execute the action associated with the rule which gener-
ated this dynamic rule, otherwise move to the next rule.
Check-state rules do not have a body. If no check-state rule is
found, the dynamic ruleset is checked at the first keep-state or
limit rule.

I got working stateful nat'ting IPFW firewalls on my FreeBSD systems which among other duties serve as the router on the LAN/WAN boundary. The firewall is operating only one NAT instance, and the setup is working without jails. I do not have any experience with jail networking, and I never ran tests with more than one NAT instance. So, for sure the following commented example may be somewhat limited in your usage case.

Example of a stateful NAT'ting IPFW firewall ruleset
  1. Actually the ipfw(4) configuration is done by a shell script
    and we may utilize sh(1) script facilities. My IPFW setup
    script starts with some definitions and initialization:
    Code:
    #!/bin/sh
    
    WAN="re0"
    LAN="bridge0"
    CMD="/sbin/ipfw -q"
    
    $CMD flush
    $CMD nat 1 config if $WAN unreg_only \
                              reset \
                              redirect_port tcp 192.168.0.26:2626 2626
    ...
    My NAT instance also does redirection on port 2626 to a service provided by a LAN machine on address 192.168.0.26.


  2. IPFW evaluates its ruleset sequentially from top to bottom, and for performance reasons it is best to put the rules matching most of the traffic at the beginning:
    Code:
    ...# Allow anything within the LAN - interface with heaviest traffic shall come first.
    $CMD add 10 allow ip from any to any via $LAN
    $CMD add 20 allow ip from any to any via lo0
    $CMD add 30 allow ip from any to any via em*
    $CMD add 40 allow ip from any to any via ng*
    
    # Don't trust 3rd party network devices (Printer, Router, TV, IoT devices, etc.),
    # that got assigned addresses of the upper private network range 192.168.0.192-255.
    $CMD add 60 deny ip from 192.168.222.192/26 to any via $WAN
    
    # Catch spoofing from outside.
    $CMD add 70 deny ip from any to any not antispoof in recv $WAN
    ...
    The above pretty much deals with everything wich does not need to pass the NAT.


  3. Now the NAT block starts. Note, how it is encapsulated by the NAT rules for incoming and outgoing traffic:
    Code:
    ...
    # NAT rule for incomming packets.
    $CMD add 100 nat 1 ip4 from any to any in recv $WAN
    $CMD add 101 check-state
    
    # Allow specific outgoing connections.
    $CMD add 500 skipto 10000 tcp from 192.168.0.26 to any 53 out xmit $WAN setup keep-state
    $CMD add 510 skipto 10000 udp from 192.168.0.26 to any 53 out xmit $WAN keep-state
    
    # Rules for outgoing traffic - allow everything that is not explicitely denied.
    $CMD add 1000 deny ip from not me to any 25,53 out xmit $WAN
    $CMD add 1010 deny ip from any to any 5353 out xmit $WAN
    $CMD add 1020 deny ip from any to any 1900,2195,2196,4488,5223,5350,5351 out xmit $WAN
    
    # Allow all other outgoing connections.
    $CMD add 2000 skipto 10000 tcp from any to any out xmit $WAN setup keep-state
    $CMD add 2010 skipto 10000 udp from any to any out xmit $WAN keep-state
    
    # Rules for incomming traffic - deny everything that is not explicitely allowed.
    $CMD add 5000 allow tcp from any to me 25,53,80,443,587,993,995,3939 in recv $WAN setup keep-state
    $CMD add 5010 allow udp from any to me 53,500,4500 in recv $WAN keep-state
    $CMD add 5020 allow udp from any to me in recv $WAN frag
    
    # Rules for allowing dial-in calls to services which are listening on a LAN interface behind the NAT
    $CMD add 6000 skipto 10000 tcp from any to any 2626 in recv $WAN setup keep-state
    
    # Catch tcp/udp packets, but don't touch gre, esp, icmp traffic.
    $CMD add 9998 deny tcp from any to any via $WAN
    $CMD add 9999 deny udp from any to any via $WAN
    
    # NAT rule for outgoing packets.
    $CMD add 10000 nat 1 ip4 from any to any out xmit $WAN
    ...

  4. Finally, the script finishes with one last rule, that allows anything else:
    Code:
    ...
    # Allow anything else – just in case IPFW is not configured as open firewall.
    $CMD add 65534 allow ip from any to any

Some notes:
  • Note, that some packets may pass the firewall twice, because they may become re-injected after NAT
  • Note, how the NAT rules apply to ipv4 traffic only, since a comparable NAT for ipv6 does not exist.
  • Note, the sensible use of the keywords recv and xmit instead of via, which makes the ruleset more effective.
  • Note, how fragments need to be allowed in general.
  • Note, how rules 9998 and 9999 inherently allow ally special traffic.
Any doubts left? Please ask!
 
Last edited:
Back
Top