PF NAT translating to expired IP address?

[mickey]

Today I discovered a strange issue with NAT on my router running FreeBSD 10.3-RELEASE-p7, and after hours of fiddling, I'm still not sure what to make of it...

The machine connects my internal LAN and wireless LAN to the internet by means of user-ppp over an ADSL line, so my etxernal interface for all intents and purposes is tun0. IP address assignment is done dynamically through PPP, so the external address changes every once in a while. I have confirmed that the iface-alias option in PPP is disabled and also ifconfig -v tun0 does not show any alias addresses on the interface:

tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1492
  options=80000<LINKSTATE>
  inet 77.180.129.49 --> 62.52.201.187 netmask 0xffffffff
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  groups: tun
  Opened by PID 575

In my /etc/pf.conf is a NAT rule quite similar to many examples:

ExtIF = "tun0"
[...]
nat on $ExtIF inet from 10.6.0.0/16 -> ($ExtIF)

Which appears as follows after loading the ruleset (notice the 'round-robin' automatically added):

nat on tun0 inet from 10.6.0.0/16 to any -> (tun0) round-robin

Now what seems to be happening is that some packets (mostly UDP from what I can tell) have their source address translated to an IP address I had almost 20hrs ago. And I really have no idea where this old expired address is coming from.

If I remove the braces around the interface name in the NAT rule like so:

nat on $ExtIF inet from 10.6.0.0/16 -> $ExtIF

then the problem seems to be gone (as is the implicit 'round-robin'). This would of course cease to work when the IP address on the interface changes the next time. A better solution seems to be this:

nat on $ExtIF inet from 10.6.0.0/16 -> ($ExtIF:0)

But that still does not explain where this expired IP address is still coming from and why it is not displayed in the output of ifconfig -v tun0. This old address just shouln't be around anymore anywhere.

 

[`Orum]

The address is retrieved by pf when your ruleset is loaded. If you've last loaded your firewall rules some time ago (when you had a different address), then the now-old address will persist until you manually reload the ruleset (e.g. pfctl -f /etc/pf.conf).

I'm not sure if there's a way to get pf to use an interface name instead of it's associated address at the time of loading, but I'd be curious to find out.

 

[mickey]

The address is retrieved by pf when your ruleset is loaded. If you've last loaded your firewall rules some time ago (when you had a different address), then the now-old address will persist until you manually reload the ruleset (e.g. pfctl -f /etc/pf.conf).

This is not the case with interface names surrounded by parentheses, which should make pf automatically adapt to any address change that happens on the corresponding interface. Specifically pf.conf(5) states:

Host name resolution and interface to address translation are done at ruleset load-time. When the address of an interface (or host name) changes (under DHCP or PPP, for instance), the ruleset must be reloaded for the change to be reflected in the kernel. Surrounding the interface name (and optional modifiers) in parentheses changes this behaviour. When the interface name is surrounded by parentheses, the rule is automatically updated whenever the interface changes its address. The ruleset does not need to be reloaded. This is especially useful with nat.

That still does not explain why after an address change it seemed like the new and the old address were being used. The old address should not be around anymore, as it is invalid from the moment the address has changed.

 

[`Orum]

This is not the case with interface names surrounded by parentheses, which should make pf automatically adapt to any address change that happens on the corresponding interface.

Ah, you are correct. However, the man page may not be, or perhaps something else may be going on.

For example, if a TCP connection was made under your old address, and then your address changes, the entries in the NAT table may not be updated. In the best case scenario, the table would be flushed, as any connection would have to be reestablished (remember an established connection is based on 5 different criteria--protocol, src IP, dst IP, src port, dst port--and when any of them change it's recognized as a new connection), but I'm not sure if this actually happens. If it didn't though, connections established prior to the IP switch would still show up as the old IP address, and ultimately time out from the TCP timers.

With UDP and pf's "statefulness" that it creates for such connections, there's no telling how long these might sit in such a translation table if it's not flushed. To be honest I don't have much experience with address changes and pf as the few machines I have running pf and DHCP have reservations set for them.

Are you still seeing the old address used for new connections (e.g. for TCP anything with SYN flag set) after the IP switches?

Edit: Also take a look at pfctl -s state post-switch as well, that should reveal a lot.

 

[mickey]

For example, if a TCP connection was made under your old address, and then your address changes, the entries in the NAT table may not be updated. In the best case scenario, the table would be flushed, as any connection would have to be reestablished (remember an established connection is based on 5 different criteria--protocol, src IP, dst IP, src port, dst port--and when any of them change it's recognized as a new connection), but I'm not sure if this actually happens. If it didn't though, connections established prior to the IP switch would still show up as the old IP address, and ultimately time out from the TCP timers.

With UDP and pf's "statefulness" that it creates for such connections, there's no telling how long these might sit in such a translation table if it's not flushed. To be honest I don't have much experience with address changes and pf as the few machines I have running pf and DHCP have reservations set for them.

Are you still seeing the old address used for new connections (e.g. for TCP anything with SYN flag set) after the IP switches?

Edit: Also take a look at pfctl -s state post-switch as well, that should reveal a lot.

When I first noticed the issue, I was tracing SIP messages on the tun0 interface flowing between my asterisk server and my ITSP. First thought was that maybe asterisk was still using the old IP address, so I reloaded it, but that didn't seem to make a difference. Later I also noticed that traffic coming from another PC in the network running a Steam client also was in part using the old address, which would rather point to an issue with NAT. These are all exclusively UDP protocols, so no idea if that also affected TCP connections.

From my observation it really seemed as if the (tun0) in the pf.conf internally resolved to something that is more than just a single IP address. Not only seemed NAT to be using some sort of round-robin scheme, but also the firewall did not block traffic using the old expired IP address, which it normally should do, if (tun0) only gave a single IP address. Only after I inserted a rule to explicitly deny traffic from the old IP address, I could see packets using the old address appear on pflog0. I would expect such a behaviour if there actually was an alias address on the interface ( ppp has a mechanism called 'iface-alias' that could preserve the old address as an alias, but I never had that feature enabled), but ifconfig -v tun0 did not show any alias addresses on the interface.

As for the states, I have a script that is run from ppp every time the ADSL link goes down. This should also remove states using the old IP address from the state table, i.e:

/sbin/pfctl -k ${myaddr} >> $log 2>&1
/sbin/pfctl -k 0.0.0.0/0 -k ${myaddr} >> $log 2>&1

But even without that, old states should just timeout, as they dont match anymore after the address on the interface has been changed.

Funny thing is I have been using this setup with only minor modifications for years and this was the first time I noticed such a behaviour. Just recently my ISP switched my line over to another DSL access concentrator using bitstream access technology, but those changes affected mostly the ppp.conf file. Only other change I could think of, was a couple months back, when I switched my pf.conf from 'if-bound' to 'floating'.

As a temporary fix I have replaced all occurances of (tun0) where the interface address is needed with (tun0:0). But this still does not make much sense to me.

 

[mickey]

So I've done some more testing on this matter. Machine was upgraded to 10.3-RELEASE-p8 i386 in between, but the problem still persists, as soon as I change the NAT line in pf.conf from:

nat on $ExtIF from 10.6.0.0/16 -> ($ExtIF:0)

to this:

nat on $ExtIF from 10.6.0.0/16 -> ($ExtIF)

With an additional 'log' option the line then shows up in pfctl -s nat as:

nat log on tun0 inet from 10.6.0.0/16 to any -> (tun0) round-robin

When I now look at the logged packets, it indeed looks like pf is translating one packet using the current interface address and the next packet using the old address in a round-robin fashion:

04:03:43.171035 IP 78.55.215.132.65466 > 130.149.17.8.123: NTPv4, Client, length 48
04:04:03.171151 IP 77.179.47.159.51596 > 192.53.103.104.123: NTPv4, Client, length 48
04:04:13.171088 IP 78.55.215.132.61548 > 130.149.17.21.123: NTPv4, Client, length 48
04:04:34.171050 IP 77.179.47.159.63581 > 192.53.103.108.123: NTPv4, Client, length 48
04:04:49.171127 IP 78.55.215.132.58160 > 130.149.17.8.123: NTPv4, Client, length 48
04:05:12.171068 IP 77.179.47.159.51845 > 192.53.103.104.123: NTPv4, Client, length 48

But I still have no idea, where PF is getting that old address from. It's not shown in ifconfig -v tun0:

tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1492
        options=80000<LINKSTATE>
        inet 77.179.47.159 --> 62.52.201.184 netmask 0xffffffff
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        groups: tun
        Opened by PID 579

Nor in pppctl /var/run/ppp/inet show iface:

tun0 (idx 7) <UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1492 has 1 address:
  inet 77.179.47.159 --> 62.52.201.184 netmask 0xffffffff

Which clearly states, that the interface has one address, not two or more. And also I cannot seem to find the old address anywhere else I could possibly think of. Obviously new states are created using the old address, as NAT does this during translation. I can reload the pf.conf as often as I like, but that doesn't change a thing. So where does PF get this address from?

 

[kpa]

You might have uncovered a bug in the tun(4) driver where it keeps reporting multiple addresses to PF in case the local address has changed and the device hasn't been closed and re-opened by the process that created the device.

 

[mickey]

You might have uncovered a bug in the tun(4) driver where it keeps reporting multiple addresses to PF in case the local address has changed and the device hasn't been closed and re-opened by the process that created the device.

That's odd, as I have been using such a setup (user ppp + pf) for many years now, and this is the first time I noticed this problem. If this was the case, wouldn't it then also report multiple addresses to other things, like ifconfig?

 

[mickey]

Now that's funny. I was just thinking that if there actually was more than one address on the interface, then it should be possible to address it with tun0:1 and/or (tun0:1) in a simple test filter rule, like so:

block out quick from tun0:1 to 192.168.1.1
block out quick from (tun0:1) to 192.168.1.1

But when passing this to pfctl -n -f /etc/pf.conf, this is what it gives:

no IP address found for tun0:1
/etc/pf.conf:187: could not parse host specification
/etc/pf.conf:188: interface tun0:1 has bad modifier

Both variants however do work with the :0 modifier. Now I'm even more confused as before.

 

[mickey]

I've been up and down my configs several times now, rebuilt world at least 2x, and still having this issue. Any rdr rule referencing (tun0) shows the described misbehaviour. I will do some more experimentation with ppp's set ifaddr command to see if that has any influence on the problem.

Any other suggestions are still highly welcome.