PF PF NAT very slow - 5s delay for all connections

[hukl]

I have servers, each with a public IP in a /28 VLAN. On these servers I have a 10.0.0.0/24 VLAN with some aliases configured.

The servers can talk via the public v4 addresses as well as internally via the 10.0.0.0/24 IPs. Now I wanted to create LAN only jails and set up a simple NAT with pf. This works BUT it is very very slow. Every connection I open from within the jail to the public internet has a ~5 second delay before things happen. A drill www.google.com @8.8.8.8 takes 5s like everything else and I'm absolutely puzzled where this delay is coming from.

set skip on lo0
scrub on em0 all

ext_if = "em0"
lan_if = "vlan0"

nat on $ext_if from {vlan0:network} to any -> ($ext_if)

pass all

In pflog I can see for example the initial drill query going through the nat and the response getting through 5 seconds later.

sudo tcpdump -eni pflog0 
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pflog0, link-type PFLOG (OpenBSD pflog file), capture size 262144 bytes
12:28:29.611433 rule 0/0(match) [uid 0]: nat out on em0: 10.0.0.2.54528 > 8.8.8.8.53: 31971+ A? www.google.com. (28)
12:28:34.835908 rule 0/0(match) [uid 0]: nat out on em0: X.X.X.X.57044 > 8.8.8.8.53: 31971+ A? www.google.com. (28)

In my rc.conf I have gateway_enable="YES" set of course. Any help / ideas would be appreciated greatly

 

[SirDice]

You have two IP addresses on $ext_if? One private range and a second global IP address? The ($ext_if) uses round-robin to use each address of that interface. So your first request is translated with a source address 10.0.0.2, which will eventually timeout because you're never going to get a response. The second request (after the first one timed out) used the global IP address as a source. That does work. The timeout on the first request is the reason for the delay.

Simple solution, don't put private LAN and internet WAN addresses on the same interface. Another option is to use ($ext_if:0) to specifically use the first address of that interface. This will work fine with static addresses but won't work reliably with DHCP (DHCP can cause the addresses to 'swap' position on the interface).

If you used VLANs make sure you're not mixing up tagged and untagged traffic somewhere.

 

[hukl]

Oh yes indeed - checked with tcpdump and that was exactly what happened! Thank you!