PF pf with nat and rdr persistent failure

[tege]

My failing setup is not complex. The machine is virtualised with xen. The physical network interfaces belong to a FreeBSD 13 guest which is intended to work as a firewall, routing traffic to a number of guests.

Networking from that FreeBSD system works perfectly. I can make TCP connections, pass ICMP packets, etc.

Here is my stripped-down /etc/pf.conf on that system:

  ext_if="bnxt0"
  ext_addr="1.2.3.4"    # real IP address suppressed 
  diener="192.168.0.2"
  nat on $ext_if inet from ! ($ext_if) to any -> ($ext_if)
  rdr  on $ext_if proto tcp from any to $ext_addr port 2201 -> $diener port 2201

The problem is that the guest "diener" gets poor access to the outside world. Its traffic is silently blocked by the FreeBSD fw system. If I try to connect to port 2201 to trigger the rdr rule, the SYN packet reaches diener as it should, and diener sends a SYN ACK, but that packet is silently tossed by the FreeBSD fw.

TCP traffic from diener to the wider world is blocked already at the first SYN.

ICMP works fine.

Could somebody please enlighten me about the idiosyncrasies with FreeBSD here? Is pf not supposed to be used? Do I need to compile a kernel which works with pf?

(I have used pf for > 20 years, but not with FreeBSD for a very long time. In the last 15 years, it has been with NetBSD. So no newbie, but thus far I have not succeeded with even the trivial pf configuration above with nat and rdr in this system with FreeBSD 13.)

 

[SirDice]

Here is my stripped-down /etc/pf.conf on that system:

Is this the entire configuration? Or did you only post the rules you think are relevant?

Its traffic is silently blocked by the FreeBSD fw system.

Nothing in your rules would block this traffic. The default of PF is to allow everything. As long as you don't have a block rule nothing will get blocked by the firewall.

 

[tege]

That's my entire stripped-down pf.conf, except that 1.2.3.4 is something different here. I have gateway_enable="YES" in rc.conf too, of course.

I do not have any block rules in pf.conf, and it is my understanding that a broad "pass all' is the default.

It seem like an implicit "block in" for all internal interfaces is in effect. This firewall protects the Internet against my machine, instead of the opposite. (I have dumped the rules and there is of course no such block rule.)

Since there is no block rule, there is no logging ("block log ..."). This stymies my debugging attempts. The incoming packets are silently dropped by something.

 

[SirDice]

It seem like an implicit "block in" for all internal interfaces is in effect.

There isn't. Not by default. Possible to enable though, but requires a recompilation of the kernel.

KERNEL OPTIONS
     The following options in the kernel configuration file are related to pf
     operation:

     PF_DEFAULT_TO_DROP  Change default policy to drop by default

But this would also block ICMP and I understood that works? Maybe the problem is on the Xen side? If you look with tcpdump(1) on the bnxt0 can you see the traffic going out?

 

[tege]

Indeed. I saw that kernel option before I posted. (I have actually spent a ridiculous amount of time trying to get this to work.)

I am using the kernel that comes with the FreeBSD release, and have the latest one which freebsd-update provided. So unless that was set by mistake in those kernels, PF_DEFAULT_TO_DROP is off.

This all started with upgrading the server hardware. The old server is 8 years old, and runs with NetBSD is the same role, i.e., a Xen guest with the hardware network interfaces mapped to the NetBSD guest.

Migrating this sort of setup is usually, if not trivial, not difficult. In this case, NetBSD does not support the 10 GbE hardware, so I switched to FreeBSD. And networking just refuses to work. Yes, the FreeBSD fw guest can access things, but its routing of packets from the other Xen domains simply refuses to work.

I'd love to hear som advice in how to debug this! If I can figure out what determines to toss packets and why it does so, then it would be simple to fix/workround.

(I am no FreeBSD newbie, although to prefer NetBSD for networking tasks. I have run FreeBSD as my primary OS since FreeBSD 1.1! The main Xen guests, i.e., web server, shell access, on these servers is FreeBSD.)

 

[SirDice]

I am using the kernel that comes with the FreeBSD release

That's the GENERIC kernel, it doesn't have this switched on.

So unless that was set by mistake in those kernels, PF_DEFAULT_TO_DROP is off.

Has never been enabled on any of the GENERIC kernels, for any release. Unless someone else recompiled the GENERIC kernel and added that option. But as I said, that would also block ICMP, it would block everything.

That diener host, where is it? Is it a jail running on the FreeBSD host? If it's a jail how's that jail set up? If it's not a jail where is it? How is it connected to the FreeBSD host?

 

[tege]

This computer currently has 3 installs:
1. "diener": A Gentoo main system which runs Xen. (Dom0 is Xen speak, but that is irrelevant here.)
2. "fw": A FreeBSD system which owns the hardware Ethernet interfaces (pci mapping trickery).
3. "shell": A FreeBSD shell access system.

(The goal is to have hundreds of guests here, but we're not there yet. The system it replaces had hundreds of guests.)

There are bridge interfaces to connect things. These show up as xn0 in FreeBSD. The setup of those is specific to Xen. All setup is inherited from the old server, i.e., not something I am "trying to get right" now.

Note that I can ping to any responding Internet ICMP echo host from diener, which is a testament to that the basic networking is sound. TCP connections are, however not possible as all *outgoing* TCP packets from diener are binned by fw.

 

[SirDice]

TCP connections are, however not possible as all *outgoing* TCP packets from diener are binned by fw.

On what interface on the FreeBSD host do these packets arrive? Have you looked at tcpdump(1) on that interface there are actually packets arriving? Even if the firewall would be blocking packets should still arrive on the interface.

 

[tege]

To clarify the configuration, here is a little picture with a networking perspective:

[ diener, Gentoo, Xen Dom0 ] -> [ fw, FreeBSD, Xen hvm guest ] -> Internet
br0 xn0 bnxt0

There is also a FreeBSD system which relies on fw for Internet access:

[ shell, FreeBSD, Xen hvm guest ] -> [ fw ... ]

Here a tcpdump from an attempt at downloading from a FreeBSD server. First two lines from xn0 and then two lines for the NAT'ed packets on bnxt0:

17:23:45.614969 IP 192.168.0.2.34580 > 147.28.184.42.443: Flags [S], seq 390285552, win 64240, options [mss 1460,sackOK,TS val 2300400230 ecr 0,nop,wscale 7], length 0
17:23:46.637624 IP 192.168.0.2.34580 > 147.28.184.42.443: Flags [S], seq 390285552, win 64240, options [mss 1460,sackOK,TS val 2300401253 ecr 0,nop,wscale 7], length 0

17:23:45.614988 IP 130.242.124.100.34580 > 147.28.184.42.443: Flags [S], seq 390285552, win 64240, options [mss 1460,sackOK,TS val 2300400230 ecr 0,nop,wscale 7], length 0
17:23:46.637639 IP 130.242.124.100.34580 > 147.28.184.42.443: Flags [S], seq 390285552, win 64240, options [mss 1460,sackOK,TS val 2300401253 ecr 0,nop,wscale 7], length 0

There is no SYN ACK reply. Somehow, all (well, all I tested!) Internet hosts dislike the NAT'ed packets.

If attempting the exact same download from fw directly, things work:

17:04:44.278194 IP 130.242.124.100.35717 > 147.28.184.42.443: Flags [S], seq 3229559594, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 3259773528 ecr 0], length 0
17:04:44.302314 IP 147.28.184.42.443 > 130.242.124.100.35717: Flags [S.], seq 1355507879, ack 3229559595, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 3993094465 ecr 3259773528], length 0
17:04:44.302331 IP 130.242.124.100.35717 > 147.28.184.42.443: Flags [.], ack 1, win 1027, options [nop,nop,TS val 3259773554 ecr 3993094465], length 0

It is not the Gentoo system diener which is uniquely naughty, attempting the same operation from "shell" fails just like downloads from diener: First two lines from xn2 (yes, "shell" is connected via another virtual network) and then two lines for the NAT'ed packets on bnxt0:

17:36:02.797423 IP 10.0.0.3.58180 > 147.28.184.42.443: Flags [S], seq 3236844768, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1462792950 ecr 0], length 0
17:36:03.855961 IP 10.0.0.3.58180 > 147.28.184.42.443: Flags [S], seq 3236844768, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1462794010 ecr 0], length 0

17:36:02.797442 IP 130.242.124.100.58180 > 147.28.184.42.443: Flags [S], seq 3236844768, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1462792950 ecr 0], length 0
17:36:03.855976 IP 130.242.124.100.58180 > 147.28.184.42.443: Flags [S], seq 3236844768, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1462794010 ecr 0], length 0

Another thing tested: I can run diener without Xen too, giving it direct access to bnxt0 (not known under that name, of course). That too works fine.

I have no idea what to do next. Of course, I could stick an older network card into the server and run the old, stable NetBSD config. I'd rather not, for several reasons.

PS. When gathering the tcpdumps above, I had a "static-port" options to the nat pf rule, that's why the port numbers are the same in the "inside" and "outside". I didn't expect static-port to make any difference, and indeed it did not.

[ I have tried to edit the tcpdump text above to the originally inserted text. Clearly, inserting text verbatim in this forum is not straightforward. ]

 

[tege]

This forum interface edits my text, making it meaningless. The text to the right if Flags in many places above lack . It also put overstrikes over lots of the text, but that I could fix by editing the message.

 

[tege]

That overstrike in the immediately preceding message was not inserted by me! This is getting pretty silly. And after the "lack" there was supposed to be a left bracket, a capital S, and a right bracket.

 

[SirDice]

This forum interface edits my text, making it meaningless

Put system output in a [code]...[/code] block, or else the forum software will indeed try to 'translate' smilies and other 'special' characters.

Formatting Guidelines

1. Please write as clearly as possible. Use English punctuation and capitalization, and avoid abbreviating things. This helps to improve the clarity of messages and make problems easier to solve. 2. Use BBCode formatting tags. As with punctuation, these tags make messages easier to read...

forums.freebsd.org