IPFW Strange behavior with IPFW Nat and haproxy tcp mode

[clinta]

There seems to be some weird interaction between haproxy tcp mode and ipfw's NAT which cause connections to be killed prematurely and I'm hoping someone can help me troubleshoot this.

On this VPS I have 2 jails, one running nginx (172.16.0.6) and one running haproxy (172.16.0.4). These addresses are on the lo1 interface. On the nginx server I have a 32k test file I'm trying to download.

The haproxy config is very basic:

frontend fe
  bind 0:80
  mode tcp
  default_backend be

backend be
  mode tcp
  server be01 172.16.0.6:80

This works just fine. I can curl http://172.16.0.4/32k.file and get the full file with no problem.

The ipfw config to nat inbound 80 and redirect to haproxy is also very simple:

#!/bin/sh
# Flush out the list before we begin.
ipfw -q -f flush
cmd="ipfw -q add"

ext_if="vtnet0"
haproxy="172.16.0.4"
nginx="172.16.0.6"

ipfw -q nat 1 config if $ext_if \
    redirect_port tcp $haproxy:80 80

$cmd 0100 nat 1 ip4 from any to any via $ext_if
$cmd 00610 allow ip from any to any

But for some reason this doesn't work. When I curl http://outside-ip/32k.file curl reports the connection closed with 16640 bytes remaining to read.

If I take haproxy out of the picture, and redirect to $nginx:80 instead, everything works fine.

Also everything works fine if I use pf instead of IPFW. This is the pf config that works.

ext_if = vtnet0
haproxy = 172.16.0.6
nginx = 172.16.0.5

nat on $ext_if from any to any -> ($ext_if)
rdr on $ext_if proto tcp from any to any port 80 -> $haproxy port 80
pass in all

Here's what I get from curling the outside address with PF, where everything works fine:

$ curl -v http://X.X.X.X/32k.file > /dev/null
*   Trying X.X.X.X...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to X.X.X.X (X.X.X.X) port 80 (#0)
> GET /32k.file HTTP/1.1
> Host: X.X.X.X
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx/1.12.0
< Date: Sat, 29 Apr 2017 14:47:30 GMT
< Content-Type: application/octet-stream
< Content-Length: 32768
< Last-Modified: Sat, 29 Apr 2017 14:42:19 GMT
< Connection: keep-alive
< ETag: "5904a64b-8000"
< Accept-Ranges: bytes
< 
{ [14224 bytes data]
100 32768  100 32768    0     0   231k      0 --:--:-- --:--:-- --:--:--  233k
* Connection #0 to host X.X.X.X left intact

But here's what I get with ipfw active where the connection is closed early:

 $ curl -v http://X.X.X.X/32k.file > /dev/null
*   Trying X.X.X.X...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to X.X.X.X (X.X.X.X) port 80 (#0)
> GET /32k.file HTTP/1.1
> Host: X.X.X.X
> User-Agent: curl/7.54.0
> Accept: */*
> 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< HTTP/1.1 200 OK
< Server: nginx/1.12.0
< Date: Sat, 29 Apr 2017 14:50:27 GMT
< Content-Type: application/octet-stream
< Content-Length: 32768
< Last-Modified: Sat, 29 Apr 2017 14:42:19 GMT
< Connection: keep-alive
< ETag: "5904a64b-8000"
< Accept-Ranges: bytes
< 
{ [1192 bytes data]
 30 32768   30  9880    0     0   8694      0  0:00:03  0:00:01  0:00:02  8689* transfer closed with 16640 bytes remaining to read
 49 32768   49 16128    0     0  13085      0  0:00:02  0:00:01  0:00:01 13080
* Closing connection 0
curl: (18) transfer closed with 16640 bytes remaining to read

 

[obsigna]

In the ipfw(8) manual under sections Bugs, there is a comment about segmentation offloading:

Due to the architecture of libalias(3), ipfw nat is not compatible with the TCP segmentation offloading (TSO). Thus, to reliably nat your network traffic, please disable TSO on your NICs using ifconfig(8).

So you want to try to disable TSO by adding the -tso flag on the respective ifconig_vtnet0 directive in /etc/rc.conf.

In some environments there are more issues with vtnetX interfaces. For example, on a Google Cloud FreeBSD 11 installation, I had to turn off a whole set of facilities (-rxcsum -txcsum -lro -tso -vlanhwtso) in order to achieve reasonable transfer rates.

 

[clinta]

Thanks. That fixed it!