TCP packets sent out-of-order via netgraph tunnel on FreeBSD 13.5— causing large delays

[robot468]

Hi all,

I'm seeing unexpected behavior when sending HTTPS (TLS) POST requests through a netgraph-based tunnel. TCP packets are sent out-of-order, which causes SACKs, delays, and poor performance.

Setup:

• Client: 10.10.0.1 — FreeBSD
• Tunnel: ng_iface ↔ ng_ksocket (UDP), interface ng0
• VPN endpoint: 10.10.0.2 — also FreeBSD, identical config
• Traffic path: macOS → 10.10.0.1 (netgraph) → 10.10.0.2 → Internet

Sending a POST request to an external server (e.g., YouTube) from 10.10.0.2 works fast.
From 10.10.0.1, the same request takes ~2 seconds or more, and here's why:

tcpdump snippet from 10.10.0.1:

12:02:56.438842 IP 10.10.0.1.51100 > ...: seq 518:598      # 80 bytes sent
12:02:56.514085 IP 10.10.0.1.51100 > ...: seq 9704:9735    # out-of-order jump
12:02:56.589029 IP ... > 10.10.0.1.51100: SACK 9704:9735   # server sees the gap
12:02:56.799401 IP 10.10.0.1.51100 > ...: seq 598:1986     # late fill-in

This continues with slow re-filling (`1986:3374`, `3374:4762`, etc.) and final ACK happens much later.

Not the cause:

• TLS — same curl binary on 10.10.0.2 runs instantly
• MTU — no IP fragmentation (MSS respected: 1400)
• Packet capture — verified via ng0, full snaplen

Suspected causes:

• netgraph delivery out-of-order
• TCP buffer or congestion window misbehavior
• delayed ACK + netgraph latency affecting stack logic

System notes:

• FreeBSD 13.5 on both hosts
• netgraph: only ng0 ↔ ng_ksocket
• net.inet.tcp.sendspace=65536
• net.inet.tcp.delayed_ack=1

Question:
Has anyone observed out-of-order TCP transmission over netgraph tunnels?
Any tunables, workarounds, or known pitfalls I should look into?

Thanks!

 

[robot468]

UPD:

Here's what I captured via pflog:

15:46:01.784956 IP 10.10.0.1 > 209.85.233.198.443: seq 597:9703, length 9106
15:46:01.785020 ICMP 127.0.0.1 > 10.10.0.1: 209.85.233.198 unreachable - need to frag (mtu 1472)

tunnel:

ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> metric 0 mtu 1472
        inet 10.10.0.1 --> 10.10.0.2 netmask 0xfffffffc
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

So TCP sent a 9106-byte segment — well above the effective tunnel MTU.
The kernel responded with a local ICMP "fragmentation needed", but by then the damage was done:
the segment had been dropped, and the server responded with SACKs for out-of-order data.

This explains all the delays, gaps, and retransmissions.

why it ignoring mtu?