Solved FreeBSD 12 - copying big files over ssh fails with broken pipe

EDIT: I just found the errata for this issue. Fixed yesterday. Updated, tested, all good. Nevermind then.

Hello.

I'm not quite sure where to start looking on this one. Basically trying to scp a big file to a FreeBSD 12 server fails, while Linux and FreeBSD 11 servers work fine.

What it looks like is, it starts copying the file (say, at 2 MB/s), the speed increases (2.2 MB/s, 3.1 MB/s, 4 MB/s, 6 MB/s, etc.), then starts going down and in the end it stalls and results in Broken pipe. Seems like it has to do with packet loss, so the more unstable/slow the network, the higher chance of it happening.

The setup is simple. Mac client, FreeBSD 12 running on AWS, on a t2.micro instance. It doesn't matter if it's t2 (old type) or t3 (new type of instance, new network driver), so it doesn't look like the new driver has anything to do with it. At least I don't think so.

Bash:
 ~ » scp tmp/500MB 35.182.213.24:
500MB              88%  443MB 492.9KB/s - stalled -packet_write_wait: Connection to 35.182.213.24 port 22: Broken pipe
lost connection

I got tcpdump captures, looked at them in Wireshark. I'm no networking expert, but this is what it looks like to me:
- At some point the server misses a packet.
- The server then floods the client with requests for the missing packet (TCP Dup ACK in Wireshark). Many times, like one packet every few microseconds (!?)
- It might take a while for the client to answer, but at some point it does. Also more than once.
- However, the server keeps requesting the missing packet and after a while resets the connection.

I have dumps, but they are rather big, around 400 MB. I can provide them if needed. But it should be easy to reproduce. Start a FreeBSD 12 on a t2.micro on AWS on the other side of the planet, scp a big file to it, 1 GB or so. If it works, repeat a few times. If it still works... I don't know, get worse internet I guess :).

Has anyone else seen that? Any idea what to do about it, what changed in FreeBSD 12?
 
You can edit your post to mark it "solved" by clicking the drop down just left of where you would type in the title.
;)
 
Back
Top