freebsd.org download only 4 kBps

PMc

PMc

Hi folks,
this is nasty. Looking at the traffic it is obvious what is going wrong:

Code: 
# tcpdump -nivtnet0 "host 139.178.72.202"
01:31:41.065196 IP 139.178.72.202.443 > XX.XX.XX.XX.12182: Flags [.], seq 29899:32617, ack 0, win 1029, options [nop,nop,TS val 648940938 ecr 249697186], length 2718
01:31:41.065345 IP XX.XX.XX.XX > 139.178.72.202: ICMP 192.168.0.1 unreachable - need to frag (mtu 1411), length 576
01:31:41.365701 IP 139.178.72.202.443 > XX.XX.XX.XX.12182: Flags [.], seq 29899:31258, ack 0, win 1029, options [nop,nop,TS val 648941238 ecr 249697186], length 1359
01:31:41.434819 IP XX.XX.XX.XX.12182 > 139.178.72.202.443: Flags [.], ack 31258, win 1029, options [nop,nop,TS val 249697561 ecr 648941238], length 0
01:31:41.439146 IP 139.178.72.202.443 > XX.XX.XX.XX.12182: Flags [.], seq 31258:33976, ack 0, win 1029, options [nop,nop,TS val 648941312 ecr 249697561], length 2718
01:31:41.439277 IP XX.XX.XX.XX > 139.178.72.202: ICMP 192.168.0.1 unreachable - need to frag (mtu 1411), length 576
01:31:41.728537 IP 139.178.72.202.443 > XX.XX.XX.XX.12182: Flags [.], seq 31258:32617, ack 0, win 1029, options [nop,nop,TS val 648941601 ecr 249697561], length 1359
01:31:41.800688 IP XX.XX.XX.XX.12182 > 139.178.72.202.443: Flags [.], ack 32617, win 1029, options [nop,nop,TS val 249697926 ecr 648941601], length 0

I'm trying to download a freebsd image (139.178.72.202 translates to some freebsd.org). It does work, but the speed is abysmal.

Here it works as it should:

Code: 
03:15:49.993884 IP YY.YY.YY.YY.12606 > 139.178.72.202.443: Flags [S], seq 2498767748, win 65535, options [mss 1371,nop,wscale 6,sackOK,TS val 3473533780 ecr 0], length 0
03:15:50.004341 IP 139.178.72.202.443 > YY.YY.YY.YY.12606: Flags [S.], seq 1190330928, ack 2498767749, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1486256421 ecr 3473533780], length 0
03:15:50.045450 IP YY.YY.YY.YY.12606 > 139.178.72.202.443: Flags [.], ack 1, win 1029, options [nop,nop,TS val 3473533835 ecr 1486256421], length 0
03:15:50.056360 IP YY.YY.YY.YY.12606 > 139.178.72.202.443: Flags [P.], seq 1:701, ack 1, win 1029, options [nop,nop,TS val 3473533840 ecr 1486256421], length 700
03:15:50.067288 IP 139.178.72.202.443 > YY.YY.YY.YY.12606: Flags [P.], seq 1:261, ack 701, win 1029, options [nop,nop,TS val 1486256484 ecr 3473533840], length 260
03:15:50.116219 IP YY.YY.YY.YY.12606 > 139.178.72.202.443: Flags [P.], seq 701:781, ack 261, win 1029, options [nop,nop,TS val 3473533905 ecr 1486256484], length 80
03:15:50.122633 IP YY.YY.YY.YY.12606 > 139.178.72.202.443: Flags [P.], seq 781:1406, ack 261, win 1029, options [nop,nop,TS val 3473533905 ecr 1486256484], length 625
03:15:50.126894 IP 139.178.72.202.443 > YY.YY.YY.YY.12606: Flags [P.], seq 261:564, ack 781, win 1029, options [nop,nop,TS val 1486256544 ecr 3473533905], length 303
03:15:50.134502 IP 139.178.72.202.443 > YY.YY.YY.YY.12606: Flags [.], seq 564:1923, ack 1406, win 1029, options [nop,nop,TS val 1486256551 ecr 3473533905], length 1359
03:15:50.134533 IP 139.178.72.202.443 > YY.YY.YY.YY.12606: Flags [.], seq 1923:3282, ack 1406, win 1029, options [nop,nop,TS val 1486256551 ecr 3473533905], length 1359
03:15:50.134550 IP 139.178.72.202.443 > YY.YY.YY.YY.12606: Flags [.], seq 3282:4641, ack 1406, win 1029, options [nop,nop,TS val 1486256551 ecr 3473533905], length 1359
03:15:50.134570 IP 139.178.72.202.443 > YY.YY.YY.YY.12606: Flags [.], seq 4641:6000, ack 1406, win 1029, options [nop,nop,TS val 1486256551 ecr 3473533905], length 1359
03:15:50.134585 IP 139.178.72.202.443 > YY.YY.YY.YY.12606: Flags [.], seq 6000:7359, ack 1406, win 1029, options [nop,nop,TS val 1486256551 ecr 3473533905], length 1359
03:15:50.134596 IP 139.178.72.202.443 > YY.YY.YY.YY.12606: Flags [.], seq 7359:8718, ack 1406, win 1029, options [nop,nop,TS val 1486256551 ecr 3473533905], length 1359
03:15:50.134607 IP 139.178.72.202.443 > YY.YY.YY.YY.12606: Flags [.], seq 8718:10077, ack 1406, win 1029, options [nop,nop,TS val 1486256551 ecr 3473533905], length 1359
03:15:50.134627 IP 139.178.72.202.443 > YY.YY.YY.YY.12606: Flags [.], seq 10077:11436, ack 1406, win 1029, options [nop,nop,TS val 1486256551 ecr 3473533905], length 1359
03:15:50.134639 IP 139.178.72.202.443 > YY.YY.YY.YY.12606: Flags [.], seq 11436:12795, ack 1406, win 1029, options [nop,nop,TS val 1486256551 ecr 3473533905], length 1359
03:15:50.213542 IP YY.YY.YY.YY.12606 > 139.178.72.202.443: Flags [.], ack 564, win 1029, options [nop,nop,TS val 3473534005 ecr 1486256544], length 0

Comparing this to the opening of the malfunctioning session:

Code: 
03:09:18.901169 IP XX.XX.XX.XX.21135 > 139.178.72.202.443: Flags [S], seq 206543814, win 65535, options [mss 1371,nop,wscale 6,sackOK,TS val 1203879749 ecr 0], length 0
03:09:18.905484 IP 139.178.72.202.443 > XX.XX.XX.XX.21135: Flags [S.], seq 2119600904, ack 206543815, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 154829626 ecr 1203879749], length 0
03:09:18.931441 IP XX.XX.XX.XX.21135 > 139.178.72.202.443: Flags [.], ack 1, win 1029, options [nop,nop,TS val 1203879779 ecr 154829626], length 0
03:09:18.938918 IP XX.XX.XX.XX.21135 > 139.178.72.202.443: Flags [P.], seq 1:701, ack 1, win 1029, options [nop,nop,TS val 1203879784 ecr 154829626], length 700
03:09:18.943533 IP 139.178.72.202.443 > XX.XX.XX.XX.21135: Flags [P.], seq 1:261, ack 701, win 1029, options [nop,nop,TS val 154829665 ecr 1203879784], length 260
03:09:18.971762 IP XX.XX.XX.XX.21135 > 139.178.72.202.443: Flags [P.], seq 701:781, ack 261, win 1029, options [nop,nop,TS val 1203879819 ecr 154829665], length 80
03:09:18.976172 IP 139.178.72.202.443 > XX.XX.XX.XX.21135: Flags [P.], seq 261:564, ack 781, win 1029, options [nop,nop,TS val 154829697 ecr 1203879819], length 303
03:09:18.978036 IP XX.XX.XX.XX.21135 > 139.178.72.202.443: Flags [P.], seq 781:1406, ack 261, win 1029, options [nop,nop,TS val 1203879819 ecr 154829665], length 625
03:09:19.024013 IP 139.178.72.202.443 > XX.XX.XX.XX.21135: Flags [.], ack 1406, win 1029, options [nop,nop,TS val 154829745 ecr 1203879819], length 0
03:09:19.043901 IP XX.XX.XX.XX.21135 > 139.178.72.202.443: Flags [.], ack 564, win 1029, options [nop,nop,TS val 1203879894 ecr 154829697], length 0
03:09:19.198399 IP 139.178.72.202.443 > XX.XX.XX.XX.21135: Flags [.], seq 564:7359, ack 1406, win 1029, options [nop,nop,TS val 154829919 ecr 1203879894], length 6795
03:09:19.198499 IP 139.178.72.202.443 > XX.XX.XX.XX.21135: Flags [.], seq 7359:14154, ack 1406, win 1029, options [nop,nop,TS val 154829919 ecr 1203879894], length 6795
03:09:19.198773 IP XX.XX.XX.XX > 139.178.72.202: ICMP 192.168.0.1 unreachable - need to frag (mtu 1411), length 576
03:09:19.198847 IP XX.XX.XX.XX > 139.178.72.202: ICMP 192.168.0.1 unreachable - need to frag (mtu 1411), length 576
03:09:19.517085 IP 139.178.72.202.443 > XX.XX.XX.XX.21135: Flags [.], seq 564:1923, ack 1406, win 1029, options [nop,nop,TS val 154830238 ecr 1203879894], length 1359
03:09:19.584880 IP XX.XX.XX.XX.21135 > 139.178.72.202.443: Flags [.], ack 1923, win 1029, options [nop,nop,TS val 1203880434 ecr 154830238], length 0
03:09:19.589097 IP 139.178.72.202.443 > XX.XX.XX.XX.21135: Flags [.], seq 1923:4641, ack 1406, win 1029, options [nop,nop,TS val 154830310 ecr 1203880434], length 2718
03:09:19.589209 IP XX.XX.XX.XX > 139.178.72.202: ICMP 192.168.0.1 unreachable - need to frag (mtu 1411), length 576
03:09:19.924575 IP 139.178.72.202.443 > XX.XX.XX.XX.21135: Flags [.], seq 1923:3282, ack 1406, win 1029, options [nop,nop,TS val 154830646 ecr 1203880434], length 1359
03:09:19.993614 IP XX.XX.XX.XX.21135 > 139.178.72.202.443: Flags [.], ack 3282, win 1029, options [nop,nop,TS val 1203880844 ecr 154830646], length 0
03:09:19.997965 IP 139.178.72.202.443 > XX.XX.XX.XX.21135: Flags [.], seq 3282:6000, ack 1406, win 1029, options [nop,nop,TS val 154830719 ecr 1203880844], length 2718
03:09:19.998073 IP XX.XX.XX.XX > 139.178.72.202: ICMP 192.168.0.1 unreachable - need to frag (mtu 1411), length 576

There is no endpoint change between the two snippets: The server 139.178.72.202 is obviousely the same. The client browser is also the same, it has not even been restarted.
The only change is the outbound router.
How to fix this?
 
No comments whatsoever? surprize...

I found the fix, but I fail to understand what is happening.
The tcpdump shows large packets being received. How can that be at all on a cross-internet connection? (Trying this again, it cannot be reproduced anymore. Now the packets all have length 1359 as they should - but speed is not improved.)

Then. when the machine downloads the file to it's own filesystem. there is no problem, throughput is >10 MByte/sec.
Only when the machine acts as an outbound router and moves the downloaded packets onwards to somewhere else, then troughput falls to ~ 4 kByte/sec.

The fix appears to be this:
ifconfig vtnet0 -rxcsum

The machine is a KVM in some compute-center. I have no idea what the underlying metal is, or what the virtualization software is (and running FreeBSD on it is not supported, as usual).

rxcsum as I understand, means: have the checksums of received packets done in hardware. What does that mean? Why should the hardware compute checksum for received packets, and what should it do with the result? How would it tell the kernel that checksum? Or would it then just silently throw away packets with mismatched checksum?

Then, vtnet0 is a virtual device. There is no hardware below to do that checksum. It gets computed on the core in either way. Also, there should not be any mismatched checksums reaching that point, because the real physical card likely has rxcsum set as well. So what is the point in this option, at all?
 
Code: 
ifconfig vtnet0 -rxcsum
Will disable checksum offloading to the hypervisor ethernet devices, so it appears that the hypervisor does not do a great jobs on the checksum offloading handling (so maybe a bug in the hypervisor ethernet driver somewhere).
 
Thank You.

Do we get any idea about what the hyperviser in place might actually be? I can see only that much:

Code: 
CPU: QEMU Virtual CPU version 2.5+ (2395.85-MHz K8-class CPU)
  Origin="AuthenticAMD"  Id=0x6d3  Family=0x6  Model=0xd  Stepping=3
  Features=0x783fbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
  Features2=0x80202001<SSE3,CX16,x2APIC,HV>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
Hypervisor: Origin = "KVMKVMKVM"
...
ACPI APIC Table: <BOCHS  BXPCAPIC>
...
virtio_pci0: <VirtIO PCI (legacy) Network adapter> port 0xc560-0xc57f mem 0xfebd1000-0xfebd1fff,0xfe000000-0xfe003fff irq 11 at device 3.0 on pci0
vtnet0: <VirtIO Networking Adapter> on virtio_pci0
vtnet0: Ethernet address: 00:16:3e:0c:93:26
vtnet0: netmap queues/slots: TX 1/256, RX 1/128
000.000754 [ 450] vtnet_netmap_attach       vtnet attached txq=1, txd=256 rxq=1, rxd=128
 
You must log in or register to reply here.
Back
Top