Slow network performance on WAN

I have multiple laptops running FreeBSD 12.1 (used as a router, only 1 is active at a time) and all are maxing out at < 100 Mbps and I should be getting 200 Mbps. I tried the same hardware booted up to Ubuntu and it is seeing > 200 Mbps, so I am sure it isn't hardware. I also have tried disabling lro, tso, tx and rx checksums, but that didn't seem to have any effect.

I should note that I since I'm using a laptop as a router, I am using a USB Ethernet adapter to provide the WAN connection. As the laptop is older, it is only USB 2.0 (480 Mbps). That should still be providing adequate bandwidth to get 200 Mbps. The LAN port is 1 Gbps.

I ran tcpdump to see what was going on and I see a high number of TCP retransmissions and duplicate acks. Additionally, when the laptop is idle, the CPU is 99+% idle, whenever I run a speed test, I see the CPU idling at as little as 35%. I am running ZFS on all the laptops. Is there perhaps some sort of interaction with ZFS, network load, and CPU usage that is causing this degradation? If the CPU is having trouble keeping up, that could explain the retransmissions or duplicate acks, it's too latent. I also tried disabling pf to see if my rules were too complex and that made no difference.
 
So you're not connecting via WLAN/WiFi, correct? You do not have a switch or hub, but instead the USB ethernet adapter connects to the cable modem with an ethernet cable?
 
Yes, this is a wired connection - I'm connecting directly from the Fiber Optic Terminal's Ethernet port to my laptop(s). I swapped cables, adapters, for whatever reason, FreeBSD is giving me < 100 Mbps whereas Ubuntu was giving me 230 Mbps and I should max out at 200 Mbps.
 
Maybe the driver negotiated simplex instead of duplex connection? What does ifconfig show? It could also be a problem on the USB side. We need more infos.
 
Ah, good point, it negotiated 100baseTX:

wan: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80008<VLAN_MTU,LINKSTATE>
ether <OMITTED>
inet <OMITTED> netmask 0xffffff00 broadcast <OMITTED>
media: Ethernet autoselect (100baseTX <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>


Walter
 
Ah, good point, it negotiated 100baseTX
Well, the maximum for 100baseTX is 100 Mbps.

By the way, you forgot to mention the most important piece of information: What ethernet hardware is that exactly, and which driver is attaching to it?
 
Ok, with the adapter that is Gigabit:

wan: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=80000<LINKSTATE>
ether <OMITTED>
inet6 <OMITTED>%wan prefixlen 64 scopeid 0x3
inet <OMITTED> netmask 0xffffff00 broadcast <OMITTED>
media: Ethernet autoselect (1000baseT <full-duplex,master>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>


It is reporting it negotiated Gigabit; however, when I run fast-cli, I am only seeing about 60 Mbps. Additionally, the internal NIC is also Gigabit and negotiated as that. The driver for this particular device is if_ure and uether.

Yes, I understand about overhead and with a 100 Mbit connection, I would be reasonably satisfied for 80 Mbps. My connection should be 200 Mbps and I am connected at 1 Gbps, so I would expect to see 200 Mbps like I saw in Ubuntu.

The device I am using is a TP-Link 10/100/1000 USB 3.0 adapter. My laptop only supports USB 2.0, so 480 Mbps:
ugen6.2: <TP-LINK USB 10/100/1000 LAN> at usbus6, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (180mA)

Still, I'm only measuring 60 Mbps.
 
I guess that’s the USB overhead, then.
FreeBSD’s USB stack is rather slow, unfortunately.
 
I am using a USB Ethernet adapter to provide the WAN connection.
I bet this is the problem. USB ethernet is not so great.
Have you tried reversing the arrangment and use onboard ethernet as WAN and USB ethernet as LAN?
That would pinpoint the problem.
 
I am not sure of what fast-cli does, but can you perhaps use ftp or some other well known protocol to measure speed? Maybe this fast-cli is just bad on FreeBSD?
Are there other USB devices active on that bus?
ZFS is slower to write than UFS so that will always hamper comparisons.
Is this machine of yours hyperthreaded?
Personally, I would enable the soreceive interface to see if that speeds up your connection:
net.inet.tcp.soreceive_stream=1
in loader.conf
You should see some speed increase. How much?
 
It was the USB Ethernet adapter - I thought I had tried the internal Ethernet port earlier, once I tried it again, I was pulling over 300 Mbps. So, I guess no more laptops for use as a router, I will use an old desktop since I have multiple Gigabit NICs I can use and they won't have the USB performance hit.

That is surprising, how can the USB stack be that slow?
 
I don't believe the USB stack is, but the Ethernet to USB is, obviously. To clarify, FreeBSD might not fully support your USB dongle. Just a guess.
 
Ok, how might I go about making it better? Would it be simple enough to make the code changes? Also, I tried the net.inet.tcp.soreceive_stream=1 in loader.conf and didn't see any marked difference. I also grepped for stream and tried setting hw.usb.xhci.streams=1. That too did nothing.
 
A quick look shows it's supported fine; well as fine as any realtek chipset can be by FreeBSD.
What's the output of usbconfig dump_device_desc ?
The name of the device?
 
add media 1000baseTX mediaopt full-duplex to your ifconfig(8) something like this:
ifconfig interface ip media 1000baseTX mediaopt full-duplex

[EDIT] from ure(4): RTL8153/1000baseTX: Set 1000baseTX operation over twisted pair. The RealTek gigE chips support 1000Mbps in full-duplex mode only
 
ugen1.5: <TP-LINK USB 10/100/1000 LAN> at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (180mA)

bLength = 0x0012
bDescriptorType = 0x0001
bcdUSB = 0x0210
bDeviceClass = 0x0000 <Probed by interface class>
bDeviceSubClass = 0x0000
bDeviceProtocol = 0x0000
bMaxPacketSize0 = 0x0040
idVendor = 0x2357
idProduct = 0x0601
bcdDevice = 0x3000
iManufacturer = 0x0001 <TP-LINK>
iProduct = 0x0002 <USB 10/100/1000 LAN>
iSerialNumber = 0x0006 <000001000000>
bNumConfigurations = 0x0002
 
I don't have particularly good experiences with USB3.0 ethernet adapters on USB2.0 ports in general. One of those used to be part of a Linux based router setup i ran a couple of years ago and it gave me nothing but pain (bad connection, randomly disappearing from the bus, ...) until i replaced it with a USB 2.0 one. I have no idea what the exact models were though.

It's not really a direct solution but maybe you could replace the router laptop with an old thinclient (Fujitsu Siemens S400, Igel H700C or something like that). Both of them have gigabit ethernet ports and a usable PCI slot (for more ports) built in. You should be able to get those at around 10€ including shipping on a certain huge auction site.
 
ugen1.5: <TP-LINK USB 10/100/1000 LAN> at usbus1, cfg=0 md=HOST spd=HIGH (480Mbps) pwr=ON (180mA)

bLength = 0x0012
bDescriptorType = 0x0001
bcdUSB = 0x0210
bDeviceClass = 0x0000 <Probed by interface class>
bDeviceSubClass = 0x0000
bDeviceProtocol = 0x0000
bMaxPacketSize0 = 0x0040
idVendor = 0x2357
idProduct = 0x0601
bcdDevice = 0x3000
iManufacturer = 0x0001 <TP-LINK>
iProduct = 0x0002 <USB 10/100/1000 LAN>
iSerialNumber = 0x0006 <000001000000>
bNumConfigurations = 0x0002
Ok, so how is it set up? You have a devd setting for this unit?
 
Uh, I don't think so. The only thing I do on my systems is rename the interfaces and configure static IP or dhcp. In this case here, I think I just renamed the interface using ifconfig_ure0_name=wan. Then, I instructed it to use dhcp.
 
Who/what instructed you to do that?
What does dmesg report when the device is plugged in?
You will note your device has multiple configurations. Have you tried usbconfig -d 1.5 set_config 1 ?
 
I like to rename my interfaces so they're intuitive to me.

ugen6.2: <TP-LINK USB 10/100/1000 LAN> at usbus6
ure0 on uhub0
ure0: <TP-LINK USB 10/100/1000 LAN, class 0/0, rev 2.10/30.00, addr 2> on usbus6
miibus1: <MII bus> on ure0
rgephy1: <RTL8251/8153 1000BASE-T media interface> PHY 0 on miibus1
rgephy1: none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto
ue0: <USB Ethernet> on ure0
ue0: Ethernet address: <OMITTED>
ue0: link state changed to DOWN
ue0: changing name to 'wan'
wan: link state changed to UP
wan: link state changed to DOWN
wan: link state changed to UP
wan: link state changed to DOWN
wan: link state changed to UP

Changing the config to 1 caused the device to disappear ...
 
Back
Top