FreeBSD 13.1 + USB 3 + ZFS is brutally slow

[Dragony]

There IS something broken in the FreeBSD USB 3 driver, but it's hard to locate.

So far I have invested 3 days in tracking down the issue, but I tend to give up and just recommend people NOT to use USB 3 with FreeBSD but stick with USB 2.

Below the bottom line it's about massive write speed problems with USB 3, which do NOT appear in USB 2. To distinguish the mode, I used usbconfig. If it's listed as "spd=SUPER (5.0Gbps)", the performance will be very slow. Only spd=HIGH works at the speed levels you expect. This leads to the absurd situation that USB 2 is a lot faster than USB 3 on FreeBSD.

So far I have tested on different system, from desktop to notebook, and with different media, from cheap to expensive usb stick, and external magnetic usb drives to avoid a flash-bottleneck. I also tested with different tools. From dd with different block sizes to a fully configured ZFS.

The results differ, which makes it hard to pin the problem.

The slowest experience I had was with the GT Voyager 32 GB and 64 GB stick in USB 3 mode + ZFS. Write speed went down to 50(!) kb/sec when doing zfs recv with a root filesystem. But there were also bursts with 30 MB/sec.

The fastest USB 3 experience was with an external hard drive, but i still was only able to reach around 5 MB/sec average when doing zfs recv. The really interesting thing with magnetic drives is that you can hear if it's working or not, because you can hear when it's writing stuff. I was able to observe that first it writes for 1 second, then it pauses for around 10 seconds, then another 1 second. This does not happen in USB 2 mode and the heads are writing data all the time.

I also observed that iostat always showed a qlen of constantly 10. Even when it's doing nothing. The hard drives doesn't write, but qlen is 10.

Again, all this does NOT happen with USB 2 and qlen varies wildly from 0 to 32.

So far I recommend NOT to use USB 3 with FreeBSD 13. This problem seems to exist for years already., but it's hard to pin down, because in the end it's working. It's just awfully slow.

Unfortunately I was unable to find an option to force the device to USB 2 mode, so if your computer only has USB 3 ports, you are screwed.

 

[diizzy]

USB 3 itself works fine, I'm not sure why you're expecting to see great performance using flash drives with unsuitable performance characteristics for file systems that utilizes random read and writes. If you have a decent USB to SATA controller (and a non SMR HDD) performance of spinning rust isn't that bad but you'll certainly wony see any performance records.

 

[Dragony]

Did you even read my posting? I said the problem is not there with USB 2. Furthermore I used a USB<->SATA converter to attach the magnetic drive and ZFS works also very well with magnetic drives.

To clarify:

USB 2 speed: linear characteristics, avg. speed 40 MB/sec. This is max. speed for USB 2.
USB 3 speed: heavy stuttering characteristics, avg. speed 5 MB/sec.

Test performed with same drive on same computer with "zfs recv backup@rootfs". That computer having USB 3 and USB 2 ports.

So by just putting the exact same hardware in just another port of the exact same computer solves the problem.

 

[diizzy]

Effective speed of USB 2.0 (480Mbit) is ~35Mbyte/s due to bus contraints etc so 40Mbyte/s avg sounds very unlikely, sounds more like broken hardware / implementation unfortunately.

 

[zirias@]

USB 3 speed: heavy stuttering characteristics, avg. speed 5 MB/sec.

Well, I can't confirm this. I'm doing backups of my server to an external USB3 drive, and although I'm not exactly excited with the speed, it's something between 50 and 100 MB/s.

So, just saying, if there's a bug, it must have a more "specific" cause, not USB3 in general.

 

[Dragony]

What USB 3 drive do you use? What filesystem?

dd with big block sizes works without problem. The problem is severe when it's about small blocks. If you do a zfs recv on zroot/ROOT/default, there will be many small transactions. On that scenario USB 2 is much faster than USB 3.

I will do some more testing today. Maybe its a geli problem. idk...

 

[Dragony]

Yeah sorry, seems it's really not USB3 fault. After a reboot of the system the behaviour of USB2 is the same as USB3. I did a lot of experiments and it might be I have triggered something in the USB stack which was only reset after a hard reboot....

Anyway.... if the following behaviour is normal, then Corsair GT sticks are not usable for ZFS:

time zpool create -o ashift=12 -O atime=off -O compression=zstd testfs da0 (da0 to make sure it isn't a problem with geli or partitions)
USB2 USB3 HyperX <1s <1s GT32 125s 86s GT64 58s 51s HDD <1s <1s

 

[zirias@]

What USB 3 drive do you use? What filesystem?

It's an external WD 12TB drive (classic spinning hdd), and backups are done using zfs send/recv. GELI is involved as well, below both pools. As I said, speed isn't great, but definitely not slower than USB2.