USB transfer speeds (was: Question about FreeBSD)

W

Woolfy021

Hello. I'm looking to install FreeBSD (Hopefully with KDE) onto my Thinkpad X1Carbon Gen 6. (Samsung 980 SSD/16 GB ram) I'm mostly familiar with linux in general and have tried different distros. I currently run Arch (With KDE and btrfs / not encrypted) and it definitely runs great on my laptop. I had Windows 11 installed prior to that and, it didn't run good at all. Fan would rev up a lot and just slow in general.

The previous distros I have tried have been, OpenSuse (Runs slow-ish with the fan revving up a lot) Ubuntu (Also runs slow) Linux Mint (slow again) and a few others I can't remember. The main problem I have with my laptop is, transferring files from my laptop to a usb thumb drive or usb HD. (usb-c as well). So the problem is that, the transfer speed is extrememly slow. I have about 5 drives in total (1 new usb-c drive and 1 new regular USB 3.0 thumb drive). The other drives work find on windows. When I goto transfer a file, it is either extremeley slow or the drive just stops transferring files. If I wait long enough, and it finishes transferring, the file will be corrupted in some way.

I've researched and tried a lot of different fixes, none of which worked. (Formatting the drives to ext4/exfat/fat32 etc etc hasn't helped) From what I have read, it seems to be some type of bug, possibly related to KDE Or linux in general. I can't say it's a hardware issue as, like I mentioned it works fine in Windows. Is this a known issue with FreeBSD at all? I want to install FreeBSD but, I don't want to go through all the trouble, if it has the same problem. I just don't understand why it's working so bad. Arch and KDE runs great on my laptop. I just really want to run FreeBSD as it's something I've always wanted to setup as my main OS.

Has anyone dealt with this problem in the past, with usb transfer speed issues? If FreeBSD doesn't work, I'm not really looking forward to going back to Windows but, I need working usb/usb-c ports. Thank you.
 
Never had any problems in that regard; neither with file-based transfers nor when dd'ing images onto USB drives. If transfer speeds did drop to abysmall slow speeds, the drives were simply failing and started to timeout on any IO shortly after.

Also keep in mind that with many DEs/graphical file explorers (IIRC also KDE + dolphin; but haven't used that bloatfest in many years...), Linux in general and *especially* Windows there's a lot of caching going on when writing to USB and hence they are flat out lying about transfer speeds. The actual time until the data is transferred to the USB device is much longer (often multiple times than what it took to "finish" the transfer in the GUI), hence the requirement to 'safely remove' devices on windows (and IIRC KDE/dolphin) or you will end up corrupting the data because it was never fully transferred.
 
usb thumb drives have low write speeds (most of them) 4-10 MB/s (the theoretical interface speed limit means nothing)
the best are in fact a mini external ssd (with a "ssd" class flash controller) but even those hit thermal management pretty soon and slow down
an external hdd should do a lot better if you transfer large files (not very good at small files)
 
sko said:
Never had any problems in that regard; neither with file-based transfers nor when dd'ing images onto USB drives. If transfer speeds did drop to abysmall slow speeds, the drives were simply failing and started to timeout on any IO shortly after.

Also keep in mind that with many DEs/graphical file explorers (IIRC also KDE + dolphin; but haven't used that bloatfest in many years...), Linux in general and *especially* Windows there's a lot of caching going on when writing to USB and hence they are flat out lying about transfer speeds. The actual time until the data is transferred to the USB device is much longer (often multiple times than what it took to "finish" the transfer in the GUI), hence the requirement to 'safely remove' devices on windows (and IIRC KDE/dolphin) or you will end up corrupting the data because it was never fully transferred.
Click to expand...
Dang. That is a lot of info I had no clue on, as far as Windows and things like KDE/Dolphin. Thank you.
 
covacat said:
usb thumb drives have low write speeds (most of them) 4-10 MB/s (the theoretical interface speed limit means nothing)
the best are in fact a mini external ssd (with a "ssd" class flash controller) but even those hit thermal management pretty soon and slow down
an external hdd should do a lot better if you transfer large files (not very good at small files)
Click to expand...
I think the last ones I tried were fairly small movie files (About 4 GB or so). It is starting to make sense though, with the issues I'm having. I may just end up trying to install FreeBSD when I have some free time, just to see how it goes after all. Appreciate it.
 
You need a proper benchmark. A program writing out a file with an fsync(2) at the end. misc/cstream can do that. It runs in Windows WSL.

Of course as long as you have outright halting of transfers and corruption you have bigger problems.

You should also check which USB level (1.1, 2.0, 3.x etc) is used for the stick. `lsusb -v`
 
Woolfy021 said:
When I goto transfer a file, it is either extremeley slow or the drive just stops transferring files. If I wait long enough, and it finishes transferring, the file will be corrupted in some way.
Click to expand...
If the file is corrupt, things don't look too good aside from the speed. I can't imagine USB drivers on Linux to be buggy (only have FreeBSD experience). Nonetheless I'd advice to take the filesystem out of the mix if you want to test.
You can run dd to test one of the sticks/drives (Warning: ALL PARTITIONS/DATA WILL BE WIPED!). It'll tell you the speed. Afterwards checking if everything can be read correctly is also very useful.
To verify one can use hexdump, no idea if hexdump exists on Linux as well. I'm also not sure if it will be present on a FreeBSD install image/livedisk.
Code: 
> dd if=/dev/zero of=/dev/<usbdev> bs=32m
(... dd output)
> hexdump -C /dev/<usbdev>
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
If hexdump doesn't return all zeroes, something went wrong catastrophically.
 
sko said:
adding status=progress to dd(1) will tell you the transfer speed during the write operation.
Click to expand...
Or hit Control T while dd is running.

And note that any benchmark that uses tools such as dd will be inherently inaccurate. But not factor of 2 or 3 inaccurate, so for sanity checks it's good enough.
 
malavon said:
If the file is corrupt, things don't look too good aside from the speed. I can't imagine USB drivers on Linux to be buggy (only have FreeBSD experience). Nonetheless I'd advice to take the filesystem out of the mix if you want to test.
You can run dd to test one of the sticks/drives (Warning: ALL PARTITIONS/DATA WILL BE WIPED!). It'll tell you the speed. Afterwards checking if everything can be read correctly is also very useful.
To verify one can use hexdump, no idea if hexdump exists on Linux as well. I'm also not sure if it will be present on a FreeBSD install image/livedisk.
Code: 
> dd if=/dev/zero of=/dev/<usbdev> bs=32m
(... dd output)
> hexdump -C /dev/<usbdev>
00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
If hexdump doesn't return all zeroes, something went wrong catastrophically.
Click to expand...
So, I've tried 2 drives with different results. The first drive is old and 2nd is new. (I used the command you mentioned plus the status=progress)
It didn't seem to like the lowercase 'm' for some reason so, I used the 32M one instead. With the first usb drive I got this:

31004295168 bytes (31 GB, 29 GiB) copied, 1133 s, 27.4 MB/s
dd: error writing '/dev/sdb': No space left on device

That drive is 32GB and formatted with exfat. It locked up towards the end for some reason and after that, I couldn't exit / ctrl + c out of it. (Had to pull the thumbdrive, and reinsert it) When I ran the 'sudo hexdump -C /dev/sdb it refused to work. (I guess because it had locked up / never finished copying)

The 2nd one was a new USB thumb drive (Also 32GB and exfat)

I got different results for this one when it finished (No issues):

30735859712 bytes (31 GB, 29 GiB) copied, 1838 s, 16.7 MB/s
dd: error writing '/dev/sdb': No space left on device
917+0 records in
916+0 records out
30765219840 bytes (31 GB, 29 GiB) copied, 1839.83 s, 16.7 MB/s

Hexdump gave me these results so, it seems to be ok!

00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
^C

I don't fully understand why the 2nd drive gave me these results or if that's normal:

917+0 records in
916+0 records out

I always thought they should both be the same, when using the dd command. (Idk for sure though, because I don't use it too often) I assume the first usb drive is bad probably? I will definitely go through the rest of my thumb drives, just to get an idea if they are good or bad. That's a very helpful command to run though! ty
 
Woolfy021 said:
I don't fully understand why the 2nd drive gave me these results or if that's normal:

917+0 records in
916+0 records out

I always thought they should both be the same, when using the dd command.
Click to expand...
The last record was never written because the drive was full; so 916 records went "out" to the disk, but 917 went "in" to dd's queue.


Regarding the first drive: did you see any errors in dmesg? Lockups (or apparent ones) mostly happen due to a drive becoming completely unresponsive and outstanding I/O has to time out. Those timeouts can add up to several minutes during which the process can become catatonic.

Otherwise the transfer speeds seem fine and normal.
 
sko said:
sko said:
The last record was never written because the drive was full; so 916 records went "out" to the disk, but 917 went "in" to dd's queue.


Regarding the first drive: did you see any errors in dmesg? Lockups (or apparent ones) mostly happen due to a drive becoming completely unresponsive and outstanding I/O has to time out. Those timeouts can add up to several minutes during which the process can become catatonic.

Otherwise the transfer speeds seem fine and normal.
Click to expand...
Click to expand...
Here is the output of dmesg. The thumb drive i'm using, has the option to plugin to regular USB or a USB-C port (I just have it plugged into USB). I don't seem to notice anything personally, in the dmesg. I may just need to get rid of these old drives. It's definitely good to know everything is ok, as far as the speed of it goes. I split the dmesg file into 2 groups, in hopes it's easier to read/make sense:

(Dmesg while dd command is running / and when it almost finishes / locks up)

Code: 
[18942.506655] usb 2-1: new SuperSpeed USB device number 6 using xhci_hcd
[18942.520015] usb 2-1: New USB device found, idVendor=154b, idProduct=00f5, bcdDevice= 1.10
[18942.520032] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[18942.520039] usb 2-1: Product: DUO LINK Type-C
[18942.520064] usb 2-1: Manufacturer: PNY
[18942.520069] usb 2-1: SerialNumber: 070A93D6609C5C63
[18942.522299] usb-storage 2-1:1.0: USB Mass Storage device detected
[18942.522918] scsi host1: usb-storage 2-1:1.0
[18943.539430] scsi 1:0:0:0: Direct-Access     PNY      DUO LINK Type-C  PMAP PQ: 0 ANSI: 6
[18944.332802] sd 1:0:0:0: [sdb] 60604416 512-byte logical blocks: (31.0 GB/28.9 GiB)
[18944.333330] sd 1:0:0:0: [sdb] Write Protect is off
[18944.333343] sd 1:0:0:0: [sdb] Mode Sense: 45 00 00 00
[18944.333723] sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[18944.347797]  sdb: sdb1
[18944.348086] sd 1:0:0:0: [sdb] Attached SCSI removable disk

(After I force remove it and plug it back in)

Code: 
[19402.821537] i915 0000:00:02.0: [drm] *ERROR* Atomic update failure on pipe A (start=1165606 end=1165607) time 235 us, min 1073, max 1079, scanline start 1072, end 1088
[20377.314083] usb 2-1: USB disconnect, device number 6
[20377.314262] xhci_hcd 0000:00:14.0: WARN Set TR Deq Ptr cmd failed due to incorrect slot or ep state.
[20377.322939] blk_print_req_error: 8093 callbacks suppressed
[20377.322948] device offline error, dev sdb, sector 59754848 op 0x1:(WRITE) flags 0x4800 phys_seg 256 prio class 2
[20377.322966] buffer_io_error: 326524 callbacks suppressed
[20377.322969] Buffer I/O error on dev sdb, logical block 7469356, lost async page write
[20377.322988] Buffer I/O error on dev sdb, logical block 7469357, lost async page write
[20377.322994] Buffer I/O error on dev sdb, logical block 7469358, lost async page write
[20377.323000] Buffer I/O error on dev sdb, logical block 7469359, lost async page write
[20377.323005] Buffer I/O error on dev sdb, logical block 7469360, lost async page write
[20377.323010] Buffer I/O error on dev sdb, logical block 7469361, lost async page write
[20377.323015] Buffer I/O error on dev sdb, logical block 7469362, lost async page write
[20377.323021] Buffer I/O error on dev sdb, logical block 7469363, lost async page write
[20377.323026] Buffer I/O error on dev sdb, logical block 7469364, lost async page write
[20377.323031] Buffer I/O error on dev sdb, logical block 7469365, lost async page write
[20377.323452] device offline error, dev sdb, sector 59746648 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 2
[20377.323586] device offline error, dev sdb, sector 59756896 op 0x1:(WRITE) flags 0x800 phys_seg 2 prio class 2
[20377.324020] device offline error, dev sdb, sector 59756912 op 0x1:(WRITE) flags 0x100000 phys_seg 38 prio class 2
[20377.324275] device offline error, dev sdb, sector 59757216 op 0x1:(WRITE) flags 0x800 phys_seg 58 prio class 2
[20377.324606] device offline error, dev sdb, sector 59757680 op 0x1:(WRITE) flags 0x100000 phys_seg 31 prio class 2
[20377.324717] device offline error, dev sdb, sector 59757928 op 0x1:(WRITE) flags 0x800 phys_seg 16 prio class 2
[20377.325212] device offline error, dev sdb, sector 59758056 op 0x1:(WRITE) flags 0x100000 phys_seg 46 prio class 2
[20377.325435] device offline error, dev sdb, sector 59758424 op 0x1:(WRITE) flags 0x800 phys_seg 10 prio class 2
[20377.325873] device offline error, dev sdb, sector 59758504 op 0x1:(WRITE) flags 0x800 phys_seg 57 prio class 2
[20377.849992] usb 1-1: new high-speed USB device number 10 using xhci_hcd
[20378.890457] ldm_validate_partition_table(): Disk read failed.
[20378.890480]  sdb: unable to read partition table
[20382.656196] usb 2-1: new SuperSpeed USB device number 7 using xhci_hcd
[20382.669865] usb 2-1: New USB device found, idVendor=154b, idProduct=00f5, bcdDevice= 1.10
[20382.669883] usb 2-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[20382.669891] usb 2-1: Product: DUO LINK Type-C
[20382.669896] usb 2-1: Manufacturer: PNY
[20382.669902] usb 2-1: SerialNumber: 070A93D6609C5C63
[20382.672265] usb-storage 2-1:1.0: USB Mass Storage device detected
[20382.672928] scsi host1: usb-storage 2-1:1.0
[20383.732978] scsi 1:0:0:0: Direct-Access     PNY      DUO LINK Type-C  PMAP PQ: 0 ANSI: 6
[20385.765920] sd 1:0:0:0: [sdb] 60604416 512-byte logical blocks: (31.0 GB/28.9 GiB)
[20385.766521] sd 1:0:0:0: [sdb] Write Protect is off
[20385.766537] sd 1:0:0:0: [sdb] Mode Sense: 45 00 00 00
[20385.766938] sd 1:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[20385.771819] sd 1:0:0:0: [sdb] Attached SCSI removable disk
 
sko said:
Also keep in mind that with many DEs/graphical file explorers (IIRC also KDE + dolphin; but haven't used that bloatfest in many years...), Linux in general and *especially* Windows there's a lot of caching going on when writing to USB and hence they are flat out lying about transfer speeds. The actual time until the data is transferred to the USB device is much longer (often multiple times than what it took to "finish" the transfer in the GUI), hence the requirement to 'safely remove' devices on windows (and IIRC KDE/dolphin) or you will end up corrupting the data because it was never fully transferred.
Click to expand...
That's why I quit using GUI programs for file operations and I always run sync command multiple times after file operation is "completed".
 
You must log in or register to reply here.
Back
Top