In that case, the appropriate benchmark is actually all the way through the file system. So if your external USB disk is /dev/da0p1, and it is mounted at /mnt/ext (I simplified your example to make it easier to type), then the correct command for testing the bandwidth of one single file would be
dd if=/dev/zero of=/mnt/ext/bigfile.test bs=1M count=16384
, which creates a 16GiB file. Why did I copy from /dev/zero and not from a pre-existing file? Because I'm trying to test the speed of the USB port, the external disk, and the file system on the external disk, not the speed of the source file system.
Note that this test partially tests the buffer cache of the file system on the external disk: when
dd
finishes, part of the output file is still in RAM cache and not on disk. I think this is a realistic test. If your copy program writes to the external disk with an fsync() call at the end, then you need to write a little C or Python etc. program that mimics the fsync, or call the
fsync(1) command after running dd (and time the whole assembly of dd + fsync).
I think in reality, you might not dump one single huge file to the external disk, but probably multiple directories with small and medium sized files. That's much harder to benchmark; you may have to simply time the actual execution.
Now, if it isn't fast enough, the question is: what is the bottleneck? That's much harder to answer. For fun, I just ran the test on my home machine. I have an external Seagate disk drive (the cheap 2.5" 4TB drive you can get for about $70 at Costco) connected via USB3. Reading directly from the raw disk drive (dd to /dev/da0p1) runs at about 110 MB/s, which is probably limited by the platter speed. Reading a single very large file through the ZFS file system runs at ~70 MB/s (which is probably limited by CPU overhead and doing smaller IOs, I have a very slow CPU, 1GHz Atom). Writing a similar large file through ZFS is even slower, about 34 MB/s; but then, on my machine the write speed of ZFS is always pretty bad (not a surprise, and doesn't bother me at all). After the write of a 16GiB file finishes, running the fsync command adds another 0.27 seconds, which tells me that ZFS didn't have much dirty data in the buffer cache.