HDD are much slower in gmirror

dvl@

dvl@

Developer
I'm trying to figure out if I've missed something. gmirror is about 21MB/s slower than the slower of the two HDD involved.

Doing the math, FWIW, they are 4K aligned: 4096008 * 512 / 4096 = 512001

Is an 18% reduction in throughput reasonable for a gmirror?

I tested these HDD with dd for simple sequential writes:

Code: 
 136 MB/s ST250DM000-1BD141

[root@floater:~] # dd if=/dev/zero of=/dev/ada1s1 count=1600000 bs=32k
1600000+0 records in
1600000+0 records out
52428800000 bytes transferred in 365.007414 secs (143637630 bytes/sec) 

111 MB/s WDC WD2500AAKX-08ERMA0

[root@floater:~] # dd if=/dev/zero of=/dev/ada2s1 count=1600000 bs=32k32k
1600000+0 records in
1600000+0 records out
52428800000 bytes transferred in 449.196279 secs (116716906 bytes/sec) 
[root@floater:~] #

When combined into a gmirror, I get only 90 MB / s and I do not know why.

Code: 
 dd if=/dev/zero of=testing count=1600000 bs=32k
1600000+0 records in
1600000+0 records out
52428800000 bytes transferred in 555.504958 secs (94380436 bytes/sec)

The gmirror looks like this:

Code: 
$ gpart show
=>       63  488397104  mirror/gm0  MBR  (232G)
         63    4095945              - free -  (2G)
    4096008  484301097           1  freebsd  [active]  (231G)
  488397105         62              - free -  (31k)

=>        0  484301097  mirror/gm0s1  BSD  (231G)
          0   12582912             1  freebsd-ufs  (6.0G)
   12582912  134217728             2  freebsd-swap  (64G)
  146800640   12582912             4  freebsd-ufs  (6.0G)
  159383552   12582912             5  freebsd-ufs  (6.0G)
  171966464  304087040             6  freebsd-ufs  (145G)
  476053504    8247593                - free -  (4G)

$ 

$ gmirror status
      Name    Status  Components
mirror/gm0  COMPLETE  ada0 (ACTIVE)
                      ada1 (ACTIVE)

$ dmesg | grep ada0
ada0 at ahcich2 bus 0 scbus3 target 0 lun 0
ada0: <ST250DM000-1BD141 KC45> ATA-8 SATA 3.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 238475MB (488397168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad8
$ 


$ dmesg | grep ada1
ada1 at ahcich3 bus 0 scbus4 target 0 lun 0
ada1: <WDC WD2500AAKX-08ERMA0 19.01H19> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 238475MB (488397168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad10

NOTE: the mirror uses ada0 and ada1. The tests at the top of the page use ada1 and ada2. These are the same units, just renumbered.
 
There is bound to be some overhead in both read and writes. But what determines it? I have a couple of spare drives I can test later.
 
Followup: these are WD Blue 500G drives on a GA-K8U motherboard, Athlon 64 3200+, SATA I ports. The drives are identical, so only one test each. Also, no partitions, just the bare drives. This system only has PCI slots, and I don't have a PCI SATA controller. I'm pretty sure these drives are actually much faster than this.

The partition layout on your mirror is kind of weird, leaving 2G empty at the beginning of the disks. Maybe that was intentional.

Single drive, bare partition: 31.3M/second
Code: 
# dd if=/dev/zero of=/dev/ada0s1 bs=64k count=16k
16384+0 records in
16384+0 records out
1073741824 bytes transferred in 32.713865 secs (32822225 bytes/sec)

Mirror, default balance algorithm, bare partition: 24.4 M/second
Code: 
# dd if=/dev/zero of=/dev/mirror/gm0s1 bs=64k count=16k
16384+0 records in
16384+0 records out
1073741824 bytes transferred in 41.999805 secs (25565400 bytes/sec)

Mirror, default balance algorithm, 6G UFS filesystem: 25.9M/second
Code: 
# dd if=/dev/zero of=/mnt/testing bs=64k count=16k
16384+0 records in
16384+0 records out
1073741824 bytes transferred in 39.478802 secs (27197933 bytes/sec)

So bare partition on a single drive is the fastest, followed by the mirror when writing into a filesystem. Offhand I don't know why writing to a filesystem is 6% faster than writing to a raw partition on a mirror. Could be measurement error, could be an advantage of the mirror.

There are different things being compared, too. A write to a bare partition should go faster than a write into a file where filesystem overhead is involved.

Another thing that could differ is the balance algorithm in the mirror. The default is load, which should be the fastest. But a lot of instructions use round-robin, which could be slower.

I could try a longer test, but it feels like the old motherboard may be a bottleneck. My faster SATA controllers are PCIe and I don't have any PCIe systems available for testing.
 
I can see that gmirror will have a perfomance cost. I'm happy to accept that cost given the benefit. I have a theory that mismatched drives impose a higher cost than matched drives. That is, if the two drives have significantly different speeds, the performance hit is higher.
I will create a gmirror with two identical drives and compare the performance against my first post above.
 
You must log in or register to reply here.
Back
Top