ZFS Striping and mirroring across multiple CPU

Phishfry

Phishfry

I am setting up a ZFS RAID10. So making mirrors then striping them.
I have dual CPU with two NVMe drives on CPU0 and two drives on CPU1.
Should I create the mirrors with the same CPU group?
CPU0
nvd1=PCIe bus 6:0:0
nvd2=PCIe bus 7:0:0
CPU1
nvd3=PCIe bus 133:0:0
nvd4=PCIe bus 134:0:0

Then stripe across CPU?
zpool create icydock mirror nvd1 nvd2 mirror nvd2 nvd3
 
Would it be wiser to bring all 4 drives into one CPU group? I could shuffle things around.
I was thinking about spreading the load over both CPU by using two drives on each CPU.
But if it has to leave the CPU then NUMA path might add latency versus CPU and QPI on die.
 
Here are some numbers for my 4 disk RAID10 with Samsung PM983 1.92TB NVMe.

With sync=standard
Code: 
root@X10DRX:~ # diskinfo -wS /dev/zvol/icydock/vol01
/dev/zvol/icydock/vol01
    512             # sectorsize
    214748364800    # mediasize in bytes (200G)
    419430400       # mediasize in sectors
    8192            # stripesize
    0               # stripeoffset
    Yes             # TRIM/UNMAP support
    Unknown         # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:    209.8 usec/IO =      2.3 Mbytes/s
       1 kbytes:    236.6 usec/IO =      4.1 Mbytes/s
       2 kbytes:    212.2 usec/IO =      9.2 Mbytes/s
       4 kbytes:    201.2 usec/IO =     19.4 Mbytes/s
       8 kbytes:    161.1 usec/IO =     48.5 Mbytes/s
      16 kbytes:    148.9 usec/IO =    104.9 Mbytes/s
      32 kbytes:    213.3 usec/IO =    146.5 Mbytes/s
      64 kbytes:    250.7 usec/IO =    249.3 Mbytes/s
     128 kbytes:    477.6 usec/IO =    261.7 Mbytes/s
     256 kbytes:    748.1 usec/IO =    334.2 Mbytes/s
     512 kbytes:   1216.5 usec/IO =    411.0 Mbytes/s
    1024 kbytes:   2119.6 usec/IO =    471.8 Mbytes/s
    2048 kbytes:   4512.4 usec/IO =    443.2 Mbytes/s
    4096 kbytes:   7825.8 usec/IO =    511.1 Mbytes/s
    8192 kbytes:  16190.0 usec/IO =    494.1 Mbytes/s

With sync=disabled
Code: 
root@X10DRX:~ # diskinfo -wS /dev/zvol/icydock/vol01
/dev/zvol/icydock/vol01
    512             # sectorsize
    214748364800    # mediasize in bytes (200G)
    419430400       # mediasize in sectors
    8192            # stripesize
    0               # stripeoffset
    Yes             # TRIM/UNMAP support
    Unknown         # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:    375.6 usec/IO =      1.3 Mbytes/s
       1 kbytes:    294.5 usec/IO =      3.3 Mbytes/s
       2 kbytes:    204.3 usec/IO =      9.6 Mbytes/s
       4 kbytes:    175.6 usec/IO =     22.2 Mbytes/s
       8 kbytes:    144.7 usec/IO =     54.0 Mbytes/s
      16 kbytes:    120.7 usec/IO =    129.5 Mbytes/s
      32 kbytes:    125.6 usec/IO =    248.8 Mbytes/s
      64 kbytes:    165.6 usec/IO =    377.4 Mbytes/s
     128 kbytes:    250.6 usec/IO =    498.8 Mbytes/s
     256 kbytes:    470.8 usec/IO =    531.0 Mbytes/s
     512 kbytes:    865.3 usec/IO =    577.9 Mbytes/s
    1024 kbytes:   1530.5 usec/IO =    653.4 Mbytes/s
    2048 kbytes:   3102.2 usec/IO =    644.7 Mbytes/s
    4096 kbytes:   6298.8 usec/IO =    635.0 Mbytes/s
    8192 kbytes:  12549.0 usec/IO =    637.5 Mbytes/s
.
With sync=always
Code: 
root@X10DRX:~ # diskinfo -wS /dev/zvol/icydock/vol01
/dev/zvol/icydock/vol01
    512             # sectorsize
    214748364800    # mediasize in bytes (200G)
    419430400       # mediasize in sectors
    8192            # stripesize
    0               # stripeoffset
    Yes             # TRIM/UNMAP support
    Unknown         # Rotation rate in RPM

Synchronous random writes:
     0.5 kbytes:    147.7 usec/IO =      3.3 Mbytes/s
       1 kbytes:    156.3 usec/IO =      6.2 Mbytes/s
       2 kbytes:    214.4 usec/IO =      9.1 Mbytes/s
       4 kbytes:    240.5 usec/IO =     16.2 Mbytes/s
       8 kbytes:    168.3 usec/IO =     46.4 Mbytes/s
      16 kbytes:    156.6 usec/IO =     99.8 Mbytes/s
      32 kbytes:    212.7 usec/IO =    146.9 Mbytes/s
      64 kbytes:    273.0 usec/IO =    228.9 Mbytes/s
     128 kbytes:    510.9 usec/IO =    244.7 Mbytes/s
     256 kbytes:    886.7 usec/IO =    281.9 Mbytes/s
     512 kbytes:   1827.8 usec/IO =    273.5 Mbytes/s
    1024 kbytes:   3668.3 usec/IO =    272.6 Mbytes/s
    2048 kbytes:   6813.1 usec/IO =    293.6 Mbytes/s
    4096 kbytes:  14627.8 usec/IO =    273.5 Mbytes/s
    8192 kbytes:  28069.4 usec/IO =    285.0 Mbytes/s


Code: 
 ~# zpool status
  pool: icydock
 state: ONLINE
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    icydock     ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        nvd1    ONLINE       0     0     0
        nvd2    ONLINE       0     0     0
      mirror-1  ONLINE       0     0     0
        nvd3    ONLINE       0     0     0
        nvd4    ONLINE       0     0     0
 
You must log in or register to reply here.
Back
Top