Hi Robert, there's a lot of good advice above. Here are my random thoughts:
- The Backblaze Hard Drive Data and Stats are a great resource, but never offer a complete view of all options, and often lag the market.
- The introduction of SMR by Western Digital was sneaky (but WD seem to have responded to the bad press). Read the datasheets of any disk you contemplate, and avoid SMR.
- SSD and NVME are great if they fit your capacity and budget needs. But at 20 TB+ might be a challenge.
- Choosing very large disks has cost advantage, but an operational down-side. Re-silvering time for a 3 to 4 TB disk is measured in days. Disk access for other applications will be severely compromised (maybe unusable) while this is happening. The smaller the disk, the less the grief (which is why vendors like IBM are still selling truck loads of 500 GB disks into the enterprise market).
- You largely get what you pay for.
- My five CMR 3 TB WD Reds (which performed poorly in the Lifetime Backblaze stats) have been quite satisfactory in my ZFS server. One infant failure, replaced under warranty, and another very recent failure at ~8 years continuous service.
- As a generalisation, RAIDZ gives you surprisingly good striping across the spindles (much better than mirrors), and (usually) better capacity than mirrors. RAIDZ2 gives you striping, plus better redundancy than mirrors. I expect that mirrors will probably perform better if you have a truly random small block I/O pattern.
- I am slowly switching from 8-year old CMR 3 TB WD Reds to 3 and 4 TB Seagate EXOS (enterprise) disks for durability and reliability (I'm laying in spares as the WD Reds are pretty old).
- Reliability is so important to me that I am switching my ZFS server from 5 x 3 TB consumer grade disks in RAIDZ1 to 7 x 3 TB enterprise grade disks in RAIDZ2.
ZFS send
), and multi-spindle redundancy with enterprise class disks are all in play.