ZFS Choosing HDDs for offline backup

Hello,

I want to buy a few HDDs for use as personal home backup. The drives will be set up as standalone (non-RAID) ZFS volumes with the option copies=2 (each file stored in two different parts of the disk) and will be stored offline most of the time.

I know that most HDDs marketed for desktop are SMR nowadays, and that SMR is known to offer poor performance with ZFS. But does it really matter in my usecase? The complaints I read about it were mostly about server usage, usually involving RAID setups and frequent random writes. Are there significant downsides of using ZFS on SMR drives in a context consisting mostly of large sequential writes and deletions? Is copies=2 more likely to cause trouble than it would on CMR drives?

If CMR drives are a better option, then what kind of drives would better fit the purpose? NAS drives (Seagate IronWolf, WD Red Plus...), CCTV drives (Seagate SkyHawk, WD Purple...)? Datacenter-grade hardware is pricey and probably overkill for redundant storage which is offline most of the time.
 
I don't think you have the option to choose between CMR or SMR drives, because most HDDs do not tell you which technology they employ.
However, I would most definitely use sets of mirrored-pairs of disks for any kind of backup instead of a single disk with 2 copies. I wouldn't want a failure in the drive electronics to make my backup unaccessable, which, in the case of a single drive with two copies would place both copies out of reach.
 
I prefer to have 2 copies on different disks than 2 copies on single one.
Every single disk is not reliable at all. All disks will fail.
In case of large amount of bad-sectors on HDD you will not have a possibility to read your data, even having 2 copies of every file on the disk.
IMHO: In most cases I will prefer WD than Seagate. Usually failed Seagate left less probability to recover data at home, without going to expensive data-recovery services. In most cases few disks of the same model will fail after the same lifetime.
My experience based on hundreds of failed non-server disks
 
I don't think you have the option to choose between CMR or SMR drives, because most HDDs do not tell you which technology they employ.
It needs a bit of research, but usually it can be figured out. Unusual big cache is a hint.
 
I have bought a 2.5" SMR drive by accident and the sequential write performance is abysmal. It works well enough as a backup drive on a Mac, though. I can't estimate how that would translate to receiving ZFS send/receive snapshots, though.
IO performance of SMR disks is highly workload and file system dependent. If used correctly, they are very fast, and significantly cheaper than CMR disks per byte of storage. Personally, I've only used them with custom file systems. I know that Linux ext4 has a lot of support for SMR available, and I know a few of the developers of those features personally. But I don't know which shipping version (what kernel version, what distro) that is available in, or whether this is only used in research and internal versions.

I've heard rumors (when having a few beers with file system developers who work at Apple) that APFS is SMR-aware and optimized, in particular when using the "time machine" mode. But given that beer was involved in those discussions, I don't trust that information very much.

On the other hand, if you use SMR disks with workloads that do lots of small updates, or with file systems that are not optimized for SMR, or (god forbid) with RAID systems that do small writes in place, then their performance can be awful.
 
It needs a bit of research, but usually it can be figured out. Unusual big cache is a hint.
Also, go to the manufacturer's web site, and pull up the specification. It will often explicitly say how recording is done (for example, my external backup disk says "perpendicular recording"), and it will give the track density, a.k.a. TPI (tracks per inch). Then compare the track density to other disks of a similar generation, and the SMR disk will jump out as having significantly higher TPI.
 
I would get refurbished enterprise drives such as

Test them hard and look at the SMART data when they arrive and you should be good to go.
I agree. Also, check new prices for other manufacturers. You might be surprised.
Anything advertised as "Enterprise Class" is not likely to be shingled (SMR), and is likely to be durable.
I would always favour getting backups off-site with as many backup disks as you can afford.
More disks is always better than more copies on one disk.
I backup all my hosts to a file system (tank/backups) on the ZFS server, using sysutils/rsnapshot.
The backups are stored as a de-duplicated time series, on-line, on site, permanently.
Thus I can always immediately recover anything (apart from loss of the ZFS server itself).
I also ZFS send the entire ZFS tank to a stand-alone backup disk (a SATA disk in a hot swap bay) on a regular basis.
I have two backup disks (12TB WD Golds). One is always off-site. The other may be on-site, off-site, or in transit.

I have spent a lot of time shopping and never found an external USB 3.x gen 2 disk enclosure that didn't have "cooling issues". So my suggestion is to use a hot swap bay, and naked SATA disks. Hot swap is less flexible than powered external enclosures, but it's always going to be the fastest and most reliable option. Speed matters when you start copying many terabytes. And you don't really need that CD drive any more... so you can replace it with a single hot swap bay, without disturbing anything else.
 
I would most definitely use sets of mirrored-pairs of disks for any kind of backup instead of a single disk with 2 copies.
I prefer to have 2 copies on different disks than 2 copies on single one.
Me too. Who said I intended to only use a single disk for a given backup? :)

I typically keep two separate HDDs with the same data, each with copies=2. A mirrored pair of disks is better than a single disk with two copies, but two standalone disks with the same data in two copies on each is better than both. In case something goes wrong during a backup process, you can't loose the two disks at the same time, while this can happen when using a mirror. Two pairs of mirrored disks would be even better, but more expensive.

Thank you for the suggestion... storage devices aren't something I've ever considered buying used, but why not after all. That's still not cheap, but I will consider the option.

most HDDs do not tell you which technology they employ
At least Seagate and WD are quite clear about it nowadays. It's much easier than finding out whether a SSD uses TLC or QLC NAND... Seagate provides a page which sums up everything: https://www.seagate.com/products/cmr-smr-list/. For WD you can check product sheets for each model and the information is there. Actually I just noticed that there are still CMR WD Blue in 3TB, 4TB, 6TB and 8TB capacity, while Seagate BarraCuda are SMR for anything bigger than 1TB (the BarraCuda Pro line has been discontinued).

So my suggestion is to use a hot swap bay, and naked SATA disks.
I use an external docking station with SATA HDDs. It does the job for my usage. I don't have huge amounts of data, speed isn't a major concern for me as long as a backup doesn't take half a day.

And you don't really need that CD drive any more...
Who told you that? I do use my optical drives a lot :)

I'm still curious about the differences between a HDD marketed for PC, another one marketed for NAS and another one marketed for CCTV. Let's say I have a WD Blue, a WD Red Plus and a WD Purple, all using CMR and offering the same storage capacity. What differences will I notice between them, in my usecase? Is one of them more, or less suitable than the others for offline backup?
 
I'm still curious about the differences between a HDD marketed for PC, another one marketed for NAS and another one marketed for CCTV. Let's say I have a WD Blue, a WD Red Plus and a WD Purple, all using CMR and offering the same storage capacity. What differences will I notice between them, in my usecase? Is one of them more, or less suitable than the others for offline backup?
I expect that others will have wider experience than mine, but I can offer some insights.

Ten years ago I built my ZFS server with 3TB CMR WD reds. There was one infant failure, replaced under warranty, and two more drives failed after about nine years of service. I was quite happy with that outcome, especially since there was no UPS, no A/C in the office, the power was unreliable, and the original case had poor ventilation. I got a UPS and a much better case, and started to replace the Reds with 4TB Seagate EXOS drives, and have enough spares in store to replace all the Reds (and then some). I'm using mirrors, and pair each 3TB Red with a 4TB EXOS. This delivers minimum performance (the Reds are quite slow) and maximum recoverability (I expect more Reds to fail at any time). Eventually, when all the Reds are replaced, I'll get the extra 1TB in each mirror.

My experience with CMR Reds is quite positive. However, Reds have been around for a long time, and who knows what changes WD have made over the years (apart from their catastrophically sneaky attempt to switch to SMR). I think that the Reds will suffer abuse more readily that the EXOS drives. They appear to me to be quite rugged, and probably well designed for the "home NAS". They rev slower and certainly run cooler.

I have never used an SMR drive, and probably never will. The ZFS server is the only place I want to use spinning disks these days. Everything else is (or should be) solid state. So, for me, a spinning disk that won't play well with ZFS is a poor investment.

The "enterprise class" drives have quite superior performance. Transfer rates and seek times are markedly superior on the EXOS drives. Also the EXOS drives have 512 byte sectors. The Reds use 4K. The Golds also use 4K.

I can't comment on blues or purples. I got my WD Golds at a good price and want their performance to run the backups as fast as possible.
 
Thank you for the suggestion... storage devices aren't something I've ever considered buying used, but why not after all. That's still not cheap, but I will consider the option.
It's not cheap, that is the problem.
Normally, for tax and accounting purposes, computer equipment is considered to have a lifetime of 5 years. This means, 5 year old equipment has served it's purpose and is calculated with a net value of zero.

But nowadays a market seems to have established that sells such equipment, which has been obtained for zero (or even charged for erasure) on ebay and amazon etc. for astronomical prices as "refurbished" - which is a blatant lie, because you cannot "refurbish" a mechanical disk: it is built by robots, and you can, sometimes, open it in cleanroom and replace a broken part in order to make it run again for some hours to recover the data, but nothing more.

It is indeed true that the better series of drives can usually work for 20 years and more, and in mirrored configurations it can make sense to use such drives. But not to buy them for such prices.

And if you look into the customer comments of that item, you find that these drives are sold in exactly the condition in which they have gathered them from the scrap heap: some dirty, some broken, but most of them somehow useable.
There are more stories about that topic: there is info that some people have managed to reset the SMART records. There is info that drives were sold as quasi new with an erased manufacturing date stamp.
I consider these sellers of used hardware a bunch of criminals. If you sell a car with a rolled back odometer, that is fraud. With computer hardware there are no such standards, instead ripoff seems to be the standard.

I'm still curious about the differences between a HDD marketed for PC, another one marketed for NAS and another one marketed for CCTV. Let's say I have a WD Blue, a WD Red Plus and a WD Purple, all using CMR and offering the same storage capacity. What differences will I notice between them, in my usecase?

Heat dissipation?
But that's not dependent on the color. Look at the 6TB purple: there are two of them, with apparently only cache size difference. But wait - there is another difference: one eats 4 Watts, the other 6 Watts. That is not explainable by the cache. But what else? That does the spec sheet not tell.
However, what the spec sheet does tell: that these drives (all of them) support 16 drive bays. Okay, so, no 17th drive bay possible. In what fashion might these drives "support" drive bays? I don't know.
Do these guys consider their customers idiots? That for certain.

Otherwise... I for my part have only one WD blue in my zoo. That one runs fine for 12 years now, mostly 24/7, and occasionally at 67 Celsius. Interesting: the spec sheet tells that only the purple can do 65 Celsius, the blue can only do 60 Celsius. How would one change design to create that difference? Maybe... not at all?

Wait, more fancy comes. Quote "Actual spindle motor rotational speed for this model is 7200 RPM; although ID Device may report 5400 to reflect previous Performance Class designation". Beg you pardon? I think the main problem here is these spec sheets are not printed on paper. Why? Because then one could wipe the ass with.

So to answer your question: yes, there will be differences, but they will likely not relate to the color. One might assume that they just sell what comes off the production belt and give it the color (and firmware) that is currently demanded. After binning, that is.
Specs in the spec sheet do regularly not match the limits in SMART, and I assume they even less match what is actually built into the drive.
 
Lets get a bit practical: two of the more interesting values are the rotational speed and the actuator speed.

There are two classes of rotational speed: 5k4/5k7/5k9 and 7k2. 7k2 gets hot, the others usually not - and the difference is quite expressed. This concerns cooling requirements, and maybe also expected lifetime. There is also a performance difference, 7k2 feeds some 200 or more MB/sec, the others usually somewhere at 160-180 MB/sec. (outer tracks only, inner tracks are always much slower). But since most workload involves seeks, this does not make as much difference as one might suppose.
(There is also a 10k and 15k class, but these are only for industry use, as SAS, and nowadays probably moved to SSD.)

There are also roughly two classes of actuator speed: 8 ms and 16 ms average seek, and this makes a big difference. Most disks today are 8 ms.
But care has to be taken: the HGST megascale, which is available as used, and which is an interesting option for archiving, because it is one of the rare models that are industry-grade and low-rpm/low-power, is usually 16 ms.

The problem with this is: none of the manufacturers of today would tell us the actuator speed. Most should be 8 ms, but for video disks which do mostly streaming, it might be imaginable that it is slower. We do not know.
WD does also not bother to tell us the rotational speed for their "purple" stuff.
 
Price difference between CMR and SMR drives is so small that it is best to avoid the latter. With Seagate and Toshiba it's simple: Ironwolf = always CMR, X300 & N300 = always CMR.

There are more stories about that topic: there is info that some people have managed to reset the SMART records. There is info that drives were sold as quasi new with an erased manufacturing date stamp.
In most drives it is very easy to clear S.M.A.R.T statistics. People who sell significant amount of these drives often don't bother to overvrite them with zeros, so you can even undelete files on these "brand new" HDDs.
Quote "Actual spindle motor rotational speed for this model is 7200 RPM; although ID Device may report 5400 to reflect previous Performance Class designation". Beg you pardon? I think the main problem here is these spec sheets are not printed on paper. Why? Because then one could wipe the ass with.
Some 7200 rpm WD drives performs so badly that they are marketed as "5400 class". You should avoid them as you are buying 5400-like performance with 7200 rpm power consumption and heat dissipation.
 
you don't really need that CD drive any more... so you can replace it with a single hot swap bay, without disturbing anything else.
Did I miss a significant change in FreeBSD code regarding how hot-swap works?
My last experience was that FreeBSD crashed every single time I attempted to hot-swap a "hot-swap" SATA disk. And I am not talking about issues with mounted volumes. I am talking about unmounted disks, with no partitioning at all. It still crashed, every time.
So I learnt to never attempt hot-swapping. I shut down FreeBSD, turn off the computer, then swap what needs to be swapped.
So, has anything changed? Why recommend a hot-swap bay otherwise?
 
Did I miss a significant change in FreeBSD code regarding how hot-swap works?
My last experience was that FreeBSD crashed every single time I attempted to hot-swap a "hot-swap" SATA disk. And I am not talking about issues with mounted volumes. I am talking about unmounted disks, with no partitioning at all. It still crashed, every time.
So I learnt to never attempt hot-swapping. I shut down FreeBSD, turn off the computer, then swap what needs to be swapped.
So, has anything changed? Why recommend a hot-swap bay otherwise?

I did it just recently. On SAS, not SATA, but there shouldn't be much of a difference.

U.2 NVMe might be more interesting.
 
Did I miss a significant change in FreeBSD code regarding how hot-swap works?
I have a 3-bay hot swap SATA enclosure (normally empty) that occupies the space of two CD drives. I routinely hot swap one of my 12TB backup disks in and out when doing the backups (zfs-send a copy of the tank to the backup disk) . It works exactly the same as hot-swapping a thumb drive. I have never had a problem, in many iterations of running the backups.

I also use the hot swap enclosure to insert a replacement and re-silver a failed disk in the "cold swap" tank (no outage required), before eventually shutting down at my leisure and swapping the dead disk with the freshly re-silvered replacement. This procedure guarantees that, after a disk drive fails, a RAID set can survive removal and replacement of the wrong drive -- a very common (and often fatal) mistake.

The ICY DOCK MB153SP-B cages were discussed by several list members, including SirDice and Phishfry.

The Silverstone FS303 also looked plausible.

Edit: I also have a StarTech USB 3.1 Gen 2 (a.k.a. USB 3.2 Gen 2x1) to SATA adapter for 2.5"/3.5" SSD/HDD that provides hot-plug on FreeBSD. It tends to be somewhat "fragile" because of the profusion of easily disturbed power and data cables, and the propensity of the disks to "walk" when they get busy. I also point a desk fan at the disks for extended duty cycle (like an over-night backup). I still routinely boot a small desktop server from release media on a thumb drive, use these adapters to build the disks for a new system, and then transport the disks to their eventual destination (much easier than going off-site to build a system). The USB docking stations may be a more robust option. My hazy recollection is that my two adapters, on separate USB ports, provide full bandwidth to each disk, and that the docking stations may share the bandwidth of a single USB port.
 
I did it just recently. On SAS, not SATA, but there shouldn't be much of a difference.
Hm. Hotplugging SCSI was always possible, as there are all independent entities and no status on the wire while idle. Hotplugging ATA was never possible, because ATA moves the controller logic from where it belongs to where it not belongs (into the peripherial device). Only with AHCI that became somehow remediated, and should work now. ("devctl detach/attach ahcichX" might be advisable)
 
as mentioned before, instead of using copies=2 I highly recommend having separate devices. Also: I keep using single disks for exactly your usecase (and not zmirror) ... I actually have one offline copy at my brothers house, and two offline copies in my office so I can rotate one disk with data updates through the locations, and I keep track of those snapshots.
 
I'm actually thinking I may get a few 6TB SAS drives and pair them with an LSI 9260. The controllers are cheap on eBay and the SAS drives are as well. The 13.2 release hardware list has the controller listed as supported. If SAS is good for ZFS I may just get a new set up myself.

Edit: I'm coming from a desktop user to FreeBSD and it's become pretty clear that FreeBSD is most skilled as a server. So I'm likely going to start moving to server hardware but still use as a workstation or desktop user. But if a system performs well on certain hardware I'm ok with making changes to get the best support.
 
I'm actually thinking I may get a few 6TB SAS drives and pair them with an LSI 9260. The controllers are cheap on eBay and the SAS drives are as well. The 13.2 release hardware list has the controller listed as supported. If SAS is good for ZFS I may just get a new set up myself.

Edit: I'm coming from a desktop user to FreeBSD and it's become pretty clear that FreeBSD is most skilled as a server. So I'm likely going to start moving to server hardware but still use as a workstation or desktop user. But if a system performs well on certain hardware I'm ok with making changes to get the best support.

Yes, the LSI works. One disadvantage of SAS is that you can never stuff the disk into an external USB enclosure should the need arise.

I'm all for server hardware. There are some great options cheap now. And ECC memory is always recommended for file storage.
 
Yes, the LSI works. One disadvantage of SAS is that you can never stuff the disk into an external USB enclosure should the need arise.

I'm all for server hardware. There are some great options cheap now. And ECC memory is always recommended for file storage.
I will likely do a Ryzen 5000 series motherboard with ecc support and I have an early 90s mid tower that has the bay space for SAS drives and active cooling panels. That way I can get the best support for storage and desktop usage.

Edit: Nevermind that. It looks like the ryzen chips that support ecc are only available in pre built systems.

Edit 2: I'm getting a bit confused about ecc support on Ryzen chips I think. I found a chart that states 5000 series support ecc on b550 and x570 boards.
 
Back
Top