Short comments:
I've thought long and hard about this. >>
good, it's "your" system.
Problem with RAID is you never have one complete set of data on one drive. Everything is scattered across drives.
>>
that isn't an actual problem
Its slower than mirror. >>
probably, (most of the time for reads); for your use case: does that really matter?
It’s harder to resilver a drive than with a mirror. >>
I don’t think so.
Takes more computing power (cpu [...] >>
Absolutely! And at the same time that hardly ever matters*!
[...] and hard drive/head movement/etc.) ] >>
I really don't think so.
Your case, as most others, means juggling with all the possibilities of allocation. So, my suggestions may not be 100% satisfactory and could be considered as variations on previous suggestions but, decide for yourself. I'm also presuming that the OS part contains only the base install and root and other management accounts; all user data is independant and stored in the data pool.
You
can use three complete drives to form a separate pool with a 3-way mirror for the OS (and other things; these not being "running data"). That is a very safely configured way to do things, IMO, based on what you have written so far about your SMB target environment: overly safe. I think that the data pool is more important than the OS pool; just as
Jose mentions. Therefore more redundancy would be required for the data pool. Based on this, I'd say that a pool with a 2-way mirror would suffice for the OS pool. When losing one disk, rebuilding takes about the same time for a 2-way mirror pool as for a 3-way mirror pool. You do not need the speed of a 3-way mirror for the OS pool, speed wise that's overkill, IMO.
Taking this reasoning a step further: for the OS pool you do not need very much space: a 2-way mirror of 2 * 512GB (SATA) SSDs would suffice. Here I'm transitioning from spindles to silicon. If you take a competent set of SSDs then they will be more reliable than spinning platters of rust. The added extra read/write speed is a pleasant side effect; you'll benefit from that when resilvering to a new replacement SSD in case of failure; keeping your vulnerability time window with a degraded pool narrower than with spinning platters. Also, in case of absolute disaster when the whole pool must be rebuilt from backups this will go fast. Added bonus: two/three extra usable physical 3.5 " slots in your drive cages; presuming the SSDs can be "tucked away" elsewhere in the system.
This brings me to the last item of the OS-pool: your local backups. I don't see a clear basis for that. Perhaps you're thinking of making a quick and efficient backup of the data (perhaps db data) from the data pool but, you're sort of squandering prime online disk space for local backup purposes. With ZFS's snapshots you'll be able to take a snapshot of (part) of a pool. After that you can use that snapshot as the source of your backup; no difficulties with a backup window or locking files: ZFS has done that already for you.
Next: the data pool. You’ll have to make an (business) assessment of how and when a possible expansion might be needed. First with the number of spindles at your disposal: a 5 or 6 drive RAIDZ2. When using 6 or more drives: RAIDZ3 for ultimate redundancy: that is the same level of redundancy as the 3-way mirror option. You'd be using your disk space a lot more efficiently; unless you have a clear use case that speed is that important.
When expansion becomes necessary in the future, you'll have basically two options. First replace each individual drive by a bigger one (say 8TB instead of 4TB). You'll be doing a replacement and resilver action on a disk-by-disk basis. That will probably take several days. The second is adding an extra vdev of new drives to the data pool. When using the same redundancy type that would mean doubling the pool that already exists.
While technically your data is indeed distributed, scattered if you prefer, I'm unsure as to why that matters to you in particular and otherwise in any practical way. With a data pool with a 3-way mirror, you only have a space utilisation of 33%. What do you think you could be doing with each separate disk and when would you likely need that? When you expand the pool by adding another 3-way mirrored vdev to the pool, this will not be the case any more. Your data is distributed over the two vdevs and while technically the files on the original vdev with the 3-way mirror are only on that vdev: they won't be of any use as a separate unit anymore once the second 3-way mirror vdev has been added.
While you can reason that there is an advantage for two separate pools for data and OS, in such a relatively small setting that would only be the separation of concerns, which in and of itself might be valid as a matter of personal preference. When combing the two pools you basically have the option of one big RAIDZn pool of 5 or 6 disks or, as
Jose mentions, one pool consisting of (a slice over) two 3-way mirrors. I've found an overview of the various options (drawing) with their available spaces, space utilisation and redundancy useful.
Finally, to your extra HD disk on standby on the shelf. That is money paid for and not used and, I think, a false sense of security to a certain extent. Because that disk added to the (data) pool adds extra redundancy, space or speed, depending how it is deployed. You only have a slight advantage in case of a failing disk in the sense that the sysadmin (that would be you alone I presume, in the SMB use case) when phisycally at the system, could let the system start with resilvering sooner. When that disk is deployed as extra redundancy (RAIDZ3 instead of RAIDZ2) and a disk fails you're falling back to the same level of redundancy that you would have had when that disk is on the shelf and RAIDZ2 deployed. The system only has to endure the extra bit of power while operating.
Weigh your options and make a decision. With your level of preparation and management, you should also be considering the two ZFS (e)books (
FreeBSD Development: Books, Papers, Slides); at least the first one. When you're concerned about speed issues: Six Metrics for Measuring ZFS Pool Performance:
Part 1 -
Part 2 -
pdf (2018-2020); by iX Systems
___
* if you’d have a very low specced CPU without any hardware instruction support for the calculations needed, that might be an issue.