ZFS Automatic Scrubbing

I had to add daily_scrub_zfs_enable="YES" to /etc/periodic.conf. The default period is 35 days. If you want to override it, add daily_scrub_zfs_default_threshold=value to the periodic.conf file. You can also choose different defaults for different pools. All this doesn't seem to be documented in any man-page -- zfs-scrub(8) still talks about systemd!
 
bakul said:
I had to add daily_scrub_zfs_enable="YES" to /etc/periodic.conf. The default period is 35 days. If you want to override it, add daily_scrub_zfs_default_threshold=value to the periodic.conf file. You can also choose different defaults for different pools. All this doesn't seem to be documented in any man-page -- zfs-scrub(8) still talks about systemd!
Click to expand...
I had no idea about that. I had been using cron to do it every month.
 
This may start another discussion how often a scrub makes sense (some may even say 'never'), but depending on the amount and kind of data - how often it's changed; I have separated pools, one for my backups and repository, and another one for store-only data [videos, disc images, PDFs, datasheets, pictures, etc. - the by far way larger amount of data] - a daily scrub can be too much.
I do it every second month (cron, every 31st of a month), which is quite enough for my purposes (By now within the 5 years I'm using my NAS scrub never found and corrected an error.)

If you are paranoid about your stored data resp. it's of extreme value and importance, or you just can afford it, you may consider using error correcting RAM.
 
There is also daily_trim_zfs_{enable,pools,flags} if you are so inclined (for SSDs, NVME and thinly provisioned virtual disks).

Maturin said:
a daily scrub can be too much.
Click to expand...
Note that it is *checked* daily for scrubbing but actual scrubbing happens once every daily_scrub_zfs_default_threshold days. For HDDS that have lots of files, it can take hours so you certainly don't want to scrub too often!
 
I want some scrubs on the first saturday morning each month:

Bash: 
$ doas crontab -l | fgrep scrub
2       8       *       *       Sat     [ $(date +%d) -le 7 ] && zpool scrub …
 
bakul said:
I want a crontab with a more flexible time spec -- basically a crontab is a repeating Tasks to do list and cron is a task scheduler!
Click to expand...
I don't understand what you are asking for, at least the "more flexible time spec" part. mro has a reasonable way to specify "the second saturday" which I don't think is natively supported in crontab format. I'm just trying to understand what you mean
 
By flexibility I mean it should be able to accept things like "second saturday in a month", "first working day after april 14", etc. as well as a particular date & time (but specified flexibly) to do the work of the at(1) command.
 
  • Like
Reactions: mer
Well, what you also could do, is not let cron execute the command directly, but let cron start a small script that contains the command.
You let cron execute the script every saturday, and the script tests if it's the second saturday of a month, and only then executes the command, else just exits.
You see where this is leading: Telling cron to execute this script every day/hour/month, and your what-ever-complex-date-check-function tests anything you like to do actually the job or just do nothing and exits again.
 
Maturin said:
You let cron execute the script every saturday, and the script tests if it's the second saturday of a month, and only then executes the command, else just exits.
Click to expand...
That is essentially how /etc/periodic/daily/800.scrub-zfs works. The idea is to factor out all "time to do something* specification in one place.
 
bakul said:
I had to add daily_scrub_zfs_enable="YES" to /etc/periodic.conf. The default period is 35 days. If you want to override it, add daily_scrub_zfs_default_threshold=value to the periodic.conf file. You can also choose different defaults for different pools. All this doesn't seem to be documented in any man-page -- zfs-scrub(8) still talks about systemd!
Click to expand...
It is rather unfortunate that there is no specific FreeBSD information mentioned or referenced in PERIODIC SCRUB of zpool-scrub(8) (while the systemd example is obviously not relevant).
However, periodic.conf(5):
Rich (BB code): 
DESCRIPTION
   [...]
	 daily_scrub_zfs_enable
	     (bool) Set to "YES" if you want to run a zfs scrub periodically.

	 daily_scrub_zfs_pools
	     (str) A space separated list of names of zfs pools  to  scrub.   If
	     the list is empty or not set, all zfs pools are scrubbed.

	 daily_scrub_zfs_default_threshold
	     (int)  Number of days between a scrub if no pool-specific threshold
	     is set.  If not set, the default value is 35,  corresponding  to  5
	     weeks.

	 daily_scrub_zfs_<poolname>_threshold
	     (int) The same as daily_scrub_zfs_default_threshold but specific to
	     the pool <poolname>.

	 daily_trim_zfs_enable
	     (bool) Set to "YES" if you want to run a zfs trim daily.

	 daily_trim_zfs_pools
	     (str) A space separated list of names of zfs pools to trim.  If the
	     list is empty or not set, all zfs pools are trimmed.
What is missing: daily_trim_zfs_flags. It seems that PR 292074 did not trigger an update/inclusion for this in periodic.conf(5) for periodic/801.trim-zfs: Fix daily-trim-zfs-flags.

Instead of the reference to settings related to systemd, an appropriate inclusion, or at least a reference to periodic.conf(5) seems in order in the SEE ALSO section of zpool-scrub(8).
 
Maturin said:
This may start another discussion how often a scrub makes sense
Click to expand...
That is a very interesting question. Meaning it's insanely complicated, and nobody knows the answer. Why? Because it's a tradeoff. Pro: Scrub finds silent data corruption (disk contains wrong content, but drive did not have a read error), and it makes sure all sectors remain readable, or at least latent read errors in hardware are found earlier, and redundancy can be restored. Con: Scrub costs lots of system resources, and if implemented badly (ZFS is OK but not great in that) can impact performance of foreground workload. Really big con: The mere act of reading data wears out a disk drive, and increases the number of read errors. The fact that scrub both "cures" read errors (by creating extra redundant copies if needed) and causes them means that the balance is subtle.

The reason the question is interesting: We do not know how much extra reads to disk drives damage data reliability. To my knowledge (and I was in the center of that industry until a few years ago), nobody in the world has good data on that, data accurate enough to build predictive models from. So we can't answer it. The only parties that today can collect enough data are the hyperscalers; they're the only ones that have both enough disks to collect good reliability statistics, and that have enough metrics of how much the disks were used (even Backblaze, bless their heart for being so open, does not have enough data). And even if someone collected enough data from a fleet of real-world installed drive (not lab bench tests), validating any model is likely outside our current capabilities.

Another factor that makes such a calculation difficult is that you need to take the cost of losing data, and the performance cost of restoring data from backup into account. In the above discussion, the cost model was "a disk block is either OK (cost = 0, utility = 1), or it is lost (cost=1, utility = 0). But with backup this gets more into shades of gray: If I can spend many $$$ and 3 days getting a backup tape out of an underground vault ... if I can get a copy of the file back over an extremely expensive network link from Singapore or Australia ... if I can tell the customer that I have a 2-week old copy of the file, whether that would be better than nothing ... that all figures into how important scrub (and the resilience of your primary storage system) really is.

From "wet finger in the wind" calculations, it seems that for common RAID codes (such as the ones used by ZFS), the optimal scrub frequency is probably roughly around 1 time per month, plus or minus a factor of 2 either way. For extremely redundant codes (some of the hyperscalers use encodings that have dozens of redundancy copies, but very wide groups), the balance is probably towards longer scrub intervals.

Maturin said:
(some may even say 'never')
Click to expand...
That is definitely wrong. Scrub really helps. Whether ZFS scrub is perfect, whether it could be improved, and what the scrub frequency should be: those are good questions. But "never" is wrong.

Maturin said:
If you are paranoid about your stored data resp. it's of extreme value and importance, or you just can afford it, you may consider using error correcting RAM.
Click to expand...
If you are paranoid about your data, you should also invest in ECC RAM. But that argument can't be turned around: If you can't use ECC RAM for some reason, that doesn't immediately mean that you should give up on RAID and scrub.
 
You must log in or register to reply here.
Back
Top