ZFS Par2

Alain De Vos

Alain De Vos

Anyone experience with par2 or something similar ?
I want to protect my external 1-disk USB data against radiation , you know alpha particles, beta particles or gamma radition...
 
1622818006218.png

i have one of these
 
Alain De Vos said:
Anyone experience with par2 or something similar ?
Click to expand...
It was quite common on usenet to provide additional files to fix corrupted or missing files from a multi part post. The poster would provide additional PAR2 files, with those you could restore those missing or corrupted parts. Used archivers/par2cmdline for that.

You could use ZFS too, but you have to set copies to 2 or higher on a single vdev pool so ZFS can actually fix errors instead of only detecting them.
 
SirDice said:
You could use ZFS too, but you have to set copies to 2 or higher on a single vdev pool so ZFS can actually fix errors instead of only detecting them.
Click to expand...
That's what I'd do.
Alain De Vos said:
I want to protect my external 1-disk USB data against radiation , you know alpha particles, beta particles or gamma radition...
Click to expand...
You only need to do that if it's an SSD. Cosmic rays can bit-flip RAM or flash memory, but they cannot flip the bits of magnetic hard drives.
 
mtu said:
That's what I'd do.
Click to expand...
Yes, it's the simplest and easiest solution because you get the protection without having to think about it. The downside is that you can only store about half as much on the USB media because every block is written twice (with copies=2).

The downside of using PAR2 files is that you have to generate them every time you have written something to the USB stick. And you have to calculate the most optimal number of extra files to generate to account for a certain amount of data loss.
 
covacat said:
View attachment 10082
i have one of these
Click to expand...
Not toally unreasonable ;) You could use a decent amount of heavy metal (couple of centimeters) above the drive (no need to shield the bottom)—but there's still a minimal chance of a muon hit (which is likely to penetrate the metal), or charged particles from secondary radiation wreaking havoc. All in all, data redundancy is probably more practical than shielding :)
 
With mtree md5 , you can detect problems.
Tuning of par2 seems to be black magic when it comes to blocksize.
 
Flash memory already uses ECC internally as even a brand new NAND flash memofy doesn't have 100% good cells. You are more likely to run into its limited write cycles than cosmic rays bitflips!
 
It doesn't make much sense, indeed none.
"1-disk USB"

Make multiple copies, on different media, rather than multiple copies on the same media.

Ideally even on optical media, which are completely insensitive to magnetic fields, typically only the temperature (and UV) can cause problems.

How to make fast backups?
Of course with zpaqfranz -checksum :) if you are really paranoid.
You can check at any moment that the files are OK, even better than with zfs
 
I created a zfs dataset on the internal harddisk with name "triple" and zfs set copies=3 and block checksum skein.
There I'll place important data which is little in size.
Unless you are a bank, important data is little in size and non-important data is big in size.
Here I'll post the half-hour incremental backup of my home directory with dedup & gzip-9 compression.
Code: 
NAME                                                USED  AVAIL     REFER  MOUNTPOINT
ZHD/usr_home_hourly@2021_06_04__16_14_00           1.75M      -     1.78G  -
ZHD/usr_home_hourly@2021_06_04__16_32_24           1.18M      -     1.78G  -
ZHD/usr_home_hourly@2021_06_04__17_02_24            856K      -     1.78G  -
ZHD/usr_home_hourly@2021_06_04__17_32_24            916K      -     1.78G  -
ZHD/usr_home_hourly@2021_06_04__18_02_24            956K      -     1.78G  -
ZHD/usr_home_hourly@2021_06_04__18_32_24            832K      -     1.78G  -
ZHD/usr_home_hourly@2021_06_04__19_02_24              0B      -     1.78G  -
Data used is marginal in our modern age.

PS1: There is also the file : /etc/periodic/daily/800.scrub-zfs.

PS2:
Difference between scrub scanned. Scanned indicates size of the metadata.
 
I am really skeptical.
If your purpose is just to make a copy of your hard drive, let's say / home, inside a USB, trivially
Code: 
zpaqfranz a /mnt/usbwhatever/mycopy.zpaq /home
Crontabbed every 1/2 hour.
Finished.
 
Why you want to scrub usb drive?
Is it a primary storage media, or a backup one?
 
After a scrubbing of the freebsd root pool linux-grub no longer wants to boot my freebsd install.
It's device naming is not ok and the grub loader does no longer detect the pool and says "no such device"
I tried "zpool import -f -d /dev/disk/by-id poolname -o cachefile=/etc/zfs/zpool.cache" in vain.
The device attached to the zpool is listed as the full drive and no longer the specific gpt partition.
I've seen this before...
Do you know the freebsd installer does not allow zfs-on-root on a partition .... Coincidence ?
Here what I'll do. Simply create a small ufs2 partition and install a freebsd-loader on it and do a grub chainload to this ufs partition. In the end I will start to know how to fix problems :)
I think it shows at least incompatibiity between freebsd zfs-kmod and linux-zfs and fragility of mixing operating-systems.
The solution of a small ufs partition with the loader worked like a charm.
(Even if linux-grub complained cross drive booting to gpt is not possible without dedicated efi-partition. "kfreebsd /boot/loader" did the trick)
 
You must log in or register to reply here.
Back
Top