Solved Frequent incremental zfs backup problems.

covacat said:
cp /backup/.zfs/snapshot/S-XX/dir/file/pippo.txt /tmp
Click to expand...
Ahem... no
If you do a send-receive on the same machine, maybe on two different drives, you're doing nothing more than an rsync
Of course you will get a replica, and therefore all the various snapshots, and therefore it will also be very quick to obtain
But it won't be much better than a mirror

If you keep .zfs files, i.e. send redirects on a file, perhaps like in the script I put above (to operate differentially) you will have a big problem

That is, the time (and disk space) needed to restore the "primary" image and then the differential one (or the incremental chain).

This can sometimes be nearly impossible (restore a zfs multiterabyte image to smaller disks)

Very often long, very long (even hours and even days, if you are dealing with numerous incrementals)

zfs replicas (send | receive, syncoid or whatever) are GREAT for replicating between TWO distinct machines, local (LAN) and/or remote (WAN) over ssh
Some sort of "superrobocopy" or "superrsync"

So it is normal to replicate a server to a remote machine with syncoid for disaster recovery

I also made a small (on this forum) guide on how to clone zroot
---
OK but now let's see a different case, namely that of the thread

I want to make a backup of my data (so the JPG files, cpp, whatever), which are versioned, snapshot style
I want to be able to go back to the various versions on the various dates I made them
In short, time machine style

The best answers, on BSD, are for me hb and zpaq
hb is great, but creates huge folders (with lots of files) and hard to manage
It's also not open source

zpaq is opensource and takes the complexity of managing backup to zero
a single line and that's it

The efficiency is extremely high, the files can be encrypted (which is good and right) and when I want to restore version 27 of /etc/pippo.txt I will do this
Code: 
zpaq x /backup/thebackup.zpaq /etc/pippo.txt -to /tmp/restored.txt -until 27

I won't need temporary disks to "receive" the .zfs file on.
I won't have to do anything, mount this and that
One line and I restored the file
---
Normally (even here I put my little guide on this forum) more strategies and more programs are adopted to do different things

You'll do (assuming SSD-NVMe)
- mirror
- snapshots
- replica locale (on different drives, bigger spinning, to retain more snapshots)
- prune snapshots
- rsync to a NAS place (fast restore)
- zpaqfranz to a local drive (fast check)
- replica of the zpaqfranz local drive to remote location1
- rsync of the zpaqfranz local drive to remote location2
- hb on a NFS-mounted NAS
- uncompressed ZIP to a NFS-mounted NAS (faster open of backup) via 7z
- snapshots of zroot (+differential, pruned) stored in .zfs files (just in case)
- replica of "everything" to a different VM (faster recovery time)
- restoring the backups to a local deduplicated drive
- comparing the restored backups with the original one, usually 3-5 copies for each day with different software and different OSs
- checking free spaces on the various media
- sending by e-mail a report
... and... and...

But if you want a backup of your home, take a snapshot, zpaq the snapshot, delete the snapshot
You can even do not snapshot, if you do not have VM running (ex. virtualbox)

Really I'm always very surprised that such a handy tool (and for years in ports) is so little used
On the other hand, the open source world is full of unknown "gems"
 
Alain De Vos said:
Does oneone has a sanoid/syncoid tutorial? How to configure for dummies ?
Click to expand...
Yep
On the github

Short version (in this case a WAN replica)
Code: 
if ping -q -c 1 -W 1 backup.francocorbelli.com >/dev/null; then
/bin/date +"%R ----------REPLICA remota server risponde PING => replica"
/usr/local/bin/syncoid  -r --sshkey=/root/script/root_backup --identifier=bakrem tank/d ro
ot@backup.francocorbelli.com:zroot/copia_rambo
/bin/date +"%R ----------REPLICA locale: fine replica su backup"
else
    /bin/date +"%R non pingato server di replica backup!"
fi

The ssh key, a "identifier" (aka: the prefix of the snapshots)
The crontab
Code: 
01 07 * * * /root/script/bak_replicaremota.sh >/dev/null 2>&1

BUT

you need this

github.com

sanoid/FREEBSD.readme at master · jimsalterjrs/sanoid

These are policy-driven snapshot management and replication tools which use OpenZFS for underlying next-gen storage. (Btrfs support plans are shelved unless and until btrfs becomes reliable.) - jim...
github.com github.com
Syncoid assumes a bourne style shell on remote hosts. Using (t)csh (the default for root under FreeBSD)
will cause syncoid to fail cryptically due to 2>&1 output redirects.

To use syncoid successfully with FreeBSD targets, you must use the chsh command to change the root shell:
root@bsd:~# chsh -s /bin/sh

FreeBSD users will also need to change the Perl shebangs at the top of the executables from #!/usr/bin/perl
to #!/usr/local/bin/perl in most cases.

Sorry folks, but if I set this with #!/usr/bin/env perl as suggested, then nothing works properly
from a typical cron environment on EITHER operating system, Linux or BSD. I'm mostly using Linux
systems, so I get to set the shebang for my use and give you folks a FREEBSD readme rather than
the other way around. =)

If you don't want to have to change the shebangs, your other option is to drop a symlink on your system:

root@bsd:~# ln -s /usr/local/bin/perl /usr/bin/perl

After putting this symlink in place, ANY perl script shebanged for Linux will work on your system too.
Click to expand...
 
fcorbelli said:
Ahem... no
If you do a send-receive on the same machine, maybe on two different drives, you're doing nothing more than an rsync
Of course you will get a replica, and therefore all the various snapshots, and therefore it will also be very quick to obtain
But it won't be much better than a mirror
Click to expand...
Ok, seriously, what are you talking about? You can access any individual file in a snapshot easily, and covacat pointed out how that works...
 
patmaddox said:
In my above post, by "restore" I meant "disaster recovery" where my local disk caught on fire and I need to restore it completely.

ZFS snapshots are without a doubt the best backup format that I've ever used. covacat already showed how to restore a file. It couldn't be easier.
Click to expand...
Not so easy
Also because, sooner or later, the snapshots will be purged
In my experience with magnetic disks already a thousand slow down considerably

Code: 
zpaqfranz l intermedie.zpaq -all


zpaqfranz v54.16i-experimental (HW BLAKE3), SFX64 v52.15, compiled Jun 15 2022
intermedie.zpaq: 
Block          1 K       58.823 (block/s)
484 versions, 484 files, 790 fragments, 35.934.705 bytes (34.27 MB)

- 2022-09-17 11:35:10                   0       0001| +1 -0 -> 872.092
- 2022-09-17 11:33:28           1.923.392 A     0001|zpaqfranz.cpp
- 2022-09-17 17:07:17                   0       0002| +1 -0 -> 151.268
- 2022-09-17 17:06:47           1.923.589 A     0002|zpaqfranz.cpp
- 2022-09-17 17:33:20                   0       0003| +1 -0 -> 13.259
- 2022-09-17 17:33:12           1.923.595 A     0003|zpaqfranz.cpp
(...)
- 2022-09-30 15:07:15           1.971.339 A     0194|zpaqfranz.cpp
- 2022-09-30 15:14:44                   0       0195| +1 -0 -> 742
- 2022-09-30 14:13:45           1.968.810 A     0195|zpaqfranz.cpp
- 2022-09-30 15:15:28                   0       0196| +1 -0 -> 13.752
- 2022-09-30 15:15:27           1.968.806 A     0196|zpaqfranz.cpp
- 2022-09-30 15:15:49                   0       0197| +1 -0 -> 101.480
- 2022-09-30 15:15:47           1.968.815 A     0197|zpaqfranz.cpp
- 2022-10-01 13:12:03                   0       0198| +1 -0 -> 136.553
- 2022-10-01 13:11:51           1.969.104 A     0198|zpaqfranz.cpp
- 2022-10-01 13:12:15                   0       0199| +1 -0 -> 112.745
- 2022-10-01 13:12:13           1.969.125 A     0199|zpaqfranz.cpp
- 2022-10-01 13:18:36                   0       0200| +1 -0 -> 112.776
- 2022-10-01 13:18:34           1.969.179 A     0200|zpaqfranz.cpp
- 2022-10-01 13:44:15                   0       0201| +1 -0 -> 112.738
- 2022-10-01 13:44:13           1.969.227 A     0201|zpaqfranz.cpp
- 2022-10-01 13:48:22                   0       0202| +1 -0 -> 112.834
- 2022-10-01 13:48:20           1.969.311 A     0202|zpaqfranz.cpp
- 2022-10-01 13:56:29                   0       0203| +1 -0 -> 112.865
- 2022-10-01 13:56:28           1.969.404 A     0203|zpaqfranz.cpp
- 2022-10-01 13:57:12                   0       0204| +1 -0 -> 112.836
- 2022-10-01 13:57:10           1.969.367 A     0204|zpaqfranz.cpp
- 2022-10-01 13:58:59                   0       0205| +1 -0 -> 112.891
- 2022-10-01 13:58:55           1.969.464 A     0205|zpaqfranz.cpp
- 2022-10-01 14:04:09                   0       0206| +1 -0 -> 112.844
- 2022-10-01 14:04:07           1.969.506 A     0206|zpaqfranz.cpp
- 2022-10-01 14:04:33                   0       0207| +1 -0 -> 123.147
- 2022-10-01 14:04:32           1.969.507 A     0207|zpaqfranz.cpp
- 2022-10-01 14:06:12                   0       0208| +1 -0 -> 13.971
- 2022-10-01 14:06:09           1.969.614 A     0208|zpaqfranz.cpp
- 2022-10-01 14:08:54                   0       0209| +1 -0 -> 135.886
- 2022-10-01 14:08:52           1.969.716 A     0209|zpaqfranz.cpp
- 2022-10-01 14:10:21                   0       0210| +1 -0 -> 14.004
- 2022-10-01 14:10:19           1.969.729 A     0210|zpaqfranz.cpp
- 2022-10-01 14:10:35                   0       0211| +1 -0 -> 14.016
- 2022-10-01 14:10:33           1.969.732 A     0211|zpaqfranz.cpp
- 2022-10-01 14:10:53                   0       0212| +1 -0 -> 14.016
- 2022-10-01 14:10:51           1.969.732 A     0212|zpaqfranz.cpp
- 2022-10-01 14:11:53                   0       0213| +1 -0 -> 14.059
- 2022-10-01 14:11:51           1.969.811 A     0213|zpaqfranz.cpp
- 2022-10-01 14:12:16                   0       0214| +1 -0 -> 14.045
- 2022-10-01 14:12:14           1.969.844 A     0214|zpaqfranz.cpp
- 2022-10-01 14:14:06                   0       0215| +1 -0 -> 14.073
- 2022-10-01 14:14:02           1.969.912 A     0215|zpaqfranz.cpp
- 2022-10-01 14:21:34                   0       0216| +1 -0 -> 15.611
- 2022-10-01 14:21:30           1.974.319 A     0216|zpaqfranz.cpp
- 2022-10-01 14:22:45                   0       0217| +1 -0 -> 47.286
- 2022-10-01 14:22:43           1.974.326 A     0217|zpaqfranz.cpp
- 2022-10-01 14:23:12                   0       0218| +1 -0 -> 35.974
(...)
- 2022-11-28 19:51:59           3.810.656 A     0482|zpaqfranz.cpp
- 2022-11-29 14:57:53                   0       0483| +1 -0 -> 191.301
- 2022-11-29 14:50:40           3.812.814 A     0483|zpaqfranz.cpp
- 2022-11-29 15:03:23                   0       0484| +1 -0 -> 40.650
- 2022-11-29 15:02:59           3.813.006 A     0484|zpaqfranz.cpp

971.514.969 (926.51 MB) of 971.514.969 (926.51 MB) in 968 files shown



Do you want this specific version  (just an example)?

- 2022-10-01 13:44:13           1.969.227 A     0201|zpaqfranz.cpp




C:\zpaqfranz\spaz>zpaqfranz x intermedie.zpaq -to z:\estratto -until 201
zpaqfranz v54.16i-experimental (HW BLAKE3), SFX64 v52.15, compiled Jun 15 2022
intermedie.zpaq -until 201:
201 versions, 201 files, 324 fragments, 14.627.901 bytes (13.95 MB)
Extracting 1.969.227 bytes (1.88 MB) in 1 files (0 folders) with 32 threads

0.047 seconds (00:00:00)  (all OK)

C:\zpaqfranz\spaz>dir z:\estratto
 Il volume nell'unità Z è RamDisk
 Numero di serie del volume: F849-FB20

 Directory di z:\estratto

30/11/2022  17:03    <DIR>          .
30/11/2022  17:03    <DIR>          ..
01/10/2022  12:44         1.969.227 zpaqfranz.cpp
               1 File      1.969.227 byte
               2 Directory  40.324.943.872 byte disponibili
 
fcorbelli said:
Not so easy
Also because, sooner or later, the snapshots will be purged
In my experience with magnetic disks already a thousand slow down considerably
Click to expand...

You do not need to receive snapshots to restore files. You just need the snapshots accessible. Ideally snapshots are accessible in three places:

1. Your local file system.
2. A low-latency backup server.
3. An offsite backup server.

So, to restore a file, you first look at your local snapshots. If the file isn't there, you look at your low-latency backup server. That's really it. If you trust the security of your offsite backup server, you can do the same thing there. The only reason I would need to pull from offsite is because I have encrypted it, and don't load the encryption key there. But that's a security choice, not a technical limitation. It's just as easy to load the encryption key, mount the files, explore the snapshots.

I'll give you a real world example. I use rsync to back up files from a Mac to a ZFS NAS, and then replicate the ZFS datasets to rsync.net. The only reason I use rsync from Mac is because I don't have ZFS running on it yet. https://openzfsonosx.org/ has an upcoming release for Ventura. When that becomes available, I will stress-test it on a secondary machine. If I'm satisfied with performance and reliability, I will add a ZFS partition to my Mac and use it instead of rsync to get the files from the Mac to the NAS.

Here's how I can restore a file. The final command is the great trick - it shows me every unique version of the file I have backed up. It's just a file system, so you can do whatever you ordinarily do with file systems.

Code: 
patmaddox$ ssh nas
Welcome to FreeBSD!

patmaddox@nas:~ $ cd istudo-pending/
patmaddox@nas:~/istudo-pending $ ls next\ EIS\ lesson/next\ EIS\ lesson.dorico
next EIS lesson/next EIS lesson.dorico

patmaddox@nas:~/istudo-pending $ ls .zfs/snapshot/*/next\ EIS\ lesson/next\ EIS\ lesson.dorico
.zfs/snapshot/autosnap_2022-11-27_15:35:03_hourly/next EIS lesson/next EIS lesson.dorico
.zfs/snapshot/autosnap_2022-11-27_16:35:03_hourly/next EIS lesson/next EIS lesson.dorico
<SNIP 68 LINES>
.zfs/snapshot/autosnap_2022-11-30_09:35:02_hourly/next EIS lesson/next EIS lesson.dorico
.zfs/snapshot/autosnap_2022-11-30_10:35:03_hourly/next EIS lesson/next EIS lesson.dorico

patmaddox@nas:~/istudo-pending $ for FILE in .zfs/snapshot/*/next\ EIS\ lesson/next\ EIS\ lesson.dorico; do SUM=$(md5sum -q "$FILE"); echo "$FILE - $SUM"; done | uniq -f 1
.zfs/snapshot/autosnap_2022-11-27_15:35:03_hourly/next EIS lesson/next EIS lesson.dorico - 30f83d10ad23486a4f652edace7e0dbb
.zfs/snapshot/autosnap_2022-11-27_16:35:03_hourly/next EIS lesson/next EIS lesson.dorico - 7e2f04af5b86daf3f49f8bd5128b9d56
<SNIP 9 LINES>
.zfs/snapshot/autosnap_2022-11-30_01:35:03_hourly/next EIS lesson/next EIS lesson.dorico - 8e4c096c657522e02771b847fd24a0b7
.zfs/snapshot/autosnap_2022-11-30_03:35:03_hourly/next EIS lesson/next EIS lesson.dorico - 6e1ac76be53ad9c99928257d60bae0da
 
Well not quite
Sooner or later you have to prune snapshots
Even enumerating a thousand snapshots takes minutes
And sooner or later free space will end

Instead I never prune
Ever, from 2014 until today

No worry for encryption


You cannot trust rsync either,for two reasons

First is codepage. Cannot copy from bsd to mac maintaining non latin charsets
Happens all the time

Second rsync without --checksum does not copy office xls and ppt ,because windows' office change internal metadata without touching the external (filesystem) metadata
Two runs needed, one for everything, one for ppt and xls
Very slow in enumerating
No encryption

Very slow to check

Therefore those are only two of at least 7 different strategies that can be used

As I say for backup of data nothing so far is better than zpaq
Second place hb
Then zip
Then zfs into file
Then rsync
 
Ps you can find zpaq into ports paq, if you want to try
 
I tuned my scripts now it works without problems.
Here is the code,
On boot,
cat once_usr_home
Code: 
export source="ZT/usr/home"
export mp="/mnt/snap_usr_home_hourly"
export mydate=`/bin/date "+%Y_%m_%d__%H_%M_%S"`
export destsmall="ZHD/backup_usr_home"
export dest=${destsmall}@${mydate}
export current=${source}@${mydate}
/sbin/zfs destroy -r -f -v ${destsmall}
/sbin/zfs create -u -v ${destsmall}
/sbin/zfs list -t snap ${source} | /usr/bin/grep ${source}@  | /usr/bin/awk '{print $1}' | /usr/bin/xargs -I {} /sbin/zfs destroy -v {}
/sbin/zfs snapshot ${current}
echo "SRC:" ${current}
echo "DST:" ${dest}
( /sbin/zfs send ${current} 2>>/var/log/messages | /sbin/zfs receive -o readonly=on -o snapdir=hidden -o checksum=skein -o compression=lz4 -o atime=off -o relatime=off -o canmount=off -o mountpoint=${mp} -F -v -u ${dest}  2>>/var/log/messages ) || /usr/bin/logger "zfs-send-receive-once failed" ${current} ${dest}

Each hour:
Code: 
export source="ZT/usr/home"
export mydate=`/bin/date "+%Y_%m_%d__%H_%M_%S"`
export current=${source}@${mydate}
export dest="ZHD/backup_usr_home"@${mydate}
export previous=` /sbin/zfs list -t snap -r ${source} | /usr/bin/grep ${source}@  | /usr/bin/awk 'END{print}' | /usr/bin/awk '{print $1}'`
/sbin/zfs snapshot             ${current}
echo "SRC:" ${previous} ${current}
echo "DST:" ${dest}
( /sbin/zfs send -i ${previous} ${current} 2>>/var/log/messages | /sbin/zfs receive -o readonly=on -v -u ${dest} 2>>/var/log/messages ) || ( /usr/bin/logger "zfs-send-receive failed" ${previous} ${current} ${dest} ; /root/Root/backup/once_usr_home )
 
patmaddox said:
I don't. I am completely happy with ZFS snapshots, for the reasons that I demonstrated. I admire your persistence though!
Click to expand...
Well, I admire your unwillingness to learn something new
And maybe find something better, that you simply don't know
This in general should be the purpose of a forum. Learn new things

In this case using incredibly fragile scripts to get a much worse result then a straight
Code: 
zpaq a /zhd/backup/myveryownbackup.zpaq /usr/home/whatIwanttoget

But everyone is free to remain in their very own Dunning-Kruger effect :)
 
Any suggestion that running zfs send/recv is inferior to some other option for replication* of a filesystem is really going to have to show some receipts.

It's as near-ideal as I've seen, once you wrap your head around exactly what is happening, and design your zfs layout to accommodate the backup routine you would like. You end up with the full filesystem you're interested in. Listing 1000 snapshots does not take minutes as you claim (3200+ took 20s on my box), and pruning them is not as arduous as you make it (hello, awk and xargs). You can also prune snapshots in the middle (temporally), and leave older ones intact.

And to your concern about MS and metadata, there's no fooling ZFS as to what is on the filesystem; it is the filesystem.

The ability to update the backup of filesystems with tens of millions (yes) of files in well under a minute** exists with ZFS, explicitly because it is the filesystem, so it knows explicitly what has changed from time point A to point B without having to crawl through the tree and look for changes.

Deduplication, compression, and verification (checksums embedded with and checked against the data) all exist in ZFS, if you want to turn them on (well, verification is on by default, and strongly recommended against turning off, but the knob is there).

* Except, obviously, if you don't run ZFS on both sides. You can use it (by saving streams) in that fashion, but I will agree it is far from ideal (and not how it is intended to be used.)

** Closer to a second if there have been no changes, obviously much longer if there have been a significant of actual changes; on my system, enumerating and sending 20GB worth of changes accumulated over a week on a filesystem system with 20M files took less than 15s. (This time will obviously depend on the types of I/O going in to creating those 20GB of changes.) Good luck doing anything from above the filesystem layer that can guarantee that any changes (even changes obfuscated by retaining file sizes and/or modification dates) of those 20M files were carried over to the destination in 15s.
 
Eric A. Borisch said:
Any suggestion that running zfs send/recv is inferior to some other option ...
...
The ability to update the backup of filesystems with tens of millions (yes) of files in well under a minute** exists with ZFS, explicitly because it is the filesystem, so it knows explicitly what has changed from time point A to point B without having to crawl through the tree and look for changes.
Click to expand...

That is a very good argument. Matter-of-fact, the head of my department (a large research group at one of those big computer companies) made exactly that argument to executives, and they gave him a blank check worth many M$ to create a new file system, which was intended to be integrated with the backup system. We built it, it worked well, we shipped it to customers, they liked it, and then the company cancelled the product (for good and logical reasons, which had nothing to do with backup integration).

Except that there is one argument that is ignored in that argument: You are relying on the correctness (being bug free) of ZFS here. If there is a ZFS bug, then the replication process will destroy BOTH copies. That gets back to the fundamental question of backup, redundancy and replication: What are you trying to accomplish? Are you trying to defend against failure of a single disk (typically, one uses RAID for that)? Against failure of the whole site (fire destroys the computer, a good defense is an off-site backup)? Or in this case, against a software bug?

On my home system, the important data is in ZFS on FreeBSD, using two mirrored disks. I know that 3 disks would be better, but I have only limited space and physical connections. The local backup copy is also on ZFS, using a 3rd disk drive which is about 2m (6 feet) away from the server, and reasonably well protected against fire. The remote backup copy is on a Mac AppleFS disk with native encryption, and many miles away. I used to have backups also on a cloud-based service (one of the big cloud providers), but my homebrew software lost that ability, and I didn't maintain it well enough. But you can see that I deliberately use more than one file system (and OS) for the backup system. You can never be too paranoid.
 
ralphbsz said:
That is a very good argument. Matter-of-fact, the head of my department (a large research group at one of those big computer companies) made exactly that argument to executives, and they gave him a blank check worth many M$ to create a new file system, which was intended to be integrated with the backup system. We built it, it worked well, we shipped it to customers, they liked it, and then the company cancelled the product (for good and logical reasons, which had nothing to do with backup integration).

Except that there is one argument that is ignored in that argument: You are relying on the correctness (being bug free) of ZFS here. If there is a ZFS bug, then the replication process will destroy BOTH copies. That gets back to the fundamental question of backup, redundancy and replication: What are you trying to accomplish? Are you trying to defend against failure of a single disk (typically, one uses RAID for that)? Against failure of the whole site (fire destroys the computer, a good defense is an off-site backup)? Or in this case, against a software bug?

On my home system, the important data is in ZFS on FreeBSD, using two mirrored disks. I know that 3 disks would be better, but I have only limited space and physical connections. The local backup copy is also on ZFS, using a 3rd disk drive which is about 2m (6 feet) away from the server, and reasonably well protected against fire. The remote backup copy is on a Mac AppleFS disk with native encryption, and many miles away. I used to have backups also on a cloud-based service (one of the big cloud providers), but my homebrew software lost that ability, and I didn't maintain it well enough. But you can see that I deliberately use more than one file system (and OS) for the backup system. You can never be too paranoid.
Click to expand...
Couldn’t agree more. The things I care most about (photos, for example) are backed up on two separate zpools (one typically detached and off-site) and in the cloud. I try not to be bleeding-edge (I don’t run the ports-tree OpenZFS) and run scrubs religiously.

But yes, a software bug could bring it all down, but so far (knocks on wood) I’ve had much better luck with ZFS than any other FS. But your reminder to inject some diversity of technologies for critical data is worth remembering.
 
I think there is a misunderstanding
"You" can use zfs as a backup mechanism
I've been doing it for about a decade (even more, from Solaris era)
Both locally and remotely
Both with scripts and with syncoid
As I explained, normally, I use at least 5 different programs and methodologies on between 7 and 10 copies for each server

What I have tried to explain, obviously failing, is that depending on the different cases there are more or less suitable tools

I understand that it is normal to develop, with work of months or even years, scripts that automate copies
I have some that are a thousand lines long

---
So I'm not saying "don't use zfs snapshots they are useless"

I use Solaris and BSD servers for this very reason (instead of Linux)

I'm saying "ADD to snapshots specific programs that do things snapshots don't"
Very easily

After 35 years of working in disaster recovery I have some experience.
Or at least I pretend to
---

The first is hb. HashBackup
It works perfectly on FreeBSD and is not complex to use
Equivalent is borg, but of the two (on BSD) I prefer the former
It does have one problem however, having a folder approach which I hate, it's too fragile to backup backups

Then there's zpaq, which is a kind of 7zip with snapshots, written by Mahoney
If you are not an operator of the branch, i.e. you don't deal with compression, backup and disaster recovery, it won't tell you anything
If you are instead of the "paq" will tell you everything you need to know
It is roughly (in compression) the same as saying "car designed by Lewis Hamilton" or "car by Ayrton Senna".
Incidentally it was technology used by Dell in its storage systems etc.

It is in the BSD port system since 2014 with an old version.
The designer has retired and no longer supports it since 2016

Then there is zpaqfranz (DISCLAIMER: my work) that is zpaq that "understand" .zfs folders, and put a bunch of very paranoid-level checks on integrity (much, much, MUCH higher than zfs')
It is just about "everything you ever dream for a backup in a single executable"

Short version: don't trust me. I steal candy bars from little boys, indeed
Then compile (or install) the paq package, the try yourself zpaq (old version).
And you horizon will widen a lot

Of course you can also say "my superduper script snapshot" is perfect
No it is not
Just like mine it has strengths and weaknesses
You can say "I've been using it for years and it suits me"
Mine do too
But you won't be able to redo zpaq's functions though
Or, at least, not with the same banality in basically zero time

zfs is great for replicas
But for backups, there are better solutions

That's all

I personally use (together, not alternately)

zfs
zfSnap
syncoid
7z
hb
zpaq
zpaqfranz
rsync
hashdeep
pgzip
rar
diff
 
Eric A. Borisch said:
Any suggestion that running zfs send/recv is inferior to some other option for replication* of a filesystem is really going to have to show some receipts.
(...)
Click to expand...
Replication is not backup
Eric A. Borisch said:
Deduplication, compression, and verification (checksums embedded with and checked against the data) all exist in ZFS, if you want to turn them on (well, verification is on by default, and strongly recommended against turning off, but the knob is there).
Click to expand...
zpaq(franz) has much more deduplication, compression and verification and checksum of zfs
Eric A. Borisch said:
* Except, obviously, if you don't run ZFS on both sides. You can use it (by saving streams) in that fashion, but I will agree it is far from ideal (and not how it is intended to be used.)
Click to expand...
And in a LOT of other situations
Eric A. Borisch said:
** Closer to a second if there have been no changes, obviously much longer if there have been a significant of actual changes; on my system, enumerating and sending 20GB worth of changes accumulated over a week on a filesystem system with 20M files took less than 15s. (This time will obviously depend on the types of I/O going in to creating those 20GB of changes.) Good luck doing anything from above the filesystem layer that can guarantee that any changes (even changes obfuscated by retaining file sizes and/or modification dates) of those 20M files were carried over to the destination in 15s.
Click to expand...
You can send as much as fast, or even faster, with zpaq(franz) with rsync (!)
Yep, you can send the changes accumulated over X time with rsync to a NAS, or a Linux server (without zfs) or anything, as fast as zfs
I do all the time with terabyte-big server

Well, for instance, you cannot send 20GB in 15 seconds, unless you have a 1GB/s bandwidth

Let's be more precise
Your files changed of X bytes, you send X bytes (with zfs) (+ 0 calculation time)
Your files changed of X bytes, you send X bytes (with zpaq) (+ calculation time, that can be 1 minute or 1 week, depends)

In fact I
=> create the .zpaq
=> replicate the .zpaq with zfs
 
One last thing

If you've never once used zpaq or hb (or whatever), how are you supposed to compare it to zfs?

Try it first, then rate it
 
Eric A. Borisch said:
But yes, a software bug could bring it all down, but so far (knocks on wood) I’ve had much better luck with ZFS than any other FS. But your reminder to inject some diversity of technologies for critical data is worth remembering.
Click to expand...
If you have a professional level of paranoia, you will use different software compiled on different machines with different filesystems and different CPUs and EVEN different endianness

That is ... you will use zpaqfranz :)

I'm kidding, but up to a point
Because, in fact, it is true
 
patmaddox said:
I think you're confusing me with someone else at this point. I doubt you'd agree I have "incredibly fragile scripts" - and we may disagree on effectiveness of the results.
Click to expand...
If you have a script more complex than a single line yes, you have an incredible fragile script, for my standard at least
Maybe your are lower
Maybe
 
I just want to point out that

zfs send |zfs receive

is not a backup, but a replica (with script, syncoid, whatever), just about like rsync
Way faster, of course, way better,
BUT
not a backup

zfs send >base.zfs
zfs send differential >differential.zfs
(...)
zfs send >base.zfs
zfs send incremental >incremental1.zfs
zfs send incremental >incremental2.zfs
zfs send incremental >incremental3.zfs (pippo.txt is here)

those are backups, just like in the script I already posted (yes, I use script too)

BUT
there is not a easy way to restore pippo.txt, you cannot simply "cp" as previously writted

You have to "restore" the base (zfs receive), the "apply" the difference, then enter into the folder, THEN you can cp
Way complex and fragile, if you have a bunch of incremental images to be restored

There is, AFAIK, an easy way to "mount" zfs' image files
If images are big, this can take hours and even days (with a lot of incremental snapshot to be applied)
BTW
hb do this "thing" (mount-on-the-fly the backups)

Sooner or later the snapshot will be purged => data is lost
That a fact of using zfs as a "backup" method

---
On checking and checksums
There is a misunderstanding on the technical level
It is impossible to get the full hash of a FILE via hash functions, based on a random access and write
It is possible via CRC-32, but it is not used for other reason
Recap: zfs CANNOT tell you what is the hash(pippo.txt), because zfs does not know, and cannot know
You have to read pippo.txt, then computer hash(pippo.txt) (as rsync, rar, 7z, zpaqfranz etc do)
Therefore there is NOT an easy way to check if your valuable data is "good", in a zfs backup
You have to trust on zfs, you cannot check yourself
I can make the "spiegone", if someone is interested in filesystem & checksums & hashing

Short version. you get a backup/replica and you PRAY it is good. You BELIEVE, you DO NOT KNOW
This is where, for instance, hashdeep kicks in (with a good script) o zpaqfranz (of course :)

Recap: a "long range" replication (aka zfs send |ssh |zfs receive) means you CANNOT compare local and remote. You must trust
A "backup" (zfs send >afiletobekeeped) means you CANNOT check what's inside, CANNOT compare. You must trust

Those are just the biggest "flaws" for zfs as a backup method
You cannot check
 
So, your argument is "it's only a backup when it's stored in some kind of archive file". Which is, of course, entirely made up, this has nothing to do with the definition of the term backup. And THEN, only based on this, you argue some tooling would be needed to still access files in dumps of serialized(!) zfs snapshots.

Sorry, that won't convince many people on here. It's a different story if you are in a situation where you MUST store your backup somewhere zfs isn't an option. *)

Furthermore, of course you won't keep old backups forever, so "purging" old snapshots is a natural thing. With a zfs pool as your backup target, no data will be lost except of course the exact state of these old snapshots you purge.

*) [edit] -- Nothing against advocating your software, in fact, I'd say that's pretty much welcome, and very likely, your tool does a good job for this use case. But starting to insult people and neglecting their different use case isn't good advocacy, please keep that in mind.
 
Hello Circular Arguing.
Everyone's specific needs and desires for data retention and recoverability are different.
Best practices for an enterprise that has specific retention records because of tax/government reasons are likely
overkill for a home user that doesn't want to lose pictures of the grandkids.

It boils down to requirements. Define what you need, then figure out how to best implement it.
Don't discount any tools, but pick one that does what you want, affords flexibility and think about the future.
Storing copies on CDs/DVDs/USB drives offers easy access (recoverability) but for how long? We've had threads recently about issues with reading CD/RW and others, all data degrades over time, so is 20yrs long enough?

So this thread started as "This is the procedure I'm using because it suits my needs, but it doesn't work reliably, can anyone tell me why" to something else.
If the OP thinks his question has been solved, maybe it's time to let it be.
But that's all my opinion, feel free to do what you want.
 
A backup-strategy must work for you and we might all have different requirements.
Mine is keep it simple. Regular local snapshots & send to a different place/location in a reliable way.
 
  • Like
Reactions: mer
You must log in or register to reply here.
Back
Top