ZFS If an incremental snapshot fails to send, will it cause problems in the future when you need to restore from a snapshot?

puppydog

puppydog

Take for instance a prod server (source) with a zfs dataset that gets snapshotted once a day, and a backup server (destination) that receives those daily snapshots via a zfs send/receive operation. In the beginning a full initial snapshot was sent, and then a daily cron job was set up to each day send an incremental snapshot based off the previous day's snapshot, so it would be something like zfs send -i @daily_20231203 data/set@daily_20231204|ssh backupserver zfs receive backup/data/set. Each day the two dates increment by one day, so the next day would be zfs send -i @daily_20231204 data/set@daily_20231205 and so forth daily.

On 3, 4, 5, 6, and 7 December, the send operation runs on the source, but the network on the destination was down on 5 and 6 December. The source cronjob doesn't do anything to check what's on the destination server, so the backup server has the snapshots for only 3,4, and 7 December.

Is there going to be any data loss if the source server needs to be restored with the 7 December snapshot? I know you wouldn't be able to roll back to the missing snapshots, but is that the only downside to missing an intermediate snapshot like this?

When Googling I found people saying both ways - some people saying there's no problem, others saying you need all incremental snapshots. It feels like there would be a problem, like if a file was created between the 4 and 5 snapshots, then the incremental between the 6 and 7 snapshot wouldn't include it. But on the other hand, maybe ZFS does something under the hood that would account for something like this.

So which is it? Is it safe to miss intermediary snapshots and all you lose is the ability to roll back to the missed dates? Or are all subsequent snapshots on a backup server potentially ruined if an intermediary is missed?
 
I think you can use -R in combination with -I (capital "i") to "replicate" the source stream with all snapshots up to the given one.
For example, I use the following to completely replicate all history (also children are included in the parent dataset):
Code: 
zfs send -LPRecv dataset@snapshot
zfs send -LPRecv -I @first_snapshot dataset@fifth_snapshot
The rest of the flags are for verbose logging, compression etc. Look them up in the documentation.
I receive with:
Code: 
zfs receive -Fu target_dataset
-F forcefully deletes any tailing differences from the target and makes the target identical to the source snapshot. -u is used to prevent mounting. I also used a -s but as I send between FreeBSD and Linux, the ZFS on Linux has never been able to successfully resume an interrupted zfs receive, so I stopped using it.
 
When sending an incremental snapshot the destination's most recent snapshot must match the source incremental snapshot, i.e. when your destination previously only has received the 20231203 and 20231204 snapshots, it couldn't also receive your 20231207 incremental snapshot in the first place, because 20231206 is missing on the destination.

To protect against missing incremental snapshots on the destination you probably need a monitoring solution to check if your snapshots were actually received. There might be automatic solution already available, I don't know.

Also remember that ZFS allows regular files as vdevs. With files you can try and test many things before doing it on the "real" metal. Let's test your usecase:

Code: 
# truncate -s1g /tmp/src.img
# truncate -s1g /tmp/dest.img
# mdconfig -u0 /tmp/src.img
# mdconfig -u1 /tmp/dest.img
# mkdir /tmp/src /tmp/dest
# zpool create -m /tmp/src srcpool md0
# zpool create -m /tmp/dest destpool md1
# echo -n 'You ' > /tmp/src/note.txt
# zfs snap srcpool@0
# zfs send srcpool@0 | zfs recv destpool/bak
# echo -n 'lose ' >> /tmp/src/note.txt
# zfs snap srcpool@1
# echo 'big' >> /tmp/src/note.txt
# zfs snap srcpool@2
# zfs send -i @1 srcpool@2 | zfs recv destpool/bak
cannot receive incremental stream: most recent snapshot of destpool/bak does not
match incremental source
# zpool export destpool
# zpool export srcpool
# mdconfig -du1
# mdconfig -du0
# rm -rf /tmp/src*
# rm -rf /tmp/dest*
 
You must log in or register to reply here.
Back
Top