Solved Strange - why is the zfs dataset deleted ?

P

Peacekeeper2000

For me, this looks a bit strange, but maybe I assume I did not understand it overall. So what I try to do is to use zfs send and zfs receive to create a backup of a dataset with movies. So everytime I added some movies, I make a snapshot and send this incremental snapshot. Overall I want to replicate the dataset.

It worked fine so far until I added the -R on the sending side and the -F on the recv. So I attach the output :
Code: 
...
send from @20221210 to zvideo/video@20221210a estimated size is 624
total estimated size is 1,22K
send from @20221209 to zvideo/video@20221210 estimated size is 624
send from @20221210 to zvideo/video@20221210a estimated size is 624
attempting destroy zmedia1/video@20220722
success
attempting destroy zmedia1/video@20220819
success
attempting destroy zmedia1/video@20221011
success
attempting destroy zmedia1/video@20221030
success
receiving incremental stream of zvideo/video@20221210 into zmedia1/video@20221210
snap zmedia1/video@20221210 already exists; ignoring
received 0B stream in 1 seconds (0B/sec)

replicator@blackbox:~ $ zfs list -t all -r zvideo
NAME                     USED  AVAIL  REFER  MOUNTPOINT
zvideo                  6,70T  2,11T   104K  /filmpool
zvideo@20220819           56K      -   104K  -
zvideo/natur            68,1G  2,11T  68,1G  /filmpool/natur
zvideo/natur@20220819     56K      -  68,1G  -
zvideo/tv                353G  2,11T   353G  /filmpool/tv
zvideo/tv@20220819        56K      -   353G  -
zvideo/video            6,29T  2,11T  6,26T  /storage/video
zvideo/video@20221104   2,85G      -  6,09T  -
zvideo/video@20221114    488K      -  6,11T  -
zvideo/video@20221124   8,97G      -  6,16T  -
zvideo/video@20221207    720K      -  6,23T  -
zvideo/video@20221208    344K      -  6,25T  -
zvideo/video@20221209       0      -  6,26T  -
zvideo/video@20221210       0      -  6,26T  -
zvideo/video@20221210a      0      -  6,26T  -
replicator@blackbox:~ $ zfs send -RceV --verbose -I @20221210a zvideo/video@20221211  | ssh replicator@silverwave zfs receive -vF zmedia1/video
send from @20221210a to zvideo/video@20221211 estimated size is 58,6K
total estimated size is 58,6K
send from @20221210a to zvideo/video@20221211 estimated size is 58,6K
cannot receive incremental stream: destination 'zmedia1/video' does not exist
cannot open 'zmedia1/video': dataset does not exist
attempting destroy zmedia1/video@20221104
success
attempting destroy zmedia1/video@20221114
success
attempting destroy zmedia1/video@20221124
success
attempting destroy zmedia1/video@20221207
success
attempting destroy zmedia1/video@20221208
success
attempting destroy zmedia1/video@20221209
success
attempting destroy zmedia1/video@20221210
success
attempting destroy zmedia1/video
success
I did this twice - the first time with @20221210 as snapshot, the second with @20221210a . So it seems that second snapshot with the "a" as the last letter was not recognised as a different snapshot. Why ?!
And then the second it just has deleted the whole dataset instead of ignoring it again . Why this ?

I was under the impression that the parameter -R is for replicating including the zfs-properties. The -F will delete all snapshots on the receiving side , that also no longer available on the source/send side.
dI am glad that zvideo/video with more than 1200 movies is still available on the sending side. But I was not expecting that zmedia1/video is destroyed and all underlying snapshots.
Where is my fault of thinking ?
 
I have had the "same" experience where intermediate snapshots on receiving side where destroyed.

Now, This is how i send incremental backups. And it works fine, just "-i" on zfs-send. Nothing on zfs-receive.
Code: 
echo "LAST:" ${last}      >> /var/log/messages
echo " SRC:" ${srcdate}   >> /var/log/messages
echo " DST:" ${dst}       >> /var/log/messages
( /sbin/zfs send -i ${last} ${srcdate} 2>>/var/log/messages |  \
  /sbin/zfs receive -o readonly=on -o snapdir=hidden -o checksum=skein -o compression=lz4 -o atime=off -o relatime=off -o canmount=off -v ${dst} >>/var/log/messages 2>&1 ) || \
/usr/bin/logger "zfs-send-receive failed" ${last} ${srcdate} ${dst}
I tried parameters/flags until it worked good. Because i found the documentation of flags/parameters very confusing...
 
I had a complete script for automatic backup by using als using zfs user -properties. Today I would rather take zxfer for tis instead of a self-developed tool. But I would really understand, why the above happend ? There is something strange happening between the "ignoring duplicate snapshot" and the next snapshot ....
 
zfs-send -R:
Code: 
-R, --replicate
           Generate a replication stream package, which will replicate the
           specified file system, and all descendent file systems, up to the
           named snapshot.  When received, all properties, snapshots,
           descendent file systems, and clones are preserved.

           If the -i or -I flags are used in conjunction with the -R flag, an
           incremental replication stream is generated.  The current values of
           properties, and current snapshot and file system names are set when
           the stream is received.  If the -F flag is specified when this
           stream is received, snapshots and file systems that do not exist on
           the sending side are destroyed.  If the -R flag is used to send
           encrypted datasets, then -w must also be specified.

zfs-receive -F:
Code: 
-F  Force a rollback of the file system to the most recent snapshot
           before performing the receive operation.  If receiving an
           incremental replication stream (for example, one generated by zfs
           send -R [-i|-I]), destroy snapshots and file systems that do not
           exist on the sending side
 
Yepp, so far so good. BUT the the sending filesystem still exists, while the received filesystem is now destroyed ! My expectations using -R and -F was, that all snapshots that no longer exists on the sending side, are also deleted on the receiving side. So if I have @snap_a, @snap_b, and @snap_c on dataset/media_org and I have @snap_a, @snap_b on dataset/media_back. I would expect that the following commands:
zfs destroy dataset/media_org@snap_a will delete the first snapshot
zfs send -RI @snap_b dataset/media_org@snap_c | zfs recv -F dataset/media_back will
  1. copy all zfs properties from media_org to media_back
  2. copy snap_c to media_back
  3. destroy dataset/media_back@snap_a
but will NOT destroy dataset/media_back, as the dataset/media_org still exists. But exactly that has happend and I wonder why
 
Yes, it has not changed the sending side ( in my example the dataset/media_org ) , but it was also my understanding, that it will not destroy the receiving dataset (as long as the original set is still there) - just the missing snapshots of the source will also be deleted on the receiving side ( so they are in sync again) . Maybe I would need to use the -d or -e Option on zfs recv ...
 
On the pratical side, I agree with you to not use -F for recv. On the thoretical side I would like to understand how to use it right and what happened, to prevent similar situations in the future.

One observation is also, that the "a" appended to the snapshot name was not recognised correctly. Another observation is, that the sending system is on FreeBSD 12.x ( ZFS Version 5) while the receiving system is on FreeBSD 13.x (ZFS Version 5) . While both ZFS Versions are the same, ZPOOL on the 13.x System is telling me, that the ZFS dataset created on the 12.x has not enabled all ZFS Option and could be upgraded. I don't get this on the 12.x system.

I was able to connect the zmedia device/pool to the 12.x system via eSATA. So my hope is, that it will not take again 3 days to copy the data. I also want now to create some playground with less example data to investigate the behaviour of -R / -F option from zfs send/recv . Meanwhile I will give zxfer a try.
 
Don't use too many flags/options.
For incremental backup i just use "-i" on sending side. That's all, no options/flags on receiving side.
I have an incremental backup each 15minutes.
 
Alain De Vos said:
Don't use too many flags/options.
For incremental backup i just use "-i" on sending side. That's all, no options/flags on receiving side.
I have an incremental backup each 15minutes.
Click to expand...
That is a possibility, but I like more to understand it. To my mind there are reasons for the flags and they also give you opportunities. Some flags ( like the -v and the -V ) seem to work not right as a combination, while they work as described when you use them "either-or". A snapshot every 15 min as a base for the incremental backup could fill up your pool when you add files ( as they will increase all snapshots ). Here I would question, how valuable are the data that is produced within 15. it might be a SAP system or other financial systems , where it could be valuable to only loose 15 min of data. For my private servers, this is not needed. Nowadays you also have to look, how actions will increase the power consumptions of your servers.
 
Peacekeeper2000 said:
On the pratical side, I agree with you to not use -F for recv. On the thoretical side I would like to understand how to use it right and what happened, to prevent similar situations in the future.
Click to expand...

There are whole other threads about if a ZFS replication is or is not a backup (depends on your definition of backup, to be sure) but receiving with -F, especially in any automated schedule, is a recipe for foot-shooting. If you accidentally (fat-finger) a sub-filesystem deletion on your source, and your backup kicks off while in this state, kiss your replication / “backup” goodbye.

So seriously look at if you can get by without -F. I would only consider -F for an infrequently updated (typically offline) tertiary backup where you periodically reconnect it, and double check it first with a dry run (-n) receive to make sure it is doing what you want.
 
Eric A. Borisch said:
There are whole other threads about if a ZFS replication is or is not a backup (depends on your definition of backup, to be sure) ...
Click to expand...
:)
Just one question
What exectly the author is seeking for?
The goal?

A backup of data (aka: files content), to be "quickly" restored in case of a problem (usually tank)?
An image (aka: everything, including metadata), to be SLOWLY restored in case of a problem (usually this is for zroot or some kind of user data with a lot of ACLs etc)
Or a quick replica, something like HA?

Because send-receive is the latter, a replica.
Just like a faster rsync or robocopy
Not a backup

Usually you will need a MANDATORY backup (well, from 2 to 10 copies)
THEN
you can "think" on a replica (aka: faster restore in case of hardware failure)

If "faster restore" doesn't make sense to you, you're making very bad decisions

FIRST you have to make a reliable backup (more than 1)
THEN whatever replica(s) you want
AND if "something" is much complex than one or two commands => it is too fragile for valuable data

Just a suggestion
 
fcorbelli said:
...
A backup of data (aka: files content), to be "quickly" restored in case of a problem (usually tank), saving the day?
An image (aka: everything, including metadata), to be SLOWLY restored in case of a HW problem (usually this is for zroot or some kind of user data with a lot of ACLs etc)
Or a quick replica, something like HA?

Because send-receive is the latter, a replica, just like a faster rsync or robocopy or copy-and-paste
Not a backup

A backup is some kind of file (.7z, tar.gz, rar, even send >file.zfs, or a folder or whatever) that you will store "somewhere else" (a local drive, USB enclosure, NAS, cloud, tape, BD...)

Usually you (as everyone else) will need a MANDATORY backup (well, from 2 to 10 copies)
THEN
you can "think" on a replica (aka: getting a faster restore in case of hardware failure)
THEN
you can "think" on a almost-real-time-replica (aka: HA)

If "faster restore" doesn't make sense to you (because you do not care on restore time, it is something to be done very rarely), you're taking very bad decisions tinkering with zfs' replicas

AND
if "something" is much complex than one or two commands => it is too fragile for valuable data
BTW
do NOT use incremental send receive. If you really want use differential send. (Of course "differential" means "incremental with a fixed initial"). The more perfect snapshot sequence needed, the higher the risk something "goes wrong"
Not a big deal for a replica, but a disaster for a backup

Just a suggestion :)

PS sorry but I am writing with the left hand, with a 10kgs cat on the knees and right arm :)
Click to expand...
Thanks for your thoughts. You may also consider the environment: ther is no business behind this, just personal memories (photo) and some private work ( ripping of CD's and cutting of movies ). So I don't need up to ten copies - too much disk device price. But I want at least 1-2 copies where one could be used instead of a corrupt original device (without a lot of setup changes ). A snapshot will also give me historical data ( e.g. a removed file, that was removed by error ) So zfs send ... and zfs recv ... would fit my backup needs very well without using any legacy backup software. Using the incremental method is also different to former backup programs as you will get always one full backup together with an additional incremental step ( which may held deleted files ).
I am using two different servers together with ssh to send the backup from one to the other. This ( | ssh...) is for a full backup too slow, but for incremental it is acceptable ( and you could use -s to get restart tokens). I am now using FreeBSD together with ZFS more than 10 yrs and I think it is very easy to manage and very reliable. During my business times I used UP-UX, MC ServiceGuard and LVM. I also started LVM on FreeBSD. But it is far too complex, when you get out of training.

I was thinking to use HAST, but then I remind my self, that my wife is the only Backup OP - and she is not willing to manage this. :D Right this weekend, ZFS has self-healed a disk ( with meta-errors) and the first basic snapshot is send&recv within two days ( via eSATA ) . So I don't want to move away from "send and recv" nor do I see ZFS as a problematic or fragile software. ( You know: the root cause is always in front of the keyboard :D )

As I am not a native speaker, my interpretation of -R and -F might be wrong. Or I also need to use -d and/or -e together with -F to get it right. I will now setup some "playground" dataset to try, error and understand this better.
But I am still open to comments, why this has happened, why the snapshot was not recognised right etc.
 
A smarter way to use send-receive
Make a local backup :), fast replicate with send-receive (syncoid, in fact) to a remote zfs-based server the backup

You can do exactly the same (just as fast) with rsync-over-ssh, at the same speed, and to NOT zfs (for example NAS, Linux, whatever) receiver

I do both

I really, really find hard to understand why "someone" want to complicate his own life by using a tool (send and receive) that is NOT designed to make backups... to make backups

Normally someone see this zfs function and say "cool, I like it, I'm going to use it even if I'll waste days trying to get it to work, and every time I'll have to pray something didn't go wrong"
nor do I see ZFS as a problematic or fragile software
Click to expand...
You are much more confident than me.
I see any software as fragile and problematic
Because they are
They really are
zfs, too
 
fcorbelli said:
You can do exactly the same (just as fast) with rsync-over-ssh, at the same speed, and to NOT zfs (for example NAS, Linux, whatever) receiver
Click to expand...
I agree with you that Solaris or Xenix is not easy to manage - and that the reason why Sun no longer exists :D I also have used rsync, but the advantage of zfs send/recv is that zfs knows which blocks have changed and is only sending these changed blocks, while rsync has to scrub the whole device (filessytem) and will send the whole file again when it has changed. So zfs send/recv is faster than rsync.

It is interesting that I ask for something specific to understand and to get some help with this and everybody try to pull me away from using it, instead of explaining the fault or wrong usage. It's clear for me, that I could use different software e.g. TrueNAS or ProxMox, but I also want to understand what I am doing and minimise the set of tools ( that can break).
 
Peacekeeper2000 said:
I agree with you that Solaris or Xenix is not easy to manage - and that the reason why Sun no longer exists :D I also have used rsync, but the advantage of zfs send/recv is that zfs knows which blocks have changed and is only sending these changed blocks, while rsync has to scrub the whole device (filessytem) and will send the whole file again when it has changed. So zfs send/recv is faster than rsync.

It is interesting that I ask for something specific to understand and to get some help with this and everybody try to pull me away from using it, instead of explaining the fault or wrong usage. It's clear for me, that I could use different software e.g. TrueNAS or ProxMox, but I also want to understand what I am doing and minimise the set of tools ( that can break).
Click to expand...
actually rsync checksums blocks of 64k or something from the file and only send changed blocks
kind of like onedrive/dropbox do
als if you rsync with --inplace remote files will be edited no copied and moved over so it's easier on remote snapshots if they exist
 
Peacekeeper2000 said:
It is interesting that I ask for something specific to understand and to get some help with this and everybody try to pull me away from using it, instead of explaining the fault or wrong usage. It's clear for me, that I could use different software e.g. TrueNAS or ProxMox, but I also want to understand what I am doing and minimise the set of tools ( that can break).
Click to expand...

TL;DR: recreate the state prior to the observed the issue, document the state on both ends, run the offending command (capturing output), and show the unexpected state after the command. With that documentation, we can try to help pick apart what went wrong.

Long version:

I'm not definitely not trying to pull you away from using zfs here, just suggesting you use -F with extreme care.

That said, if you would like help debugging what is happening, we need:
  1. zfs listings ( zfs list -rt snap zvideo/video of both the source and dest) before the send/recv in question
  2. The full send/recv command line
  3. The output from the send/recv process (I would disable the verbose on the send side; it just pollutes the log -- if you want the send side logs, you can ~recreate them with a send -nv [...] > /dev/null)
  4. The listing of the dest (assuming source is unchanged, as it should be from a send) after the process completes showing the unexpected state
Noting what version of FreeBSD you are on is also helpful; perhaps I missed it. There are a number of features available in 13's ZFS that aren't in 12's.

From your logs above the problems start with "cannot receive incremental stream: destination 'zmedia1/video' does not exist"; without a listing of zmedia1 on the destination side prior to the send/recv in question, we (or at least, I) can't help further debug what went wrong. It looks like things actually go south on the earlier send, here:

receiving incremental stream of zvideo/video@20221210 into zmedia1/video@20221210
snap zmedia1/video@20221210 already exists; ignoring
Click to expand...

Is there something creating timed (@YYYMMDD) snapshots on the destination pool separately from the send/recv process? Make sure that nothing else is creating new snapshots on the dest filesystem(s) other than the send/recv process. (You can, but only with snapshot names that do not conflict with the incoming snapshot names.) As an aside, also set dest filesystems to be readonly=on -- you can still zfs recv into a (filesystem layer) readonly=on destination.

I will say again, avoid -F unless you (1) really want what it does and (2) have run your process successfully multiple times without -F (you should never need -F, and only use it as a convenient way to prune, and again, I would caution against using it at all) and it's in a script so you know you're running it the same each time. You can use zfs-hold(8) to help provide some level of protection against accidental deletion of important snapshots. (I think that will cause recv -F to throw errors if it would otherwise delete the snapshot —as it won't be able to delete it — can't recall for certain.)
 
You must log in or register to reply here.
Back
Top