Incremental ZFS backup errors

xy16644 · Jan 26, 2014

I am trying to setup automated backups for my root ZFS pool. Currently I am just trying to get it working by manually running the commands. On my root ZFS pool I have the following datasets:

Code:

zroot                           /
zroot/tmp                       /tmp
zroot/usr                       /usr
zroot/usr/home                  /usr/home
zroot/usr/ports                 /usr/ports
zroot/usr/ports/distfiles       /usr/ports/distfiles
zroot/usr/ports/packages        /usr/ports/packages
zroot/var                       /var
zroot/var/crash                 /var/crash
zroot/var/db                    /var/db
zroot/var/db/pkg                /var/db/pkg
zroot/var/empty                 /var/empty
zroot/var/log                   /var/log
zroot/var/mail                  /var/mail
zroot/var/run                   /var/run
zroot/var/tmp                   /var/tmp

In preparation for the backups I did the following:

Code:

zfs create zstore/dailysnapshots
zfs set mountpoint=/dailysnapshots zstore/dailysnapshots

I changed the mountpoint so that I can access all my snapshots in /zbackups/dailysnapshots.

I then took the first full backup as follows:
zfs snapshot -r zroot@YESTERDAY
zfs send -R zroot@YESTERDAY | zfs receive -Fduv zstore/dailysnapshots

I then run the first incremental backup:
zfs snapshot -r zroot@TODAY
zfs send -R -i zroot@YESTERDAY zroot@TODAY | zfs receive -duv zstore/dailysnapshots

At this point all is great and I don't receive any errors. Except one. I can't see var, usr and tmp mounted in zstore/dailysnapshots. So I thought ok, I'll mount those with zfs mount -a. Now I can see var, usr and tmp in /zbackups/var, /zbackups/usr and /zbackups/tmp respectively. This isn't mounted in the correct location so I run:

 zfs set mountpoint=/dailysnapshots/tmp zstore/dailysnapshots/tmp

zfs set mountpoint=/dailysnapshots/var zstore/dailysnapshots/var

zfs set mountpoint=/dailysnapshots/usr zstore/dailysnapshots/usr

Now I can see everything as I want it in zstore/dailysnapshots and I can see my snapshots in the .zfs/snapshots directory for the relevant dataset.

But when I try to run the next incremental backup with:

zfs send -R -i zroot@TODAY zroot@TOMORROW | zfs receive -duv zstore/dailysnapshots I get:

Code:

receiving incremental stream of zroot@TOM into zstore/dailysnapshots@TOM
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/usr@TOM into zstore/dailysnapshots/usr@TOM
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/usr/ports@TOM into zstore/dailysnapshots/usr/ports@TOM
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/usr/ports/packages@TOM into zstore/dailysnapshots/usr/ports/packages@TOM
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/usr/ports/distfiles@TOM into zstore/dailysnapshots/usr/ports/distfiles@TOM
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/usr/home@TOM into zstore/dailysnapshots/usr/home@TOM
received 312B stream in 1 seconds (312B/sec)
receiving incremental stream of zroot/tmp@TOM into zstore/dailysnapshots/tmp@TOM
cannot receive incremental stream: destination zstore/dailysnapshots/tmp has been modified
since most recent snapshot
warning: cannot send 'zroot/var/log@TOM': Broken pipe

Why is it saying zstore/dailysnapshots/tmp has been modified when is hasn't? FYI: I have atime turned OFF on all pools/datasets.

xy16644 · Jan 28, 2014

One of the ZFS gurus on the forum must have some ideas

I don't understand why the incremental backups break when I mount all chile datasets?

gesperon · Jan 31, 2014

I ran into this issue a while back, if I recall correctly I added -F to zfs receive. Hope it helps

xy16644 · Jan 31, 2014

gesperon said:
I ran into this issue a while back, if I recall correctly I added -F to zfs receive. Hope it helps

Many thanks! I am still stumped by this problem. I shall give you recommendation a try. With this option were all the datasets/filesystems mounted after the incremental backup ran? Were you backing up root on ZFS?

reading zfs() I see it says -F does the following:

Code:

   -F
	       Force a rollback of the file system to the most recent snapshot
	       before performing the receive operation. If receiving an incre-
	       mental replication stream (for example, one generated  by  "zfs
	       send -R -[iI]"), destroy snapshots and file systems that do not
	       exist on the sending side.

Won't this destroy all my older snapshots? I want to be able to keep all my snapshots so that I can refer to them in case I want to do a restore. I plan on having a year or twos worth of backups.

gesperon · Jan 31, 2014

I was backing up a bunch of jails under rpool/JAILS/[jail0..N]. On the destination they were mounted.

I did a quick and dirty test. Seems to work even after modifying the destination ( mkdir /mnt/2/a):

Code:

root@fbsd9:~ # zfs list
NAME                   USED  AVAIL  REFER  MOUNTPOINT
rpool                 13.3G  18.0G   144K  none
rpool/1               87.5K  18.0G  87.5K  /mnt/1
rpool/2               87.5K  18.0G  87.5K  /mnt/2
...
root@fbsd9:~ # zfs snapshot rpool/1@a
root@fbsd9:~ # zfs send -R rpool/1@a | zfs receive -Fuv rpool/2
receiving full stream of rpool/1@a into rpool/2@a
received 41.7KB stream in 1 seconds (41.7KB/sec)

root@fbsd9:~ # mkdir /mnt/2/a

root@fbsd9:~ # zfs snapshot rpool/1@b
root@fbsd9:~ # zfs send -R -i a rpool/1@b | zfs receive -Fuv rpool/2
receiving incremental stream of rpool/1@b into rpool/2@b
received 312B stream in 1 seconds (312B/sec)
root@fbsd9:~ # ls -l /mnt/2
total 0

gesperon · Jan 31, 2014

xy16644 said:
gesperon said:

I ran into this issue a while back, if I recall correctly I added -F to zfs receive. Hope it helps

Click to expand...

Many thanks! I am still stumped by this problem. I shall give you recommendation a try. With this option were all the datasets/filesystems mounted after the incremental backup ran? Were you backing up root on ZFS?

Reading zfs() I see it says -F does the following:

Code:

-F Force a rollback of the file system to the most recent snapshot before performing the receive operation. If receiving an incre- mental replication stream (for example, one generated by "zfs send -R -[iI]"), destroy snapshots and file systems that do not exist on the sending side.

Won't this destroy all my older snapshots? I want to be able to keep all my snapshots so that I can refer to them in case I want to do a restore. I plan on having a year or twos worth of backups.

The man is a little confusing. -F will destroy whatever is on the destination that doesn't exist on the source. If the source have snapshot 1, 2 and 3, the same snapshots are going to be kept on the destination.

xy16644 · Feb 1, 2014

I ran the incremental backups last night using my script and your suggestion seems to have done the trick! I'll monitor it over the next few days to make sure but this is great news.

xy16644 · Feb 2, 2014

I found an issue with using -F with zfs receive. It does delete my older snapshots on the backup pool.

So, how do keep ALL the snapshots that are received to the backup pool AND run incremental backups so that I can't receive the error that the destination dataset has been modified?

xy16644 · Feb 2, 2014

I think I may have the incremental ZFS backups working now. Thought I'd share my config to see if I've done it properly.

First I create the datasets:

Code:

zfs create zstore/dailysnapshots
zfs create zstore/dailysnapshots/tmp
zfs create zstore/dailysnapshots/var
zfs create zstore/dailysnapshots/usr
zfs create zstore/backuptofile

Then I change the mount paths:

Code:

zfs set mountpoint=/dailysnapshots zstore/dailysnapshots
zfs set mountpoint=/dailysnapshots/tmp zstore/dailysnapshots/tmp
zfs set mountpoint=/dailysnapshots/var zstore/dailysnapshots/var
zfs set mountpoint=/dailysnapshots/usr zstore/dailysnapshots/usr
zfs set mountpoint=/backuptofile zstore/backuptofile

Then I take the first full backup and mount all filesystems:

Code:

zfs snapshot -r zroot@a
zfs send -R zroot@a | zfs receive -Fduv zstore/dailysnapshots
zfs mount -a

Take the first incremental backup:

Code:

zfs snapshot -r zroot@b
zfs send -R -I zroot@a zroot@b | zfs receive -Fduv zstore/dailysnapshots

Note that zfs recv used -F.

Then I do the next incremental:

Code:

zfs snapshot -r zroot@c
zfs send -R -I zroot@b zroot@c | zfs receive -du zstore/dailysnapshots

Here is the bit that is different: For the second (and all subsequent) incremental backups I REMOVE the -F option from the zfs recv and the backups seem to run fine with no errors.

I'm going to keep my eye on them this week and see how it goes. If you use -F with the first incremental backup it deletes all the snapshots on the backup pool which is not what I wanted.

Comments are welcome.

xy16644 · Feb 3, 2014

Argh, after running my backup script last night I got the dreaded error again:

Code:

cannot receive incremental stream: destination zstore/dailysnapshots/usr has been modified
since most recent snapshot
warning: cannot send 'zroot/usr@2014.02.03': Broken pipe

What am I doing wrong with the incremental backups? How do other people backup root on ZFS using snapshots to another pool on the same machine?

Is it even possible to only have one snapshot on the source pool but have many snapshots on the destination pool? I only ever want to have one or two snapshots on the source pool but I want to have up to a years worth of snapshots on the destination pool. Is this possible?

I seem to be running out of options here.

xy16644 · Feb 4, 2014

Can anyone help me here?

I just can't seem to ge this right. All I am trying to do is use incremental backups with my root ZFS.

Is it possible for me to only have one snapshots on the live root ZFS pool but have up to a years worth of snapshots on the backup pool? Whenever I run my zfs recv it deletes ALL my snapshots on the backup pool (except the two most recent which are on the live root ZFS pool).

How do other people handle their incremental ZFS backups?

I want to keep up to a years worth of snapshots on the backup pool but I only want up to two snapshots on the root ZFS pool.

Can this be done? I've read the zfs man() page and tried so many zfs recv command switches that my head is spinning now.

Can anyone help? :q

t1066 · Feb 4, 2014

How about using zfs diff to see what has changed in the receiving end?

xy16644 · Feb 5, 2014

t1066 said:
How about using zfs diff to see what has changed in the receiving end?

So I ran another incremental and it failed:

Code:

(/support)$ zfs send -R -i zroot@a zroot@b | zfs receive -duv zstore/dailysnapshots
receiving incremental stream of zroot@b into zstore/dailysnapshots@b
snap zstore/dailysnapshots@b already exists; ignoring
received 0B stream in 1 seconds (0B/sec)
receiving incremental stream of zroot/usr@b into zstore/dailysnapshots/usr@b
cannot receive incremental stream: destination zstore/dailysnapshots/usr has been modified
since most recent snapshot
warning: cannot send 'zroot/usr@b': Broken pipe

I then ran zfs diff zroot/usr@b zstore/dailysnapshots/usr@b and it said:

Code:

cannot open '': invalid dataset name
Not an earlier snapshot from the same fs: invalid name

t1066 · Feb 6, 2014

You should run
zfs diff zstore/dailysnapshots/usr@b zstore/dailysnapshots/usr
to see what had changed in zstore/dailysnapshots/usr

Savagedlight · Feb 6, 2014

There's one thing I find confusing here. Why do you create a secondary structure called 'dailysnapshots'? With ZFS send/recv, you can transfer live file systems up to the latest snapshot without interruption. You don't have to copy stuff from the original source to some secondary structure to transfer. Unless those files are not stored on ZFS to begin with; although they seem to be.