Best way to use snapshots as incremental backups?

bsdolith · Jun 6, 2012

Hi folks,

I need some help and suggestions on how to manage my backups wisely. This is my setup.

I have made two servers called bokesha and kevinisha as fileserver and backup server.

Bokesha has 4 2TB SATA disks in RAIDZ. FreeBSD is installed on a 160GB SATA disk. I make rsync backups from various laptops to bokesha. I also share one folder from bokesha on my LAN to laptops in my family. In this NFS share I store the family album, music, films etc. I have made a zpool called storage and I have put /home and /bu in this pool.

Code:

[johan@bokesha ~]$ zpool status storage
  pool: storage
 state: ONLINE
 scan: scrub repaired 0 in 0h14m with 0 errors on Thu May 10 21:50:17 2012
config:

	NAME           STATE     READ WRITE CKSUM
	storage        ONLINE       0     0     0
	  raidz1-0     ONLINE       0     0     0
	    gpt/disk1  ONLINE       0     0     0
	    gpt/disk2  ONLINE       0     0     0
	    gpt/disk3  ONLINE       0     0     0
	    gpt/disk4  ONLINE       0     0     0

errors: No known data errors
[johan@bokesha ~]$ df -h
Filesystem                Size    Used   Avail Capacity  Mounted on
/dev/ada0p2               142G    1.7G    129G     1%    /
devfs                     1.0k    1.0k      0B   100%    /dev
storage/bu                4.8T    361G    4.4T     7%    /bu
storage/home              4.9T    440G    4.4T     9%    /home
storage                   4.4T    209k    4.4T     0%    /storage

Kevinisha has 6 1TB SATA disks in a straight, non redundant ZFS pool also called storage. I have put /bu in this pool. FreeBSD is installed on a 160GB SATA disk.

Code:

[johan@kevinisha ~]$ zpool status storage
  pool: storage
 state: ONLINE
 scan: scrub repaired 0 in 0h0m with 0 errors on Mon May 14 20:30:42 2012
config:

	NAME         STATE     READ WRITE CKSUM
	storage      ONLINE       0     0     0
	  gpt/disk1  ONLINE       0     0     0
	  gpt/disk2  ONLINE       0     0     0
	  gpt/disk3  ONLINE       0     0     0
	  gpt/disk4  ONLINE       0     0     0
	  gpt/disk5  ONLINE       0     0     0
	  gpt/disk6  ONLINE       0     0     0

errors: No known data errors
[johan@kevinisha ~]$ df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ada0p2    142G    1.7G    129G     1%    /
devfs          1.0k    1.0k      0B   100%    /dev
storage/bu     5.4T     31k    5.4T     0%    /bu
storage        5.4T     31k    5.4T     0%    /storage

My backup plan is to make rsync backups when needed of the family computers to the /home folder on bokesha. The NFS share also resides in /home so this folder gets updated continuously. To track the changes I want to take daily snapshots of the /home folder using this script or something similar.

Code:

#!/usr/local/bin/bash
##########
#
# Test snapshot of bu
#
##########

# Create archive filename
day=$(date +%d)
month=$(date +%b)
year=$(date +%Y)
snapshot="$day-$month-$year"

zfs snapshot storage/home@$snapshot

I will manually delete old snapshots to not fill up the pool. My plan is also to rsync the snapshots to /bu on kevinisha daily. So now it's time for my questions:

Do I need to make an initial snapshot from where the other ones are based upon to make it possible to delete old snapshots not needed anymore?
Is it enough to basically just rsync everything from /home on bokesha to /bu on kevinisha? Is it then possible to restore data on kevinisha if needed?
Is this an OK setup or is there a smarter way to do it?

Thanks for your help!

phoenix · Jun 6, 2012

Do not use rsync to send data between two ZFS pools. Use zfs send, which is designed for this specific purpose.

We were originally using rsync to send snapshots between two ZFS pools, because zfs send (actually, zfs recv) was *extremely* slow back in the ZFSv7 days (like KB/s slow). The process would take 6-8 hours normally.

We've since switched to using zfs send, which takes under 1 hour to do the same thing!

rsync has to read the metadata for every file in the snapshot, then compare that to the metadata in the filesystem on the other end, then compare file data, then transfer data. There's a lot of filesystem activity required. zfs send works below the filesystem, and just grabs a list of data blocks that are in the snapshot, and only transfers those.

Here's the very simple script I'm using to transfer snapshots between hosts:

Code:

#!/bin/sh

cat=$( which cat )
date=$( which date )
ssh="/usr/local/bin/ssh"
sort=$( which sort )
zfs=$( which zfs )

dateopts="+%b %d, %Y %H:%M"
sshhpnopts="-oNoneEnabled=yes -oNoneSwitch=yes -oHPNBufferSize=8192"

rsbpath="/root/rsb"
rsbsites="${rsbpath}/rsbackup.sites"
rsbkey="${rsbpath}/defaults/rsbackup.rsa"
rsbuser="rsbackup"

localpool="storage"
localfs="backups"
rempool="storage"
remfs="backups"

alphadrive="alphadrive.mysite"
betadrive="betadrive.mysite"
omegadrive="omegadrive.mysite"

yesterday=$( ${date} -v -1d "+%Y-%m-%d" )
lastsnap=""

for site in $( ${sort} ${rsbsites} ); do
        # Get the last snapshot date from omegadrive
        echo -n "$( ${date} "${dateopts}" ) ${site}:  Checking ${omegadrive} for latest snapshot ... "
        lastsnap=$( ${ssh} -i ${rsbkey} ${rsbuser}@${omegadrive} "${zfs} list -o name -t snapshot -r ${rempool}/${remfs}/${site} | tail -1" | awk -F'@' '{ print $2 }' )
        echo "${lastsnap}"

        # Don't bother if we're already up-to-date
        if [ "${lastsnap}" = "${yesterday}" ]; then
                echo "    Already up to date ... skipping."
                echo ""
                continue
        else
                # Send all snapshots between ${lastsnap} and ${yesterday}
                echo "$( ${date} "${dateopts}" ) ${site}:  Transferring all snapshots from ${lastsnap} to ${yesterday}."
                ${zfs} send -I ${localpool}/${localfs}/${site}@${lastsnap} ${localpool}/${localfs}/${site}@${yesterday} | ${ssh} ${sshhpnopts} -i ${rsbkey} ${rsbuser}@${omegadrive} "${zfs} recv -d -v ${rempool}"
        fi

        echo ""
done

You do the initial transfer manually, to get the first snapshot onto the other system:
# zfs send <pool>/<filesystem>@<snapname> | ssh somehost "zfs recv -d -v <poolname>"

After that, the script will keep things up-to-date, even if you skip a day.

bbzz · Jun 6, 2012

At least use raidz with your backup server, don't miss out on some redundancy.

I have very simple script which allows me to create snapshots, and hold them with certain label (eg. "last"), so that it can't be destroyed by error. Whenever I create a new snapshot it gets another label (eg. "new"). Then whenever I do incremental sending, scripts checks which snapshots have "last" and "new" label and then do incremental send between the two. This is done for all filesystems. Then the "new" gets "last" label, while "new" gets deleted.

While a more automated approach is sometimes better, this can be done if you don't have access all the time to backup server. Anyway I can post if you need it.

bsdolith · Jun 6, 2012

I will have to dig a little bit deeper into your script to fully understand it and to implement it. But if I understand your script correctly it will only send the snapshot to the pool on the remote server. I also want to take snapshots locally on "bokesha" and only use "kevinisha" as a mirror. Then it is possible for me to restore a snapshot on "bokesha" if needed even if "kevinisha" would go corrupt.

But just to clarify (so that I fully understand), if I would do daily snapshots on "bokesha" like this:

[CMD=]#zfs snapshot storage/home@day-month-year[/CMD]

I will make a list of snapshots referencing to changes made on storage/home. If I then start to delete the oldest snapshot after lets say 100 days will this work or are the younger snapshots depending on the oldest/older snapshots?

Then as an extra security I will use zfs send to send my snapshots to "kevinisha".

Thanks a lot for your help!

bsdolith · Jun 6, 2012

Yes bbzz, I think you are right. I will do this before I start to send backups to "kevinisha"

fgordon · Jun 7, 2012

As the subject of this topic seems perfect

I''d like to ask something about snapshots too ..... I would like to know the difference between "recursive" and "non-recursive"

Does -r mean I have to keep older snapshots? And without -r it's an "independent" snapshot? Or are there other differences?

I take daily snapshots without the -r option and though I deleted ~ 100 Gbyte and copied some stuff the used space of the snaphots seems extremly low.

Or is my understanding of "recursive" wrong?

I can see and use the many gigabytes of deleted files in the snapshots (made the base-snapshot directory visible and shared it) - so they are "there", but I get this information:

Code:

[root@mainserver ~]# zfs list -t snapshot
NAME              USED  AVAIL  REFER  MOUNTPOINT
tank@2012-06-03  18,3K      -  13,3T  -
tank@2012-06-04  18,3K      -  13,3T  -
tank@2012-06-05  1,29M      -  13,4T  -
tank@2012-06-06  18,3K      -  13,3T  -
tank@2012-06-07  18,3K      -  13,3T  -

But I do get:

Code:

[root@mainserver ~]# zfs list -ro space tank
NAME  AVAIL   USED  USEDSNAP  USEDDS  USEDREFRESERV  USEDCHILD
tank  2,84T  13,5T      127G   13,3T              0       127M

phoenix · Jun 8, 2012

[cmd=]zfs snapshot -r poolname@some-snap-name[/cmd] means:

create a snapshot of poolname, using the name some-snap-name
create a snapshot of every filesystem below poolname, using the name some-snap-name

It's a shortcut for:

Code:

# zfs snapshot poolname@some-snap-name
# zfs snapshot poolname/filesysystem1@some-snap-name
# zfs snapshot poolname/filesystem2@some-snap-name
# ...
# zfs snapshot poolname/filesystem30/sub-fs12@some-snap-name

fgordon · Jun 9, 2012

So the -r is recursive to zfs pools that are snapshotted and not to the method how the snapshots itself are made - I thought it might be incremental snapshots.

Thanks