Very confused about zfs snapshots

I have a zpool that has multiple subvolumes, which I am attempting to snapshot and do zfs send/receive backups of. However, when I go to these subvolumes and look at their snapshots in the .zfs directories, they're empty!

Snapshot method:
zfs snapshot -r zpool@[date]
zfs send -Ri [previous snapshot] [current snapshot] | zfs receive -dvu backups

I've been following the zfs administration documentation, but it doesn't appear the data is showing up properly when trying to verify backups are working as they should.
 
I have a zpool that has multiple subvolumes, which I am attempting to snapshot and do zfs send/receive backups of. However, when I go to these subvolumes and look at their snapshots in the .zfs directories, they're empty!
Then you didn't make correct snapshots.

Keep in mind that you don't make a snapshot from a ZFS pool. A pool is a totally different entity than a ZFS filesystem.

Snapshot method:
zfs snapshot -r zpool@[date]
That is indeed not the right way to do this. Sort off, it depends where zpool was mounted. Basically you only made a snapshot from zpool but not the underlying filesystems. For that to work you'd need to use the -r parameter.

ZFS is a file system which can set up many different file systems which all share one specific storage pool. Although the pool itself is also often a file system itself you always need to keep the two separated.

On my system I have 2 ZFS pools:

Code:
breve:/home/peter $ zpool list
NAME    SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
zdata   464G  91.6G   372G         -    10%    19%  1.00x  ONLINE  -
zroot  33.8G  18.6G  15.1G         -    52%    55%  1.00x  ONLINE  -
zroot is used for the system itself whereas zdata... well, the name should make it obvious enough.

SO when looking at zroot a bit more:

Code:
breve:/home/peter $ zfs list -r zroot
NAME                    USED  AVAIL  REFER  MOUNTPOINT
zroot                  23.2G  9.55G  3.33G  /
zroot/doc               334M  9.55G   334M  /usr/doc
zroot/home             2.09G  9.55G  1.87G  /home
zroot/local            5.09G  9.55G  5.02G  /usr/local
zroot/ports            3.86G  9.55G   209M  /usr/ports
zroot/ports/distfiles  2.14G  9.55G  2.14G  /usr/ports/distfiles
zroot/ports/packages   1.52G  9.55G  1.52G  /usr/ports/packages
zroot/src              1.86G  9.55G  1.86G  /usr/src
zroot/swap             4.13G  12.4G  1.26G  -
zroot/tmp              10.3M  9.55G  10.3M  /tmp
zroot/var               799M  11.2G  39.2M  /var
zroot/var/db            731M  11.2G   180M  /var/db
zroot/var/db/mysql      482M  11.2G   417M  /var/db/mysql
zroot/var/db/pkg       49.3M  11.2G  45.9M  /var/db/pkg
zroot/var/log          3.55M  11.2G  2.32M  /var/log
zroot/var/mail         74.5K  11.2G  61.5K  /var/mail
zroot/var/run           312K  11.2G    81K  /var/run
zroot/var/tmp            62K  11.2G    29K  /var/tmp
I understand that this could be confusing because zroot is also mounted on / and therefor a file system of its own, but it is important to note that the filesystem zroot is different than the ZFS pool called zroot.

So if I were to make a snapshot from zroot and all underlying filesystems (see above) I'd be using: # zfs snapshot -r zroot@snap. This would create a snapshot for zroot and everything below it. Easily viewable with using zfs list -rt snapshot zroot.

(edit, just to clarify a bit better)

And if I wanted to make a snapshot from /var which, as seen above, is provided by the zroot/var filesystem then I'd use: # zfs snapshot zroot/var@snap.

My advice to you: zfs list -rt snapshot zpool and see what it shows.
 
Does zfs get -r used backups show the snapshots that you expect to have transferred?

Yes I do see the snapshots that I expect to transfer.

ShelLuser, I do use the -r parameter when I do zfs snapshot -r zpool@[date], and looking at my snapshots, it appears that each zfs filesystem inside my zpool is snapshotted accordingly, so I think at this point we can safely assume I am properly following your explanation of snapshots. So then my next major question is if I'm doing these snapshots, what would be the best way for me to do incremental backups and restoration? If I dig through the directories on my backup drive the same way I do on my host, the data isn't there which tells me my zfs filesystems aren't being recursively sent/received with my commands
 
it appears that each zfs filesystem inside my zpool is snapshotted accordingly, so I think at this point we can safely assume I am properly following your explanation of snapshots. So then my next major question is if I'm doing these snapshots, what would be the best way for me to do incremental backups and restoration?
"Best" is very objective obviously, it heavily depends on the situation and personal preference.

Personally I wouldn't try to incorporate sent backups onto another ZFS filesystem unless you're using all that as some kind of shadow system which should pick up the moment the main system fails. There's no direct need because the snapshots on your main system will serve as the primary source for backups, and it also makes it harder to retrieve those backups when you need to do a rollback. If needed you can rollback an entire snapshot using # zfs rollback (see zfs(8)) or you can retrieve individual files by checking the hidden .zfs directory in the root of the ZFS file system you need to recover.

For example: I have my backups (snapshots) scripted, each day a snapshot is made named after the current date and the oldest is destroyed. So if I needed to retrieve an individual file from a snapshot for /home (zroot/home) I'd be looking into /home/.zfs/snapshot/140817 (for example).

But bottom line: you can only restore backups once they've been imported into your current (active) file system.

So... incremental. When done daily you could simply make a snapshot and then send that to the remote host. Then every once in a while you could send a full replication stream, basically consisting of the current file system plus all snapshots made to that date.

Also: sending of snapshots is just that. A snapshot, ergo done on a per-file system (or dataset if you will) basis.
 
Okay I see what you're saying and duly noted. I don't have a hot/warm backup solution since this is simply for home use. So basically doing a snapshot then backing up the snapshot is the most ideal solution then. I'd like to do it that way because I want to disable the services manipulating data on that zpool, take a snapshot and then start the services again and backup that snapshot. I like the zfs send/receive method because it's fast and clean, but its also very cumbersome to do a send/receive of each zfs filesystem that was snapshotted. So what would you suggest then for efficiently backing up a zpool full of different zfs filesystems?
 
I think you're doing things correctly, at least assuming you want to back all filesystems on zpool, and I'm still not clear why they're not showing up. (I don't understand the earlier comments saying it's not the right way to do snapshots, but that's another discussion...)

I notice you're receiving with the unmounted option. Is it possible that your receive data sets are not mounted at all? Or do they (ack) have the same mountpoints as your source file systems which would mean you don't want them mounted at the same time?

What are the outputs of these commands?

zfs list -o space
zfs list -ro canmount,mounted,mountpoint,name
 
So basically doing a snapshot then backing up the snapshot is the most ideal solution then.
Absolutely. The frequency heavily depends on your needs. If the system is using RAID and this is a home environment I could well imagine that a full dump every week could be more than sufficient to keep you safe (also because, as mentioned earlier, your local snapshots also serve as a backup).

I'd like to do it that way because I want to disable the services manipulating data on that zpool, take a snapshot and then start the services again and backup that snapshot.
Don't bother. Just create a snapshot and you will be fine. A snapshot is made in an instant so there's no need to disable anything. Also keep in mind that Unix doesn't really know the Windows concept of files being in use (sort off). Of course, it's not a bad practice (it's always safe to do this) but I never stopped anything prior to making a snapshot and I've been in some very tight spots already (one of the reason I favor ZFS so much; it's quite robust).

I like the zfs send/receive method because it's fast and clean, but its also very cumbersome to do a send/receive of each zfs filesystem that was snapshotted. So what would you suggest then for efficiently backing up a zpool full of different zfs filesystems?
Well, using a shell script of course. That's the 'glue' which you normally use to string everything together.

Now, in all fairness there might be other options available in the Ports collection. I notice mention of ready-to-use ZFS backup options a few times but I never pay much attention to those because to me a customized setup is always much better than relying on a premade solution which author could give up on at any given time. Also because it's usually pretty easy to set up.

Having said that, maybe this can help... Its a script I wrote to sort out my snapshots, it checks the current date and builds a snapshot based on it and also removes outdated snapshots (based on the current date minus the preferred threshold). Not perfect mind you (I never got around to adding support for multiple ZFS pools for example), but it might give you an idea to set up something of your own.

Code:
#!/bin/sh

## Snapshot.ZFS v1.0
##
## A script which will manage ZFS snapshots on the
## filesystem(s) of your choosing.

### Configuration section.

# ZFS pool to use.
POOL="zroot";

# Filesystem(s) to use.
FS="/home /local /var"

# Retention; how many snapshots should be kept?
RETENTION=5

# Recursive; process a filesystem and all it's children?
RECURSE=yes;

## System settings

PATH=$PATH:/sbin:/usr/sbin

### Script definitions <-> ** Don't change anything below this line! **

CURDAT=$(date "+%d%m%y");
PRVDAT=$(date -v-${RETENTION}d "+%d%m%y");
PROG=$(basename $0);

if [ ${RECURSE} == "yes" ]; then
        OPTS="-r";
fi

### Script starts here ###

if [ "$1/" == "/" ]; then
        echo "Error: no command specified."             > /dev/stderr;
        echo                                            > /dev/stderr;
        echo "Usage:"                                   > /dev/stderr;
        echo "${PROG} y : Manage snapshots."            > /dev/stderr;
        echo                                            > /dev/stderr;
        exit 1;
fi

if [ "$1" == "y" ]; then

        # Make & clean snapshot(s)
        for a in $FS; do
                ZFS=$(zfs list -r ${POOL} | grep -e "${a}$" | cut -d ' ' -f1);
                if [ "$ZFS/" == "/" ]; then
                        echo "${PROG}: Can't process ${a}: not a ZFS filesystem." >/dev/stderr;
                else
                        $(zfs snapshot ${OPTS} ${ZFS}@${CURDAT} > /dev/null 2>&1) || echo "${PROG}: Error creating snapshot ${ZFS}@${CURDAT}" > /dev/stderr
                        $(zfs destroy ${OPTS} ${ZFS}@${PRVDAT} > /dev/null 2>&1) || echo "${PROG}: Error destroying snapshot ${ZFS}@${PRVDAT}" > /dev/stderr
                fi
        done;
else
        echo "Error: wrong parameter used."             > /dev/stderr;
        echo                                            > /dev/stderr;
        echo "Usage:"                                   > /dev/stderr;
        echo "${PROG} y : Manage snapshots."            > /dev/stderr;
        echo                                            > /dev/stderr;
        exit 1;
fi
I call this thing snapshot.zfs but it doesn't really matter how you name this because the script basically detects & uses its own name in its error reporting.

Hm, you could even simply add some of your own requirements in there. Lets say you want to sent yesterdays snapshot to the remote server (current one should be easy enough but..). You could add:

SENDDAT=$(date -v-1d "+%d%m%y");

(so: get a variable set to yesterdays date stamp)

And then add something like this:

$(zfs send -i ${SENDDAT} ${ZFS}@${CURDAT} | ssh backup@remote "dd of=/opt/backups/${CURDAT}.zfs" > /dev/null 2>&1) || echo "${PROG}: Error sending snapshot ${ZFS}@${PRVDAT}" > /dev/stderr

Obviously this would require a passwordless SSH session, I rely on keybased authentication for that. But this would create an incremental stream build on the differences between yesterdays snapshot and todays.
Maybe this can give you some ideas?
 
I notice you're receiving with the unmounted option. Is it possible that your receive data sets are not mounted at all? Or do they (ack) have the same mountpoints as your source file systems which would mean you don't want them mounted at the same time?

Your latter assumption is correct. And I am seeing I do never want to do that again because I couldn't export my device after backing up because what I'm running started to use my external drive? So I had to shut down my server, disconnect my external HDD and then boot and all was back to normal

Maybe this can give you some ideas?

I think I am getting some ideas on this. For one, I can't wait to get extra time to set up my extra NUC with FreeBSD so I can do sends over ssh as you show. sending the contents of .zfs is different than I thought too. I'm still trying to wrap my mind around this coming from using BTRFS. I love the reliability and actual functioning features of ZFS, but I just have a hard time separating my mind from BTRFS and the traditional filesystem :p maybe I just need a really in-depth book to get this stuff more
 
Your latter assumption is correct. And I am seeing I do never want to do that again because I couldn't export my device after backing up because what I'm running started to use my external drive? So I had to shut down my server, disconnect my external HDD and then boot and all was back to normal



I think I am getting some ideas on this. For one, I can't wait to get extra time to set up my extra NUC with FreeBSD so I can do sends over ssh as you show. sending the contents of .zfs is different than I thought too. I'm still trying to wrap my mind around this coming from using BTRFS. I love the reliability and actual functioning features of ZFS, but I just have a hard time separating my mind from BTRFS and the traditional filesystem :p maybe I just need a really in-depth book to get this stuff more

Ok. I think we're getting somewhere.

Some ideas:
(1) set your backup datasets readonly with zfs set readonly=on backups. You can still zfs receive into a read-only filesystem. (You can also set a pool read-only with zpool, but then you can't receive into it.) This is just good practice for a backup filesystem, where you don't want it to diverge from the snapshot state, and can also help avoid use of -F on the receive. (Not that you are, just something to avoid if at all possible unless you've really read and re-read what -F does.)

(2) Set your mountpoints on the backup datasets such that they can be mounted at non-conflicting locations. You might want to export it and import it with an altroot while you are first doing this process to make sure you don't accidentally mount over yourself. (Note altroot is not across reboots!) You can set these properties on a read-only dataset. That may seem odd, but note the contents of a dataset don't change just because the mountpoint did. Once they are set locally, they will stick (not overwritten by new received properties.)

(3) shutting down services (or putting them in a known good filesystem state) is important primarily for databases; sysutils/zfstools (a rolling snapshots tool) has flavors and directions for flushing/locking databases during the (quick!) snapshot process. If you have installed other services that are of importance to you (revision control servers, for example) can choose how you want to proceed based on your usage patterns and risk tolerance. The base OS services (ntpd, for example) tend to only have log files and ~disposable state files getting modified while running; I've never bothered to shut them down, and haven't seen any recommendations to do so with snapshots.

(4) Other people are certainly trying to do these same things, so leverage that! Google zfs snapshot replication script. Look for the ones you find in ports (a good sign that someone else liked them) and check them out.

(5) Don't use .zfs/snapshot mounts (not clear if that's what you meant) as your source for backing up, unless you have to operate with something non-zfs... send/recv is awesome, use it!

Once you've done this, you should be able to have your backups online and mounted (readonly) elsewhere such that you can wander through/read them (as well as historical snapshots in their .zfs directories) without concern that you may modify them. (Especially if you don't have atime off. Also set atime=off on your sources unless you need it; this avoids cluttering up your snapshots and wastes IO.)

One caveat: if you use beadm, beware that newly created and activated BEs will be received with their mountpoint=/ property. There are bug reports filed about changing the behavior (to have beadm set canmount=noauto) ... so be aware of that and handle it appropriately to avoid a nasty mounting over /. (It will be readonly in this setup and quickly become obvious, but it still likely requires a reboot to untangle.)
 
Thank you so much for all that information! So, here is my thought. I'm going to redo my backups and start with destroying the zpool on my backup drive and recreating it with zpool create backups /dev/ada3 and then if I'm understanding you right, additionally run zfs set readonly=on backups or will I have to create the zpool and then create a zfs filesystem on that? Then do an import of that zpool doing zpool import altroot=/mnt/backups backups. Once that's done, I'll create my snapshot via something like zfs snapshot -r zdata@`date` and then send the initial snapshot like this zfs send zdata@`date` | zfs receive backups, or should I receive it to /mnt/backups?

Finally, for incrementals I would do another snapshot and do something like zfs send -i zdata@snap1 zdata@snap2 | zfs receive backups

Then I would do my typical zpool import/export when handling the drive, using the altroot mount. Then if I wanted to restore my recent snapshot backup, say, backups@2017-08-17 what command would I do? Would I do zfs send -R backups@2017-08-17 | zfs receive zdata If I have the backup part done correctly, I'd like to know if I would be restoring it correctly too
 
That looks like it is coming together. I think it's good to go through it manually once to really understand what you are getting, and to make sure you're getting what you want, but something like zap (where others have spent countless hours ironing out the kinks) is worth considering.

As long as there are multiple datasets and you want to back them all up, you will likely want -R on all of your sends, and just send from the base of the tree. Also look at -I (rather than -i) if you want intermediate snapshots replicated.

You receive into a dataset; its moutpoint is irrelevant.

I like to put in at least a first level filesystem (backup/zdata); it gives you more flexibility in the future (if you have another pool to backup; or want to start a completely new set of backups but retain the first. You will likely need a -F on the initial seeding unless recieving with -e; be sure to do a test recv with -nvF to know what is happening. (Recving zdata@foo into backup with -e will cause backup/zdata to spring into existence and no -F will be required to write over an existing dataset... if I recall; when in doubt, check with -nv)

Using altroot is fine for initial setup, and potentially if this is an external drive that is only occasionally attached; but if it stays attached, you will want to modify the mountpoint properties of the backup such that it can coexist (sans altroot) with zdata. Receive with -u any time you are concerned and double-check mountpoints before mounting. Recall you can still modify the properties (metadata) of a read-only (at the posix/data layer) dataset.

You'll have to export it after creation before you can import altroot, or you can add the altroot to the create, if I recall... (it is never a persistent property, however.)

As far as "restoring" backups, that depends on the use case. A few things to remember:

1) unless you delete them, your snapshots are still available on zdata. The easiest "restore" is to copy files you need out of the appropriate .zfs snapdir. (Shameless plug: I like my own tool https://github.com/eborisch/zfs_versions for doing that.) Or rollback to the snapshot if you want to discard all changes since then. Be mindful of what state your backup is in (what snapshots exist on it) if you rollback; you may need to rollback the backup as well to a common starting point for the next incremental update.

2) if you've deleted the snapshots on zdata, but retain them on backup, you can pull individual files of out backup's snapdirs. (Or use a tool like rsync for entire directory trees, also potentially within a snapdir if you want an older version.)

3) Sending from your backup would typically only be called for if zdata has a catastrophic failure and/or the drive is replaced / pool recreated.

It (send from backup) can also be used if you want to revert everything in a filesystem to an older (but deleted on zdata) snapshot; you would need to rollback zdata the the newest existing-on-both-sides snapshot older than the desired, and then do an incremental update to the desired sent from backup. In my experience this is rare; you likely either want to rollback to a recent (and still existing on the source; no need to send, just rollback) snapshot, or only want to retrieve a few specific files / directories from an old state.

Check your sends first with -nv, and then check your recvs with -nv (turn it off on the send, obviously) the first few times to make sure you know what is happening before you have it actually "do" anything and you'll have this down in no time!

I think I covered everything. Good luck!
 
I will take a look at zap, I probably will want a well-refined tool once I learn this (or I'll just use it and toy with ZFS on my extra machine). I still ultimately want to thoroughly understand all of this anyways since it is part of my job to learn this deeply and present why ZFS will also be good for an organization's data archive server. You have given so much information and I am incredibly grateful, this is all information that I have searched for for at least a year now and haven't had it explained so well to me. I will be continually referencing what you have written as I set up everything :) many thanks!
 
Back
Top