ZFS Backup strategy

I'm in the middle of reorganizing my servers and network structure. So I thought it's a good opportunity to rethink my backup strategy. At the moment I just send incremental ZFS snapshots to an other box. (Which worked fine)

Now I have an additional storage server and I'm thinking about to use something different than FreeBSD (not that I'm unhappy with FreeBSD but diversity is a nice thing) Linux or maybe NetBSD. These leaves a some options how I can backup my files on this server.

1. With Linux I can just use ZFS on Linux
2. Something like rsnapshot
3. Something entirely different

ZFS on Linux sounds like a neat solution but a ZFS bug could potently corrupted all my data on the other hand is the rsnapshot much worse than ZFS.

Any recommendations? How do you do your backups?

Important for my is space efficiency and incremental since I need to backup over shitty home networks. Nice to have would be features like encryption.
 
I fail to see why would you ask this question here on FreeBSD forum if you a planning to use Linux or NetBSD. ZFS has built backup system (ZFS replication). IIRC there was a tool in FreeBSD ports which will allow you to do UFS mirroring HAMMER stream style. I don't use UFS so I would not know.

First off the question you should ask yourself is what do you really want: journaling, backup, or arhiving? The only journaling file systems which are fully functional are ZFS and HAMMER1. ZFS appears to be more robust and it is no brainier for people who have any significant amount of data (more than 20TB). ZFS runs on Linux and some people swear by it but I am not the one of them. Journaling on ZFS is done using periodic snapshots. HAMMER has a built in fine grained journaling (hammer history). HAMMER does support snapshots as well but they have a slightly different role than ZFS snapshots (they are not needed for mirror streaming unlike ZFS snapshots which are needed for ZFS replication). Backup with ZFS and HAMMER is trivial. ZFS uses replication for backup while HAMMER has mirror-stream command. Both are network aware. Both are well tested and in the case of ZFS used by some of the large data centres in U.S. As I mentioned earlier IIRC there is a port which can do something similar with UFS. Obviously RAID 1 gives you the illusion of backup. I like better hardware RAID. Archiving I am not sure. We don't archive much data in my Lab. I really like sysutils/duplicity if you ask me as archiving tool. Many people appear to use duplicity with Amazon glacier. I should mentioned that our own Colin Percival has cloud archiving tool http://www.tarsnap.com/ which uses Amazon to store encrypted data.

The only thing I think ZFS seems ill suited (somebody will correct me with this one if I am wrong) is being backed up on the tape. FreeNAS people have always in the past removed tape drivers from their version of the kernel. I have seen very few thread where people describe backing up ZFS to the tape.

In real life besides OpenBSD and FreeBSD I also run RHEL on over two dozen large computing nodes (60 cores + 1TB of RAM) and about 2 dozen or more of desktops in my Lab. I use sysutils/rsnapshot to backup user directories (XFS file system) onto the remote server which happens to run FreeBSD. The tool works perfectly. I use it also at home to backup my OpenBSD desktop onto an OpenBSD file server. We don't backup RHEL computing nodes. In the past I had one or two file servers running RHEL and we were relaying on the hardware RAID 6 as I backup strategy. I would strongly discourage such practice. We used high end LSI cards but I also like very much Areca cards. I honestly can't think of anything in Linux world which comes close to ZFS replication. For the record LSI RAID6+XFS is robust industry solution and I would not hesitate twice to put 200-300 TB on it. Maybe you should look misc/amanda or something similar.I think I was running net/rsync cronjob to backup RHEL file servers but that is not very robust.

NetBSD is on a dead bed if you ask me. It has WABPL which is really cool but other than that and npf there has been nothing original coming out of that camp for a long time. They have no fancy file system. Their softraid stack is 10 year old research code (I should know as it came out of CMU where I work). I am not sure if you can use any serious RAID cards on NetBSD except of course Areca which is open hardware. Again I would not know how you could monitor your hardware RAID. Again RAID is not backup so that would not solve anything anyway.
 
I don't particularly see any major reason to not use FreeBSD on the backup box. I generally trust ZFS on FreeBSD just as much as on any other OS. I've got no numbers but I expect the number of ZFS bugs that affect ZFS itself, and thus all the supported systems is probably higher than the number of ZFS bugs that are FreeBSD only. I can understand the idea of using another OS that has ZFS support though just for extra diversity.

I personally much prefer backing up to a secondary ZFS pool if my primary is ZFS. I get pretty much automated backup versioning by just keeping the snapshots (and can browse/restore from these as easy as changing directory), assured backup integrity due to the checksums (with the ability to verify that regularly via scrub), and it's a hell of a lot quicker than anything else. My mail server with around 1.5M files backs up in less than 20 minutes now it's on ZFS, versus 3-4 hours with rsync. Same story for everything else I've moved to ZFS. I use RAID-Z2 on backup servers so I can have a disk fail and not have to worry about any read/checksum errors messing up files before I can replace it.

I think you'd be unlucky to have pools on two systems die at the same time. I've been using ZFS since it was on v15 in FreeBSD 7 and haven't yet had a pool fail.

The only downside to ZFS send is that it's not resumable, although I wouldn't expect that to be a problem over a LAN. They are working on adding this feature though. One nice thing about rsync is that as it's file level, it will automatically just skip the files it's already done if it stops abnormally.
 
I'm thinking about to use something different than FreeBSD (not that I'm unhappy with FreeBSD but diversity is a nice thing) Linux or maybe NetBSD.
I'll say the same thing Oko said. I don't understand this. Diversity doesn't help here. If anything, it throws a "quirk" into the system. Something different than everything else which will only lead to a mistake made later on when someone forgets which system they're on.
 
The main reason I personally use ZFS is that it pays attention to bitrot and heals around it. Personally, I use ZFS and then I back that up to crashplan using a Linux VM. It's a tad convoluted, but I couldn't get crashplan to work in the headless config.

Anyways, ZFS is just a filesystem, it's a very reliable filesystem, but it was created by people so there are always the possibilities of obscure bugs. That being said, I'd be careful about what FS you use with Linux, I've had ext4 happily eat entire filesystems if it wasn't dismounted properly and more recently it seems to have no particular problem with files disappearing.

I don't agree with you about diversity. Having a small number of systems that are running different OSes is just going to increase the workload. Now, in an enterprise department where you've got dozens or hundreds or more computers, it's not as big of a deal, but I wouldn't personally want to spend that much time keeping current on other OSes just to have diversity.

As far as backups go, there are a few ways of going about it. I'm personally running crashplan in a Linux VM. It's not ideal, but I have yet to figure out how to get it set up headless and they don't provide a UI that works with FreeBSD. If I expected to have a lot of open files, I'd probably have it backup snapshots rather than the system at large. I will eventually be doing that so that I've got some protection against fat fingering things.
 
Crashplan backup from a ZFS dataset is working for you? I emailed Code42 regarding this and I got a fairly blunt response from Code42 stating that ZFS was unsupported and they had no interest in looking into my query any further.
Though I actually don't think my problem is ZFS related at all. I believe the crashplan service is avoiding backing up a dataset that I selected by its mount point - eg Pool/HomeDir is mounted into /Users/HomeDir, and I've selected /Users/HomeDir as the backup source. I have a vague recollection of it not liking this when I did it once before with an ext3 volume.
 
The only thing I think ZFS seems ill suited (somebody will correct me with this one if I am wrong) is being backed up on the tape. FreeNAS people have always in the past removed tape drivers from their version of the kernel. I have seen very few thread where people describe backing up ZFS to the tape.
Since someone else awakened the zombie...

I'm doing both replication to a second ZFS pool and backup to tape, with a fair amount of data (17TB, give or take). On a 10 Gigabit Ethernet, the pool replicates at between 500 and 750 Mbyte/second using sysutils/zrep and sysutils/bbcp. Sending the complete 17TB pool takes a little over 7 hours. Tape backup is done with archivers/gtar (for multivolume support) and misc/mtx (for tape library support).

I have a detailed writeup here, which includes the shell scripts I use, benchmarks, and other useful info. The whole article is probably of some interest to the same audience, as it describes building a 128TB FreeBSD-based fileserver in a 3RU chassis.
 
  • Thanks
Reactions: Oko
dump(8) doesn't work on ZFS filesystems. You get:
Code:
dump: /data: unknown file system

Not sure why people keep saying this. dump works perfectly well on ZFS on 9.3-RELEASE-p21. And yes, I have tested the restores and can retrieve files. It seems to have been broken sometime after 9.3 as I have not been able to make it work in 11.0.

Given that dump/restore is a very good solution to the backup problem, I am not sure why the functionality was removed. Trying to get a good backup solution out of ZFS has been highly frustrating.
 
Not sure why people keep saying this. dump works perfectly well on ZFS on 9.3-RELEASE-p21. And yes, I have tested the restores and can retrieve files. It seems to have been broken sometime after 9.3 as I have not been able to make it work in 11.0.
It didn't work in 8.x and doesn't work in 10.3 (I just checked). So it doesn't seem to be a viable strategy as the expected EoL for 9.x is at the end of this month. I did a little checking in the code (/usr/src/sbin/dump/main.c) and the S_ISDIR(2) is failing here:
Code:
        if (snapdump == 0) {
                if ((diskfd = open(disk, O_RDONLY)) < 0)
                        err(X_STARTUP, "Cannot open %s", disk);
                if (fstat(diskfd, &sb) != 0)
                        err(X_STARTUP, "%s: stat", disk);
                if (S_ISDIR(sb.st_mode))
                        errx(X_STARTUP, "%s: unknown file system", disk);
        }
When dumping a UFS filesystem, disk has been translated to the raw disk name via lookup in /etc/fstab, but on a ZFS filesystem it is just the same string as was passed to dump(8).
 
  • Thanks
Reactions: Oko
It looks like the issue is that 9.x created /dev entries for ZFS:
/dev/zvol/zfs/home 4058062 143446 3589972 4% /home
whereas 11.X does not:
zj6/usr/home 1865392204 520672 1864871532 0% /usr/home

I'm wondering if this functionality could be restored by once again creating them?

Also, I'm concerned that I'd lose this functionality if I upgrade from 9.X?
 
It looks like the issue is that 9.x created /dev entries for ZFS:
/dev/zvol/zfs/home 4058062 143446 3589972 4% /home
whereas 11.X does not:
zj6/usr/home 1865392204 520672 1864871532 0% /usr/home

Can anyone else confirm this behavior? Device nodes in /dev/zvol are created for ZFS volumes, which are a particular kind of ZFS dataset. Create a ZFS volume, format that volume to UFS, mount it on /home, and yeah, you can use dump(8) to back up /home, because dump never even touches the ZFS layer.

The paths given in the quote above are for two different kinds of datasets: the first, a ZFS volume dataset, which is given a device node in the actual filesystem hierarchy; the second, a ZFS filesystem dataset, with a path that merely represents a ZFS filesystem and doesn't actually exist in the filesystem hierarchy (there is no zj6/usr/home directory or device node). I'm just having a hard time believing that there was one very brief moment in FreeBSD history that the two dataset types were handled in the same way. It seems much more likely to me that you've been running dump on a UFS filesystem written on a ZFS volume.
 
The filesystem in your first example is a UFS filesystem on top of a ZVOL (ZFS volume). ZFS volumes are block devices that show up under /dev/zvol.

As far as I was aware, the FreeBSD installer has never used volumes when creating a ZFS system. Are you sure that home filesystem on the 9.x system wasn't something created manually?

When asked to create a ZFS system, the installer will put all the filesystems on normal ZFS datasets - using the ZFS filesystem (ZPL). dump()/restore() will not work with these.
 
You could of course create a ZVOL for your data, format it UFS and mount it somewhere (e.g /home). Seems a bit ugly though and accessing files from snapshots would be more awkward.

send/recv really is the best option for backing up ZFS pools. Sending to a file is not a great idea though as a single bit wrong in the backup would cause the entire backup file to be useless. I did mention a possible way around this in another post, although it is a bit of a hack -

1) Create a sparse file big enough to store the backup using truncate truncate -s 1T mybackup.pool
2) Mount that as a memory disk mdconfig -a -t vnode -f /path/to/mybackup.pool
3) Create a pool using that memory disk zpool create mybackup mdX
4) Send snapshots of your live datasets to it
5) Export the pool and remove the memory device(s)

This way you end up with a backup file that contains an actual ZFS pool. You can connect this backup file to a memory disk and import the pool in the future, and if there's any corruption you can actually run a scrub and let it identify the data that is corrupt. (Obviously bad corruption could render the pool completely faulted, but in some cases it could be just a single file that is damaged, whereas a file containing purely a send stream would be trashed).
 
Yes, the above is correct, UFS on top of ZFS appears to be what I have and it works fine. Thank you for clearing up my confusion.
 
Trying to focus on the original questions:
1 multi-os implementation
2 copies of incremental vs something else
My 5cents
@1: the core of zfs, i.e. the filesystem part, has a single source, namely openzfs.org. All distributions want to use this same source. So there is little gain, on the data-integrity level, to use different implementations of the same core.
@2: copies are not backups - this is both good and bad. A backup warrants that you can access the data in the same way in the same environment. Your concern, however, sounds like: "I might not be able to access X after I upgrade Y to Z". The only solution is to snapshot/backup Y too. In simple terms, at every snapshot of "my_valuable_data" you also need to make a snapshot of the systems you use to access it (your worstation).
Also, backups typically access a storage system at a lower level then files/directories. This means that backups (ie ufsdump) are actually more prone to silent datacorruption then a zfs send/receive cycle.
A snapshot/copy of the data, on the other hand, has the advantage that, if done right, might be usable in the next release. By definition a copy of a file is an accessible file and not a solidified representation on an awkward device supported only by 3rd party drivers.

From your scripts I see you use 10 levels with a fixed interval, so the lifespan of your copy/backup is predetermined. As long as you also keep copies of the accessing sytems with the same timespan you should be safe and have the benefits like old-school backups with the storage efficiency of snapshots.
 
Back
Top