ZFS Seeking assistance in dealing with ZFS backups

[wayne47]

Looking to replace a group of FreeBSD machines running UFS with newer version of FreeBSD and ZFS. The big thing I'm having an issue with is replacing the existing backup system with ZFS:

• A level 0 dump is run every 15 days.
• A level 1 dump is run nightly.
• Only 2 level 0 dumps and 14 level 1 dumps are retained. So nothing is ever available for more that 30 days (policy).
• All of this gets ssh'ed to a big file server with only non-root access that does not run ZFS (so far so good).
• The problem: Restores need to handle both full restore (currently, this is use the last level 0 and the last level 1) as well as individual files (I can not figure out how to do this with ZFS). There is no way to use ZFS tools on the file server nor is root an option.

So far, the only way I can figure out to do this is to put a UFS file system on top of the ZFS system and run dump/restore. But that's truly ugly. I have a hard time believing that anyone created a file system with no real remote backup/restore capability so figured I would ask for advice here.

 

[ShelLuser]

If all you can come up with is use UFS on top of ZFS (ugh!) then pardon me for saying so but then I think you didn't read up very much on this subject. I mean, even the handbook covers ZFS usage.

First things first: ZFS is different than UFS in many ways, so obviously you should be careful with trying to make ZFS fit into your existing model. Ensure that it actually fits and if not re-adjust the model.

Snapshots are your friend here. If this system is utilizing RAID then you can easily rely on snapshots for your daily backups. Takes roughly a second or so to create one and it allows for both a full rollback as well as per-file access. You could then continue making those and retain them for 15 days or such.

The main issue are remote backups. My suggestion would be to treat this strategy separately from the snapshots. Just pick intervals in which you deem an offline backup to be required and then dump either a subset of snapshots or the full sequence to the remote host. See zfs(8), in specific the snapshot, send and receive options.

I could imagine sending a replication every 15 days and relying on incremental streams in between.

 

[wayne47]

I read the manual and I get snapshot, send and receive. I can't figure out how to restore just a single file using ssh. As I understand the man page, every 15 days I'd need to destroy all snapshots locally, then create an new snapshot and zfs send it to the file server. On the intervening days, create a incremental snapshot from the latest full. That gets it backed up but I can not figure out how to replace restore's functionality.

 

[ShelLuser]

When you make a snapshot of a file system it'll be accessible through the hidden .zfs directory (located in the root of the file system) which gives you direct access to your snapshots. This is the easiest way to access / restore single files:

breve:/home/.zfs/snapshot $ ls -l
total 3
drwxr-xr-x  5 root  wheel  5 Jun 18  2015 110817/
drwxr-xr-x  5 root  wheel  5 Jun 18  2015 120817/
drwxr-xr-x  5 root  wheel  5 Jun 18  2015 130817/
drwxr-xr-x  5 root  wheel  5 Jun 18  2015 140817/
drwxr-xr-x  5 root  wheel  5 Jun 18  2015 150817/

Also keep in mind that if you're messing with remote backups then you'll need to import the whole thing before you can actually restore (individual) things from it. So zfs rollback needs a snapshot but one which is already present on your system. As such you'd be performing 2 steps: zfs receive and then zfs rollback (or getting individual files from it). This is why it makes more sense to rely on snapshots first and remote backups second.

As I understand the man page, every 15 days I'd need to destroy all snapshots locally, then create an new snapshot and zfs send it to the file server.

No, I'd advice against that because there's no need. It would also create a dangerous situation because you'd basically be removing your backups.

Set up a threshold and only destroy the oldest snapshot while leaving the rest. So: always keep several snapshots in place on your system because that will be your first backup solution to turn to. When you need to create a full offline backup then simply use the -R parameter to dump the full set (the file system and all available snapshots) to the remote storage.

For incremental backups you can simply send individual snapshots.

Just keep in mind that a major part of your backups will be stored locally.

 

[Datapanic]

You may want to look at sysutils/zap for automating your snapshop creation and retention. No need to do that part manually.

 

[wayne47]

Thank you, the above two posts were quite helpful. One really difficult thing about ZFS is that it is such a different paradigm from UFS that it requires a different approach to addressing issues. While I understood about snapshots, I was not thinking of using them for local restores since I am so used to getting them remotely. So, ignoring the whole incremental issue for the moment (that's just script logic) as well as snapshot management (looks like zap will handle that), backups would consist of:

• Create snapshot (likely using zap)
• zfs send <snapshot> | gzip -9 | ssh backup_server dd of=zipped_snapshot

and restores would be (as suggested above)

• ssh backup_server dd if=zipped_snapshot | gunzip | zfs receive <snapshot>
• zfs rollback <snapshot>

which would seem to handle most problems other than bare-metal restores. What is the recommended information to backup (off-machine) to simplify recreating a server starting with blank drives and a FreeBSD DVD? I'm thinking that having these would be sufficient:

• zspool list
• zpool history
• zfs list
• ifconfig -a

but may have missed something. It's MUCH easier to have a little extra information available when in the midst of a recovery. I'm thinking that restores in this case would be

• Boot FreeBSD DVD
• use the saved ifconfig to set IP address
• use the saved zpool history to create the pool and datasets
• receive & rollback each dataset from the backup server as above

 

[sko]

zfs send <snapshot> | gzip -9 | ssh backup_server dd of=zipped_snapshot

Compressing the stream with gzip is unnecessary, especially if you already use compression on the pool/datasets - just use the -c flag with zfs send and ZFS will send a compressed stream. For very slow links and/or powerful senders, generating a deduplicated stream with the -D option might also be feasible.

All of this gets ssh'ed to a big file server with only non-root access that does not run ZFS (so far so good).

You can transfer your ZFS replication streams into a file on that server. This file WON'T be protected by any ZFS mechanism! If this system suffers from bitrot and damages one of the stream files, the replication stream will be inconsistent and you won't be able to use it to recover your pool or datasets!
I'd suggest using a local staging system (dedicated backup server or a local file server) to transfer these ("unsafe") external backups and for keeping local backups within a live ZFS pool. This way you could also restore files, datasets or the whole pool locally without much delay or bandwidth restrictions in case your production system dies.

 

[wayne47]

Thanks for the tip on compression - will test the difference on the test setup.

This solution is for lots of small servers backing up to one huge datastore (think of it as "fast_tape"). Duplication of each small server is really not viable. Backups have a lifetime of 4 weeks, after that period all backups are removed so bitrot over time is a minor concern.

Current test script (run once daily) looks something like:

• Take a snapshot
• If the latest full backup is 14 days old send a full copy, else send an incremental from latest full copy.
• If 3 full backups exist, delete the oldest one.
• Remove all incremental backups older than 14 days
• Save a copy of "[FONT=Courier New]zpool list[/FONT]", "[FONT=Courier New]zpool history[/FONT]", "[FONT=Courier New]zfs list[/FONT]" and "[FONT=Courier New]ifconfig -a[/FONT]" as part of the backup to make bare-metal restores from FreeBSD DVD easier.

 

[sko]

Do you really run backups individually as scripts on every single one of these small servers/VMs?
Have you considered using a backup tool like amanda to centralize backup configuration and reporting on this "huge datastore"?
It can easily model such requirements like yours and works well with ZFS and snapshots. It can essentially combine aggregating all backups from your local systems and then uploading everything to your external backup server instead of everything being a single job on every machine (which is rather fragile...).

I'm using it since ~2013 in our infrastructure and apart from adding/removing hosts to the backup cycles and adding ZFS to its capabilities, I haven't touched it since.

 

[wayne47]

We ran Amanda for many years. Eventually outgrew it. This is actually a single program built using what we learned from Amanda that runs on the backup server. It reaches over to each of the small servers in the farm, in parallel, doing the "right thing". All of the logic to manage dump levels, disk space and reporting has been rock solid for over a decade and properly solves the correct problem.
The goal here is to simply rip out dump and replace it.
I was at a BSD user group last night - the one additional suggestion was to save a copy of the ZFS properties, which I am now doing.

 

[phoenix]

Are you running ZFS on the backups server? The servers being backed up? Both/neither?

My suggestion would be to run ZFS on the backups server. Create a separate ZFS filesystem for each server. And just rsync from the remote server into the ZFS filesystem. Then snapshot the filesystem. Before the backups run, count the number of existing snapshots; if it's greater than X, delete the oldest one. Voila! Done. No need to mess with "full" vs "incremental". Every rsync run is an "incremental" backup, but every ZFS snapshot provides a "full" backup to restore either individual files or complete systems from.

It really sounds overly complicated what you are trying to do.

 

[wayne47]

From the original post:

There is no way to use ZFS tools on the file server nor is root an option.

This is one of the things that really bothers me about ZFS, the attitude that "You should run it everywhere." The entire point of this post is to determine how to use ZFS in a mixed environment.

 

[wayne47]

Compressing the stream with gzip is unnecessary, especially if you already use compression on the pool/datasets - just use the -c flag with zfs send and ZFS will send a compressed stream.

ZFS on 11.0 does not support -c:

% zfs send -c <snapshot>
invalid option 'c'

Leaving off gzip costs ~10% more space to be used

 

[sko]

ZFS on 11.0 does not support -c:

% zfs send -c <snapshot>
invalid option 'c'

Sorry, my mistake - I looked at the zfs manpage on a 12.0-CURRENT (TrueOS) system.

For backups over the local network gzip might still impose a bottleneck, especially because it is single-threaded. I'd benchmark it against lz4 and pbzip (parallel bzip).

This is one of the things that really bothers me about ZFS, the attitude that "You should run it everywhere." The entire point of this post is to determine how to use ZFS in a mixed environment.

This is the most common scenario and absolutely workable - however, it is easier if most/all systems higher up the stack (backup infrastructure wise) are running ZFS.
As already said: you can put an zfs send stream into a file and store it on the non-ZFS backup server, but handling these backups is quite painful and needs multiple times the storage space and bandwidth than a ZFS pool with snapshots and incremental send/receive.
If you pay by used space and net transfer to/from the server (e.g. the payment model for amazon glacier) even on the short run this could cost (much) more than a ZFS based solution. Duration of the backup/upload might also pose a problem at one point.
Storing incremental streams in files would need you to incrementally restore the pool from your backups - anyways this might be a better solution as it reduces storage and bandwidth needs.
Regardless of the details, it is always beneficial to use ZFS at the top of your local backup infrastructure and as a key element of your backup strategy.
With ZFS I managed to get the nightly backup window reduced from ~4-5h to ~1h (except for systems that still rely on rsync based backups, which run on a different schedule), mainly because the total amount of data to be transferred has been reduced to a tiny fraction of what once were all the rsync'ed or dump'ed filesystems and full-snapshots of qcow VM images (which took an eternity and a half to snapshot beforehand).
When the BBU on one HW-RAID controller inside the old backup server decided to die, the controller disabled r/w caches, performance of the LVM pool (which isn't particulary great anyways) went completely downhill and backups were far from finished at 8 in the morning when everyone tried to get work done...


To be clear: I don't say you shouldn't or can't do this or stop using ZFS at all just because at one point you have to use or access another FS. In fact I've also started to put whole 'zfs send' streams into my tarsnap backups. It spoils the deduplication feature of tarsnap, but it greatly simplifies backup and recovery on my side compared to file-based backups. And because pricing for tarsnap is _so_ low, the few additional MB due to inefficient deduplication don't bother me at all...

 

[Terry_Kennedy]

I have a hard time believing that anyone created a file system with no real remote backup/restore capability so figured I would ask for advice here.

There are a fair number of ostriches here.

Snapshots can help, but are not a solution on their own. For one thing, a snapshot file that has not been received onto a pool is susceptible to catastrophic loss from something potentially as small as a single bit flip.

I go into this in a fair amount of detail here, which is an expansion of an earlier article here. You may find some useful ideas / scripts there.

 

[phoenix]

This is one of the things that really bothers me about ZFS, the attitude that "You should run it everywhere." The entire point of this post is to determine how to use ZFS in a mixed environment.

No, what I said was to use ZFS on the backups server(s).

We use it in a mixed environment. Remote servers run Linux, FreeBSD, and Windows. Some are bare metal, some are VMs. Some run XFS, others ext4, some still use ReiserFS. Only the backup servers run ZFS.

The Windows machines save to Samba shares. Windows VMs use file-based disks. Everything is transferred to the backups servers using rsync into separate ZFS datasets per server. A very few use push services to get the data onto the backups servers (OpenVMS system, for example). Snapshots are created for each dataset after the rsync finishes.

The following morning, each ZFS dataset is transferred to a remote ZFS server via send/recv over bbcp links.

Doesn't get more mixed than that. Like I said, you are making things more complicated than they need to be.

 

[phoenix]

We restore individual files for teachers and students in a weekly basis, sometimes multiple versions of the same file going back over multiple days.

We also use our backup server to transfer accounts and files between buildings (even cities) instead of doing it from site-to-site (minimise impact on school Internet connections).

And we do bare-metal restores of VMs throughout the year, and for school servers. We actually have a staging server where we restore from backup every morning to a degraded RAID10 (3 of 6 drives) for the largest schools, so that recovering from a dead server only takes an hour (time to build new server and drive it to the location) instead of multiple days transfer across the network.

All of this is simple and easy (and very inexpensive!) because we use ZFS on the backups storage servers, and rsync to do the actual backups/restores.

We're very much a KISS shop. And it doesn't get any simpler than dealing with files in snapshot directories from the shell prompt