Backup of ZFS mirror

X

xy16644

Currently I have an encrypted mirrored ZFS root on two SSD drives:

Code: 
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Jul 28 03:00:53 2014
config:

        NAME           STATE     READ WRITE CKSUM
        zroot          ONLINE       0     0     0
          mirror-0     ONLINE       0     0     0
            da0p1.eli  ONLINE       0     0     0
            da1p1.eli  ONLINE       0     0     0

errors: No known data errors

What I would like to do it buy a third SSD drive (of the same size) to use as a backup of my system. I was thinking of creating a 3 way mirror (ie: adding the new drive to the existing mirror), letting it sync/resilver and then removing the drive to keep offsite somewhere. The commands:

zpool attach zroot da1p1.eli da2p1.eli (create 3 way mirror)
zpool detach zroot da2p1.eli (detach new drive after syncing has completed)

Power down down server, remove da2 drive and then power up again. So I should have a 2 way mirror again at this stage with the original two disks.

Does this look right?

If in a couple months I want to "update" what is on the 3rd disk, do I just re-add it to the zroot pool and repeat the steps above?
 
You can also use zfs send to save a (compressed) snapshot of the system to the third disk:
Code: 
# full backup
zfs send -R zpool/filesystem@snapshot1 > /third_ssd_mount_point/full_backup.zfs
# compressed full backup
zfs send -R zpool/filesystem@snapshot1 | gzip -9 > /third_ssd_mount_point/full_backup.gz
This way, if your pool utilization is not too high, you can have multiple full backups. You can also do incremental backups, which are obviously smaller and faster than a full backup:
Code: 
# incremental backup
zfs send -R -i zpool/filesystem/@snapshot1 zpool/filesystem/@snapshot2 > /third_ssd_mount_point/incremental_backup.zfs
# compressed incremental backup
zfs send -R -i zpool/filesystem/@snapshot1 zpool/filesystem/@snapshot2 | gzip -9 > /third_ssd_mount_point/incremental_backup.gz
 
The downside of storing a ZFS snapshot in a file is that a single error in the file will cause the entire backup to fail during restore. I don't know if recent versions of ZFS can cope with errors during send/recv but I'd say it's a risk.

The detach command removes the device from the pool completely. This command isn't designed to make the detached device available as a standalone copy of the data. You could use zpool split to achieve something similar. (This removes one disk from the mirror and makes a new independent pool with it).

I would just make a new pool on the 3rd SSD, and use zfs send/recv to send a copy of the live filesystem to it. Once you have a copy on the backup disk, you can use the incremental send options to send just the changes since the last backup. You'd end up using a process something like the following for each backup:

Code: 
-- insert disk --
# zpool import backup
# zfs send -Ri last-backup-snapshot zroot@new-snapshot | zfs recv -F backup
-- optionally remove any snapshots from backup you no longer want --
# zpool export backup
-- remove disk --

Using split you would have something like this:

Code: 
-- insert disk --
# zpool attach zroot da0p1.eli da2p1.eli
-- wait for resilver to finish --
# zpool split zroot backup
-- remove disk --

Once the split has finished, zpool import should show a pool called backup available to be imported. I would expect the first method to be quicker as you're just sending the differences during each backup, rather than rebuilding a copy of the entire pool.
 
Thanks @usdmatt

Unfortunately you can't use zfs split with a root pool (ie: you can't split a mirrored ZFS root pool):

http://docs.oracle.com/cd/E19253-01/819 ... index.html

I can't create a new pool as this wouldn't work for me. Basically what I am trying to achieve is this: Let's say say I "lose" my server due to a hardware failure and that the disks have failed with it. What I want to be able to do is, take my third SSD drive, plug it into a server and boot off it. I also want to be able to update the third SSD drive occasionally. Can this be done?

What about removing the second drive from the live root mirror and keeping this somewhere safe and then adding the third drive to the live mirrored pool?

Yes, I do have backups but I want the third SSD on standby as it enables a very fast recovery from a catastrophic failure.
 
Last edited by a moderator:
Interesting that you can't split a root pool, although you can detach disks from it. I'd be interested to know if that actaully applies to OpenZFS or just the Oracle version. It isn't mentioned anywhere in the FreeBSD man page for zpool.

The problem as I see it is that there isn't really any way to remove a disk from a pool, whilst leaving both the original pool intact and the old disk containing a live, working copy of the same pool. You could just shutdown the server and pull the live disk, but your pool would always be in a degraded state (although it would still have the 2 disk redundancy). While it would work, I wouldn't consider having a permanently degraded pool a valid option for production use.

The closest I think of is as already mentioned, to have a second pool containing a copy of the system. The only issue is it might not be as simple as you want to recover from. (You may need to boot a live cd to import the backup pool, check mount points, etc). However, if modern FreeBSD releases are able to boot without the zpool.cache file and use the bootfs property instead of loader.conf, you may be able to get very close to being able to just boot the backup pool. That's something you'd probably have to try out (especially with GELI involved)
 
usdmatt said:
You could just shutdown the server and pull the live disk, but your pool would always be in a degraded state (although it would still have the 2 disk redundancy). While it would work, I wouldn't consider having a permanently degraded pool a valid option for production use.
Click to expand...

What I was thinking was:

  1. Power down server
  2. Connect third SSD drive and power server back on
  3. Create three way mirror of zroot and wait for resilvering to complete
  4. Power down server and remove one of the drives from the zroot three way mirror
  5. Power server back on
  6. Remove degraded (or "failed") drive from three way mirror so that it is back to a two may mirror

Would this work? I'm not sure why you said the pool would always be in a degraded state? I assume I could use the third disk I removed from step four on another server to boot up in the event of a catastrophe?
 
What I meant was, if you remove the third disk and reboot, you'll be left with a 3 disk mirror with one disk missing. At this point the pool will show up as degraded. If ZFS will now let you zpool detach the disk that is missing then yes, it should go back to a fully online two disk mirror.

You should also be able to boot that third disk on its own, because as far as it's concerned, it's still part of the active 3 way mirror. It'll just show up as a three disk pool with two disks missing.

You'd want to test it though as I'm not 100% sure what ZFS will do when you come to do the next backup. You'll need to boot up with all three disks attached, and at this point you'll have two disks in the 'live' pool and one in the 'backup' pool. Both these pools will have exactly the same name & pool GUID. I don't know how ZFS will act with multiple identical pools in one system. You can just hope that it'll import the first it finds meaning you'll probably be OK if the backup disk is never the first on the controller.
 
usdmatt said:
What I meant was, if you remove the third disk and reboot, you'll be left with a 3 disk mirror with one disk missing. At this point the pool will show up as degraded. If ZFS will now let you zpool detach the disk that is missing then yes, it should go back to a fully online two disk mirror.

You should also be able to boot that third disk on its own, because as far as it's concerned, it's still part of the active 3 way mirror. It'll just show up as a three disk pool with two disks missing.

You'd want to test it though as I'm not 100% sure what ZFS will do when you come to do the next backup. You'll need to boot up with all three disks attached, and at this point you'll have two disks in the 'live' pool and one in the 'backup' pool. Both these pools will have exactly the same name & pool GUID. I don't know how ZFS will act with multiple identical pools in one system. You can just hope that it'll import the first it finds meaning you'll probably be OK if the backup disk is never the first on the controller.
Click to expand...

I'm assuming you can detach the third degraded disk and convert a three way mirror back into a two way mirror. But yes, I agree, this approach needs to be tested 100%.

Reconnecting the third (backup) disk to the live pool to update it is where I get worried. Which disks get priority when the resilvering starts? I was thinking that the safer way is to just format the third disk and start from the beginning when it comes to updating it. ie: follow the steps I outlined in my previous post as though you are adding it as a new disk again.

Remember, there is no backup pool. There is only one ZFS pool during all this: zroot.
 
Remember, there is no backup pool. There is only one ZFS pool during all this: zroot.
Click to expand...

That was my point in the last reply; Once you have done the first backup, and come to do the second backup, you'll need to plug the third disk in and boot the machine. At this point you have the following:

  1. 2 disks that think they are the only members of pool 'zroot' with GUID '123', which the machine should boot off.
  2. A third disk that still thinks it's a member of the same pool with the same GUID.

That's why I said you'd effectively have 'two' pools when you boot up. The backup disk still thinks it's part of a pool called zroot, and has the exact same ID as the pool which the system is supposed to import on boot. Hopefully on boot ZFS will just leave the third disk alone as it already has a zroot pool imported.

I don't think you'll have a problem with priority which resilvering the pool. You manually tell it which disk you are attaching, and which disk you are attaching to, in the attach command. You'll just need to use the -F option as the third disk thinks it's part of an active pool.
 
xy16644 said:
I can't create a new pool as this wouldn't work for me.
Click to expand...

Yes, it would. And it's really the only workable option that doesn't require jumping through umpteen hoops.

Create a new pool. Use zfs send/recv to send backups to it.

When the original server and pool dies, you can boot off a LiveCD, import the backup pool under the name of the dead pool, and then reboot the server using the renamed backup pool. Simple as pie.

Use the system the way it's meant to be used, and you'll save yourself a lot of headaches. :)
 
Any chance this can get written up in the wiki? A full set of backup/restore steps sounds like a great thing to have easily available.
 
Very interesting.
I'm really interested how 3rd drive will behave in such situation:
1. Create ZFS pool of 2 disks (1,2)
2. Attach 3rd
3. Add it to pool, so we have (1,2,3) pool after resilver
4. Remove it, reboot we have pool (1,2)
5. Use pool (1,2) for some time - so this is other that 3 disk.
6. Connect 3 disk to have (1,2,3) - now we have 1,2 the same data and 3 out of sync
7. ??? Now resilver and what data will finally be on (1,2,3) ??? from (1,2) or from 3??

I will test it in VirtualBOX
 
You must log in or register to reply here.
Back
Top