ZFS How to backup / without shooting in your foot.

D

Dragony

Hi!

Today I tried backing up zroot, but something horrible happened.

I did "zfs send -Rv zroot@today | zfs receive offline_backup/myserver". For testing purposes I pressed CTRL-C in mid-process just to find out my box became totally unusable. Entering "zfs" didnt work anymore. It said something about library problem. I had to reboot the box. After doing zfs list I found out that the backup copied the mountpoints as well, and of course they were all like / /usr /var etc. So I basically killed my root-fs by overlaying.

What can I do to avoid that mountpoint problems? This happened more than once to me that an automatically mounted zfs in a newly attached zpool has killed some important mount points.

PS: I am completely new to ZFS.
 
I don't use -R as I tend to find it's very easy to get in a mess, but a good start would probably be to use -u on the recv end so it doesn't actually mount any datasets that have been received (assuming that works with a full replication stream).
 
Backing up zroot without -R seems to be a very error-prone task. If you dont use an automated script, it seems next to impossible?
 
I highly recommend getting the 2 ZFS ones as a foundation for ZFS .. It will really make your life better. .. chapter 4 in the advanced zfs book goes over replication in detail . have a look here https://mwl.io/nonfiction/os. I think you can get the PDF's on tiltedwindmill for 20-30$ .. guarantee it will be the best few dollars you spend.

it will also help you in your quest to implement ZFS without shooting your body parts :)
 
Sounds good. I will buy the Storage Bundle. But I need to backup zroot now. What can I do right now so I have a backup before I have studied everything about ZFS? Since I have just killed my server FS once again by using -R, I think I won't use that option anymore.... :( Any hints?

I think the most important problem here is that replicated datasets can have mountpoint="/", which must be avoided at all cost. Everything must be under /offline_backup. Even after restart. I have tried altroot, but this property doesn't survive a reboot. The pool containing /offline_backup is an USB-HDD.
 
Lets see how I do it... I am using some zfs send -R -PLcepv ... | zfs recv -Fu ...

Then, on the destination machine (this is a remote copying and runs every few minutes) I have created this as /etc/rc.d/zfsnomount-local
Code: 
#!/bin/sh
#

# PROVIDE: zfsnomount
# REQUIRE: zfsbe
# BEFORE: zfs

. /etc/rc.subr

name="zfsnomount"
desc="block ZFS backup pools from mounting"
rcvar="zfs_enable"
start_cmd="zfsnomount_start"
stop_cmd="zfsnomount_start"
required_modules="zfs"

zfsnomount_start()
{
        echo "Inhibit backup ZFS pool from mounting"
        local fs
        zfs list -o name -t filesystem | grep "^backup/" | while read fs; do
                zfs set canmount=off "$fs"
        done
}

load_rc_config $name
run_rc_command "$1"

No problems so far. The only issue is that you need to have the correct name prefix for the grep (should actually be an rc.conf variable...
 
OK I have purchased the book package about ZFS and the only chapter covering zfs send|recv is in the Advanced ZFS book Chapter 4. Unfortunately the whole chapter does not cover the problem how to backup a whole zroot device securely. Or I just did not find it.

To clarify: I am having a backup USB drive where I want to backup several machine's zroot system, so that in the end I am having a folder structure like:

<pre>
Name Mountpoint
/offline_backup/machine1_zroot/ROOT/default /
/offline_backup/machine1_zroot/etc /etc
/offline_backup/machine1_zroot/home /home
/offline_backup/machine1_zroot/usr /usr
...
/offline_backup/machine2_zroot/ROOT/default /
/offline_backup/machine2_zroot/etc /etc
/offline_backup/machine2_zroot/home /home
/offline_backup/machine2_zroot/usr /usr
...
</pre>
(how to do <pre>?)

The problem here is the Mountpoint, killing the host OS as soon as you plug in the USB-drive.
PMc method seems to solve this issue, but it overwrites the mountpoint-property of the datasets, destroying the exact replication of the datasets. This means that when I send the datasets back to a (failed) machine for recovering, the mount-properties are not the same. Furthermore the mount-properties are not always yes or no. They are mixed up. On the live system, a few are "yes", while others are "no". What I mean: Touching properties of a backup is against the fundamental philosophy of a backup. Isn't there any smoother solution?
 
Why even bother to replicate the whole thing btw? The only advantage you might get is having quick access to files, but the snapshot would cover for that anyway since you can access those through the hidden .zfs directory.

What I always do is store ZFS filesystems as files: # zfs send zroot/home | ssh shell@backup "dd of=/opt/backups/main_root.zfs, command typed from mind; should be legit enough though.

There are several advantages here.. In case you need to restore stuff you don't have to bother with fabricating a whole new zfs send command again, you simply dump the file onto stdout and pipe that back to your main server where you can use zfs recv .... This should also help reduce some (probably minor) delays because your backup system doesn't have to construct a whole new ZFS datastream, all it has to do is dump the binary file.

Another advantage is that this also makes the backup images a lot more portable; thus making it much easier to copy them to other places if required (I usually keep an image on both my backup server as well on a cloud storage just in case). Which leads up to another advantage: even a small server using UFS with enough free diskspace can do as backup location. I'm using an OpenBSD environment for this myself (which doesn't support ZFS, only using UFS here).

And of course; no more messing with mountpoints and all that stuff.
 
I have thought about > file as well. The biggest disadvantage come to play later when you need to backup all again. Will you do an incremental backup and save the stream to ">increment1" and then ">increment2"? ? Of course you will! But you will never know if you can really import that file. In case you did something wrong with the incremental settings, zfs recv always complains instantly, but "> file" will never complain. You could have a non-working backup without even knowing about it.
 
that is the test of a true back up .. can you restore it??!.. its safe to say if any file system on earth can .. it's going to be zfs .. the better question is .. what schema works best for you.

nothing is stopping you from replicating out as many copies / snapshots as you have disk space for ..
then all you need to do is configure something like sysutils/zfsnap with a schema that works for you.

by using the proper tools like zfs send/receive on other zpools there is very (and I mean VERY) little chance you will ever lose a single byte of data.. If you want to dump to a file as Shel mentioned your even safer.. In my case I just expire snapshots once per week, in turn brining my base up once per week well allowing me to rollback upto 7 days.
 
Dragony said:
I have thought about > file as well. The biggest disadvantage come to play later when you need to backup all again. Will you do an incremental backup and save the stream to ">increment1" and then ">increment2"? ? Of course you will! But you will never know if you can really import that file. In case you did something wrong with the incremental settings, zfs recv always complains instantly, but "> file" will never complain. You could have a non-working backup without even knowing about it.
Click to expand...
No, I didn't do incremental backup, just backup full dataset if I need a backup.
And..for your case, I think ZFS Administration Guide not give a example of send pool and recieve in dataset.
Here is example in guide:
# zfs snapshot -r users@today
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
users 187K 33.2G 22K /users
users@today 0 - 22K -
users/user1 18K 33.2G 18K /users/user1
users/user1@today 0 - 18K -
users/user2 18K 33.2G 18K /users/user2
users/user2@today 0 - 18K -
users/user3 18K 33.2G 18K /users/user3
users/user3@today 0 - 18K -
# zfs send -R users@today > /snaps/users-R
# zfs destroy -r users
# zfs receive -F -d users < /snaps/users-R
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
users 196K 33.2G 22K /users
users@today 0 - 22K -
users/user1 18K 33.2G 18K /users/user1
users/user1@today 0 - 18K -
users/user2 18K 33.2G 18K /users/user2
users/user2@today 0 - 18K -
users/user3 18K 33.2G 18K /users/user3
users/user3@today 0 - 18K -


I think you should try send pool and receive it in pool first, not receive it in a dataset.

Update:
After try it myself (put my old boot disk in usb-sata case and plugin), altroot is necessary when import pool... :(
 
Dragony said:
PMc method seems to solve this issue, but it overwrites the mountpoint-property of the datasets, destroying the exact replication of the datasets.
Click to expand...

No it does not.
It set's the canmount property, which is a non-inheritable local entity for just that purpose:
Code: 
$ zfs get mountpoint,canmount backup/disp/usr/local
NAME                   PROPERTY    VALUE       SOURCE
backup/disp/usr/local  mountpoint  /usr/local  inherited from backup/disp/usr
backup/disp/usr/local  canmount    off         local

Isn't there any smoother solution?
Click to expand...

This is the smoother solution. ;)
 
The canmount property does not have to be set. It's set to no at destination in one instance since it tries to mount on same path as the host OS.
 
Hi ShelLuser,

as much as I like your approach, I understand that zfs send . . . | zfs receive . . . checks that a snapshot has been received correctly. How do you ensure that in your scenario?

Kindest regards,

M
 
You must log in or register to reply here.
Back
Top