Backing Up Jails Offsite - Symbolic Links

Flash619

Flash619

Greetings,

When backing up a jail to an off-site location that does not support ZFS transfers, what is the best course of action for handling symbolic links? For instance, I use rclone to backup pertinent files to Backblaze B2 Cloud Storage. B2 does not support symbolic links, and rclone will not copy symbolic links for this reason. This breaks links such as /dev/null, /dev/zero, /dev/stdin, /dev/random, /etc/unbound, etc... A fill list can be seen below.

I will note that I am using IOCage to manage jails.

Code: 
2017/05/06 10:19:29 NOTICE: tags/mcsjail: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/mcsjail_opt: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/home: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/sys: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: releases/11.0-RELEASE/root/sys: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/dev/log: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/dev/random: Can't transfer non file/directory
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/dev/urandom: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/dev/null: Can't transfer non file/directory
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/dev/zero: Can't transfer non file/directory
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/dev/stdin: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/dev/stdout: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/dev/stderr: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/dev/zfs: Can't transfer non file/directory
2017/05/06 10:19:29 ERROR : jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/root/.history: Failed to copy: failed to open source object: open /iocage/jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/root/.history: permission denied
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/proc/curproc: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/dev/pts/0: Can't transfer non file/directory
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/dev/pts/3: Can't transfer non file/directory
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/dev/pts/1: Can't transfer non file/directory
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/etc/aliases: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/etc/rmt: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/etc/termcap: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/etc/unbound: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:29 NOTICE: jails/a9536aae-e104-40d3-a6e4-f87376d547bb/root/sbin/nologin: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:37 NOTICE: releases/11.0-RELEASE/root/etc/aliases: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:37 NOTICE: releases/11.0-RELEASE/root/etc/rmt: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:37 NOTICE: releases/11.0-RELEASE/root/etc/termcap: Can't follow symlink without -L/--copy-links
2017/05/06 10:19:37 NOTICE: releases/11.0-RELEASE/root/etc/unbound: Can't follow symlink without -L/--copy-links

-L will only make rclone copy the data on the other side of the symlink as well which is undesirable as it could cause recursive loops or duplicate files.

This being said, whats the best way to backup the raw files of a jail in a way where it can easily be restored? Is there any method to restore the jail, then re create the symbolic links? This is my first time facing this issue.
 
Phishfry said:
How about tar'ing up the whole jail and then rclone. This would maintain the symbolic links.
ftp://ftp.gnu.org/old-gnu/Manuals/tar/html_node/tar_115.html

Here is an example of taring up a whole drive and copying to another drive.
https://github.com/freebsd/crochet/blob/master/board/BeagleBone/overlay/root/copy-to-emmc.sh
Click to expand...

Phishfry said:
I see the ez-jail manual page uses archive to backup:
https://www.freebsd.org/doc/handbook/jails-ezjail.html

There are many ways to backup a jail and this is just one.
Click to expand...

Thanks for the input! I was hoping for something a bit more incremental so it could automatically run every so often.

I suppose I could write a script that first deletes the original tar, makes a new tar, then uploads that file. That would however temporarily require more space on the local disk (for the tar) and wouldn't be incremental as each backup the entire file would need to be deleted off the remote and re-sent. I imagine a jail won't change too often though. (and it is only like 300mb).
 
Flash619 said:
When backing up a jail to an off-site location that does not support ZFS transfers, what is the best course of action for handling symbolic links?
Click to expand...
Why would the remote site need to support those in the first place?

You could still use zfs send to dump the entire thing (assuming the jail resides on its own file system) and in case you need to perform a restore then zfs receive can help. If you check zfs(8) you'll notice that zfs supports the usage of stdin as input.

Which means that something like this should be feasible to use: # ssh backup@backup-server "cat /opt/backups/snapshot.zfs" | zfs receive -Fs. Give or take, this is from the top of my head. Still, the point being is that you could let ZFS worry about making incremental snapshots and only use the remote storage for what it is: storage.
 
ShelLuser said:
Why would the remote site need to support those in the first place?

You could still use zfs send to dump the entire thing (assuming the jail resides on its own file system) and in case you need to perform a restore then zfs receive can help. If you check zfs(8) you'll notice that zfs supports the usage of stdin as input.

Which means that something like this should be feasible to use: # ssh backup@backup-server "cat /opt/backups/snapshot.zfs" | zfs receive -Fs. Give or take, this is from the top of my head. Still, the point being is that you could let ZFS worry about making incremental snapshots and only use the remote storage for what it is: storage.
Click to expand...

This is actually a perfect fix for the time being. Again, I'm not sure how I would go about incrementally backing up small data changes made within the jail. I.e. a database may change every so often in the jail from user interaction. I'm looking for a good way to automate that backup to an off site location. I'm sure I could do a initial backup of the entire jail, then just do a database export as a different revision each time but I'm trying to avoid having to copy and entire file every time.

I am aware zfs can incrementally export changes between snapshots with the -i flag. Would it be possible to export with the -i flag every so often, then eventually read them all one at a time in order to collect all the changes and restore a file system? Or would that be a bit inefficient?

:EDIT:

This just in, iocage has a export command. The export command will create a zfs snapshot, send it to a file in the /iocage/images dataset, and zip the exported zfs dataset.
 
Flash619 said:
Again, I'm not sure how I would go about incrementally backing up small data changes made within the jail. I.e. a database may change every so often in the jail from user interaction. I'm looking for a good way to automate that backup to an off site location.
Click to expand...
ZFS snapshots are your friend, but obviously this is assuming our jail resides on its own ZFS filesystem. For example, here's my home directory (/home) which is set up through zroot/home (the ZFS filesystem):
Code: 
breve:/home/peter $ zfs list -rt snapshot zroot/home
NAME                USED  AVAIL  REFER  MOUNTPOINT
zroot/home@020517  44.2M      -  4.26G  -
zroot/home@030517  42.5M      -  4.26G  -
zroot/home@040517  35.4M      -  4.26G  -
zroot/home@050517  34.9M      -  4.26G  -
zroot/home@060517  43.1M      -  4.26G  -
zroot/home@070517  34.6M      -  4.26G  -
zroot/home@080517  27.6M      -  4.26G  -
As you can see I have 7 backups / snapshots, I set this up using a script I wrote. All this basically does is run #zfs snapshot zroot/home@090517 (I get the date timestamp from using date(1)) and it'll destroy snapshots older than 7 days ago (date is a wonderful program; it also allows you to perform calculations, for example: date -v-8d would show you the date 8 days ago).

Once you make a ZFS snapshot it doesn't use up any space, that only happens when changes are applied to the actual filesystem. So, keeping this offsite could be done using: # zfs send zroot/home | ssh backup@backup-server "dd of=/opt/backups/snapshot.zfs" and afterwards using incremental backups.

So opt to keep multiple snapshots and send the incremental data between them. To only get the data generated between yesterday and now I'd use something like this: # zfs send -i zroot/home@070517 zroot/home@080517. You may also want to use the -R option if you go this route (generate a replication stream).

I'd definitely suggest to check up on zfs(8) before setting this up, definitely perform test cases.

Note that restoring data like this isn't the same as (for example) retrieving stuff using an UFS snapshot. When I use restore(8) to retrieve files from an UFS snapshot (made using dump) then I actually retrieve those files and place them onto my active filesystem.

Not so much with ZFS. When you retrieve such a snapshot then the snapshot will get imported into your ZFS file system after which you can retrieve the files from there. This is why it might be best not to rely on offsite snapshots alone, because retrieving them would also mean importing the whole thing again.

I don't know how familiar you are with all this, so another example... When someone on my system makes a mistake and needs a file back which they deleted 4 days ago (time of writing being 080517, so we're talking 040517) then I'd simply use: cd /home/.zfs/snapshot/040517 and then pick out the file(s) which I need and copy them over. This is also why I keep a 7 day snapshot retention.

Hope that this can give you some ideas.

Flash619 said:
I am aware zfs can incrementally export changes between snapshots with the -i flag. Would it be possible to export with the -i flag every so often, then eventually read them all one at a time in order to collect all the changes and restore a file system? Or would that be a bit inefficient?
Click to expand...
I should read the whole message first, and respond afterwards ;) Oh well, I'm a fast typist anyway and felt like it.

This would be possible. I can't really comment on the efficiency part because when it comes to backups then that's a difficult subject. If something may seem inefficient yet it gets the job done (and done reliably) then how important would that be?

Still, having said that, personally I'd rely on sending whole snapshots (depending on data amount of course) apart from sending the whole filesystem over as well from time to time. Snapshots are pretty much useless without the actual filesystem itself.

Flash619 said:
This just in, iocage has a export command. The export command will create a zfs snapshot, send it to a file in the /iocage/images dataset, and zip the exported zfs dataset.
Click to expand...
Can't comment on iocage but I can say that this should be easily scripted. You could even compress the stream while sorting it all out....

(used my previously shared setup as an example here):

# zfs send zroot/home@040517 | gzip -c | ssh backup@backup-host "dd of=/opt/backups/stream.gz".

This would even help you limit the amount of data which needs to be send over.
 
You must log in or register to reply here.
Back
Top