rsbackup: FreeBSD + ZFS + Rsync

phoenix

Administrator
Staff member
Administrator
Moderator

Thanks: 1,164
Messages: 3,985

#1
Background

The original version of this backup system is detailed in this other thread.

This thread will describe the newly-refactored and streamlined rsbackup system that we're using. See the original thread for all the gory hardware and OS details.


rsbackup v3.2
The latest version of rsbackup is attached to this message later in this thread.

We've upgraded our systems to 9.2-RELEASE, are using a split head node/storage node for a couple of the servers (better expandability down the line, up to 360 TB of raw storage using 2 TB harddrives), and using ZFS-on-Root for all systems now (easier to upgrade using Boot Environments). We've also upgraded to the latest version of ZFS, and have removed dedupe from 2 of the systems, instead using LZ4 compression.

rsbackup v3.0
The main reason for this refactoring was to simplify things, especially the configuration file setup. The second reason was to implement the ability to run a command on the remote system before and/or after running the backup. We've implemented several virtual servers recently, and needed a way to snapshot the VMs before running the backup. Relying on crontab timing for this was not reliable enough.

Earlier versions of rsbackup required several wrapper scripts. Now, everything is in one script, with proper command-line options.

We've also greatly improved the comments, so it should be easier to see how things are done, why they're done that way, etc.

The only bit that has changed since the original version was posted is that we're now running running 7-STABLE with ZFSv13 support, the ARC has been configure to cache metadata only, and 1 raidz2 vdev has been updated with 1.5 TB drives. We're waiting on delivery of a 30 GB SSD which will be configured as an L2ARC. We're still not sure whether to have the SSD be a data or metadata cache.

Enjoy!
 
OP
OP
phoenix

phoenix

Administrator
Staff member
Administrator
Moderator

Thanks: 1,164
Messages: 3,985

#2
rsbackup

Rather than take up multiple posts, I've attached all the files with .txt extensions. Just remove the .txt if you want to test this. These are all heavily commented (perhaps too much so?).

  • rsbackup: the script itself
  • rsbackup.conf: the global configuration file for rsbackup, must be in the same directory as rsbackup. This lists all the directories used by rsbackup, the log file location, default rsync options, etc.
  • rsbackup.sites: a list of directories to scan for remote server config files, relative to sites/ folder
  • defaults/exclude.freebsd: default exclude file for rsync, customised for a FreeBSD system
  • defaults/exclude.linux: default exclude file for rsync, customised for a Linux system

The default directory hierarchy is now (relative to where the rsbackup script is located):
  • defaults/: holds the rsbackup ssh keys and default exclude files for each OS
  • sites/: holds all the directories for each remote site
  • sites/<sitename>/: holds the config files for all the systems at a remote site

Here's the help blurb from the script:
Code:
rsbackup 3.0
Copyright School District #73 (Kamloops/Thompson) 2008-2010, all rights reserved
Released under the BSD License.

Usage: rsbackup [-h] [-m force|start|stop|restart|status|snapshot] [-m one -d site -c server]

  -m:  tells it which mode to run.  Modes include:
         force:    starts the backup process, even if another one is already running
         start:    start the backup process if no other backups are running
          stop:    stops any running rsbackup processes
       restart:    does stop, waits, then does start
        status:    shows stats on any running rsbackup processes
      snapshot:    create a ZFS snapshot of the backups directory
          help:    shows this help blurb

           one:   does a backup for a single server, requires two more arguments
                 -d:  tells it which site directory to scan for a config file
                 -c:  tells it which config file to use

  -h:  shows this help blurb

Examples:
  rsbackup -h
  rsbackup -m force
  rsbackup -m start
  rsbackup -m stop
  rsbackup -m restart
  rsbackup -m status
  rsbackup -m snapshot
  rsbackup -m help
  rsbackup -m one -d remotesite -c servername
 

Attachments

OP
OP
phoenix

phoenix

Administrator
Staff member
Administrator
Moderator

Thanks: 1,164
Messages: 3,985

#3
Server Configuration Files

These have been greatly simplified. For a server that fits all the defaults listed in rsbackup.conf, only a single line is needed: rsync_server=hostname.of.server All other settings are optional.

If a <servername>.exclude file exists in the same directory as <servername>.conf, that file is used as a supplemental exclude file for the rsync of that server.

<servername>.conf:
Code:
# Server-specific options
#
# Usage is as follows:
#
# rsync_server=		hostname of server (used for naming logs, connecting, etc)
# rsync_exclude=	name of default exclude file to use from ${confdir}
# rsync_port=		port to use to connect, if different from default
# rsync_opt_override=	override all rsync options with ones specific to this server (be careful)
# rsync_opt_append=	append the following to the list of default rsync options
#
# pre_cmd=		a command to run on the remote server before starting the rsync process
# post_cmd=		a command to run on the remote server after  ending   the rsync process
#
# sitedir=		the directory under ${backupdir} to use for all backups at this site
# serverdir=		the directory under ${backupdir/${sitedir} to use for this server
#
# If neither of the above (sitedir/serverdir) are set, they are derived from rsync_server as follows:
#    serverdir.sitedir.wherever.tld

rsync_server=server.somewhere.net
 

dennylin93

Aspiring Daemon

Thanks: 108
Messages: 783

#4
A simpler way to remove leading slashes using parameter expansion:
Code:
--- rsbackup.orig       2010-05-16 16:49:27.000000000 +0800
+++ rsbackup    2010-05-16 16:49:57.000000000 +0800
@@ -360,12 +360,7 @@
        local today=$( ${date} "+%Y-%m-%d" )
        local dataset

-       # Remove any leading slashes from the backup directory, to get the ZFS dataset name.
-       if [ $( echo ${backupdir} | ${cut} -c 1 ) = "/" ]; then
-               dataset=$( echo ${backupdir} | ${cut} -c 2- )
-       else
-               dataset=${backupdir}
-       fi
+       dataset=${backupdir#/}

        # Create the ZFS snapshot using the date as the name
        ${zfs} snapshot ${dataset}@${today}
 
OP
OP
phoenix

phoenix

Administrator
Staff member
Administrator
Moderator

Thanks: 1,164
Messages: 3,985

#5
Neat. I'll have to give that a try.

Yeah, that works perfectly. Even works when there's no leading slash.

Thanks.
 

dennylin93

Aspiring Daemon

Thanks: 108
Messages: 783

#6
Found a small problem on line 214:
Code:
rsync_options+=${rsync_opt_append}
The "+=" doesn't work for Bourne shell. This should fix it:
Code:
rsync_options="${rsync_options} ${rsync_opt_append}"
 
OP
OP
phoenix

phoenix

Administrator
Staff member
Administrator
Moderator

Thanks: 1,164
Messages: 3,985

#7
Ah yes, I did find this when I went to actually use the rsync_opt_* variables.

Too much time in ports Makefiles where this works, I think. :) Confusing my limited make foo with my limited sh foo. :D
 

pmeunier

New Member


Messages: 4

#8
Re: rsbackup: ZFS + FreeBSD + Rsync

I'm interested in rsbackup to replace a Dirvish installation. It's been almost four years, are you still using it and happy with it, and have there been more changes since?
 
OP
OP
phoenix

phoenix

Administrator
Staff member
Administrator
Moderator

Thanks: 1,164
Messages: 3,985

#9
Re: rsbackup: ZFS + FreeBSD + Rsync

Yes, we continue to use this on a daily basis, spread across three primary backups servers. And we've switched to use zfs send to send our daily snapshots to a single off-site storage server.

There have been a few small fixes and changes to the code, but nothing major. I can post the code if you're really interested in it.
 
OP
OP
phoenix

phoenix

Administrator
Staff member
Administrator
Moderator

Thanks: 1,164
Messages: 3,985

#11
Re: rsbackup: ZFS + FreeBSD + Rsync

I have time scheduled next week for working on the hardware side of the storage servers. I'll take that time to clean-up and consolidate the rsbackup code, and then post it here.

If you don't see anything by next Thursday, drop me a reminder PM. :)
 
OP
OP
phoenix

phoenix

Administrator
Staff member
Administrator
Moderator

Thanks: 1,164
Messages: 3,985

#12
Re: rsbackup: ZFS + FreeBSD + Rsync

Attached is a tarball of the latest in-use version of rsbackup (3.2).

The big difference with this version is the "-m one" mode has been replaced with "-m site" to backup all servers at a single remote site and "-m server" to backup a single server at a single remote site.

There's also a "setexpiry" script that can be used to add a custom property to snapshots that includes an expiry date for that snapshot. And a companion "rsprune" script that can be run on a daily basis to remove snapshots that have expired.
 

Attachments

logan893

New Member


Messages: 7

#13
I'm very excited to try this backup solution of yours! Many thanks for sharing this, phoenix!

As it has been two years since the last update of this thread, have there been any major changes since then which you can share? Perhaps a posting on github would be in order, as suggested by that0n3guy.

Going through the configuration file and the code I do find a few things I'll have to modify to suit my environment. Your script assumes that the ZFS dataset name ($backupdir) is mounted under root (/$backupdir). I have my dataset mounted in /mnt/<zfspoolname>/<datasetname>, and as this machine is running FreeNAS I'd like to keep it this way.

In the original 3.2 codebase, unless I set $backupdir to the full path of the mounted dataset it will not work for rsync. And if I do set the full path (backupdir = mnt/zfspool/dataset) while it will work for rsync, it will instead not work for snapshots.

Here's a quick and dirty patch which allows the use of a new variable in the rsbackup.conf file, "backupdir_mountprefix". Leaving this variable empty should preserve the original behaviour just fine.

Code:
--- rsbackup.original-3.2  2014-03-17 19:37:54.000000000 +0100
+++ rsbackup  2016-04-07 00:45:03.450179927 +0200
@@ -290,7 +290,7 @@
  --exclude-from="${defaultsdir}/${rsync_exclude}" ${rsync_exclude_server} \
  --log-file="${logdir}/${rsync_server}.log" \
  --rsync-path="${rsync_path}" --rsh="${ssh} ${sshopts} ${hpnopts} -p ${rsync_port} -i ${defaultsdir}/${rsync_key}" \
-  ${rsync_user}@${rsync_server}:${basedir}/ /${backupdir}/${sitedir}/${serverdir}/${basedir}/
+  ${rsync_user}@${rsync_server}:${basedir}/ ${backupdir_mountprefix}/${backupdir}/${sitedir}/${serverdir}/${basedir}/
  logit "====>" "Ending rsync process for ${rsync_server}."

  # Check if there's a post_cmd to run
 
OP
OP
phoenix

phoenix

Administrator
Staff member
Administrator
Moderator

Thanks: 1,164
Messages: 3,985

#14
Nothing in the actual script itself or the config file format has changed. Why break what works by modifying it? ;) We've changed our processes around it for syncing data with the off-site storage servers, and split our backups across multiple servers, but the actual backup process hasn't changed.
 
Top