rsbackup: ZFS + FreeBSD + Rsync [Archive]

View Full Version : rsbackup: ZFS + FreeBSD + Rsync

phoenix

March 5th, 2010, 00:37

Background

The original version of this backup system is detailed in this other thread (http://forums.freebsd.org/showthread.php?t=3689&highlight=rsbackup).

This thread will describe the newly-refactored and streamlined rsbackup system that we're using.

See the original thread for all the gory hardware and OS details. The only bit that has changed is that we're now running running 7-STABLE with ZFSv13 support, the ARC has been configure to cache metadata only, and 1 raidz2 vdev has been updated with 1.5 TB drives. We're waiting on delivery of a 30 SSD which will be configured as an L2ARC. We're still not sure whether to have the SSD be a data or metadata cache.

rsbackup v3
The main reason for this refactoring was to simplify things, especially the configuration file setup. The second reason was to implement the ability to run a command on the remote system before and/or after running the backup. We've implemented several virtual servers recently, and needed a way to snapshot the VMs before running the backup. Relying on crontab timing for this was not reliable enough.

Earlier versions of rsbackup required several wrapper scripts. Now, everything is in one script, with proper command-line options.

We've also greatly improved the comments, so it should be easier to see how things are done, why they're done that way, etc.

Enjoy!

phoenix

March 5th, 2010, 00:39

rsbackup

Rather than take up multiple posts, I've attached all the files with .txt extensions. Just remove the .txt if you want to test this. These are all heavily commented (perhaps too much so?).

rsbackup: the script itself
rsbackup.conf: the global configuration file for rsbackup, must be in the same directory as rsbackup. This lists all the directories used by rsbackup, the log file location, default rsync options, etc.
rsbackup.sites: a list of directories to scan for remote server config files, relative to sites/ folder

defaults/exclude.freebsd: default exclude file for rsync, customised for a FreeBSD system
defaults/exclude.linux: default exclude file for rsync, customised for a Linux system

The default directory hierarchy is now (relative to where the rsbackup script is located):

defaults/: holds the rsbackup ssh keys and default exclude files for each OS
sites/: holds all the directories for each remote site
sites/<sitename>/: holds the config files for all the systems at a remote site

Here's the help blurb from the script:
rsbackup 3.0
Copyright School District #73 (Kamloops/Thompson) 2008-2010, all rights reserved
Released under the BSD License.

Usage: rsbackup [-h] [-m force|start|stop|restart|status|snapshot] [-m one -d site -c server]

-m: tells it which mode to run. Modes include:
force: starts the backup process, even if another one is already running
start: start the backup process if no other backups are running
stop: stops any running rsbackup processes
restart: does stop, waits, then does start
status: shows stats on any running rsbackup processes
snapshot: create a ZFS snapshot of the backups directory
help: shows this help blurb

one: does a backup for a single server, requires two more arguments
-d: tells it which site directory to scan for a config file
-c: tells it which config file to use

-h: shows this help blurb

Examples:
rsbackup -h
rsbackup -m force
rsbackup -m start
rsbackup -m stop
rsbackup -m restart
rsbackup -m status
rsbackup -m snapshot
rsbackup -m help
rsbackup -m one -d remotesite -c servername

phoenix

March 5th, 2010, 00:46

Server Configuration Files

These have been greatly simplified. For a server that fits all the defaults listed in rsbackup.conf, only a single line is needed: rsync_server=hostname.of.server All other settings are optional.

If a <servername>.exclude file exists in the same directory as <servername>.conf, that file is used as a supplemental exclude file for the rsync of that server.

<servername>.conf:
# Server-specific options
#
# Usage is as follows:
#
# rsync_server= hostname of server (used for naming logs, connecting, etc)
# rsync_exclude= name of default exclude file to use from ${confdir}
# rsync_port= port to use to connect, if different from default
# rsync_opt_override= override all rsync options with ones specific to this server (be careful)
# rsync_opt_append= append the following to the list of default rsync options
#
# pre_cmd= a command to run on the remote server before starting the rsync process
# post_cmd= a command to run on the remote server after ending the rsync process
#
# sitedir= the directory under ${backupdir} to use for all backups at this site
# serverdir= the directory under ${backupdir/${sitedir} to use for this server
#
# If neither of the above (sitedir/serverdir) are set, they are derived from rsync_server as follows:
# serverdir.sitedir.wherever.tld

rsync_server=server.somewhere.net

dennylin93

May 16th, 2010, 09:51

A simpler way to remove leading slashes using parameter expansion:

--- rsbackup.orig 2010-05-16 16:49:27.000000000 +0800
+++ rsbackup 2010-05-16 16:49:57.000000000 +0800
@@ -360,12 +360,7 @@
local today=$( ${date} "+%Y-%m-%d" )
local dataset

- # Remove any leading slashes from the backup directory, to get the ZFS dataset name.
- if [ $( echo ${backupdir} | ${cut} -c 1 ) = "/" ]; then
- dataset=$( echo ${backupdir} | ${cut} -c 2- )
- else
- dataset=${backupdir}
- fi
+ dataset=${backupdir#/}

# Create the ZFS snapshot using the date as the name
${zfs} snapshot ${dataset}@${today}

phoenix

May 16th, 2010, 18:06

Neat. I'll have to give that a try.

Yeah, that works perfectly. Even works when there's no leading slash.

Thanks.

dennylin93

June 13th, 2010, 09:31

Found a small problem on line 214:

rsync_options+=${rsync_opt_append}

The "+=" doesn't work for Bourne shell. This should fix it:

rsync_options="${rsync_options} ${rsync_opt_append}"

phoenix

June 16th, 2010, 03:04

Ah yes, I did find this when I went to actually use the rsync_opt_* variables.

Too much time in ports Makefiles where this works, I think. :) Confusing my limited make foo with my limited sh foo. :D