ZFS Help with backup details

M

mefizto

Greetings all,

I have finally centralized my backup to a single FreeBSD server. Since I have a heterogeneous network, inherited Synology running BTRFS, Windows 10, FreeBSD, I am pulling the data to be backed up via rsync, and then taking a snapshot via Sanoid.

I have two concerns:

1. (a) how do I ensure that the rsync session completed successfully; and
(b) how to resume/restart the rsync session that did not complete successfully; and
2. how do I ensure that the next rsync session does not start until the previous rsync session finishes. e.g., because of larger data to back-up or resume/restart of the previous session?

Kindest regards,

M
 
Are you driving this from crontab?

Resuming in rsync needs no special attention. When it aborts it deletes it temporary files and you can just splatter the next rsync over. If you have a sudden crash (power fail, panic), temporary files are left, but they are in places that do no harm except take space (unless you use rsync options that prohibit that).
 
Hi cracauer@,

thank you for the answer.

Since I do not quite know how to handle the described situation, I run the rsync sessions with the --progress options and start the next session after I see the preceding session ending. But I would like to automate it, so I do not have to baby-sit it and run it at nigh with no/less activity.

Kindest regards,

M
 
mefizto said:
Hi cracauer@,

thank you for the answer.

Since I do not quite know how to handle the described situation, I run the rsync sessions with the --progress options and start the next session after I see the preceding session ending. But I would like to automate it, so I do not have to baby-sit it and run it at nigh with no/less activity.

Kindest regards,

M
Click to expand...
The canonical solution to not have cronjobs overlap is like this:
Code: 
#! /bin/sh
rsync -a -S -H foo bar:/baz/. &
echo $! > /var/run/mybackjob.pid

And testing for it like this:
Code: 
#! /bin/sh
if [ -f var/run/mybackjob.pid ] ; then
    kill -0 `cat /var/run/mybackjob.pid`  && exit
fi
[actual backup code]
 
Hi cracauer@,

thank you for posting the code. After consulting "How to read code for morons(tm)", I figured the code. I added some Czechs checks and balances and ended with a pre-production code:

Code: 
#!/bin/sh

#==================================================================================#
# Set strict error handling for the foreground setup
#==================================================================================#

set -eu

#==================================================================================#
# Configuration variables
#==================================================================================#

# Remote rsync daemon source
SOURCE="username@IP_address::Backup/source"

# Local ZFS destination path
DESTINATION="/Backup/target"

PIDFILE="/var/run/rsync_backjob.pid"

#==================================================================================#
# Root privilege check (Safeguards the tracking file path)
#==================================================================================#

if [ "$(id -u)" -ne 0 ]; then
    echo "Error: This script must be run as root to write to $PIDFILE" >&2
    exit 1
fi

#==================================================================================#
# Securely prompt you for the password in the foreground
#==================================================================================#

printf "Enter rsync daemon password for %s: " "$SOURCE"
stty -echo  # Turn off terminal echoing so your typing is hidden
read -r RSYNC_PASSWORD
stty echo   # Turn terminal echoing back on
echo ""

#==================================================================================#
# Launch optimized rsync directly into the background
#==================================================================================#

# Key Optimization Assets:
# -c         : Forces cryptographic checksum verification to catch NTFS bit rot
# -a, -S, -H : Preserves attributes, sparse files, and hard links cleanly
# --partial  : Keeps partially transferred files to resume instantly on interruption
# --timeout=120 : Hard drop if the network stalls or goes unresponsive for 2 minutes
# --contimeout=30: Fails fast if the target server's daemon won't answer the initial call

rsync -a -S -H --partial --timeout=120 --contimeout=30 "$SOURCE" "$DESTINATION" &

#==================================================================================#
# Securely track the process ID
#==================================================================================#

PID=$!
echo "$PID" > "$PIDFILE"
echo "Backup engine started safely in background with PID: $PID"

Clearly, there are some rough edges, e.g., handling of the password entry is rather a hack, still missing check for the exit code, but considering that it is my about 6th script it is a start.

If someone would be willing to review the code and suggest improvements, e.g., using
sysutils/pass for the password entry (mer ?), checking the exit code, and any others, I would appreciate it.

Also is there any AI specialized in reviewing a code?

Kindest regards,

M
 
Pretty much any LLM will review code just fine these days.

The thing I would do differently is the auth. I would use an ssh agent, symlink $SSH_AUTH_SOCK to a fixed location known to the script. Then after each boot of the machine you add the key to this agent only once and the script can run on it as often as it wants - including from cron.
 
You must log in or register to reply here.
Back
Top