I wanted a second opinion - is my backup methodology here sane?

I have two servers, both running FreeBSD 14.0-RELEASE and both hosting a number of jails. One is a dedicated server with a commercial hosting provider, and one is in my house. I refer to these as my dedicated and home servers respectively.

For my dedicated server until now my backup solution has just been to occasionally offload everything to my home server with tar and rsync, but since it is more permanent now than I had originally planned, I decided I needed something more regular and standard. I decided to set up a little scheme using ZFS replication which seems to be working just fine, but I wanted to get a second set of eyes on and make sure that I'm not overlooking something here, or if I've accidentally left a huge gaping hole that I don't see since this is my first time using ZFS for something like this.

On my home server, I set up a jail just to receive backups. There is a ZFS dataset at home/jails/backups/receive which has the jailed=on property set. The backup jail configuration uses exec.created="zfs jail backups home/jails/backups/receive"; to be able to use it on jail restart, and the jail runs ssh and my home router is forwarding a nonstandard port to the jail's ssh instance. The jailed ssh service is properly secured with pki required for login, though I do allow root login over ssh on the jail in order to accommodate the backup process (I tried to use a dedicated backup user and zfs allow but I was having trouble making it work well, so I figured allowing root on a jail only was an acceptable risk).

On the dedicated server, I generated a keypair that allows access to the jail (but not the host). My thinking here is that, if the key were to be compromised, then the only thing an attacker is getting access to is a backup of the server they already apparently have access to, so there's no great increase in risk (assuming they don't have some exploit for breaking out of a jail, but that's an angle that's hard to address, and one that's unlikely to be a problem anyway).

On the dedicated server's root crontab, I have two commands:
0 1 * * * zfs snapshot -r remote@daily_$(date "+\%Y\%m\%d")
0 2 * * * zfs send -RI remote@initial_20231205 remote@daily_$(date "+\%Y\%m\%d")|ssh -p xxxxx root@backupserver zfs receive -Fu home/jails/backups/receive/remote

The end goal here is a a daily snapshot of my entire remote filesystem being sent to my home server (there is adequate storage space), from which I can either restore my remote server if necessary, or even use it to move jails from the dedicated server to home server if decided to do so. The receiving end, I believe, should match the snapshot structure exactly because of the -F option, meaning if I delete old snapshots on the dedicated server then at the next send they will be deleted on the backup server as well.

The only thing I still intend to do that I haven't done is update the host's firewall to block all network traffic to and from the jail's IP besides over its ssh port, in order to reduce the risk to my home server and network if the jail were to be compromised.

I might also adjust the sshd configuration on the backup jail so that it only runs the zfs receive command on connection and can't actually be used to gain a shell, but that's not something I've had time to do yet.

So far it appears to be working just fine, but am I overlooking any potential problems here?
 
Last edited:
Something to consider: unintended data deletion by user

Most folks (myself included) are well protected against hardware failures.
What is more insidious is unnoticed data loss.

In my case, it is 9 months of unnoticed lost photos from decades of saved images.
If I had progressive incremental backups I could go back and recover, but I do not.

Due to the immense size, I only have a "current" backup including the lost images.
Folders are present, files are gone.

In retrospect, I should have been taking all incremental backups so only those new or changed files are stored in that backup file.
This will keep the backup(s) file size(s) down, but still keep copies of new data as added.
 
I was going to say pretty much the same thing. And point out that this is a common issue with backups.

Most people build a backup system that protects against what they are most scared of, which is "the unknown". Threats that are outside the control of human computer users and administrators. That is typically disk errors (a few read errors), disk failure, and destruction of the whole computer (for example power surge or fire). Your backup system seems to be doing a fine job against that. In particular, that's important because your "dedicated" server is hosted, and it always possible that it will vanish (with all its data) if there is some issue with the hosting company.

What people tend to ignore is protection against the known. That includes "insider risk" (a person on the inside might become a saboteur and intentionally destroy data, by deleting or overwriting), and simple human errors. Bgavin described a common example, where the human admin deletes some data by mistake (unintentionally), then nobody notices that it has vanished, and after a short while the deleted data is purged from backups as they age.

For this reason, my personal backup system never loses any information automatically: deleted files are kept forever in the backup, as are earlier modified versions of files. The reason this works reasonably well for me is that my home server is to a large extent a data archive, where files are stored and kept forever. On a file system that has lots of churns (short-lived files being created and deleted again), this would not work well; and if files change content all the time yet live for a long time, it would be disastrous.

One simple idea: Remove the "-F" switch from your zfs receive, so old snapshots hang around, and then see how much disk space the backup uses. It might just fit.
 
My 24tb ZFS system is devoted entirely to movie storage, and is always running at 80% full, so I can't do snapshots.
All of my "work" is in the Windows arena.. so I wrote a backup program (wrapper) using WinRAR as the engine.
The wrapper does all the free space and aged file management, etc, including setting the Archive file attribute after spawning WinRAR to do the back up.

I used to take a Full backup at month-end, but this is creating enormous amounts of backup file data.
I am better off skipping Full and taking only Incrementals which I can keep far longer.
This would have preserved my missing JPG photo files in one of those incrementals.

Consolidating all the existing Full backups into a single baseline archive requires a huge amount of temp space.
For this, I picked up a WD Red 18tb when Newegg had them at firesale pricing... they have since gone waaay up in price.
 
1. I would do pull-based here, so home SSHs to dedicated. The reason is that dedicated is on public internet, in a remote data center, and so is more vulnerable. I also don't want that machine getting access to my home network.

2. I would use sanoid / syncoid to do automated time-based backups, automated thinning, and sync snapshots between machines.
 
I keep daily backups on a different machine than my LAN data server.
I use the RAR command line engine to create the archive files because it allows clearing the ARCHIVE file attribute via command line switch.
RAR configuration allows for fast no-compression storing of specific file types, i.e. JPG, 7Z, etc.

In my PULL arrangement, the full backup created by RAR was crashing with corrupt file headers.
This is not a RAR issue, but something in the network or the Server 2012 that holds the data files.

I have since changed from a PULL to a PUSH, first creating the backup on the server, then PUSHing it to the separate machine.
I keep the maximum file size of both INCR and FULL backup types, and thin old backup files for this amount of required free space before each backup.
Using the RAR file type gives me an industry-standard file type and avoids proprietary types.
 
Back
Top