rsync.net is good for personal use if you're using their basic SSH plan because they can have small volumes. Note that their minimum for zfs is 5TB which is $60/mo. If that fits your budget, cool, just be aware of it. Also they set up a VM for you, they upgrade it at their own pace, blow away anything that's not on your zpool, and you can't really run other services on it. It's intended as a zfs snapshot target only. They're pros though, so if you just need a place to send snapshots and can afford it, I say go for it. Also would have no hesitation with their basic rsync service. I keep meaning to set up a hosthatch server so I can run my own VM with additional service, but it's low on priority list.
If you do go with zfs, my preference is sanoid for policy-based snapshots and zelta for snapshot replication. zelta is more of a "do the right thing for backups" out of the box than sanoid. Sanoid is excellent and capable, but it's a sharper tool. It will let you receive datasets that share a mountpoint with /, which means your next commands on the receiver might be pretty wacky. That's not a shortcoming of sanoid at all - it's just how zfs works - but for a minimum of fuss, zelta gets the job done.
What I personally do is set up a
zroot/SAFE dataset, and anything that I want to back up descends from there and is mounted into the right place. For things under
/var/db I do
zfs create -o mountpoint=/var/db canmount=off zroot/SAFE/var/db (you'll need a similar one for just
/var first). Then I create datasets for the things that I want to back up, e.g.
zfs create zroot/SAFE/var/db/postgres.
This way my sanoid and zelta only need to deal with
zroot/SAFE. Anything I want in the backup descends from that dataset. I can snapshot anything else but that's just local snapshots so I can recover files easily if needed, not the long-term robust backup plan.
Should I zfs snapshot various datasets and then rsync those (and if so, which datasets)? Or should I forego snapshots and just rsync the mount points directly (and if so, which mount points)?
You should do an offsite backup of whatever data you want to be able to recover in case your server is lost. For me, that's data, not OS installation, applications, or config. All of that can be reproduced from other sources (e.g. install from upstream, config in version control).
There are two ways you can use snapshots in this sort of scenario. The first is to replicate the snapshots directly using something like zelta. The other is to make a snapshot, and then rsync the files from that snapshot. This avoids the problem of files changing while rsync is running. To do that you would do something like
rsync -avz --delete /.zfs/snapshots/foo-snapshot-name remote-server@/backups or maybe use a backup tool like restic or borg, but sourcing your latest snapshot dir. It's a nifty way to use zfs in an environment where you don't want to / can't receive snapshots. Also note that this approach is NOT recursive, so if you look at
/.zfs/snapshots that will just be the stuff on your bectl root, you'll need to do the same for
/home, /var, etc.
Personally I prefer replicating snapshots, it's very simple.
One last thing: it's a good idea to test your backups automatically. You may need to restore from them one day, and it'll be a lot more comfortable if you know it works rather than crossing your fingers when the time comes. For example, you can zfs snapshot the postgres data dir and get a crash consistent backup (equivalent to if you pulled the plug on the server). It's usually reliable but at $JOB we've seen < 0.1% of the backups not be restorable. We know this because we verify every snapshot by mounting it into a jail and starting postgres on it to make sure it recovers.
Am I wrong ?
