What is your 3-2-1 backup strategy?

blazingice

blazingice

I'm interested to hear how people implement a 3-2-1 backup strategy when using a FreeBSD ecosystem. That is 3 copies of the data, in 2 different storage medias, 1 of them being off site (e.g. online). Obviously the nature and the amount of data will dictate the scale of a backup solution (i.e. personal vs. enterprise solutions). Nevertheless I would be interested to hear what hardware you are using, what procces you follow and the kind of tools within FreeBSD ecosystem you are using to achieve it.

As an example for my personal data (mostly photos and home videos). I have the following:

  1. 24/7 FreeBSD home server. In order to keep the energy consumption low, I am using an old Dell Laptop with i5-4300U and 16G RAM. Two external hard disks creating a mirror zfs pool.
  2. Same data is kept in sync with a windows machine using net/unison
  3. Windows machine uploads it to Microsoft OneDrive, which allows me to access specific files on the go.
  4. Using sysutils/restic and net/rclone I backup some folders into OneDrive. This is a duplication of point 3, but just making use of the unused space that OneDrive offers.
  5. When traveling I use net/unison to sync data to an external ssd drive to be used for downloading and editing photos on the go, which is then synced back using net/unison when I get back home.

Steps 2,3,4 are a bit clunky. I would love to use a more streamlined solution, perhaps using Amazon Glacier storage solution.

What are others using?
 
I end up doing 4-2-1:
  • Primary (live and writable) copy of the data on my home server is on a pair of mirrored disk drives, using ZFS as the RAID layer and file system.
  • Backup (on which nothing is ever deleted and is not writable) is on a separate physical drive, still connected to the server (via a 2m long USB cable), but physically in a fire-resistant safe.
  • Remote backup is a combination of being uploaded into cloud storage, and copied to a portable disk drive that is stored far away.
 
I only have to backup my own files, no specific threats here at home. My strategy:
  1. copy all system *.conf files etc. for all boxes in my personal admin directory, I copy them directly after making changes;
  2. backup on two different USB external HDD, one incremental, one mirror (with rsync '--delete' in the command below). incidental backup on a large USB thumbdrive to take with me when on holidays etc. (off site);
  3. simple root alias for net/rsync with an exclude file for skipping *.iso, trash, cache and stuff, write backup date to a log file on the backup medium to keep track;
  4. possible improvement: backup more frequent.
alias bak_all 'rsync -az --info=progress2 --exclude-from=/media/da0/ exclude-list-rsync.txt /home/ /media/da0/HOME_backup_all/ ; ( echo incremental backup @FreeBSD ; date ; echo ---------- ) >> /media/da0/backup. bsd_all.log'
 
These days storage is cheap. My time is precious:
  • all media on all hosts are redundant, except USB sticks used to boot appliances;
  • USB boot media have a dd copy saved on the ZFS server after each major upgrade;
  • all systems (including appliances) get a monthly 100% backup to the ZFS server;
  • every few months, the ZFS server gets a 100% backup to a large capacity hot swap SATA disk;
  • hot swap disks get rotated off site.
I use rsnapshot(1) to pull file system backups from each client (FreeBSD and Linux) to the ZFS server. Rsnapshot is just a wrapper for rsync(1), which is used to de-duplicate successive backups. Only the changed files are copied to the server. When a file is unchanged from the previous backup, a hard link to the previous copy is used to "create" a new copy of it on the backup server. Hence the backups run fast because only the deltas need backup. And they are compact, because the entire time sequence of successive backups from each host is de-duplicated.

I keep the most recent 20 backups on line, on the ZFS server.

Once a year I archive a full set of backups for every host (I just move one branch of the backup tree to a safe place on the ZFS server).

In addition, my FreeBSD daily driver (which has all the stuff I really value) is duplicated in two locations once a week, and I use a USB stick to keep the two copies on my home directory in sync. [My notebook broke, and I will revert to syncing a "travelling copy" of my FreeBSD daily driver to a notebook when the new Framework 16" laptops become available.]
 
With the home data set I do a differential backup:
Code: 
{beastie} Ψ FreeBaSeD-T430 Ψ /home/beastie
   Ψ zfs snapshot tank1/zhome@`hostname`_2023-09-11                                                                                                    
{beastie} Ψ FreeBaSeD-T430 Ψ /home/beastie
   Ψ zfs send -I tank1/zhome@base-FreeBaSeD-T430_2023-06-30 tank1/zhome@FreeBaSeD-T430_2023-09-11 | xz -9 > /mnt/bkp/zhome/zhome_12023-09-11.xz

And with the root data set I just export the boot environment:
Code: 
root@FreeBaSeD-T430:~ # bectl export default | xz -9 > /mnt/bkp/zroot/"zroot-`freebsd-version`-`uname -v | awk -F '-' '{print $3}'`"

After both compression finish, I copy the compressed files to a private cloud storage.

PS.: I use Stable/13.
 
Given that almost everything here runs ZFS; our backup strategies are completely built on ZFS snapshots:

- physical servers are all running ZFS with mirror vdevs, so single drive failures in anything remotely critical (or annoying to rebuild) don't bother me... The only exceptance are some VPN routers running OpenBSD, which have their config stored in git repos. (i.e. they can be rebuild very quickly from a vanilla OpenBSD installation)

- our fileserver and backups host is collecting snapshots from all hosts in varying intervals (anything from every 30 minutes to once a day, depending on the host/dataset/VM) in a dedicated pool.

- NAS in another building is replicating the backups-pool in 3h intervals

- the most important datasets are also replicated to another host at one of our branches (the one that has decent enough bandwidth).

- configurations for hosts (rc.conf, pf.conf, iocell, vm-bhyve etc) as well as services (DHCP, DNS, RADIUS, mailserver etc pp) are all in git repos, which are synced between a local gitlab and remote git server (and on my workstation and laptop where I edit these configs). So basically the whole infrastructure could also be rebuilt from scratch.

This has proven to work when the HBA in our then-only main server died and corrupted the ZFS pool; the replacement systems were rebuild via a mix of zfs send|rcv datasets for jails or VMs and configuration in the git repositories.
I could have rebuild the old host 1:1 by just send|recv'ing the whole backup of that host, but that wasn't the goal back then.


At home my approach is a bit tuned down:
- home server collects snapshots from my desktop & laptop

- important stuff gets send off to tarsnap

- some stuff is also backed up manually to external storage from time to time (primarily my music library, which in theory could be rebuild by converting all my CDs from ~25 years into flac again...)
 
I think that the key to multi-method backup is the discipline to hold important files in certain locations, so that they are easy to backup often. I have been reasonably successful at that and can do hourly backup via rsync to multiple off-site machines for those portions that are important and small. Otherwise I use a whole backup server via ZFS snapshots and one-shot backup disks for full backups (usually SAS because they are cheaper).
 
For my personal backups I use NextCloud on a tiny headless debian home server. Not really a backup method in the strict sense but I do have 3 or more computers connected to the same account and they are never connected at the same time so in case of synced accidental deletion on one of them a full backup is available on the 2 others. If the server dies I also have full copies on the computers. Nextcloud has also a way of recovering deleted files if the data area has not yet been overwritten so it is possible to recover deleted files.
This is mainly for my personal work documents, music, photos and family videos.
There is a Nextcloud desktop app for all major platforms including FreeBSD and a mobile app on both Playstore and Appstore. The phone app has the option of sending all recently taken photos and videos to the server when connected to wifi and those will be synced to the desktop when the desktop app connects. It works like icloud only free. I value this very much.
I have not set up nextcloud server on FreeBSD yet but I do use the desktop app on my FreeBSD computer. My photos are handled by Shotwell and the photos folder is always inside the Nextcloud folder so when I open up Shotwell all my newly taken pictures are there ready after syncing is completed and no matter if I am using the FreeBSD computer or any of my Linux's everything looks fresh and no need to do anything.
This is also very useful when working on the road and from home because everything will be synced and when back home or to the office, given that you have the desktop app installed on those computers, every file you where working on is in it's place there.
With all this I however do copy the nexcloud folder on my desktop computer to separate drive once a year maybe, just in case.
 
You must log in or register to reply here.
Back
Top