Simple backup protocol for home user.

I'm new to FreeBSD and have just installed a 12.1 release 32-bit version with Mate desktop, OpenOffice, Thunderbird and Firefox are all installed, and even the printer is working. So before I inevitably mess things up, I am looking for some pointers on how best to go about backing up the system, apps and files and some pitfalls to avoid. I would like to :-

1) Do an initial full system backup to a USB drive in case I seriously mess things up and have to start from scratch with a freshly formatted hard drive.

2) Do periodic routine backups of just my files.

I'm reading through the Handbook 19.10 Backup Basics and will likely follow the instructions on dump & restore, but I wanted to check in with the forum and see if there were any pearls you can offer me first. A GUI option would be nice but I'm happy to use the terminal.
 
No need to backup the whole system. It's typically quicker to do a fresh install than restoring a full system backup. Do make a backup of your important configuration files (/etc/rc.conf for example). If you want, create a list of installed packages and backup that list (everything can be quickly (re)installed from that list). Definitely backup /home or /usr/home. Depending on your situation you may want to backup each user's home directory separately.

Backups can be as simple as a regular tar(1) job, just store the tarball on a separate machine or disk (external USB disk for example).
 
http://www.wonkity.com/~wblock/docs/html/backup.html gives good hints. If your home is not in a separate file system you could use dump(8) and restore(8) and mask directories with nodump to save space and unnecessary backup. Additionally you should try from time to time if you can restore any of your backups, just in case. Helpful notes for that should be on paper.

A tar() archives of the configuration files in /etc and /usr/local/etc have been often helpful in order to fix some mess I did more or less unintentionally. From my experience one need that more often than a full backup :rolleyes:.
 
I normally just use net/rsync to sync my user's /home to my NAS. I also back up a few customized files as SirDice mentioned. All I really care about is my data. It takes about 10 minutes for me to install FreeBSD so for my use case, backing up the OS is kind of pointless. If you do a lot of custom port builds and whatnot, there are I am sure other things you need to back up so you don't have to go through a lot of work to get back up and running. I normally use packages and feed a list of packages I use to pkg so that part is automated, sort of...
 
Rsync is the tool you want to use. Put it in your crontab and forget about it (unless you get an email that it failed)
Do you have to configure any mail daemon or logging to receive notifications if it failed or will it mail the 'root' account on failure?
 
Do you have to configure any mail daemon or logging to receive notifications if it failed or will it mail the 'root' account on failure?
You should alias root to a real mail account in /etc/mail/aliases, and test to see if you can send mail to the real account echo 'This is only a test' | mail -s 'This is a test' me@example.com. Don't forget to run newaliases(1) after you add the alias.

If this doesn't work, you could try dma(8) as Mjollnir suggests here.
 
You'll need to use -q on rsync otherwise you'll get mailed a bunch of status data every time it runs.

Follow the instructions above to alias root's mail to an actual address and test it.
 
  • Like
Reactions: a6h
If you installed your system on ZFS, you have the boot environments. See bectl(8) or beadm(1) (I recommend the latter).

This won't save you in case of disk failure but from your errors. See here for a comprehensive description: https://vermaden.files.wordpress.com/2018/11/nluug-zfs-boot-environments-reloaded-2018-11-15.pdf

But if you chose UFS, sorry...

Thanks, but I'm not using ZFS - though I did read some interesting stuff about it on an Oracle Solaris Admin website, so, it might definitely be something worth considering at a later date.
 
An example:
Code:
#!/bin/sh
mkdir -p /backup/usr/local
rsync -a /etc /backup
rsync -a /root /backup
rsync -a /usr/local/etc /backup/usr/local
rsync -a /usr/home /backup/usr

Thanks, this is great. I think I'll follow everyone's advice to some degree. Somewhere I'll need to include code to mount the ext4 USB partition whether at the beginning of vigole's script if I opt to run the script manually (where I might put the USB drive in for backup and then later remove it), or have ext4 mount done automatically at boot up if I'm going to leave the USB drive in and have the script run automatically at times indicated in crontab. Would I be correct in assuming that if set up a crontab to run the script at regular intervals but the USB drive was for some reason unavailable during one instance, that I would just get a nasty email?
 
There are two layers to distinguish here.

First, rsync is an (incredibly complex and powerful) simple copy program. OK, so I just called it both "complex" and "simple", does that mean that I've gone insane? No, it means that rsync has a simple external interface: it doesn't run by itself on a regular schedule, it doesn't e-mail root, it doesn't handle serious error by retrying. It actually doesn't even know what "backup" is, it simply copies things from point A to point B. It has lots of internal complexity (like recursing down directory trees, knowing which files don't have to be copied ...).

Second, cron knows how to run a simple program, run it on a regular schedule, send mail to root when the program "fails", and all that. The problem that msplsh mentions above with his suggestion of using "rsync -q" is: One of the things that cron does: if the program creates any output on stdout or stderr, it gets mailed to root. In the particular case of a backup program, that seems pretty silly; most people would prefer to not get an e-mail if the backup ran successfully and on schedule.

But: the suggestion of using "-q" throws the baby out with the bathwater. This means that if something goes wrong, it gets harder to debug: All you get is the final error message. And it gets harder to check whether the backup is doing the "right thing". Here would be my suggestion: wrap the rsync commands (without -q) in a little script, which saves the normal output of rsync somewhere (we'll discuss where next). Then the script checks whether the rsync commands succeeded; if yes, it ends completely silently (nothing on stderr or stdout), and nobody gets any mail. If rsync fails, the script gives sensible and human-readable output, and also fails with an error code; this is intended to cause cron to send the sensible output to the sys admin. For example:
Code:
Dear Charlie Root,

on Wednesday, 2020 05 27 at 14:27, the daily backup job failed. The last line of output from the rsync command was:
  E_ELEPHANT: A purple flying elephant ate the backup disk.
The log of the rsync command has been saved in /var/log/daily_backup.log.
Please review the log file soon, because newsyslog will rotate it at the end of the day, and scrub it after one week.

Respectfully,
    your /usr/local/sbin/do_daily_backup script.

Finally, where to store the logs? If they are not too outrageously long, /var/log/ is actually a good place for them. Sys admins will expect to find them there, and you can easily configure newsyslog to compress them, and only save the last few. Once you have that set up, it might actually a good idea to run rsync with the "-v" switch, and see more detail.

Oh, and setting up mail so it works correctly for root goes without saying. That's just necessary.
 
If you use ZFS it's very easy to do incremental system backups. You just use zfs send to make a binary copy of your whole file system in a file or as a dataset. Then each time you want to back up incrementally, you use zfs send -I.
Restoring is as simple as booting from a rescue medium (if your main one does not work) and using zfs receive to overwrite the last changes. You could even recreate your system from the backup if necessary - again with zfs receive.
 
the suggestion of using "-q" throws the baby out with the bathwater.

No, because if you get mail, then you know you need to go run the command manually without the -q. If you don't get mail, then everything is fine. This is the [unix] way. ;)

Yes, running a script would be "nicer" but it also adds complexity and failure modes. It's not too much to ask them to understand cron and run rsync with -a and simple target options.
 
I recently started using rsync. The one thing I cannot get working is the -p or --progress flag.
Does FreeBSD's rsync have this option?
My command usage looks like this:
rsync -Wrvac src dest
I see rsync also has a server/client arrangement.
At first I was so excited I wondered why this tool is not in base. Then I looked at the license and saw it was GPL3.
I was also messing with cpio for simple backups.
 
Hm, I think -P worked for me the last time I used it. I used a much simpler command-line, though: rsync -aP rsync://linuxhost.example.com/video .

I was running rsyncd on the Linux host.

BTW, the -a option specifies -r, --recursive. No need to add it to command line, though it doesn't hurt to do so.
 
I reckon that the first thing is to figure out is a strategy that meets your goals and how much capacity you need to store the backups.

I have to backup both FreeBSD and Linux systems. Not all of them use ZFS.

I also value a backup system that gives me access to a time series of individual files immediately, on-line, on demand.

So I built a ZFS server. It uses rsnapshot(1) to centrally manage the backup process.

rsnapshot(1) uses rsync(1) for data transport, and It's capable of both full and incremental backups.

You must configure an account on the backup server with ssh root access to the fleet (best to plan that early using group membership, and don't use the "root" account itself).

The ZFS file system used to store the backups on the backup server is compressed.

An incremental backup of eight hosts generally takes less than an hour (the incremental backup of an unchanged file is implemented as a hard link to the previous copy of that file on the backup server)

Periodically, I zfs send the entire ZFS backup file system to an external disk for off-site storage.
 
An example:
Code:
#!/bin/sh
mkdir -p /backup/usr/local
rsync -a /etc /backup
rsync -a /root /backup
rsync -a /usr/local/etc /backup/usr/local
rsync -a /usr/home /backup/usr

I finally managed to get a 4th UFS partition for backup of my FreeBSD on a USB drive that has 3 partitions reserved for linux .

I followed your example above but have problems with the last step as it slows to a crawl if not actually halts. I thought it was too much data on the Desktop which contains some heavy folders so I excluded that using:-
Code:
rsync --exclude=/myhome/Desktop -arv /usr/home/myhome /media/disk/backup/usr/home

However, it still hangs after so many files are transferred - most of them cache files it seems. I tried using cp instead of rsync, but it does the same. I think maybe the USB drive is overheating with too many successive write operations.

When I kill the window, I need to use
Code:
 fsck -y /dev/da0s4
to fix the the partition before mounting the partition again after reboot.

Any thoughts?
 
I hope there's nothing wrong with your HDD/USB. Maybe large amount of small files! I clear my browser's cache folder, before executing any backup operation:
Code:
rm -fr ~/.mozilla/firefox/PROFILE/cache2
rm -fr ~/.cache/chromium
and always have an eye on ~/.cache. In my case, and it may not apply to your situation, and you should not blindly do the same things, I always clear the content of ~/.cache before each non-daily manual backups. And there are some minor consequences, such as having to rebuild the fontcache and I had to quit the X mode, because of existence of some sockets in .cache folder.
rsync --exclude=/myhome/Desktop -arv /usr/home/myhome /media/disk/backup/usr/home
-a is equivalent to -rlptgoD
You do not need to add '-r' option to rsync command. '-a' is enough.
 
I hope there's nothing wrong with your HDD/USB. Maybe large amount of small files! I clear my browser's cache folder, before executing any backup operation:
Code:
rm -fr ~/.mozilla/firefox/PROFILE/cache2
rm -fr ~/.cache/chromium
and always have an eye on ~/.cache. In my case, and it may not apply to your situation, and you should not blindly do the same things, I always clear the content of ~/.cache before each non-daily manual backups. And there are some minor consequences, such as having to rebuild the fontcache and I had to quit the X mode, because of existence of some sockets in .cache folder.

-a is equivalent to -rlptgoD
You do not need to add '-r' option to rsync command. '-a' is enough.
If you have /etc, and /root, and /usr/local/etc rsync'd do you really need /usr/home backed up? I mean if everything under /usr/home/myhome is either my-files that are backed up elsewhere anyway, and cache records, would the restored system be functional without them?

~/.cache is still stalling on me even when I rsync the sub-folders of /usr/home individually. I was hoping I might get away without it.

BTW, is restoring the system from the backup after a clean install simply a case of rsyncing the same directories but reversing the source and target destinations? Will rsyncing in reverse without a clean install repair most inadvertent damage to the system.
 
What about restic and syncthings ?
I use scp to backup «config» files and syncthings to get my smartphone photo.
I will test restic for my home dir.
 
be aware that rsync is a sync tool - if you have an important file and then you do echo "" > important.txt you end up with an empty file that gets synced to your backup. I have probably used _all_ major backup tools within the last 20 years to find the best tool (yeah, I am a little obsessed with backups), and I recommend borg backup in the first place, restic on the second. (keep your keys for your borg backup in a save place!)
 
be aware that rsync is a sync tool - if you have an important file and then you do echo "" > important.txt you end up with an empty file that gets synced to your backup.

That's written a little bit unclear: rsync doesn't blank your backup file untile you say "sync this file" - there is no automatic sync. And a sync is exact what most users want a backup to do: Please update my backup, so it is equal to waht I have on my harddrive. Also rsync knows some params with which you could avoid blanked files from being transfered. But: Overwriting a fine backup with accidental damaged files can happen. That's why I'm using svn for some software development even when I'm the only deveoper (rollback), two separated backup hardware and no automatic execution of a backup run.

Actually, there is no general recommendation for a backup, the scenarios are quite too different - even in this case for a "home user": A backup is more than a piece of software, it is a concept. But the following has proven more than once itself, and may be usefull for someone else here…:


1) Backup the package list:

pkg info > packages_all.txt
pkg query -e '%a = 0' %o > packages_base.txt
pkg_cutleaves -l > packages_nodeps.txt


The first contains all installed packages, the second contains those I have explicitly installed, and the third one contains only those packages that are not referenced by any other package; If I need to reinstall my computer I only need to reinstall the packages on the last list - and be done. The other two lists are more for control purposes. To use "pkg_cutleaves" you need to install it first.


2) Backup the system configuration:

tar cf - /etc/ | gzip -f9 > etc.tar.gz
tar cf - /usr/local/etc/ | gzip -f9 > usr_local_etc.tar.gz


For this you need root privileges; I recommend using tools like doas, super or sudo, so you can do such backups as user.


3) Backup databases:

If you are using databases like PostgreSQL, Mariadb etc. make a dump of each database. It makes sense to compress such a dump with "gzip" afterwards.


4) Backup user data:

"rsync" is the tool of choice for me. First I recommend to create a exclude file - because temporary and caching stuff will be automatically rebuilt, and thus it does not need to be backuped; Here is a little from my exclude file:

Code:
.cache/
cache/
Cache/
application_cache/
opcache/
.dbus
.fontconfig/
.fullcircle/
.thumbnails/
tmp/
lost+found/
.snap/
Downloads/

A typical rsync call to backup a users home would be:

rsync -avz --exclude-from=/path/to/excludefile --delete --safe-links /home/whoever/ /path/to/target/

I myself use the parameters "-rtDv" instead of "-avz"; A little reading in the man page of rsync does no harm if you want to use it ;)


Also the tool "nice" deservs a mention here, which can lower the priority of such a backup command (so my computers runs as fast as ever while doing backups).

And at least: Everything can be packed together into a small shell script, and your ready. Very important: At the beginning always check your backups, later still check them every now and then. And do not forget to check the file permissions in the backup.

However, when using USB drives the file permissions are lost to the "classic home user" (and often causes error messages during the backup); In my opinion a NAS which can handle SSH or NFS (and not only Samba) is preferable (build your own, and you win full control over its operating system, and maybe you can do this without a fan). If you do such a backup regularly it runs quite fast; And it doesn't matter that a cheap NAS doesn't process 100MB/sec, or that the WLAN is slower than the USB cable. Otherwise you have to reformat the USB drive so that it is able to understand things like "rw-r--r--".


"GUI" was also asked: I once wrote a small tool for myself to do backups, with which my surrounding is also happy - and it is of course able to handle the above stuff completely:


Called with the parameter "-d" it can also remind you if you have not made a backup for a longer time (configurable). But actually I should rework this tool…
 
I use rsync with good results, but two caveats: 1) it seems to like 16G memory at least
probably due to concurrency while backing up... 2) some files on the destination are identical
to newly backed up files, but ending in a tilde, like text.txt~, and remain past newer
backups, vs the expected behavior, being deleted,
making the backup area larger than the source. Not a problem in this
use case, but if someone knows if it is a switch rsync needs/doesn't need that I use
it would be useful to know.
 
Back
Top