A reliable backup strategy for a newbie

C

Cedric62

Well, I'm the sort of newbie who seems to enjoy breaking his own installation of FreeBSD, and then spending the next few hours making it work again. While I've learned a few things from doing this over and over, I've decided that it's now a good time to come up with reliable strategy for recovering from my own mistakes, that way I can hopefully spend my time learning more productive things with FreeBSD.

At anyrate, I came across the below thread, which seems to provide a backup scenario which someone at my skill level should be able to handle. Also, I've read about using mksnap_ffs to create snapshots, but the process of using the snapshots to recover missing or corrupted data seems a little bit murky to me at this point, so I'm leaning towards the method outlined in the below link.

B

Thread 'How to do a full system backup of freebsd so i can boot from it if my current system fails'

Hi I'm very new to FreeBSD. Anyone know how I would go about doing a full system backup of FreeBSD that I could keep to boot from and restore my system in case my install gets corrupted?
I have been looking for what utilities to use and the steps to use them to create an exact replica of my FreeBSD system in case I need to restore to new hardware.
I have the "data" backed up already by other means, but I would like to know if there is a way to create an image of the current system that could be restore in the event of a disaster.
Disk space is not a concern, I have terabytes of disk space...

However, there is something I don't understand about the below set of commands, which I've copied and pasted from the above thread.

Code: 
mkdir -p /backup/boot
mkdir -p /backup/usr/local
rsync -a /boot/loader.conf /backup/boot
rsync -a /etc /backup
rsync -a /usr/local/etc /backup/usr/local
rsync -a /usr/home /backup/usr

It appears that the above set of commands will create back up directories for /boot and /usr, but how come there does not appear to be a command listed to create a back up directory for /etc? Also, would the above commands referencing /usr automatically create a back of the files located under /usr/local/etc/X11/xorg.conf.d/nvidia-conf?
 
If you just want to guard against fatfingering something then the easiest is to install on ZFS and pick one of the automatic snapshot makers from ports.

Learning rsync will pay off for you. It can be intimidating reading the manpage, though. But most options you don't need.
 
cracauer@ said:
If you just want to guard against fatfingering something then the easiest is to install on ZFS and pick one of the automatic snapshot makers from ports.

Learning rsync will pay off for you. It can be intimidating reading the manpage, though. But most options you don't need.
Click to expand...
I regred not picking ZFS during the installation process, and I now realize that I've kind of painted myself into a corner by going with UFS. I will probably try installing FreeBSD using ZFS on another old hard drive I have sitting on the shelf, and see how that works out. I know that this may sound foolish, but I feel that I have something invested into my current installation using UFS, and even though I picked the wrong file system, I feel like that I want to learn as much as I can from it. Maybe I'm wrong, maybe I should scrap it and move onto ZFS, but I will try UFS a while longer.
 
Cedric62 said:
I regred not picking ZFS during the installation process, and I now realize that I've kind of painted myself into a corner by going with UFS. I will probably try installing FreeBSD using ZFS on another old hard drive I have sitting on the shelf, and see how that works out. I know that this may sound foolish, but I feel that I have something invested into my current installation using UFS, and even though I picked the wrong file system, I feel like that I want to learn as much as I can from it. Maybe I'm wrong, maybe I should scrap it and move onto ZFS, but I will try UFS a while longer.
Click to expand...

It's not foolish at all to stick to some installation and make it do what you want. It is how you learn. Nothing is learned from constantly reinstalling on the first difficulty.

Learn rsync is my recommendation.
 
UFS is definitely not a "bad" option so there's nothing "wrong" in sticking with it for now. It's worth looking at ZFS, definitely, but the world isn't going to end because you are using UFS.

And +1 to rsync.

Your questions above - the rsync of the etc directory will create the sub-directory in backup - so you'll end up with /backup/etc. Just like it doesn't create /backup/usr/local/etc - the commands create /backup/usr/local, and the rsync will make /backup/usr/local/etc. And yes, rsync with the -a flag is recursive, so it will walk directories.

Just give it a whirl, especially if this is a test system and you are just learning.
 
I'm nowhere near a power user or a server admin, I just use FreeBSD on a desktop computer at home. My setup consists of zfs mirror of both / and /home, so I have 4 physical disks and one external disk, which receives automatic backups, with daily, weekly, monthly snapshots. I am using syncoid via my own backup script and zfs-auto-snapshot functionality via cron to achieve that.
I don't remember who is the original author of that solution, definitely not me though, so if you want any additional info, I will try to answer.
 
Frankly, without zfs and its boot environments, any upgrade would be hell for me. You never know what is going to break, even if you can predict some and take suited countermeasures.

It is the same when I break myself the system by ignorance. When I can't repair, I return to the last boot environment.
 
matt_k said:
I'm nowhere near a power user or a server admin, I just use FreeBSD on a desktop computer at home. My setup consists of zfs mirror of both / and /home, so I have 4 physical disks and one external disk, which receives automatic backups, with daily, weekly, monthly snapshots. I am using syncoid via my own backup script and zfs-auto-snapshot functionality via cron to achieve that.
I don't remember who is the original author of that solution, definitely not me though, so if you want any additional info, I will try to answer.
Click to expand...
Maybe down the road, but for the moment I think that I will stick with the above method which at the moment seems better than nothing. I have to admit that I really don't know very much about disk mirroring, but will my bios have to be RAID compatible if order for me to try what you're describing?
 
bakul said:
https://imil.net/blog/posts/2016/migrate-freebsd-root-on-ufs-to-zfs/ describes an example of how to migrate a ufs root to zfs. Once the conversion is complete and verified, you can reuse the old disk as another disk in a zfs mirror (provided sizes match).
Click to expand...
Wow! Very interesting, and probably much better than doing a fresh reinstall. So, I would essentially be cloning my existing UFS hard drive to a second drive, which has already been prepared with compatible ZFS partitions. I actually think that I might try this in another week or two, and I just happened to have a pair of two gigabyte Seagate barracuda drives which are not doing anything but collection dust. But first I want to understand how this operating system work a little better before I try that.
 
Well, the process of using rsync -a to back up certain directories seemed to go very well, but I had to make a minor adjustment for my home directory. See below.

Code: 
mkdir -p /backup/boot
mkdir -p /backup/usr/local
mkdir -p /backup/home  (New line)
 
rsync -a /boot/loader.conf /backup/boot
rsync -a /etc /backup
rsync -a /usr/local/etc /backup/usr/local
rsync -a /usr/home /backup/usr  (This line does not work)
rsync -a /home /backup/home  (New line)

Also, below is a screen shot of what I actually ended up with in /backup


backup.jpg


Also, the size of my backup folder ended being something like 2.9 GiB, which is way larger than I had expected.

However, now I'm wondering how I can transfer my backup directory to a USB thumb drive, and have it readable to a fresh installation of FreeBSD which doesn't yet have a desktop installed on it. Assuming that my USB thumb drive is formatted as Fat 32, could I use the below command to mount it?

Makefile: 
mount /dev/da0p1 /media/usb

Assuming that I'm able to read the above USB in FreeBSD, could I then use a command similar to the below to restore my directories?

Code: 
rsync -a /media/usb/backup/boot/loader.conf /boot

Also, I tried formatting a USB thumb drive with the UFS file system, but somehow that is beyond me at just the moment.

Any advice greatly appreciated.
 
matt_k said:
I'm nowhere near a power user or a server admin, I just use FreeBSD on a desktop computer at home. My setup consists of zfs mirror of both / and /home, so I have 4 physical disks and one external disk, which receives automatic backups, with daily, weekly, monthly snapshots. I am using syncoid via my own backup script and zfs-auto-snapshot functionality via cron to achieve that.
I don't remember who is the original author of that solution, definitely not me though, so if you want any additional info, I will try to answer.
Click to expand...
You sound like a pretty good power-user/sysadmin to me with that!

cracauer@ said:
It's not foolish at all to stick to some installation and make it do what you want. It is how you learn. Nothing is learned from constantly reinstalling on the first difficulty.

Learn rsync is my recommendation.
Click to expand...
Reinstalls for me teach me what not to change on the next install, leading to a perfect set-up!

I totally did a UFS format at first, got into it, then wiped on a whim to test ZFS :p (enough reinstalls gave me the experience to be able to get back up from wiped-disks in about 20 mins)

Not sure what I'd recommend for a back-up strategy, but one idea is to not be concerned with one until you know or wished you had a back-up of something :p (to have a better idea as to what's crucial to back-up vs mass-backing up everything)

I back-up a password database and game saves mainly, but my OS config is on plaintext notes. If I lost loader.conf, I have it noted! I'd rather re-create my set-up vs restoring it from back-up.
 
Use caution with rsync -a. If used by a normal user, it doesn't get permissions quite right.

Having spent quite a while trying to get rsync to do everything exactly the way I want, here are my options for its use. I'm just showing the options I use here, not providing a complete script:

# --dry-run, -n (very useful)
#
# --recursive, -r
# --mkpath create destination's missing path components
# --links, -l copy symlinks as symlinks
# --hard-links, H preserve hard links
# --perms, -p preserve permissions
# --owner, -o preserve owner (super-user only)
# --group, -g preserve group
# --acls, -A preserve ACLs (implies --perms)
# --xattrs, -X preserve extended attributes
# --fileflags preserve file-flags (aka chflags)
# --devices preserve device files (super-user only)
# --specials preserve special files
# --times, -t preserve modfication times
# --atimes, -U preserve access (use) times
# NOT SUPPORTED: --crtimes, -N preserve create times (newness)
# --compress, -z compress file data during transfer
# --exclude=".cache/" exclude files matching pattern
# --itemize-changes, -i ouput a change-summary for all updates
# --super receiver attempts super-user activities
# --verbose, -v be verbose
# --checksum, -c skip based on checksum, not mod-time & size
# --cvs-exclude, -C auto-ignore files matching pattern - list below
# RCS SCCS CVS CVS.adm RCSLOG cvslog.* tags TAGS .make.state
# .nse_depinfo *~ #* .#* ,* _$* *$ *.old *.bak *.BAK *.orig
# *.rej .del-* *.a *.olb *.o *.obj *.so *.exe *.Z *.elc *.ln
# core .svn/ .git/ .hg/ .bzr/


# Note: the 'c' (--checksum) option requests that the transfer be based on a checksum
# not on modification time and size. This is needed because the
# chown command above changes the modification time. Without 'c'
# all files get transferred every time.
#
# Use the 'i' (--itemize-changes) to debug what gets sent and what
# does not get sent.
# See that section in the man page for descriptions of the various flags.
#
# cd++++++++++ jpb/
# <f++++++++++ jpb/.ICEauthority
# <f++++++++++ jpb/.Xauthority
# ...
# etc.
#
# READ THIS NOTE:
# If you are testing, you must allow rsync to completely finish as certain operations
# such as ownership and group assignments only complete when rsync finishes a recursive
# copy to the remote system. There - I just saved you 5 hours of headbanging.
#
# You're welcome.
#
----------------------------

Best current practice for backups is to follow the 3-2-1 rule:

3 backups, on 2 different media, with 1 copy offsite.

In the corporate world, "offsite" is usually specified at least 300 miles (480 kilometers) away. If it's just for yourself, send a copy to your uncle Bob.

-----------

If you are doing software development using git or another source code control tool, do not use --cvs-exclude (-C) option. You actually do want all those source code history files to transfer.

Enjoy - Jim B.
 
Well, I finally figured out how to format a USB thumb drive with UFS, after that, I was able to use the below commands to copy some of my settings to the USB I had recently formatted.

Copy settings and home directory:
Code: 
mkdir -p /media/da0p1/backup
pkg prime-list > /media/da0p1/backup/pkglist
mkdir -p /media/da0p1/backup/boot
mkdir -p /media/da0p1/backup/usr/local
mkdir -p /media/da0p1/backup/home
mkdir -p /media/da0p1/backup/etc

rsync -avh --progress /boot/loader.conf /media/da0p1/backup/boot
rsync -avh --progress /usr/local/etc /media/da0p1/backup/usr/local
rsync -avh --progress /home /media/da0p1/backup/
rsync -avh --progress /etc /media/da0p1/backup/


rysnc_usb.jpg


Above is a screenshot of what I ended up with on my USB after carrying out the above commands.

So, now that I have the above directories stored on my USB, I'm planning on using the below commands to restore them to my hard drive, if I need to. However, I'm not certain that I'm using the below commands correctly, and I welcome any feedback regarding any mistakes I may have made. See below:


Code: 
rsync -avh --progress /media/da0p1/backup/boot/loader.conf /boot
rsync -avh --progress /media/da0p1/backup/usr/local/etc /usr/local
rsync -avh --progress /media/da0p1/backup/home /
rsync -avh --progress /media/da0p1/backup/etc /
cat pkglist | xargs pkg /media/da0p1/backup/install

Any feedback greatly appreciated.
 
UFS is not bad,in that case
maybe for only your home directory using rsync in a cron job every x minutes to copy your home folder to another location is fine
if only are the essential files (loader.conf , rc.conf , sysctl.conf,etc) , rsync works fine too
if you looking for full OS backups need more elaborated solution
 
Cedric62 said:
Maybe down the road, but for the moment I think that I will stick with the above method which at the moment seems better than nothing. I have to admit that I really don't know very much about disk mirroring, but will my bios have to be RAID compatible if order for me to try what you're describing?
Click to expand...
You want your BIOS not to mess around with your disks. If it even has RAID capabilities, you want to switch it to "JBoD" or "Just a Bunch of Disks", so it does not do anything with those disks .
The mirroring will be managed entirely by ZFS. If you have more disks, you may also try different solutions, like RAIDZ1, RAIDZ2, etc. It works reliably and is easy to get up and running.

Not related, but you are free to format your USB thumb drive to FreeBSD UFS, you don't have to use FAT32. If you use FAT32, you will be limited by FAT32's drawbacks, so when you copy files back from it, the longer file names will be truncated, weird symbols in file names would be preserved not consistently, permissions and ownership of files will not be preserved, etc.
 
Backups are the very most important task to be set up first on any system before you actually start to invest time. While time means anything you create productively with your computer(s): creating/editing data: texts, sourcecodes, spreadsheets, pictures, music... - files, and configuring your system, which again also means files, config files.
Anything you can simply copy from another existing source (e.g. the OS itself) does not neccasserily needed to be backupped, because it ain't no loss when it's gone, because it can be reproduced identically anytime effortless with trivial procedures.
So, what a backup does, is to make not trivial reproducebale files to be quickly trivial restored.

As clear as this is, unfortunately a realization of a backup plan is something very individual, and not trivial, because questions need answered like:
🔎 From what do you want to be protected?
own idiocy (#1 reason for to restore from backups) and fatfingering 👉 redundant copy of files
hardware failure (can happen anytime, even on new HW) 👉 redundant hardware
external influence (malware, burglary, desaster) 👉 redundant copies on redundant hardware at another location (encryption?)
🔎 What amount of data shall be stored at which storage location of what size, frequency (which data changes how often? [in fact you only need to update your backups to changes made]), and what is the bandwidth [B/s] of the channel(s) (SATA port? slow internet?)
Which brings additional questions like compression, incremental backups.

Also it's not done by realizing one backup plan. To test, reconsider, revalidate, and adapt the backup plan to your changing system and growing experience is a constant task accompanying you computers maintenence work, like updating, or cleaning up.
Everytime when you face a situation when you may lose, or actually lost data, and the restoration of your system/data ain't no trivial task simply done by ordinary routine procedures, ("could be automated somehow"), it's a sure sign for your backup plan is not sufficient, and needs to be adapted (urgently).
Doing backups is a process being learned.

My advice was to start by attaching an additional storage device to your machine, and write a small sh script, which does a simple
rsynch -auq --delete /home/myhome /newstorage/bu/ (to keep an exact copy of a directory) of your /home/, /etc/, and maybe other directories to that drive, and let cron execute this periodically.
This may be not the final solution (as I said, it needs to be revised every now and then.) But it was some start, and you have at least some backup until you worked out, and realized a better, more detailed bu plan suiting your needs better, like maybe for example adding snapshots, compression, encryption, additional redundancy backups - backing up backups (*.tar.gz) - whatever.
 
Cedric62 said:
a USB thumb drive
Click to expand...
I wouldn't trust a USB flashdrive to be a reliable backup storage, especially not, if periodic writings are done to it (rsynch by cron). Those things can wear out quickly. Particulary cheap ones are not made for constant usage.
I highly recommend you better use a real HDD/SSD for your backups.
A trap with automated backups is the automation:
Once set up, you don't care about it anymore, you don't check, and forget, and when you find out, your backup storage device is toast first when you need it to restore from it... that's bad.
 
It depends on your requirements.

Fear of file loss by erasure
Fear of file malware or encryption
Fear of physical device theft or loss.

For me it was accidental erasure.
I no longer use a mirror or raid, which replicates the erasure to other drives.

I wrote a one way push from drive to drive which ensures an erasure on Source is not propagated to Destination.

This results in clutter as files change directory locations. I wrote a separate utility to clean up older locations when superseded by newer locations when files are identical.
 
bgavin said:
For me it was accidental erasure.
Click to expand...
That's #1 reason to restore from backups.
bgavin said:
I no longer use a mirror or raid, which replicates the erasure to other drives.
Click to expand...
Mirrors or raid do protect you from drive failures.
And of course you need to check out, how you do your backups, and how often.
If for example you have only one copy of your /home/, which is synched every ten minutes, and you figure out:"F#c4! I accidently deleted that directory, half an hour ago" then it will be deleted in your backup, too. Which e.g. does the --delete option in rsynch I mentioned above: to delete files in the target you deleted in your source. Without it, also deleted files stay in the synched target directory. But then sooner or later your backup location drowns in old garbage - which - not finding a file anymore in a heap of garbage - can be as worse as having it deleted.
One way was to do redundant backups: Do backups of your backup dir (tar it for example.) Before you update it to the current state of your /home/ (or whatever dir we are talking), you pack the current one into a (compressed) tar. Doing a loop, which renames all former tar-files, something likle that:
Code: 
for (i=n_maxBU, i=0, i--)
    mv bu_(i-1).tar bu_(i).tar

tar /backupdir to bu_0.tar
backup /home... to /backupdir
This way provides you with as many old versions, as you like, for example 10, or 100. So, your deleted file will be found in one of the older tar-files.

Additionally, not only but above all, when we talk source code, or script files, version control is highly recommened. Which will provide also a source to bring you back deleted files, and former editing states.
 
If it only has to be reliable at home, I can recommend a random laptop, 1 or 2 USB SATA disk readers, a bunch of loose harddisks of 1-3 GB and a non-permanent textmarker. Most reliable disks are 5400RPM Seagates to my experiemce.
For making the backup itself, I use midnight commander.
 
matt_k said:
you are free to format your USB thumb drive to FreeBSD UFS
Click to expand...
I finally figured how to format a USB thumb drive with UFS, I've been able to copy some of my important system directories to that USB, but now I'm wondering if a fresh installation of FreeBSD will have permission to even read it?

So, my plan is to test my backup strategy by doing a fresh install of Freebsd on another bare hard drive, which I have sitting on a shelf, and then to use the command, rsync -avh --progress, to put all of my settings into the new installation. I'm also planning to use the below command to reinstall all of my packages. Also, I do have a file named install located at the below specified location.
Code: 
cat pkglist | xargs pkg /media/da0p1/backup/install

I know that this if far from having a backup of the entire operating system, for someone like me, who is still very new to FreeBSD, this might be a workable solution to counter some of my fat-fingering mistakes.
 
Maturin said:
own idiocy (#1 reason for to restore from backups) and fatfingering
Click to expand...
At this point my own idiocy and fat fingering seem to be my biggest enemies. At this point I don't have any important data such as spread sheets, documents, or photos to worry about, so I'm mainly concerned about making my system unstable by experimenting too much. Earlier I made my system unstable by installing the wrong video drivers, and each time I made such a mistake it took me a several hours to figure out what was wrong, and how to fix it. I want to try to minimize the amount of time I spend fixing my own mistakes, and spend more time using and learning about the FreeBSD operating system. At this time it looks like my backups are only going to be two or three gigabytes in size, so for the moment I don't have to worry about size, but I know that I will in the future when I adopt a better method.

Maturin said:
My advice was to start by attaching an additional storage device to your machine, and write a small sh script, which does a simple
rsynch -auq --delete /home/myhome /newstorage/bu/ (to keep an exact copy of a directory) of your /home/, /etc/, and maybe other directories to that drive, and let cron execute this
Click to expand...
This method is still slightly beyond my finger tips at the moment. I think that I understand the concept, but at this point I still don't know about how scripts work in FreeBSD. I can only guess that they are probably similar to batch files in MS DOS, and for the moment I'm stuck manually typing things into the terminal until I know more. I still have a lot to learn, but I'm making progress.
 
You must log in or register to reply here.
Back
Top