A reliable backup strategy for a newbie

Maturin said:
which is synched every ten minutes, and you figure out:"F#c4! I accidently deleted that directory, half an hour ago
Click to expand...
Oh no! That is exactly the type of mistake I enjoy making on a regular basis. Well, maybe not on a regular basis, but I could see this disaster happening to me. For the moment, I will have to stick to methods I can understand.
 
Cedric62 said:
Does Midnight Commander have the ability to make a disk image of my entire / partition, and then store it on an external drive?
Click to expand...
I do that with the dd command. 1 disadvantage may be that it breaks on any failure.
dd if=<input device/partition> of=<output file> bs=100M
Make sure to read "man dd" and test a backup recovery procedure.
You can also recover specific content inside a backup file by loading it with mdconfig and mounting it somewhere to copy files from it.
 
I'm just wondering if the below set of commands will help me to repair my master boot record, in case something were to happen to it?

Code: 
mkdir -p /media/da0p1/backup/boot/efi/efi/freebsd
rsync -avh --progress /boot/efi/efi/freebsd/loader.efi /media/da0p1/backup/boot/efi/efi/freebsd
And then to restore:
Code: 
rsync -avh --progress /media/da0p1/backup/boot/efi/efi/freebsd/loader.efi /boot/efi/efi/freebsd

Last year I gave up on FreeBSD 14.1 because I accidentally destroyed my mbr when I tried to make FreeBSD dual boot with Ubuntu. I want to come up with a plan in case I make that mistake again. Below is what I have stored at
 

Attachments

  • efi_backup_example.jpg
    efi_backup_example.jpg
    64.3 KB · Views: 11
Cedric62 said:
At this time it looks like my backups are only going to be two or three gigabytes in size, so for the moment I don't have to worry about size, but I know that I will in the future when I adopt a better method.
Click to expand...
Yeah, that's a good point, you don't have to backup easily replaceable system files. Only your configuration is unique. /home and some system files along with pkg-list and you're done.
Cedric62 said:
Oh no! That is exactly the type of mistake I enjoy making on a regular basis. Well, maybe not on a regular basis, but I could see this disaster happening to me. For the moment, I will have to stick to methods I can understand.
Click to expand...
Yeah, that's where the snapshots come in. Even manually deleting a file would not delete it from a snapshot, it will remain in weekly and monthly snapshots, until it is cleared (I clear/prune my monthly snapshots after 6 months or a year? I don't even know). But to do snapshots properly, you'd have to think a bit in advance and prepare your filesystem accordingly. Complementarising the filesystem into different ZFS datasets and then setting different backup and comprimation strategies (for example) is beneficial. For example, I do not backup my /home/user/tmp and /home/user/films and /home/user/.cache directories, they are on a different dataset, which is set not to backup. There's a lot to play with and you end up with a system, which almost requires it's own documentation, because it's been years since you set it all up and it's quite a bit of work. Thankfully, FreeBSD rarely changes functionality, so it keeps on working even if you don't exactly remember how did you set it up (which I believe I'm in a minority here among mostly sysadmins, who know better).
 
Cedric62 said:
At this time it looks like my backups are only going to be two or three gigabytes in size
Click to expand...
My music library's the largest core files I restore around 2GB. My core stuff is some text files, password database, bookmarks file, user pic, and a couple wallpapers; I can do basically everything any-OS with just those, but the music's cool in-case I can't stream it :p
 
Cedric62 said:
my plan is to test my backup strategy
Click to expand...
That's exemplary.
All effort you spend on the most sophisticated backup plan is worthless, when you don't know how to restore your data from it - especially when you're under stress, which is very probable when the desaster strikes.
Cedric62 said:
own idiocy and fat fingering seem to be my biggest enemies.
Click to expand...
Anybody experienced with computers and backups will tell you: This will not change. 😁
Hence most primitive backups aka simple redundant copies (or snapshots) first, before you think of any desaster, or hackers.
Cedric62 said:
At this point I don't have any important data
Click to expand...
But one day you will.
Then you better have an already working backup plan you can rely on, because you tested it.
I see, you already doing it right:
First set up a system. Which is not simply done by installation and configuration, only, but by learning it, which means: Become not afraid to lose, and reinstall it. Because you learned a reinstallation is no big deal. And you have backups you can restore your config and data from.
Cedric62 said:
At this time it looks like my backups are only going to be two or three gigabytes in size, so for the moment I don't have to worry about size, but I know that I will in the future when I adopt a better method.
Click to expand...
Of course.
But you will also learn your method can be adapted easily to larger amounts of storage space if needed. That's a benefit of a backup plan you know, because you created yourself: you know how it works, and you can adapt it to changing needs.

My concern wasn't about size, but about hardware reliability.
Those USB thumb drives can become faulty if not complete toast suddenly, and very quickly.
Better use a real drive for that.
Cedric62 said:
This method is still slightly beyond my finger tips at the moment. I think that I understand the concept, but at this point I still don't know about how scripts work in FreeBSD. I can only guess that they are probably similar to batch files in MS DOS, and for the moment I'm stuck manually typing things into the terminal until I know more.
Click to expand...
If you know MS-DOS batch files, you already grasp the core idea of script files: write the commands you operate in the CLI one by one in a textfile, and let them be executed automatically. While unix-shells are older, and way more sophisticated, and by magnitudes more powerful than MS-DOS. :cool:
Basic idea is, if you realize you're repeating the same commands again and again, write a script.
There is no minimum length of lines a script has to have.

Example:
Copy the four "rsync"-lines from your post #15 in a textfile which you may name e.g. bu.sh, then, in the same directory, in the shell you do chmod +x bu.sh to make it executable, and simply type ./bu.sh. Voilá, all four commands are executed one by one. Your script runs.
Adding #!/bin/sh ("shebang") as first, and exit 0 as last line, you already have a real script. 😁
More sophisticated things like tests, variables, control flow (branches, loops) etc. come when you need them.

crontab -e opens your editor to edit your crontab.
Add a line @hourly /path/to/bu.sh, save and close the editor, and your script is automated, executed every hour.
This way you now only need to edit the script, add additional rsynch lines, or edit those, like e.g. add a -q to the options. Otherwise your mailbox will flooded with garbage. When some process produces terminal messages with cron those are send by an mail to the user. With (tested) cronjobs you may want to suppress such messages.
Just let it run in the background, and care about other topics.

As I said above, this was for sure not the perfect final solution, but it's a start you already have some BU running automatically at all, so no need to care about that topic until you realize you need to backup other/more things.
As long as you don't get a bottleneck I dare say with rsync (also check out the --exclude-from option), snapshots, simple sh scripts, cron, and maybe tar you can realize all your backup needs.
 
Cedric62 said:
I'm just wondering if the below set of commands will help me to repair my master boot record, in case something were to happen to it?
Click to expand...
No.
rsync works on filesystem's level, which in your example may save the file(s) from the efi partition, which is not the MBR. For this I simply would use cp - while it works though, rsync was a bit overkill to just copy single files. 😁
To deal with MBRs you need to do things some level deeper than filesystem. You can use dd (you find examples to save/restore MBRs with it). But on FreeBSD already in basic default you already have better, more sophisticated tools like gpart(8) to deal way better and less dangerous with such issues.
 
matt_k said:
But to do snapshots properly, you'd have to think a bit in advance and prepare your filesystem accordingly. Complementarising the filesystem into different ZFS datasets
Click to expand...
I think that is where I made one of my big mistakes during this installation. I picked UFS, and I also put everything within my root directory, instead of making separate volumes for everything. However, I'm really happy with the way that my installation is running right now, so maybe I should just live with it for the time being. There are a lot of things I don't understand yet, so I think that I have to go back and read chapter three of the handbook again.
 
Cedric62 said:
I think that is where I made one of my big mistakes during this installation. I picked UFS, and I also put everything within my root directory, instead of making separate volumes for everything. However, I'm really happy with the way that my installation is running right now, so maybe I should just live with it for the time being. There are a lot of things I don't understand yet, so I think that I have to go back and read chapter three of the handbook again.
Click to expand...
With ZFS you don't need to do any of that (no preparing, no partitions), you can add or delete datasets whenever and however you want.
 
Cedric62 said:
I think that is where I made one of my big mistakes during this installation. I picked UFS, and I also put everything within my root directory, instead of making separate volumes for everything. However, I'm really happy with the way that my installation is running right now, so maybe I should just live with it for the time being. There are a lot of things I don't understand yet, so I think that I have to go back and read chapter three of the handbook again.
Click to expand...
In that case, it might be worth reinstalling with ZFS if you have the means to shuffle the files. Having things like /tmp on the same filesystem as other things can increase the risk of corruption.
 
Maturin said:
Because you learned a reinstallation is no big deal
Click to expand...

Maturin said:
Copy the four "rsync"-lines from your post #15 in a textfile which you may name e.g. bu.sh, then, in the same directory, in the shell you do chmod +x bu.sh to make it executable, and simply type ./bu.sh. Voilá, all four commands are executed one by one. Your script runs.
Adding #!/bin/sh ("shebang") as first, and exit 0 as last line, you already have a real script. 😁
More sophisticated things like tests, variables, control flow (branches, loops) etc. come when you need them.
Click to expand...
Okay, that is starting to make sense. However, I really need to get a hold of some training material, and I need to go back and re read chapter three of the handbook. There are some basics regarding BSD which I don't have a full understanding yet.
 
matt_k said:
With ZFS you don't need to do any of that (no preparing, no partitions), you can add or delete datasets whenever and however you want.
Click to expand...
Oh how I wish that I would have picked ZFS during my installation process! But here is a quick question. What if I use ZFS while installing FreeBSD on another blank hard drive I have, and then then could I use the directories I backed up to my thumb drive using the rsync command, to then import my old settings into the new ZFS installation?
 
Cedric62 said:
I think that is where I made one of my big mistakes during this installation. I picked UFS, and I also put everything within my root directory, instead of making separate volumes for everything. However, I'm really happy with the way that my installation is running right now, so maybe I should just live with it for the time being. There are a lot of things I don't understand yet, so I think that I have to go back and read chapter three of the handbook again.
Click to expand...
You didn't make any mistake by choosing the UFS. In fact it's much it's preferred file system when you have hardware raid controller or when you use a single disk or virtual machine.

Anyway here's a little guide how to create a full back of the UFS using dump on locally attached dedicated hard disk or remotely over SSH. The second part will be how to test the backup for actual restore.

To be able to make a backup on the live (mounted) file system you need to disable the soft updates journaling using tunefs(8)
To do this you need to reboot into single user mode and then run
tunefs -j disable /
After that you can reboot again in multi user mode

When you make a full backup you can exclude some directories to save disk space. For example you don't need to backup the /usr/ports which can always be downloaded again after the restore of the backup.

To do this change the nodump flag of the directory usin chflags(1)
chflags nodump /usr/ports
When you are creating a full backup and want to include also the directories with nodump flag then you can use -h0 (honor level 0)

Now the actual backup process:
First you need to have a print copy of the content of the following:
gpart show and /etc/fstab
This will help you when you are restoring the system on a new hard disk or recovering the information from the damaged disk with missing partition table as it's much easy when you know the exact partition geometry and it's super blocks.

Here's the example of disk da0 (RAID 5) with FreeBSD and disk da1(RAID1) on the same system for backups.

Code: 
camcontrol devlist
<HP RAID 5 OK>                     at scbus0 target 0 lun 0 (pass0,da0)
<HP RAID 1(1+0) OK>                at scbus0 target 1 lun 0 (pass1,da1)
<AHCI SGPIO Enclosure 2.00 0001>   at scbus6 target 0 lun 0 (ses0,pass2)
<AHCI SGPIO Enclosure 2.00 0001>   at scbus13 target 0 lun 0 (ses1,pass3)

Code: 
=>        40  1757614608  da0  GPT  (838G)
          40      409600    1  efi  (200M)
      409640   104857600    2  freebsd-ufs  (50G)
   105267240  1635778560    3  freebsd-ufs  (780G)
  1741045800    16568320    4  freebsd-swap  (7.9G)
  1757614120         528       - free -  (264K)

=>        40  1953459552  da1  GPT  (931G)
          40  1953459552    1  freebsd-ufs  (931G)

The full backup of the disk da0 is done to the locally mounted /dev/da1 into /backup using the dump(8) command and compressing the backup on the fly with gzip and replacing the root.date.dump.gz with your current date/time name for example root.20260202.dump.gz

dump -C16 -b64 -0uanL -h0 -f - / | /usr/bin/gzip > /backups/root.date.dump.gz

Or if you want to make a backup to remote system over SSH (port 2222)


/sbin/dump -C16 -b64 -0Lau -h0 -f - / | gzip | ssh -p 2222 user@example.com dd of=/home/user/root.date.dump.gz


To browse/extract or restore the archive to a new hard disk you can use restore(8)

For interactive restore/extract you can use "-i"
zcat /backups/root.dump.gz | restore -ivf -

Code: 
Available commands are:
        ls [arg] - list directory
        cd arg - change directory
        pwd - print current directory
        add [arg] - add `arg' to list of files to be extracted
        delete [arg] - delete `arg' from list of files to be extracted
        extract - extract requested files
        setmodes - set modes of requested directories
        quit - immediately exit program
        what - list dump header information
        verbose - toggle verbose flag (useful with ``ls'')
        help or `?' - print this list
If no `arg' is supplied, the current directory is used

For full restore to a new hard disk you need to format it first using bootable live cd like following for UEFI:

let say your new disk is /dev/da0 and your backup disk (/dev/da1) is mounted in /media on the live cd (usb)

First you need to create the partition scheme, ESP partition and the empty root partition on which you are going to restore the backup. The same procedure can be use when you are migrating from a small disk to a larger disk.

Code: 
gpart destroy -F da0
gpart create -s gpt da0
gpart add -t efi -s 300M da0
newfs_msdos -F 32 -c 1 /dev/da0p1
mkdir /tmp/ESP
mount -t msdosfs /dev/da0p1 /tmp/ESP/
mkdir -p /tmp/ESP/EFI/BOOT
cp /boot/loader.efi /tmp/ESP/EFI/BOOT/BOOTx64.efi
umount /tmp/ESP
gpart add -t freebsd-ufs -s 50G da0
gpart add -t freebsd-ufs -s 5G da0
gpart add -t freebsd-swap -s 4G da0
newfs -U /dev/da0p2
newfs -U /dev/da0p3
mkdir /tmp/root
mkdir /tmp/temp
mount -o rw /dev/da0p2 /tmp/root
mount -o rw /dev/da0p3 /tmp/temp
setenv TMPDIR /tmp/temp
cd /tmp/root
zcat /media/root.dump.gz | restore -rvf -
rm /tmp/root/restoresymtable

And before the reboot check and edit the fstab of the restored system

ee /tmp/root/etc/fstab

Then create the UEFI boot variable using

efibootmgr --create --activate --label "FreeBSD" --loader /tmp/ESP/EFI/BOOT/BOOTx64.efi


Then you can reboot.
 
Cedric62 said:
Oh how I wish that I would have picked ZFS during my installation process! But here is a quick question. What if I use ZFS while installing FreeBSD on another blank hard drive I have, and then then could I use the directories I backed up to my thumb drive using the rsync command, to then import my old settings into the new ZFS installation?
Click to expand...

Yes.

In fact that is a good test of your backup procedure.
 
Cedric62 said:
Will the partitions on the ZFS drive have to be the same size as the ones on the UFS drive?
Click to expand...
It's sort of a meaningless question. ZFS by default doesn't really care about how the files are distributed within the pool so long as the total doesn't exceed limits. It will keep track of how much is allocated to what and you won't really care unless you either run out of space or hit whatever limits you choose to put in place.

It is worth noting that since ZFS is a Copy On Write filesystem with no ability to defrag, that it's a good idea to ensure that you have a good chunk empty at all times or file operations can bog down to the extreme. I'm not sure what the current recommendations are, but when I got started the recommendation was to leave roughly 20% unused as a minimum free space.
 
Maturin said:
Copy the four "rsync"-lines from your post #15 in a textfile which you may name e.g. bu.sh, then, in the same directory, in the shell you do chmod +x bu.sh to make it executable, and simply type ./bu.sh. Voilá, all four commands are executed one by one. Your script runs.
Click to expand...
I think that I almost have it.
 
You must log in or register to reply here.
Back
Top