• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

What to backup?

dpx

Member

Thanks: 4
Messages: 58

#1
My hourly backup uses https://github.com/laurent22/rsync-time-backup and it works really great, keeping hourly backups for 24 hours, daily backups for the last week and monthly backups as long as there is disk space. Inspired by time machine on Mac so worth checking out if you need something like that.

Anyway, coming from Linux I am backing up /home and /etc. Tweaking FreeBSD I can see some settings are in /boot and /usr/local/etc. I am trying to complete the list of 'important' folders that will help me maintain reasonably usable backup of all the system settings that I may tweak now and in the future.

Are there other folders that should be included?
 

tman904

New Member

Thanks: 2
Messages: 8

#2
Hello,

The easiest way I can think of is using the dump and restore commands.

Check these man pages.

dump
https://www.freebsd.org/cgi/man.cgi?dump(8)

restore
https://www.freebsd.org/cgi/man.cgi?query=restore&sektion=&n=1

After using the dump command you should be able to backup the resulting dump file with the rsync command you referenced above.

If you have to do disaster recovery you can grab the backup file with rsync and then use the restore command to recover the system state. The only downside is that you will only have the data that was on the system when the last snapshot was taken. Meaning any files made after won't be included in the snapshot.

Make sure the dump file isn't just stored on the same drive it was made from. I don't mean that in a mean way it's just that I've made that mistake a few times lol:) I figured that tip would save some hair pulling:).

I'm not quite sure if this is what you were looking for but hopefully it helps.
 

ralphbsz

Daemon

Thanks: 651
Messages: 1,114

#3
When you restore after a catastrophic failure, do you want to:
  1. Install a distribution version of the OS onto a blank disk, perform all your configurations and tunings, then restore just the user data? Fundamentally ending up with a new install, just the same user files?
  2. Install a distribution version of the OS onto a blank disk, the restore your backup, and have all your configurations and tunings of the OS install also ready, but the system has no history (it thinks it was just installed), and port management and updates of the new system may be wonky?
  3. Like 2, except the system has the history of how it got there, and port management and updates of the new system continue working as if nothing had happened?
  4. Copy stuff from the backup to a blank disk, boot, and go, and you are fundamentally on the same old system just with a different disk drive?
  5. Just put the backup disk into a new server, boot, and go?
Let's talk about these levels of goals in order, since 1 is the easiest, and 5 the hardest. My personal goal is somewhere between 1 and 2.

To begin with, I keep a complete log of all administrative actions, in a set of files that are stored in /root (the user "root"'s home directory). I'm organized enough, whenever sudo or su, I type it into my personal log. This makes option 1 pretty easy: Just install a fresh FreeBSD, do the same configuration steps, restore user data, go. For this reason, I back up /root in addition to /home (which contains *all* home directories, none of that /usr/home stuff).

I also back up directories /etc and /usr/local/etc. That would allow me to check that I got all the configurations right: just compare the resulting files with what was backed up.

I *think* (never actually had to try it), if you back up /etc/ and /usr/local/etc and restore them, that option 2 would actually work, if you installed exactly the same OS version, and installed exactly the same ports or packages. It sounds extremely dicey though. There are two problems: First, you have to get exactly the right set of packages installed: having a saved copy of /usr/local/etc/foo.conf doesn't help if the new system doesn't have the foo package installed. Second, there is problems with versions and prerequisites: /usr/local/etc/foo.conf for foo version X may not work well with foo version Y. The feat of these problems is what prevents me from aiming for goal 2 or 3.

There is another problem: If your machine is a server, it may have things spooled, like mail and print jobs that have been spooled and not printed yet. If that's true, you also need to backup /var/mail, /var/spool, and friends. Furthermore, if you want to continue upgrading the restored OS, you need to take care of package management, which requires at least /var/db/pkg, /var/db/ports, /var/db/freebsd-update, and such things. Furthermore, after a restore you might want to see the history of the system before the disk crashed, so you also need /var/log. At this point, you might as well back up all of /var (perhaps skipping known cache and crash dump directories). That might in theory get you to goal 3. Good luck, you'll need it.

I know that goals 4 and 5 are achievable (I know quite a few people who have worked on products that enable this kind of stuff), but I don't know how to do it myself, and I can't be bothered to find out.
 

Datapanic

Active Member

Thanks: 123
Messages: 227

#4
What to back up, you want to know? Well, delete everything and then restore it from backups you already have. That Plus What's missing is "what to backup" It's that simple!
 

hedwards

Active Member

Thanks: 21
Messages: 207

#5
My hourly backup uses https://github.com/laurent22/rsync-time-backup and it works really great, keeping hourly backups for 24 hours, daily backups for the last week and monthly backups as long as there is disk space. Inspired by time machine on Mac so worth checking out if you need something like that.

Anyway, coming from Linux I am backing up /home and /etc. Tweaking FreeBSD I can see some settings are in /boot and /usr/local/etc. I am trying to complete the list of 'important' folders that will help me maintain reasonably usable backup of all the system settings that I may tweak now and in the future.

Are there other folders that should be included?
That's more or less the bare minimum. It's also a good idea to have a list of all the ports that you've installed and any packages that you've compiled to get the restore done quickly.

Make sure to backup your encryption keys if you're using an encrypted filesystem, those tend to be in /boot and somewhere in /var, IIRC. The last thing you want to find is that you can't access your encrypted filesystem and have to guess whether or not your backups got all the files you wanted backed up.

But, it all depends on what you're trying to protect against. If in doubt, err on the side of including things you don't need rather than excluding things you don't. It's far easier than worrying about whether or not you've included anything.

I'd go a step further and make friends with mtree for verifying that there hasn't been any file corruption since the files were last touched.
 

dpx

Member

Thanks: 4
Messages: 58

#6
Install a distribution version of the OS onto a blank disk, perform all your configurations and tunings, then restore just the user data? Fundamentally ending up with a new install, just the same user files?
Basically this, I am backing up local desktop machine so if anything happens I would probably restore /home with all the files I have in backup. I would keep /etc and similar for reference, not for automatic restore. Since I noticed FreeBSD has 'configuration' files elsewhere too (like /boot) I am curious where are all the other locations that may contain files that I may need to tweak in the future. Does this make sense?

This all works for local desktop, I would approach backup of server in a entirely different way, probably using sysutils/duplicity and moving files to other server or encrypted S3 bucket.

So, if I backup
/home
/etc
/boot
/usr/local/etc
is there anything else I should backup 'for reference', for local desktop machine?
 

metsuke

Member

Thanks: 4
Messages: 35

#7
FWIW, I back up the same directories, even on my server:
/etc
/usr/local/etc
/boot
/data (where all non-config data resides)

If you are using ZFS (and why wouldn't you be?) then you can ZFS send to a tar.gz dump and upload that somewhere.
I subscribe to CrashPlan for the great price, unlimited storage, unlimited versioning, and zero-knowledge encryption. I use a samba share to mount all my data and backups to a VM on bhyve that has CrashPlan running 24/7. I've restored one of my ZFS dataset's data very quickly with no hiccups.
 

dpx

Member

Thanks: 4
Messages: 58

#9
On server, yes, I would probably backup almost everything and exclude what's not needed. On desktop I only pick what's needed and even those have more granular exclude rules for each individual included folder. Since rsync-time-backup uses rsync it follows all the exclude rules I set, for all the included folders.

For example here is my exclude file for rsync-time-backup backed up /home folder:
- .local/
- .cache/
- .mozilla/
- Downloads/
- VirtualBox*
- Vagrant/
- Wine/
- firefox.core

Firefox latest crashes a lot so I don't want core dumps in my backup for instance.
 

dpx

Member

Thanks: 4
Messages: 58

#10
That is holding the Firefox profile and for sure you do not want to miss that one completely.
I don't need it at all. Most of my browsing in in w3m and only sometimes in firefox if I have to see something. I don't keep history or bookmarks, logging in every time I need to.

I'd do *.core, but as I almost never find firefox.core I suggest find the reason for this.
How?
 

Sensucht94

Well-Known Member

Thanks: 304
Messages: 334

#11
My hourly backup uses https://github.com/laurent22/rsync-time-backup and it works really great
rsync is great but making snapshots is nonetheless. I'm used to relying on both rsync and snapshots at the same time (and possibly cloning file system once every 6 months). If you're on UFS, see mksnap_ffs(8), as well as 17.10 File System Snapshots.

ZFS is well known for full-featured and easy to use and manage FS snapshots support, though I've never tried it

Tweaking FreeBSD I can see some settings are in /boot and /usr/local/etc
Every 3rd party application's data file and configuration (so almost everything you install from ports'tree, which does not belomg to base system) is stored either in /usr/local/share or /usr/local/etc respectively.. As opposite since in Linux everything is 3rd party, whichever software you install shall be stored inside /usr/etc+/usr/share


Are there other folders that should be included?
I'd backup /var/db too, especially /var/db/pkg ;)
 

ralphbsz

Daemon

Thanks: 651
Messages: 1,114

#12
As I said, I would add /root (which is like /home, but for the root user), and I would add either all or or most of /var.

I still deliberately exclude certain directories in /home, namely ripped copies of DVD movies. But with disk capacities getting very large for home users, the whole question of what to include and exclude is becoming more silly: for a few hundred dollars, one can get a 10TB disk drive now; that's cheaper than investing a lot of effort into trying to squeeze the backup into smaller drives. On the other hand, bandwidth has not been increasing, so backups may take longer.
 

Snurg

Aspiring Daemon

Thanks: 325
Messages: 794

#13
3. Like 2, except the system has the history of how it got there, and port management and updates of the new system continue working as if nothing had happened?
If there are only a limited number of things you use, it will work to just install the packages and then do most of the setup by copypasting and if necessary, adapting the relevant stuff from the old configuration.
I do this regularly, for example when I install my desktop system anew about every 500 days.
This also saves a lot of time when setting up again after a hardware crash.

/etc
/usr/local/etc
/var/db/pkg
/var/db/ports
/var/db/freebsd-update
I'd add
/opt/conf (recommended in rc.conf manpage)
some dot files and directories in ~ login cshrc xinitrc .fvwm/
and some stuff by default even wants to have its config in /var/, like unbound.
and then there is also /usr/[local/]share where you find your important fonts etc

A small script that extracts the most important things and information you need to set up a similar, not necessarily identical system, can save you a lot of time.
Not only in case of crash, but also when you need to set up similar environments for team work, like fonts being used.
I think the OP's idea is not bad.
 

Sensucht94

Well-Known Member

Thanks: 304
Messages: 334

#14
I'd add
/opt/conf (recommended in rc.conf manpage)
Sorry to step in between, but I wanted to point out that AFAIK /opt (Optional) is a directory to store custom 3rd party software-related data, that is supposed to be shared with other users; see /opt (Linux FileSystem Hierarchy). I remember Open Solaris always made use of /opt and so does OpenIndiana now, while in my experience /opt in Linux is hardly created by any package, though sooner or later it may happen still

As opposite, I don't believe /opt will ever be created by any Makefile in FreeBSD, as Berkeley Unix uses /usr/local for 3rd party software.
So unless it's you creating /opt, folder shouldn't exist ;). That said, when I successfully compile additional software not included amidst FreeBSD ports, I still have the habit of placing it under /opt (/opt/local/bin; /opt/local/share; /opt/etc; /opt/var),instead of $HOME (which is what many Linux user do instead), mainly because I got used to it in Solaris and in OS X, where MacPorts use /opt by default.

If you look up rc.conf(5) once again, you should notice that /opt/conf is mentioned only as possible additional directory containing init scripts, specified by user in /etc/rc.conf under local_startup="path1; path2...." variable; any folder added to local_startup, shall be scanned for a $local_startup/rc.conf.d/jail folder -containing jail cobfigurations- to be loaded too.

Instead of /opt/conf, normally the system's services/daemons that are supposed to be loaded at startup, are stored as scripts in /etc/rc.d folder, while /usr/local/etc/rc.d is used by services provided by software installed through ports/packages, and that's where you have to put your custom scripts to make them be executed at boot by rc(8), after having enabled them in /etc/rc.conf. Deprecated /etc/rc.local can be still used though, like OpenBSD still does by default: rc(8)_OpenBSD.
 

Trihexagonal

Daemon

Thanks: 501
Messages: 1,014

#15
Install a distribution version of the OS onto a blank disk, perform all your configurations and tunings, then restore just the user data? Fundamentally ending up with a new install, just the same user files?
This is basically all I do. I save all documents I want to keep from the current run of the machine to a USB stick where I save all my important docs, images and my /etc directory.

I had a HDD failure a couple months ago. Not only did I have almost all the docs that were on the drive on Flash Drive, I have 4 other machines running FreeBSD that duplicate almost everything as I use the same drive to populate them all.

I can replicate the build from ports by memory with virtually the same number of packages (553 on the one I'm using now and 554 on the one beside it), mount my USB stick, copy the important files from /etc instead of typing them out, pull my personal files off in folders and in one day be up and running like it never happened.
 

Snurg

Aspiring Daemon

Thanks: 325
Messages: 794

#16
Another thing for the to-backup-list:
Your kernel configuration file in /usr/src/sys/...

(I have no choice to use a custom kernel, because the GENERIC kernel has the VESA driver enabled, and includes the vt console, which pulls in the VESA driver.
However, as the VESA driver breaks suspend/resume when using Nvidia cards, I cannot use the pure GENERIC kernel. Every time I delete the source tree and do svn ... /usr/src I forget to backup this, and have to comment out these 4 lines again...)
 

Sensucht94

Well-Known Member

Thanks: 304
Messages: 334

#17
Another thing for the to-backup-list:
Your kernel configuration file in /usr/src/sys/...

(I have no choice to use a custom kernel, because the GENERIC kernel has the VESA driver enabled, and includes the vt console, which pulls in the VESA driver.
However, as the VESA driver breaks suspend/resume when using Nvidia cards, I cannot use the pure GENERIC kernel. Every time I delete the source tree and do svn ... /usr/src I forget to backup this, and have to comment out these 4 lines again...)
Are you tellling me I cannot suspend/resume with my Nvidia laptop because of vt invoking VESA? I didn't see it under kldstat, and never thought about the possibility of it being build inside GENERIC, since I assumed it would have conflicted with KMS or DRM-KMS drivers if loaded. Snurg would you care about providing me some more info?
 

Snurg

Aspiring Daemon

Thanks: 325
Messages: 794

#18
(Sorry for long OT post)
Are you tellling me I cannot suspend/resume with my Nvidia laptop because of vt invoking VESA?
Yes. Reason is that the VESA driver depends on the BIOS state save/restore functions, which are not implemented by Nvidia BIOSes; these simulate this only, with actually doing nothing. Thus, on Nvidia cards xorg must do this task. But it fails to do if it believes that the BIOS already restored the video cards' status. (Other BIOSes like AMD/ATI have working state save/restore functions, so with these the VESA driver causes no problems.)

The lines from GENERIC I always comment out are these:
Code:
options         VESA                    # Add support for VESA BIOS Extensions (VBE)
device          vt
device          vt_vga
device          vt_efifb
If you comment out the options VESA line only, then kldstat will show vesa.ko being loaded. It gets pulled in by the device vt*.

Thus my /boot/loader.conf contains these lines:
Code:
# vesa_load="YES"
# vt still buggy with X and modern nvidia driver
# kern.vty="vt"
kern.vty="sc"
hw.vga.textmode="YES"
I use text mode because it is fast and eye-friendly. You can select other resolutions than 80x25, too. But it is a bit tricky and depends on the BIOS actually support them, which I did not yet test. But maybe when I am in the mood some day I'll look whether mode 0x010a (132x43) can be used, too.

In addition I have in my kernel config these lines:
Code:
options SC_NORM_ATTR=(FG_BLACK|BG_GREEN)
options SC_NORM_REV_ATTR=(FG_BLACK|BG_CYAN)
options SC_KERNEL_CONS_ATTR=(FG_BLACK|BG_RED)
options SC_KERNEL_CONS_REV_ATTR=(FG_BLACK|BG_YELLOW)
options SC_HISTORY_SIZE=25000
Reasons: The white on black default "colors" are really painful to my eyes.
And, the possibility to set the scrollback buffer size is just cool. I love the ScrollLock mode. With the setting above it's possible to scroll back even long, long outputs. I don't care for the 32 megabytes lost for the buffer, as I find much more annoying if I lose important output just because of too few buffered lines. The vt console does not have such a setting afaik.

In /etc/rc.conf I have these lines:
Code:
moused_enable="YES"
moused_nondefault_enable="YES"
keybell="visual"
This enables using the mouse for marking/pasting on the console like in X. I also prefer the screen flash instead of beeping.
 
Top