Backups

G

Gambler

Currently, I'm trying to create a hot backup of a certain low-volume FreeBSD server. Which is kind of like restoring a system after a hardware failure. The problem is, it's not going very well. Which means in case of real failure, things will be even worse. After realizing this, I've begun to search the web for information on backup procedures with FreeBSD. I've read a number of articles and threadsabout backup/restoration, and I still don't see the backup procedure that is straightforward and suitable for my setup.

Here are some constraints and requirements:
1. I don't have a tape drive or hot-swappable HDD's.
2. Database dumps and web-related files are automatically backed up off-site. But that's only useful when you have a working system with the proper software installed. Effectively, I need to backup my software setup, not variable data.
3. Preferably, backups should be simple files, which could be burned to a CD, transferred over SFTP, etc. Not something complex and ephemeral.

I tried doing dump/restore. Dump went fine, I have 3 dumps of my partitions on the original server's harddrive. The restore was a mess, I still didn't brought a single partition online on a new system.

First, there is a problem of transferring files to a new system. Using USB stick is horribly slow, and when I do a restore while booted from fixit CD, the utility seems to just freeze over. I tried using -v flag, but it didn't help much in determining whether it's just very slow or frozen.

Restore is a problem, because it's done on a new system, using fixit CD. I don't have any disk space, since the disk is used as a target of restore. I don't have any scripts/programs aside from the CD content, since I'm on a new system.

Also, the restoration procedure is very involved and there are many ways to go wrong. For example, is it okay to have files on the partitions that are being uses as a target of restore? Plus, restore needs a tmp/ diretory. Is it okat to just mount the appropriate partition from dev/ folder?

Is there something simpler? It would be much easier to get the backup snapshots directly over sftp/ftp from some other serer. It would be even better if snapshots contained partitioning information, and did not require mounting, newfsing, and so on.

One thing I can probably do is to make an extra partition on the target system, which will hold the backups. Install FreeBSD, transfer backups to that partition via SFTP, reload with fixit CD, mount it the partition with backups, and so on. But that's a lot of extra steps.

Is there any way to do fusefs w/ ssh from something that runs off a CD? That would be an ideal solution, I guess.
 
Gambler said:
2. Database dumps and web-related files are automatically backed up off-site. But that's only useful when you have a working system with the proper software installed. Effectively, I need to backup my software setup, not variable data.
Click to expand...
Ok. So all the data is already being backed up?

3. Preferably, backups should be simple files, which could be burned to a CD, transferred over SFTP, etc. Not something complex and ephemeral.
Click to expand...
Just backup a few of the configuration files like /etc/rc.conf and whatever else you use (/usr/local/etc/apache*/httpd.conf perhaps?).

To restore just do a clean install (this will only take a couple of minutes), install the packages you need, copy the configuration files and restore the data backups.
 
why won't you make gpt (or any other kind) partitions and keep 1 partition for /tmp?
I'm using Backup/restore all the time (form flash)
my backups are about 1.1G only (gzipped), for me it's pretty fast.

are you using single disk (slice/partition) for everything?
[I mean for example /dev/da0 for everything, without slices/partitions etc? that's not the best idea...]
 
What version of FreeBSD is your server running?

I usually don't bother with dumps. I'll just do a minimal install and rsync my backups over. I use rsync on some and rdiff on others. Is there a reason why you can't just backup the files you need instead of doing complete partition dumps? You will save more space thus leaving room to keep older backups by using an incremental backup application.
 
Vye said:
What version of FreeBSD is your server running?

I usually don't bother with dumps. I'll just do a minimal install and rsync my backups over. I use rsync on some and rdiff on others. Is there a reason why you can't just backup the files you need instead of doing complete partition dumps? You will save more space thus leaving room to keep older backups by using an incremental backup application.
Click to expand...

you know dump can also do incremental backups ;)
 
&quot said:
To restore just do a clean install (this will only take a couple of minutes), install the packages you need, copy the configuration files and restore the data backups.
Click to expand...
I used to do it this way, but it got to the point where identifying all the configurations to keep and installing all the packages takes way too much time. I would very much like to keep this an automated process.

&quot said:
why won't you make gpt (or any other kind) partitions and keep 1 partition for /tmp?
Click to expand...
I'm not sure what you mean about 1 partition for /tmp.

GPT sounds interesting, but this is the first time I hear about it. I might try it on a workstation, but on a server I prefer to keep things simple.

are you using single disk (slice/partition) for everything?
Click to expand...
I have the default configuration FreeBSD suggests. ad0s1a for root, ad0s1b for swap, and so on for war/, temp/, and usr/.

I'm using Backup/restore all the time (form flash)
my backups are about 1.1G only (gzipped), for me it's pretty fast.
Click to expand...
My usr/ is 4.2G unzipped.

&quot said:
What version of FreeBSD is your server running?
Click to expand...
6.2. (I'm planning to upgrade to 7.2, hence the need for hot backup.)

Is there a reason why you can't just backup the files you need instead of doing complete partition dumps?
Click to expand...
Depends on what you mean by the files I need. The appeal of a dump is that it gives you a snapshot of a complete working system, regardless of what you changed.

Rsync requires running a server somwhere. I guess that if it can synchronize entire file system with one command, it can be used to keep a working hot backup server up-to-date.

---

Here is what I get on error console of fixit:

umas0: BBB bulk-in clear stall failed, IOERROR
umas0: BBB bulk-out clear stall failed, IOERROR

I guess there is something wrong with the USB drive.
 
my /usr is about the same size (unzipped)
but when i run dump i run
Code: 
dump -0Lauf - /usr | gzip >> /tmp/usr.dump.gz

and to restore
Code: 
newfs -nU /dev/ad0s1d
mount /dev/ad0s1d /mnt/dst
cd /mnt/dst
gzcat /path/to/usr.dump.gz | restore -rf -
for this you need /tmp mounted...

if you can't mount any fs to use it as tmp (ye, sometimes i use empty filesystems) you can do this:
Code: 
mdconfig -at swap -s 128M -o noreadonly -u 1
mount /dev/md1 /tmp
I'm not sure how much swap is needed, you can test and see ;)
this will be allocated on ram until you have enough free ram....

you can also add -o compress to mdconfig if you want to



if that is to hard to remember you can write simple sh script
 
Thanks for the suggestions. mdconfig might come in handy some day.

...

I just re-dumped my root, wrote it to USB again, and then the restore procedure got stuck on the same file:
Code: 
extract file ./boot/kernel/kernel

Same error messages as before, plus
umass0: BB reset failed, TIMEOUT

Hm...
 
weird.....
can you do it again, only this time copy root dump to some fs (if possible)
perhaps it usb issue....
I really have never had problems like this

...root can be restored without mounting tmp (from fixitcd)
If it's about as same size as mine (about 120M uncompressed)
 
Gambler said:
Thanks for the suggestions. mdconfig might come in handy some day.

...

I just re-dumped my root, wrote it to USB again, and then the restore procedure got stuck on the same file:
Code: 
extract file ./boot/kernel/kernel

Same error messages as before, plus
umass0: BB reset failed, TIMEOUT

Hm...
Click to expand...
I have that problem whenever I try to transfer huge files over USB. I think there may be some sort of bug in FreeBSD's code that this sort of thing rubs against.
 
Gambler said:
I used to do it this way, but it got to the point where identifying all the configurations to keep and installing all the packages takes way too much time. I would very much like to keep this an automated process.
Click to expand...
I meant install pre-compiles packages not build them from source. Installing 300+ packages doesn't take that long and I'll doubt your server will contain that much.

As for automating the process, look into creating your own release. See release(7) for more info.
 
Some updates and questions.

I used another USB drive, and the root restore went fine. However, that drive was only 2 Gig. I couldn't fit my usr/ directory there even with bzip2 compression and ports tree being marked with [font="Courier New"]chflags nodump[/font].

After that I made a minimal install on the target PC, and allocated 8 gig to tmp/ directory. With this setup I could just SFTP the dumps to that directory - I would mount it from fixit shell anyway. (From now on all my FreeBSD installs will have 8Gig tmp/ partition, heh. Just in case.)

Restoring usr/ directory right now... Investigating other backup options in parallel.

BTW, killasmurf86, what advantages gpt has in terms of making backups?
 
none
only advantages is that you can have have
/dev/ad0p1 - is used as boot partition, if it's your sys disk
/dev/ad0p2 - must be root, if it's your sys disk
/dev/ad0p3
/dev/ad0p4
/dev/ad0p5
/dev/ad0p6
...
/dev/ad0p127

I label all my fs so it's easy for me to restore
newfs -nUL usr /dev/ufs/usr
mount ... restore....


IMPORTANT TIP: one must be careful, when changing partition layout, because if your swap has been moved to different partition you can overwrite some data :D, for this reason i keep my swap on p3 for sys disk, and p1 on data disk. This helps avoiding problems.
Normally I would label swap, but since i'm using encrypted swap, it's not possible.

I have 9 partition on my sys disk +1 for boot (/dev/ad0p10) = 10 partitions
I very much prefer using gpt, because i don't have to mess with slices and bsd labels
downside is that you can't use sysinstall to install sys on gpt, but who needs it anyway, if you can:
http://daemonforums.org/showthread.php?t=1538


you can also chflags nodump /usr/src, just don't forget to save your kernel config...
Also you can chflags nudump /usr/obj (you might need to create it later, i'm not sure if it's auto created when you rebuild world/kernel)
 
You must log in or register to reply here.
Back
Top