UFS Backup

graudeejs

graudeejs

Ok, this is defiantly worth writing... especially for new users
Here i will cover how to backup/restore (to file) FreeBSD using native utilities called dump and restore

note: dump and restore works only for UFS (aka FFS)

Backing up system
To backup system you need to use dump utility

backup:
Code: 
$ dump -0Lauf /path/to/backups/ad0s1d.dump /dev/ad0s1d
where /dev/ad0s1d is any of your partitions, slices or disks, formated with UFS

Backup and compress on the fly
Code: 
$ dump -0Lauf - /dev/ad0s1d | bzip2 > /path/to/backups/ad0s1d.dump.bz2

-0 - means to backup entire filesystem
-f name - output to file/device, or to stdout if you use -
-a - you need this if you output to file. (tells dump not to worry about the backup media volume)
-u - Means to update dumpdates file
-L - needed if you backup mounted filesystem


Restoring system
to restore system restart in single user mode
format filesystem that you want to restore
in backup example, we backed up /dev/ad0s1d, so let's format it now
Code: 
$ newfs -U /dev/ad0s1d

now you need to mount it
Code: 
$ mkdir /mnt/target
$ mount /dev/ad0s1d /mnt/target
Let's imagine you backed up files to usb stick (da0, in root directory)
we need to mount it
Code: 
$ mount -t msdosfs /dev/da0 /mnt/usb


Important note: you need space in /tmp to be able to restore
if you run out of space in /tmp, mount some filesystem somewhere and
create symbolic links from /tmp to that mount point


now to restore from backup you need to cd to dir where you mounted partition that you want to restore
Code: 
$ cd /mnt/target

to restore from uncompressed backup
Code: 
$ restore -rf /mnt/usb/ad0s1d.dump

to restore from compressed backup
Code: 
$ bzcat /mnt/usb/ad0s1d.dump.bz2 | restore -rf -


And that is it
now you can delete file dumpdates (or something like that, check for weird file in target directory, in our case /mnt/target)

now unmount filesystems and reboot

Some notes
you can do incremental backups - backup everything and then backup only files that have changed since (on current backup level) see manual for more info

you can use dump/restore to clone your system to other PC's
you will probably need to copy Master Boot Record (MBR) as well

to backup MBR:
Code: 
$ dd if=/dev/ad0 of=/path/to/mbr.img bs=512 count=1

to restore MBR:
Code: 
$ dd if=/path/to/mbr.img of=/dev/ad0 bs=512 count=1

Tips
* I prefer to compress backup, you can guess why

* if you backup /usr you may delete content of ports directory
this will speed up backup process, and reduce size of backup...
It's good thing because by the time you will restore /usr from backups
/usr/ports will be outdated, and you will need to update them anyway.
And portsnap works very well (fast) in fetching ports

* I prefer to do full backups, that way you can be 100% sure, there won't
be any confusing situations

* if you want to do backups while using filesystem, make sure you haven't
deleted .snap directory, on partition that you want to backup

* if you have backed up encrypted drive, you need to somehow encrypt backups
because if someone gets these files, he can restore them to his pc, and read your files at will. (I used this method in FreeBSD + Geli guide, to encrypt drive, but process can be reversed)


Resources
dump(8)
restore(8)


Update 1
Moving system
You can move system from disk to disk on fly with
Code: 
$ newfs -U /dev/ad2s1a
$ mount /dev/ad2s1a.... /target
$ cd /target
$ dump -0Lauf - /dev/ad1s1a  | restore -rf -

you can do the same using sudo
Code: 
$ sudo echo
$ sudo dump -0Lauf - /dev/ad1s1a  | sudo restore -rf -

Update 2
as OpenBSD suggests, using gzip instead of bzip2 will seed up compression at cost of larger (very little) archives

so now i suggest using gzip to compress and zcat to uncompress on fly

I've tested it, and i was amazed.
No more Bzip2 for me

Update 3
Just wanted to remind, that you don't need to use -u flag if you're using dump from fixit...

Update 4
You can dump UFS filesystem and restore to any other FS. Dump is FS specific, while restore isn't.
Note that you probably can't use linux restore to restore dumpfile created on FreeBSD and wise versa, because dumpfile formats ar probably different

Update 5
To increase compression ratio you can use xz(1) (which I personally prefer lately) instead of gzip(1)
 
This strategy will probably fail for every server being used more than marginally. Especially dumping databases that are in use (such as Postgres or mySQL data directories) will yield inconsistent results and most likely result in non-working databases after recovery.

While I am aware that important databases are to be replicated live onto backup servers, I want to illustrate that this dump-while-in-use strategy is best used for desktops or low-profile servers, not for heavily-used systems.

How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.
 
thortos said:
This strategy will probably fail for every server being used more than marginally. Especially dumping databases that are in use (such as Postgres or mySQL data directories) will yield inconsistent results and most likely result in non-working databases after recovery.

While I am aware that important databases are to be replicated live onto backup servers, I want to illustrate that this dump-while-in-use strategy is best used for desktops or low-profile servers, not for heavily-used systems.

How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.
Click to expand...

Thanks for your reply
I use FreeBSD as desktop, so this is more desktop-oriented guide
You made some very good points....
 
thortos said:
How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.
Click to expand...
I use a script called automysqlbackup. It works quite well and it suits my needs.

http://sourceforge.net/projects/automysqlbackup/

Yours,

Zbigniew Szalbot
 
The attached script, runs a weekly full backup, and incrementals 1-6 for the other days. It can compress locally (the machine being backed up has faster CPU then the backup machine) or remotely.

All this, from the daily periodic. Primarily useful for desktops that are on during the night or where the owner has chosen a different time for daily to run.

The full back up can take a very long time, naturally depending on ammount of data, CPU speed for compression and network transfer speed.
 

Attachments

  • 201.backup-disks.sh.txt
    3.8 KB · Views: 1,504
Mel_Flynn#
# dd is necessary, because bzip2 cannot "compress STDIN to
#named file"
Click to expand...

if i understand you right, there's what i say about it:
you can compress stdin to file (simplified)
Code: 
dump -0Lauf - /dev/da0s1a | bzip2 > /path/to/backup.img.bz2
 
killasmurf86 said:
Mel_Flynn#


if i understand you right, there's what i say about it:
you can compress stdin to file (simplified)
Code: 
dump -0Lauf - /dev/da0s1a | bzip2 > /path/to/backup.img.bz2
Click to expand...

Yes, but this doesn't really work well with all shells. At least I had problems with it a few years back when i wrote it. Things may have improved since then, but I kept it to see the difference in transfer speed that dump and dd report:
Code: 
  DUMP: finished in 68 seconds, throughput 1796 KBytes/sec
  DUMP: level 3 dump on Wed Nov 19 03:22:39 2008
  DUMP: DUMP IS DONE
53795+1 records in
53795+1 records out
27543432 bytes transferred in 269.804762 secs (102087 bytes/sec)
 
well, you used
#!/bin/sh
in your script, which means it MUST work everywhere the same, unless someone have replaced sh with something else.

and it doesn't matter under which shell you launch script, because it'll be run in SH
 
killasmurf86 said:
well, you used
#!/bin/sh
in your script, which means it MUST work everywhere the same, unless someone have replaced sh with something else.

and it doesn't matter under which shell you launch script, because it'll be run in SH
Click to expand...
Only if launch it as binary. When you run it like "source ./myscript" it will be parsed by current shell ;)
 
Thank you for the post, I just need a system backup solution.

I think backup is an important part of the system administration, though you may never need the backup data.
 
fender0107401 said:
Thank you for the post, I just need a system backup solution.

I think backup is an important part of the system administration, though you may never need the backup data.
Click to expand...

as a FreeBSD desktop user, i experiment a lot. And backups saves my tons of time.
 
killasmurf86 said:
as a FreeBSD desktop user, i experiment a lot. And backups saves my tons of time.
Click to expand...

I am desktop user too, and I never experiment any terrible things (except several kernel panic for my mp3-player, but other usb device is normal).

Maybe the reason is the time that I use it is very short (since june) and I prefer the security_release branch. :e
 
This my script for dumpfilesystems. It run every day at 4:00 AM.
Code: 
cat /root/dumpfs.sh 
#!/bin/sh
fl=`date "+%d-%m-%Y"`
path="/backup/dumpfs"

#root file system
/sbin/dump  -0 -L -f - /dev/ad4s1a > $path/rootfs/dump_ad4s1a_${fl}.img
tar cfz $path/rootfs/dump_ad4s1a_${fl}.tar.gz $path/rootfs/dump_ad4s1a_${fl}.img
rm -f $path/rootfs/dump_ad4s1a_${fl}.img
chmod 400 $path/rootfs/dump_ad4s1a_${fl}.tar.gz

#home
/sbin/dump  -0 -L -f - /dev/ad4s1f > $path/home/dump_ad4s1f_${fl}.img
tar cfz $path/home/dump_ad4s1f_${fl}.tar.gz $path/home/dump_ad4s1f_${fl}.img
rm -f $path/home/dump_ad4s1f_${fl}.img
chmod 400 $path/home/dump_ad4s1f_${fl}.tar.gz

#usr
/sbin/dump  -0 -L -f - /dev/ad4s1e > $path/usr/dump_ad4s1e_${fl}.img
tar cfz $path/usr/dump_ad4s1e_${fl}.tar.gz $path/usr/dump_ad4s1e_${fl}.img
rm -f $path/usr/dump_ad4s1e_${fl}.img
chmod 400 $path/usr/dump_ad4s1e_${fl}.tar.gz

#var
/sbin/dump  -0 -L -f - /dev/ad4s1d > $path/var/dump_ad4s1d_${fl}.img
tar cfz $path/var/dump_ad4s1d_${fl}.tar.gz $path/var/dump_ad4s1d_${fl}.img
rm -f $path/var/dump_ad4s1d_${fl}.img
chmod 400 $path/var/dump_ad4s1d_${fl}.tar.gz
And this script for backup MYSQL databses.
Code: 
cat /root/backup_db.sh 
#!/bin/sh
passwd_root_mysql='password'
fl=`date "+%d-%m-%Y"`
#billing database
/usr/local/bin/mysqldump -Q --add-locks -u root --default-character-set=cp1251 --password=${passwd_root_mysql} bill > /backup/db/bill/bill_${fl}.sql
tar cfz /backup/db/bill/bill_${fl}.tar.gz /backup/db/bill/bill_${fl}.sql
rm -f /backup/db/bill/bill_${fl}.sql
chmod 400 /backup/db/bill/bill_${fl}.tar.gz
#all databases
/usr/local/bin/mysqldump --set-charset --all-databases -u root --password=${passwd_root_mysql} > /backup/db/all/all_databases_${fl}.sql
tar cfz /backup/db/all/all_databases_${fl}.tar.gz /backup/db/all/all_databases_${fl}.sql
rm -f /backup/db/all/all_databases_${fl}.sql
chmod 400 /backup/db/all/all_databases_${fl}.tar.gz
#old_base
/usr/local/bin/mysqldump -Q --add-locks -u root --default-character-set=cp1251 --password=${passwd_root_mysql} old_base > /backup/db/old_base/old_base_${fl}.sql
tar cfz /backup/db/old_base/old_base_${fl}.tar.gz /backup/db/old_base/old_base_${fl}.sql
rm -f /backup/db/old_base/old_base_${fl}.sql
chmod 400 /backup/db/old_base/old_base_${fl}.tar.gz
 
killasmurf86 said:
well, you used
#!/bin/sh
in your script, which means it MUST work everywhere the same, unless someone have replaced sh with something else.

and it doesn't matter under which shell you launch script, because it'll be run in SH
Click to expand...

No. The shell redirect is on the target machine and passed on from ssh's command line parsing. All I remember is that it wouldn't work to a BSDi 4.1 host, nor an AIX host, but I can't for the life of me remember the error message.

echo foo|ssh host "cat - >/tmp/out"

works now, didn't work then.
Come to think of it, it's possible it was caused by a shell wrapper.
 
UPDATE 2
as OpenBSD suggests using gzip instead of bzip2 will seed up compression at cost of larger (very little) archives

so now i suggest using gzip to compress and zcat to uncompress on fly

I've tested it, and i was amazed.
No more Bzip2 for me :D



P.S. can admin/moderator integrate this in original post (#1)
 
I would not backup MBRs like you suggested, except you expect to restore things on the same drive again. MBR stores the drive geometry and partitioning information.

When you restore to a fresh drive, after a drive failure for example, it is a better idea to use fdisk, bsdlabel and eventually boot0cfg, in case you want a boot manager.

It is also possible to use gpart now. These is my preferred way to partition drives at the moment. For more information, how to use GPT partitions on i386 and amd64 and boot from them, read the article on my website: http://m8d.de/news/freebsd-on-gpt.php. It's a bit tricky, but you rather have to understand what I do there, not repeat the steps line by line.
 
nakal said:
I would not backup MBRs like you suggested, except you expect to restore things on the same drive again. MBR stores the drive geometry and partitioning information.

When you restore to a fresh drive, after a drive failure for example, it is a better idea to use fdisk, bsdlabel and eventually boot0cfg, in case you want a boot manager.

It is also possible to use gpart now. These is my preferred way to partition drives at the moment. For more information, how to use GPT partitions on i386 and amd64 and boot from them, read the article on my website: http://m8d.de/news/freebsd-on-gpt.php. It's a bit tricky, but you rather have to understand what I do there, not repeat the steps line by line.
Click to expand...

ye, thank you for reminding me.... (i really forgot about this)
btw, i don't backup my MBR, if anything i use sysinstall to rebuild partitions on drive and then press "w" in fdisk editor.
It will write partition table to disk and ask for loader, pick MBR or FreeBSD loader, and exit sysinstall.
Then i just use bsdlabel to rebuild labels and that is it.
After that newfs and restore
 
thortos said:
How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.
Click to expand...

I backup my servers using rsnapshot from my archive server:

On each client server, a nightly cron makes a snapshot of each filesystem and mounts them on /snapped_fs (/snapped_fs/, /snapped_fs/usr/, /snapped_fs/var/ etc). So I always have yesterday's complete filetree, mounted and frozen in time. When my archive server connects for the nightly rsnapshot, it syncs the frozen tree, not the live tree. Filesystem snapshots are supposed to be consistent.

Just to be sure, another nightly cron also runs pg_dumpall. PostgreSQL dumps are point-in-time, consistent dumps which don't require the server to stop or lock any tables. I keep the last 15 dumps, and these are of course part of the filesystem snapshot so they get copied with rsnapshot.

It's getting late, I wonder if that makes sense lol!

/sim
 
You must log in or register to reply here.
Back
Top