UFS Backup

graudeejs · #1 November 17th, 2008, 00:42

Ok, this is defiantly worth writing... especially for new users
Here i will cover how to backup/restore (to file) FreeBSD using native utilities called dump and restore

note: dump and restore works only for UFS (aka FFS)

Backing up system
To backup system you need to use dump utility

backup:

Code:

$ dump -0Lauf /path/to/backups/ad0s1d.dump /dev/ad0s1d

where /dev/ad0s1d is any of your partitions, slices or disks, formated with UFS

Backup and compress on the fly

Code:

$ dump -0Lauf - /dev/ad0s1d | bzip2 > /path/to/backups/ad0s1d.dump.bz2

-0 - means to backup entire filesystem
-f name - output to file/device, or to stdout if you use -
-a - you need this if you output to file. (tells dump not to worry about the backup media volume)
-u - Means to update dumpdates file
-L - needed if you backup mounted filesystem

Restoring system
to restore system restart in single user mode
format filesystem that you want to restore
in backup example, we backed up /dev/ad0s1d, so let's format it now

Code:

$ newfs -U /dev/ad0s1d

now you need to mount it

Code:

$ mkdir /mnt/target
$ mount /dev/ad0s1d /mnt/target

Let's imagine you backed up files to usb stick (da0, in root directory)
we need to mount it

Code:

$ mount -t msdosfs /dev/da0 /mnt/usb

Important note: you need space in /tmp to be able to restore
if you run out of space in /tmp, mount some filesystem somewhere and
create symbolic links from /tmp to that mount point

now to restore from backup you need to cd to dir where you mounted partition that you want to restore

Code:

$ cd /mnt/target

to restore from uncompressed backup

Code:

$ restore -rf /mnt/usb/ad0s1d.dump

to restore from compressed backup

Code:

$ bzcat /mnt/usb/ad0s1d.dump.bz2 | restore -rf -

And that is it
now you can delete file dumpdates (or something like that, check for weird file in target directory, in our case /mnt/target)

now unmount filesystems and reboot

Some notes
you can do incremental backups - backup everything and then backup only files that have changed since (on current backup level) see manual for more info

you can use dump/restore to clone your system to other PC's
you will probably need to copy Master Boot Record (MBR) as well

to backup MBR:

Code:

$ dd if=/dev/ad0 of=/path/to/mbr.img bs=512 count=1

to restore MBR:

Code:

$ dd if=/path/to/mbr.img of=/dev/ad0 bs=512 count=1

Tips
* I prefer to compress backup, you can guess why

* if you backup /usr you may delete content of ports directory
this will speed up backup process, and reduce size of backup...
It's good thing because by the time you will restore /usr from backups
/usr/ports will be outdated, and you will need to update them anyway.
And portsnap works very well (fast) in fetching ports

* I prefer to do full backups, that way you can be 100% sure, there won't
be any confusing situations

* if you want to do backups while using filesystem, make sure you haven't
deleted .snap directory, on partition that you want to backup

* if you have backed up encrypted drive, you need to somehow encrypt backups
because if someone gets these files, he can restore them to his pc, and read your files at will. (I used this method in FreeBSD + Geli guide, to encrypt drive, but process can be reversed)

Resources
dump(8)
restore(8)

Update 1
Moving system
You can move system from disk to disk on fly with

Code:

$ newfs -U /dev/ad2s1a
$ mount /dev/ad2s1a.... /target
$ cd /target
$ dump -0Lauf - /dev/ad1s1a  | restore -rf -

you can do the same using sudo

Code:

$ sudo echo
$ sudo dump -0Lauf - /dev/ad1s1a  | sudo restore -rf -

Update 2
as OpenBSD suggests, using gzip instead of bzip2 will seed up compression at cost of larger (very little) archives

so now i suggest using gzip to compress and zcat to uncompress on fly

I've tested it, and i was amazed.
No more Bzip2 for me

Update 3
Just wanted to remind, that you don't need to use -u flag if you're using dump from fixit...

Update 4
You can dump UFS filesystem and restore to any other FS. Dump is FS specific, while restore isn't.
Note that you probably can't use linux restore to restore dumpfile created on FreeBSD and wise versa, because dumpfile formats ar probably different

Update 5
To increase compression ratio you can use xz(1) (which I personally prefer lately) instead of gzip(1)

dave · #2 November 17th, 2008, 02:53

Super helpful!

abarmot · #3 November 17th, 2008, 04:46

yeah, thanks a lot for how-to!!

thortos · #4 November 17th, 2008, 09:47

This strategy will probably fail for every server being used more than marginally. Especially dumping databases that are in use (such as Postgres or mySQL data directories) will yield inconsistent results and most likely result in non-working databases after recovery.

While I am aware that important databases are to be replicated live onto backup servers, I want to illustrate that this dump-while-in-use strategy is best used for desktops or low-profile servers, not for heavily-used systems.

How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.

graudeejs · #5 November 17th, 2008, 12:07

Quote:

Originally Posted by thortos

This strategy will probably fail for every server being used more than marginally. Especially dumping databases that are in use (such as Postgres or mySQL data directories) will yield inconsistent results and most likely result in non-working databases after recovery.

While I am aware that important databases are to be replicated live onto backup servers, I want to illustrate that this dump-while-in-use strategy is best used for desktops or low-profile servers, not for heavily-used systems.

How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.

Thanks for your reply
I use FreeBSD as desktop, so this is more desktop-oriented guide
You made some very good points....

zszalbot · #6 November 18th, 2008, 08:12

Quote:

Originally Posted by thortos

How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.

I use a script called automysqlbackup. It works quite well and it suits my needs.

http://sourceforge.net/projects/automysqlbackup/

Yours,

Zbigniew Szalbot

soko1 · #7 November 18th, 2008, 08:30

Poor /sbin/dump that does not support uzip (geom_uzip.ko) = (

graudeejs · #8 November 18th, 2008, 10:21

read my 1st post again

Code:

$ bzcat /mnt/usb/ad0s1d.dump.bz2 | restore rf -

Mel_Flynn · #9 November 18th, 2008, 19:28

The attached script, runs a weekly full backup, and incrementals 1-6 for the other days. It can compress locally (the machine being backed up has faster CPU then the backup machine) or remotely.

All this, from the daily periodic. Primarily useful for desktops that are on during the night or where the owner has chosen a different time for daily to run.

The full back up can take a very long time, naturally depending on ammount of data, CPU speed for compression and network transfer speed.

graudeejs · #10 November 18th, 2008, 21:29

Mel_Flynn#

Quote:

# dd is necessary, because bzip2 cannot "compress STDIN to
#named file"

if i understand you right, there's what i say about it:
you can compress stdin to file (simplified)

Code:

dump -0Lauf - /dev/da0s1a | bzip2 > /path/to/backup.img.bz2

fxp · #11 November 19th, 2008, 12:43

Mysql backup:

Code:

Stop mysql
make snapshot
Start mysql
do dump

abarmot · #12 November 19th, 2008, 13:25

fxp, do not need to stop mysql.
mysqldump lockes tables while dumping...

Mel_Flynn · #13 November 19th, 2008, 19:15

Quote:

Originally Posted by killasmurf86

Mel_Flynn#

if i understand you right, there's what i say about it:
you can compress stdin to file (simplified)

Code:

dump -0Lauf - /dev/da0s1a | bzip2 > /path/to/backup.img.bz2

Yes, but this doesn't really work well with all shells. At least I had problems with it a few years back when i wrote it. Things may have improved since then, but I kept it to see the difference in transfer speed that dump and dd report:

Code:

  DUMP: finished in 68 seconds, throughput 1796 KBytes/sec
  DUMP: level 3 dump on Wed Nov 19 03:22:39 2008
  DUMP: DUMP IS DONE
53795+1 records in
53795+1 records out
27543432 bytes transferred in 269.804762 secs (102087 bytes/sec)

graudeejs · #14 November 19th, 2008, 21:19

well, you used
#!/bin/sh
in your script, which means it MUST work everywhere the same, unless someone have replaced sh with something else.

and it doesn't matter under which shell you launch script, because it'll be run in SH

gelraen · #15 November 20th, 2008, 01:55

Quote:

Originally Posted by killasmurf86

well, you used
#!/bin/sh
in your script, which means it MUST work everywhere the same, unless someone have replaced sh with something else.

and it doesn't matter under which shell you launch script, because it'll be run in SH

Only if launch it as binary. When you run it like "source ./myscript" it will be parsed by current shell

graudeejs · #16 November 20th, 2008, 07:13

Quote:

Originally Posted by gelraen

Only if launch it as binary. When you run it like "source ./myscript" it will be parsed by current shell

now, why would you like to do that?

fender0107401 · #17 November 21st, 2008, 04:01

Thank you for the post, I just need a system backup solution.

I think backup is an important part of the system administration, though you may never need the backup data.

graudeejs · #18 November 21st, 2008, 06:01

Quote:

Originally Posted by fender0107401

Thank you for the post, I just need a system backup solution.

I think backup is an important part of the system administration, though you may never need the backup data.

as a FreeBSD desktop user, i experiment a lot. And backups saves my tons of time.

fender0107401 · #19 November 21st, 2008, 09:31

Quote:

Originally Posted by killasmurf86

as a FreeBSD desktop user, i experiment a lot. And backups saves my tons of time.

I am desktop user too, and I never experiment any terrible things (except several kernel panic for my mp3-player, but other usb device is normal).

Maybe the reason is the time that I use it is very short (since june) and I prefer the security_release branch.

blackjack · #20 November 21st, 2008, 13:14

This my script for dumpfilesystems. It run every day at 4:00 AM.

Code:

cat /root/dumpfs.sh 
#!/bin/sh
fl=`date "+%d-%m-%Y"`
path="/backup/dumpfs"

#root file system
/sbin/dump  -0 -L -f - /dev/ad4s1a > $path/rootfs/dump_ad4s1a_${fl}.img
tar cfz $path/rootfs/dump_ad4s1a_${fl}.tar.gz $path/rootfs/dump_ad4s1a_${fl}.img
rm -f $path/rootfs/dump_ad4s1a_${fl}.img
chmod 400 $path/rootfs/dump_ad4s1a_${fl}.tar.gz

#home
/sbin/dump  -0 -L -f - /dev/ad4s1f > $path/home/dump_ad4s1f_${fl}.img
tar cfz $path/home/dump_ad4s1f_${fl}.tar.gz $path/home/dump_ad4s1f_${fl}.img
rm -f $path/home/dump_ad4s1f_${fl}.img
chmod 400 $path/home/dump_ad4s1f_${fl}.tar.gz

#usr
/sbin/dump  -0 -L -f - /dev/ad4s1e > $path/usr/dump_ad4s1e_${fl}.img
tar cfz $path/usr/dump_ad4s1e_${fl}.tar.gz $path/usr/dump_ad4s1e_${fl}.img
rm -f $path/usr/dump_ad4s1e_${fl}.img
chmod 400 $path/usr/dump_ad4s1e_${fl}.tar.gz

#var
/sbin/dump  -0 -L -f - /dev/ad4s1d > $path/var/dump_ad4s1d_${fl}.img
tar cfz $path/var/dump_ad4s1d_${fl}.tar.gz $path/var/dump_ad4s1d_${fl}.img
rm -f $path/var/dump_ad4s1d_${fl}.img
chmod 400 $path/var/dump_ad4s1d_${fl}.tar.gz

And this script for backup MYSQL databses.

Code:

cat /root/backup_db.sh 
#!/bin/sh
passwd_root_mysql='password'
fl=`date "+%d-%m-%Y"`
#billing database
/usr/local/bin/mysqldump -Q --add-locks -u root --default-character-set=cp1251 --password=${passwd_root_mysql} bill > /backup/db/bill/bill_${fl}.sql
tar cfz /backup/db/bill/bill_${fl}.tar.gz /backup/db/bill/bill_${fl}.sql
rm -f /backup/db/bill/bill_${fl}.sql
chmod 400 /backup/db/bill/bill_${fl}.tar.gz
#all databases
/usr/local/bin/mysqldump --set-charset --all-databases -u root --password=${passwd_root_mysql} > /backup/db/all/all_databases_${fl}.sql
tar cfz /backup/db/all/all_databases_${fl}.tar.gz /backup/db/all/all_databases_${fl}.sql
rm -f /backup/db/all/all_databases_${fl}.sql
chmod 400 /backup/db/all/all_databases_${fl}.tar.gz
#old_base
/usr/local/bin/mysqldump -Q --add-locks -u root --default-character-set=cp1251 --password=${passwd_root_mysql} old_base > /backup/db/old_base/old_base_${fl}.sql
tar cfz /backup/db/old_base/old_base_${fl}.tar.gz /backup/db/old_base/old_base_${fl}.sql
rm -f /backup/db/old_base/old_base_${fl}.sql
chmod 400 /backup/db/old_base/old_base_${fl}.tar.gz

Mel_Flynn · #21 November 21st, 2008, 14:04

Quote:

Originally Posted by killasmurf86

well, you used
#!/bin/sh
in your script, which means it MUST work everywhere the same, unless someone have replaced sh with something else.

and it doesn't matter under which shell you launch script, because it'll be run in SH

No. The shell redirect is on the target machine and passed on from ssh's command line parsing. All I remember is that it wouldn't work to a BSDi 4.1 host, nor an AIX host, but I can't for the life of me remember the error message.

echo foo|ssh host "cat - >/tmp/out"

works now, didn't work then.
Come to think of it, it's possible it was caused by a shell wrapper.

graudeejs · #22 December 12th, 2008, 10:39

UPDATE 2
as OpenBSD suggests using gzip instead of bzip2 will seed up compression at cost of larger (very little) archives

so now i suggest using gzip to compress and zcat to uncompress on fly

I've tested it, and i was amazed.
No more Bzip2 for me

P.S. can admin/moderator integrate this in original post (#1)

nakal · #23 December 14th, 2008, 23:38

I would not backup MBRs like you suggested, except you expect to restore things on the same drive again. MBR stores the drive geometry and partitioning information.

When you restore to a fresh drive, after a drive failure for example, it is a better idea to use fdisk, bsdlabel and eventually boot0cfg, in case you want a boot manager.

It is also possible to use gpart now. These is my preferred way to partition drives at the moment. For more information, how to use GPT partitions on i386 and amd64 and boot from them, read the article on my website: http://m8d.de/news/freebsd-on-gpt.php. It's a bit tricky, but you rather have to understand what I do there, not repeat the steps line by line.

graudeejs · #24 December 15th, 2008, 05:40

Quote:

Originally Posted by nakal

I would not backup MBRs like you suggested, except you expect to restore things on the same drive again. MBR stores the drive geometry and partitioning information.

When you restore to a fresh drive, after a drive failure for example, it is a better idea to use fdisk, bsdlabel and eventually boot0cfg, in case you want a boot manager.

It is also possible to use gpart now. These is my preferred way to partition drives at the moment. For more information, how to use GPT partitions on i386 and amd64 and boot from them, read the article on my website: http://m8d.de/news/freebsd-on-gpt.php. It's a bit tricky, but you rather have to understand what I do there, not repeat the steps line by line.

ye, thank you for reminding me.... (i really forgot about this)
btw, i don't backup my MBR, if anything i use sysinstall to rebuild partitions on drive and then press "w" in fdisk editor.
It will write partition table to disk and ask for loader, pick MBR or FreeBSD loader, and exit sysinstall.
Then i just use bsdlabel to rebuild labels and that is it.
After that newfs and restore

sim · #25 December 20th, 2008, 02:18

Quote:

Originally Posted by thortos

How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.

I backup my servers using rsnapshot from my archive server:

On each client server, a nightly cron makes a snapshot of each filesystem and mounts them on /snapped_fs (/snapped_fs/, /snapped_fs/usr/, /snapped_fs/var/ etc). So I always have yesterday's complete filetree, mounted and frozen in time. When my archive server connects for the nightly rsnapshot, it syncs the frozen tree, not the live tree. Filesystem snapshots are supposed to be consistent.

Just to be sure, another nightly cron also runs pg_dumpall. PostgreSQL dumps are point-in-time, consistent dumps which don't require the server to stop or lock any tables. I keep the last 15 dumps, and these are of course part of the filesystem snapshot so they get copied with rsnapshot.

It's getting late, I wonder if that makes sense lol!

/sim

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

The Following 38 Users Say Thank You to graudeejs For This Useful Post:
aleksb (November 19th, 2008), artcurmudgeon (December 8th, 2009), asavah (February 20th, 2013), Caliante (March 17th, 2011), cbrace (February 21st, 2010), chalbersma (January 1st, 2010), ctg (November 19th, 2008), dazza (July 8th, 2012), dennylin93 (May 30th, 2009), draco003 (September 25th, 2011), edhunter (January 9th, 2009), emka81 (May 27th, 2010), estrabd (December 4th, 2008), fender0107401 (November 21st, 2008), Fireball (October 17th, 2012), Free (August 25th, 2009), fronclynne (July 18th, 2010), hansivers (October 13th, 2010), hiatek (May 23rd, 2010), inky (December 6th, 2011), konstantin (January 25th, 2010), krynn (September 2nd, 2009), ohdediku (April 19th, 2011), pamdirac (April 23rd, 2009), rbelk (February 10th, 2009), rexpretor (April 3rd, 2011), rocky (March 27th, 2009), SIFE (October 26th, 2009), sniper007 (March 5th, 2010), SPlissken (March 17th, 2009), susanth (November 21st, 2008), swa (August 19th, 2011), thorwin (June 4th, 2009), xserg86 (December 19th, 2008), y2s82 (January 14th, 2011), YZMSQ (December 12th, 2011), z0ran (April 21st, 2009), z662 (July 12th, 2011)

The Following User Says Thank You to zszalbot For This Useful Post:
thortos (November 20th, 2008)

The Following User Says Thank You to Mel_Flynn For This Useful Post:
michaelb (December 21st, 2008)

The Following User Says Thank You to blackjack For This Useful Post:
chalbersma (January 1st, 2010)

The Following User Says Thank You to graudeejs For This Useful Post:
chalbersma (January 1st, 2010)

The Following User Says Thank You to nakal For This Useful Post:
graudeejs (December 15th, 2008)