• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

UFS Backup

graudeejs

Son of Beastie

Thanks: 674
Messages: 4,611

#1
Ok, this is defiantly worth writing... especially for new users
Here i will cover how to backup/restore (to file) FreeBSD using native utilities called dump and restore

note: dump and restore works only for UFS (aka FFS)

Backing up system
To backup system you need to use dump utility

backup:
Code:
$ dump -0Lauf /path/to/backups/ad0s1d.dump /dev/ad0s1d
where /dev/ad0s1d is any of your partitions, slices or disks, formated with UFS

Backup and compress on the fly
Code:
$ dump -0Lauf - /dev/ad0s1d | bzip2 > /path/to/backups/ad0s1d.dump.bz2
-0 - means to backup entire filesystem
-f name - output to file/device, or to stdout if you use -
-a - you need this if you output to file. (tells dump not to worry about the backup media volume)
-u - Means to update dumpdates file
-L - needed if you backup mounted filesystem


Restoring system
to restore system restart in single user mode
format filesystem that you want to restore
in backup example, we backed up /dev/ad0s1d, so let's format it now
Code:
$ newfs -U /dev/ad0s1d
now you need to mount it
Code:
$ mkdir /mnt/target
$ mount /dev/ad0s1d /mnt/target
Let's imagine you backed up files to usb stick (da0, in root directory)
we need to mount it
Code:
$ mount -t msdosfs /dev/da0 /mnt/usb

Important note: you need space in /tmp to be able to restore
if you run out of space in /tmp, mount some filesystem somewhere and
create symbolic links from /tmp to that mount point



now to restore from backup you need to cd to dir where you mounted partition that you want to restore
Code:
$ cd /mnt/target
to restore from uncompressed backup
Code:
$ restore -rf /mnt/usb/ad0s1d.dump
to restore from compressed backup
Code:
$ bzcat /mnt/usb/ad0s1d.dump.bz2 | restore -rf -

And that is it
now you can delete file dumpdates (or something like that, check for weird file in target directory, in our case /mnt/target)

now unmount filesystems and reboot

Some notes
you can do incremental backups - backup everything and then backup only files that have changed since (on current backup level) see manual for more info

you can use dump/restore to clone your system to other PC's
you will probably need to copy Master Boot Record (MBR) as well

to backup MBR:
Code:
$ dd if=/dev/ad0 of=/path/to/mbr.img bs=512 count=1
to restore MBR:
Code:
$ dd if=/path/to/mbr.img of=/dev/ad0 bs=512 count=1
Tips
* I prefer to compress backup, you can guess why

* if you backup /usr you may delete content of ports directory
this will speed up backup process, and reduce size of backup...
It's good thing because by the time you will restore /usr from backups
/usr/ports will be outdated, and you will need to update them anyway.
And portsnap works very well (fast) in fetching ports

* I prefer to do full backups, that way you can be 100% sure, there won't
be any confusing situations

* if you want to do backups while using filesystem, make sure you haven't
deleted .snap directory, on partition that you want to backup

* if you have backed up encrypted drive, you need to somehow encrypt backups
because if someone gets these files, he can restore them to his pc, and read your files at will. (I used this method in FreeBSD + Geli guide, to encrypt drive, but process can be reversed)


Resources
dump(8)
restore(8)


Update 1
Moving system
You can move system from disk to disk on fly with
Code:
$ newfs -U /dev/ad2s1a
$ mount /dev/ad2s1a.... /target
$ cd /target
$ dump -0Lauf - /dev/ad1s1a  | restore -rf -
you can do the same using sudo
Code:
$ sudo echo
$ sudo dump -0Lauf - /dev/ad1s1a  | sudo restore -rf -
Update 2
as OpenBSD suggests, using gzip instead of bzip2 will seed up compression at cost of larger (very little) archives

so now i suggest using gzip to compress and zcat to uncompress on fly

I've tested it, and i was amazed.
No more Bzip2 for me

Update 3
Just wanted to remind, that you don't need to use -u flag if you're using dump from fixit...

Update 4
You can dump UFS filesystem and restore to any other FS. Dump is FS specific, while restore isn't.
Note that you probably can't use linux restore to restore dumpfile created on FreeBSD and wise versa, because dumpfile formats ar probably different

Update 5
To increase compression ratio you can use xz(1) (which I personally prefer lately) instead of gzip(1)
 

thortos

New Member

Thanks: 3
Messages: 11

#4
This strategy will probably fail for every server being used more than marginally. Especially dumping databases that are in use (such as Postgres or mySQL data directories) will yield inconsistent results and most likely result in non-working databases after recovery.

While I am aware that important databases are to be replicated live onto backup servers, I want to illustrate that this dump-while-in-use strategy is best used for desktops or low-profile servers, not for heavily-used systems.

How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.
 

graudeejs

Son of Beastie

Thanks: 674
Messages: 4,611

#5
thortos said:
This strategy will probably fail for every server being used more than marginally. Especially dumping databases that are in use (such as Postgres or mySQL data directories) will yield inconsistent results and most likely result in non-working databases after recovery.

While I am aware that important databases are to be replicated live onto backup servers, I want to illustrate that this dump-while-in-use strategy is best used for desktops or low-profile servers, not for heavily-used systems.

How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.
Thanks for your reply
I use FreeBSD as desktop, so this is more desktop-oriented guide
You made some very good points....
 

zszalbot

New Member

Thanks: 1
Messages: 1

#6
thortos said:
How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.
I use a script called automysqlbackup. It works quite well and it suits my needs.

http://sourceforge.net/projects/automysqlbackup/

Yours,

Zbigniew Szalbot
 

Mel_Flynn

Well-Known Member

Thanks: 78
Messages: 379

#9
The attached script, runs a weekly full backup, and incrementals 1-6 for the other days. It can compress locally (the machine being backed up has faster CPU then the backup machine) or remotely.

All this, from the daily periodic. Primarily useful for desktops that are on during the night or where the owner has chosen a different time for daily to run.

The full back up can take a very long time, naturally depending on ammount of data, CPU speed for compression and network transfer speed.
 

Attachments

graudeejs

Son of Beastie

Thanks: 674
Messages: 4,611

#10
Mel_Flynn#
# dd is necessary, because bzip2 cannot "compress STDIN to
#named file"
if i understand you right, there's what i say about it:
you can compress stdin to file (simplified)
Code:
dump -0Lauf - /dev/da0s1a | bzip2 > /path/to/backup.img.bz2
 

Mel_Flynn

Well-Known Member

Thanks: 78
Messages: 379

#13
killasmurf86 said:
Mel_Flynn#


if i understand you right, there's what i say about it:
you can compress stdin to file (simplified)
Code:
dump -0Lauf - /dev/da0s1a | bzip2 > /path/to/backup.img.bz2
Yes, but this doesn't really work well with all shells. At least I had problems with it a few years back when i wrote it. Things may have improved since then, but I kept it to see the difference in transfer speed that dump and dd report:
Code:
  DUMP: finished in 68 seconds, throughput 1796 KBytes/sec
  DUMP: level 3 dump on Wed Nov 19 03:22:39 2008
  DUMP: DUMP IS DONE
53795+1 records in
53795+1 records out
27543432 bytes transferred in 269.804762 secs (102087 bytes/sec)
 

graudeejs

Son of Beastie

Thanks: 674
Messages: 4,611

#14
well, you used
#!/bin/sh
in your script, which means it MUST work everywhere the same, unless someone have replaced sh with something else.

and it doesn't matter under which shell you launch script, because it'll be run in SH
 

gelraen

Member

Thanks: 4
Messages: 35

#15
killasmurf86 said:
well, you used
#!/bin/sh
in your script, which means it MUST work everywhere the same, unless someone have replaced sh with something else.

and it doesn't matter under which shell you launch script, because it'll be run in SH
Only if launch it as binary. When you run it like "source ./myscript" it will be parsed by current shell ;)
 

fender0107401

Well-Known Member

Thanks: 11
Messages: 325

#17
Thank you for the post, I just need a system backup solution.

I think backup is an important part of the system administration, though you may never need the backup data.
 

graudeejs

Son of Beastie

Thanks: 674
Messages: 4,611

#18
fender0107401 said:
Thank you for the post, I just need a system backup solution.

I think backup is an important part of the system administration, though you may never need the backup data.
as a FreeBSD desktop user, i experiment a lot. And backups saves my tons of time.
 

fender0107401

Well-Known Member

Thanks: 11
Messages: 325

#19
killasmurf86 said:
as a FreeBSD desktop user, i experiment a lot. And backups saves my tons of time.
I am desktop user too, and I never experiment any terrible things (except several kernel panic for my mp3-player, but other usb device is normal).

Maybe the reason is the time that I use it is very short (since june) and I prefer the security_release branch. :e
 

blackjack

Member

Thanks: 2
Messages: 23

#20
This my script for dumpfilesystems. It run every day at 4:00 AM.
Code:
cat /root/dumpfs.sh 
#!/bin/sh
fl=`date "+%d-%m-%Y"`
path="/backup/dumpfs"

#root file system
/sbin/dump  -0 -L -f - /dev/ad4s1a > $path/rootfs/dump_ad4s1a_${fl}.img
tar cfz $path/rootfs/dump_ad4s1a_${fl}.tar.gz $path/rootfs/dump_ad4s1a_${fl}.img
rm -f $path/rootfs/dump_ad4s1a_${fl}.img
chmod 400 $path/rootfs/dump_ad4s1a_${fl}.tar.gz

#home
/sbin/dump  -0 -L -f - /dev/ad4s1f > $path/home/dump_ad4s1f_${fl}.img
tar cfz $path/home/dump_ad4s1f_${fl}.tar.gz $path/home/dump_ad4s1f_${fl}.img
rm -f $path/home/dump_ad4s1f_${fl}.img
chmod 400 $path/home/dump_ad4s1f_${fl}.tar.gz

#usr
/sbin/dump  -0 -L -f - /dev/ad4s1e > $path/usr/dump_ad4s1e_${fl}.img
tar cfz $path/usr/dump_ad4s1e_${fl}.tar.gz $path/usr/dump_ad4s1e_${fl}.img
rm -f $path/usr/dump_ad4s1e_${fl}.img
chmod 400 $path/usr/dump_ad4s1e_${fl}.tar.gz

#var
/sbin/dump  -0 -L -f - /dev/ad4s1d > $path/var/dump_ad4s1d_${fl}.img
tar cfz $path/var/dump_ad4s1d_${fl}.tar.gz $path/var/dump_ad4s1d_${fl}.img
rm -f $path/var/dump_ad4s1d_${fl}.img
chmod 400 $path/var/dump_ad4s1d_${fl}.tar.gz
And this script for backup MYSQL databses.
Code:
cat /root/backup_db.sh 
#!/bin/sh
passwd_root_mysql='password'
fl=`date "+%d-%m-%Y"`
#billing database
/usr/local/bin/mysqldump -Q --add-locks -u root --default-character-set=cp1251 --password=${passwd_root_mysql} bill > /backup/db/bill/bill_${fl}.sql
tar cfz /backup/db/bill/bill_${fl}.tar.gz /backup/db/bill/bill_${fl}.sql
rm -f /backup/db/bill/bill_${fl}.sql
chmod 400 /backup/db/bill/bill_${fl}.tar.gz
#all databases
/usr/local/bin/mysqldump --set-charset --all-databases -u root --password=${passwd_root_mysql} > /backup/db/all/all_databases_${fl}.sql
tar cfz /backup/db/all/all_databases_${fl}.tar.gz /backup/db/all/all_databases_${fl}.sql
rm -f /backup/db/all/all_databases_${fl}.sql
chmod 400 /backup/db/all/all_databases_${fl}.tar.gz
#old_base
/usr/local/bin/mysqldump -Q --add-locks -u root --default-character-set=cp1251 --password=${passwd_root_mysql} old_base > /backup/db/old_base/old_base_${fl}.sql
tar cfz /backup/db/old_base/old_base_${fl}.tar.gz /backup/db/old_base/old_base_${fl}.sql
rm -f /backup/db/old_base/old_base_${fl}.sql
chmod 400 /backup/db/old_base/old_base_${fl}.tar.gz
 

Mel_Flynn

Well-Known Member

Thanks: 78
Messages: 379

#21
killasmurf86 said:
well, you used
#!/bin/sh
in your script, which means it MUST work everywhere the same, unless someone have replaced sh with something else.

and it doesn't matter under which shell you launch script, because it'll be run in SH
No. The shell redirect is on the target machine and passed on from ssh's command line parsing. All I remember is that it wouldn't work to a BSDi 4.1 host, nor an AIX host, but I can't for the life of me remember the error message.

echo foo|ssh host "cat - >/tmp/out"

works now, didn't work then.
Come to think of it, it's possible it was caused by a shell wrapper.
 

graudeejs

Son of Beastie

Thanks: 674
Messages: 4,611

#22
UPDATE 2
as OpenBSD suggests using gzip instead of bzip2 will seed up compression at cost of larger (very little) archives

so now i suggest using gzip to compress and zcat to uncompress on fly

I've tested it, and i was amazed.
No more Bzip2 for me :D



P.S. can admin/moderator integrate this in original post (#1)
 

nakal

Active Member

Thanks: 34
Messages: 245

#23
I would not backup MBRs like you suggested, except you expect to restore things on the same drive again. MBR stores the drive geometry and partitioning information.

When you restore to a fresh drive, after a drive failure for example, it is a better idea to use fdisk, bsdlabel and eventually boot0cfg, in case you want a boot manager.

It is also possible to use gpart now. These is my preferred way to partition drives at the moment. For more information, how to use GPT partitions on i386 and amd64 and boot from them, read the article on my website: http://m8d.de/news/freebsd-on-gpt.php. It's a bit tricky, but you rather have to understand what I do there, not repeat the steps line by line.
 

graudeejs

Son of Beastie

Thanks: 674
Messages: 4,611

#24
nakal said:
I would not backup MBRs like you suggested, except you expect to restore things on the same drive again. MBR stores the drive geometry and partitioning information.

When you restore to a fresh drive, after a drive failure for example, it is a better idea to use fdisk, bsdlabel and eventually boot0cfg, in case you want a boot manager.

It is also possible to use gpart now. These is my preferred way to partition drives at the moment. For more information, how to use GPT partitions on i386 and amd64 and boot from them, read the article on my website: http://m8d.de/news/freebsd-on-gpt.php. It's a bit tricky, but you rather have to understand what I do there, not repeat the steps line by line.
ye, thank you for reminding me.... (i really forgot about this)
btw, i don't backup my MBR, if anything i use sysinstall to rebuild partitions on drive and then press "w" in fdisk editor.
It will write partition table to disk and ask for loader, pick MBR or FreeBSD loader, and exit sysinstall.
Then i just use bsdlabel to rebuild labels and that is it.
After that newfs and restore
 

sim

Active Member

Thanks: 5
Messages: 157

#25
thortos said:
How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.
I backup my servers using rsnapshot from my archive server:

On each client server, a nightly cron makes a snapshot of each filesystem and mounts them on /snapped_fs (/snapped_fs/, /snapped_fs/usr/, /snapped_fs/var/ etc). So I always have yesterday's complete filetree, mounted and frozen in time. When my archive server connects for the nightly rsnapshot, it syncs the frozen tree, not the live tree. Filesystem snapshots are supposed to be consistent.

Just to be sure, another nightly cron also runs pg_dumpall. PostgreSQL dumps are point-in-time, consistent dumps which don't require the server to stop or lock any tables. I keep the last 15 dumps, and these are of course part of the filesystem snapshot so they get copied with rsnapshot.

It's getting late, I wonder if that makes sense lol!

/sim
 
Top