UFS Backup

Would you like to share some of your solutions for certain problems? Tips or tricks? Post here. All new topics are automatically moderated.

UFS Backup

Postby graudeejs » 16 Nov 2008, 23:42

Ok, this is defiantly worth writing... especially for new users
Here i will cover how to backup/restore (to file) FreeBSD using native utilities called dump and restore

note: dump and restore works only for UFS (aka FFS)

[SIZE="5"]Backing up system[/SIZE]
To backup system you need to use dump utility

backup:
Code: Select all
$ dump -0Lauf /path/to/backups/ad0s1d.dump /dev/ad0s1d

where /dev/ad0s1d is any of your partitions, slices or disks, formated with UFS

Backup and compress on the fly
Code: Select all
$ dump -0Lauf - /dev/ad0s1d | bzip2 > /path/to/backups/ad0s1d.dump.bz2


-0 - means to backup entire filesystem
-f name - output to file/device, or to stdout if you use -
-a - you need this if you output to file. (tells dump not to worry about the backup media volume)
-u - Means to update dumpdates file
-L - needed if you backup mounted filesystem


[SIZE="5"]Restoring system[/SIZE]
to restore system restart in single user mode
format filesystem that you want to restore
in backup example, we backed up /dev/ad0s1d, so let's format it now
Code: Select all
$ newfs -U /dev/ad0s1d


now you need to mount it
Code: Select all
$ mkdir /mnt/target
$ mount /dev/ad0s1d /mnt/target

Let's imagine you backed up files to usb stick (da0, in root directory)
we need to mount it
Code: Select all
$ mount -t msdosfs /dev/da0 /mnt/usb


[color="Red"]
Important note: you need space in /tmp to be able to restore
if you run out of space in /tmp, mount some filesystem somewhere and
create symbolic links from /tmp to that mount point[/color]


now to restore from backup you need to cd to dir where you mounted partition that you want to restore
Code: Select all
$ cd /mnt/target


to restore from uncompressed backup
Code: Select all
$ restore -rf /mnt/usb/ad0s1d.dump


to restore from compressed backup
Code: Select all
$ bzcat /mnt/usb/ad0s1d.dump.bz2 | restore -rf -



And that is it
now you can delete file dumpdates (or something like that, check for weird file in target directory, in our case /mnt/target)

now unmount filesystems and reboot

[SIZE="5"]Some notes[/SIZE]
you can do incremental backups - backup everything and then backup only files that have changed since (on current backup level) see manual for more info

you can use dump/restore to clone your system to other PC's
you will probably need to copy Master Boot Record (MBR) as well

to backup MBR:
Code: Select all
$ dd if=/dev/ad0 of=/path/to/mbr.img bs=512 count=1


to restore MBR:
Code: Select all
$ dd if=/path/to/mbr.img of=/dev/ad0 bs=512 count=1


[SIZE="5"]Tips[/SIZE]
* I prefer to compress backup, you can guess why

* if you backup /usr you may delete content of ports directory
this will speed up backup process, and reduce size of backup...
It's good thing because by the time you will restore /usr from backups
/usr/ports will be outdated, and you will need to update them anyway.
And portsnap works very well (fast) in fetching ports

* I prefer to do full backups, that way you can be 100% sure, there won't
be any confusing situations

* if you want to do backups while using filesystem, make sure you haven't
deleted .snap directory, on partition that you want to backup

* if you have backed up encrypted drive, you need to somehow encrypt backups
because if someone gets these files, he can restore them to his pc, and read your files at will. (I used this method in FreeBSD + Geli guide, to encrypt drive, but process can be reversed)


[SIZE="5"]Resources[/SIZE]
[man=8]dump[/man]
[man=8]restore[/man]


[color="red"][SIZE="5"]Update 1[/SIZE][/color]
[SIZE="5"]Moving system[/SIZE]
You can move system from disk to disk on fly with
Code: Select all
$ newfs -U /dev/ad2s1a
$ mount /dev/ad2s1a.... /target
$ cd /target
$ dump -0Lauf - /dev/ad1s1a  | restore -rf -


you can do the same using sudo
Code: Select all
$ sudo echo
$ sudo dump -0Lauf - /dev/ad1s1a  | sudo restore -rf -


[color="red"][SIZE="5"]Update 2[/SIZE][/color]
as OpenBSD suggests, using gzip instead of bzip2 will seed up compression at cost of larger (very little) archives

so now i suggest using gzip to compress and zcat to uncompress on fly

I've tested it, and i was amazed.
No more Bzip2 for me

[color="red"][SIZE="5"]Update 3[/SIZE][/color]
Just wanted to remind, that you don't need to use -u flag if you're using dump from fixit...

[color="red"][SIZE="5"]Update 4[/SIZE][/color]
You can dump UFS filesystem and restore to any other FS. Dump is FS specific, while restore isn't.
Note that you probably can't use linux restore to restore dumpfile created on FreeBSD and wise versa, because dumpfile formats ar probably different

[color="red"][SIZE="5"]Update 5[/SIZE][/color]
To increase compression ratio you can use [man=1]xz[/man] (which I personally prefer lately) instead of [man=1]gzip[/man]
User avatar
graudeejs
Style(9) Addict
 
Posts: 4591
Joined: 16 Nov 2008, 23:23
Location: Riga, Latvia

Postby dave » 17 Nov 2008, 01:53

Super helpful!
User avatar
dave
Member
 
Posts: 331
Joined: 17 Nov 2008, 00:13

Postby abarmot » 17 Nov 2008, 03:46

yeah, thanks a lot for how-to!!
User avatar
abarmot
Junior Member
 
Posts: 22
Joined: 16 Nov 2008, 18:05

Postby thortos » 17 Nov 2008, 08:47

This strategy will probably fail for every server being used more than marginally. Especially dumping databases that are in use (such as Postgres or mySQL data directories) will yield inconsistent results and most likely result in non-working databases after recovery.

While I am aware that important databases are to be replicated live onto backup servers, I want to illustrate that this dump-while-in-use strategy is best used for desktops or low-profile servers, not for heavily-used systems.

How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.
User avatar
thortos
Junior Member
 
Posts: 12
Joined: 17 Nov 2008, 08:27
Location: Germany

Postby graudeejs » 17 Nov 2008, 11:07

thortos wrote:This strategy will probably fail for every server being used more than marginally. Especially dumping databases that are in use (such as Postgres or mySQL data directories) will yield inconsistent results and most likely result in non-working databases after recovery.

While I am aware that important databases are to be replicated live onto backup servers, I want to illustrate that this dump-while-in-use strategy is best used for desktops or low-profile servers, not for heavily-used systems.

How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.


Thanks for your reply
I use FreeBSD as desktop, so this is more desktop-oriented guide
You made some very good points....
User avatar
graudeejs
Style(9) Addict
 
Posts: 4591
Joined: 16 Nov 2008, 23:23
Location: Riga, Latvia

Postby zszalbot » 18 Nov 2008, 07:12

thortos wrote:How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.

I use a script called automysqlbackup. It works quite well and it suits my needs.

http://sourceforge.net/projects/automysqlbackup/

Yours,

Zbigniew Szalbot
zszalbot
Junior Member
 
Posts: 1
Joined: 18 Nov 2008, 07:08
Location: Poland

Postby soko1 » 18 Nov 2008, 07:30

Poor /sbin/dump that does not support uzip (geom_uzip.ko) = (
User avatar
soko1
Junior Member
 
Posts: 14
Joined: 16 Nov 2008, 18:06
Location: Belarus/Minsk

Postby graudeejs » 18 Nov 2008, 09:21

read my 1st post again
Code: Select all
$ bzcat /mnt/usb/ad0s1d.dump.bz2 | restore rf -
User avatar
graudeejs
Style(9) Addict
 
Posts: 4591
Joined: 16 Nov 2008, 23:23
Location: Riga, Latvia

Postby Mel_Flynn » 18 Nov 2008, 18:28

The attached script, runs a weekly full backup, and incrementals 1-6 for the other days. It can compress locally (the machine being backed up has faster CPU then the backup machine) or remotely.

All this, from the daily periodic. Primarily useful for desktops that are on during the night or where the owner has chosen a different time for daily to run.

The full back up can take a very long time, naturally depending on ammount of data, CPU speed for compression and network transfer speed.
Attachments

[The extension txt has been deactivated and can no longer be displayed.]

Mel_Flynn
Member
 
Posts: 379
Joined: 18 Nov 2008, 18:02
Location: Drachten, Netherlands

Postby graudeejs » 18 Nov 2008, 20:29

Mel_Flynn#
# dd is necessary, because bzip2 cannot "compress STDIN to
#named file"


if i understand you right, there's what i say about it:
you can compress stdin to file (simplified)
Code: Select all
dump -0Lauf - /dev/da0s1a | bzip2 > /path/to/backup.img.bz2
User avatar
graudeejs
Style(9) Addict
 
Posts: 4591
Joined: 16 Nov 2008, 23:23
Location: Riga, Latvia

Postby fxp » 19 Nov 2008, 11:43

Mysql backup:
Code: Select all
Stop mysql
make snapshot
Start mysql
do dump
fxp
Junior Member
 
Posts: 12
Joined: 17 Nov 2008, 07:58

Postby abarmot » 19 Nov 2008, 12:25

fxp, do not need to stop mysql.
mysqldump lockes tables while dumping...
User avatar
abarmot
Junior Member
 
Posts: 22
Joined: 16 Nov 2008, 18:05

Postby Mel_Flynn » 19 Nov 2008, 18:15

killasmurf86 wrote:Mel_Flynn#


if i understand you right, there's what i say about it:
you can compress stdin to file (simplified)
Code: Select all
dump -0Lauf - /dev/da0s1a | bzip2 > /path/to/backup.img.bz2


Yes, but this doesn't really work well with all shells. At least I had problems with it a few years back when i wrote it. Things may have improved since then, but I kept it to see the difference in transfer speed that dump and dd report:
Code: Select all
  DUMP: finished in 68 seconds, throughput 1796 KBytes/sec
  DUMP: level 3 dump on Wed Nov 19 03:22:39 2008
  DUMP: DUMP IS DONE
53795+1 records in
53795+1 records out
27543432 bytes transferred in 269.804762 secs (102087 bytes/sec)

Mel_Flynn
Member
 
Posts: 379
Joined: 18 Nov 2008, 18:02
Location: Drachten, Netherlands

Postby graudeejs » 19 Nov 2008, 20:19

well, you used
#!/bin/sh
in your script, which means it MUST work everywhere the same, unless someone have replaced sh with something else.

and it doesn't matter under which shell you launch script, because it'll be run in SH
User avatar
graudeejs
Style(9) Addict
 
Posts: 4591
Joined: 16 Nov 2008, 23:23
Location: Riga, Latvia

Postby gelraen » 20 Nov 2008, 00:55

killasmurf86 wrote:well, you used
#!/bin/sh
in your script, which means it MUST work everywhere the same, unless someone have replaced sh with something else.

and it doesn't matter under which shell you launch script, because it'll be run in SH

Only if launch it as binary. When you run it like "source ./myscript" it will be parsed by current shell ;)
gelraen
Junior Member
 
Posts: 32
Joined: 16 Nov 2008, 20:01
Location: Ukraine, Kyiv

Postby graudeejs » 20 Nov 2008, 06:13

gelraen wrote:Only if launch it as binary. When you run it like "source ./myscript" it will be parsed by current shell ;)


now, why would you like to do that?
User avatar
graudeejs
Style(9) Addict
 
Posts: 4591
Joined: 16 Nov 2008, 23:23
Location: Riga, Latvia

Postby fender0107401 » 21 Nov 2008, 03:01

Thank you for the post, I just need a system backup solution.

I think backup is an important part of the system administration, though you may never need the backup data.
Just be yourself.
User avatar
fender0107401
Member
 
Posts: 300
Joined: 17 Nov 2008, 03:57
Location: China, Beijing.

Postby graudeejs » 21 Nov 2008, 05:01

fender0107401 wrote:Thank you for the post, I just need a system backup solution.

I think backup is an important part of the system administration, though you may never need the backup data.


as a FreeBSD desktop user, i experiment a lot. And backups saves my tons of time.
User avatar
graudeejs
Style(9) Addict
 
Posts: 4591
Joined: 16 Nov 2008, 23:23
Location: Riga, Latvia

Postby fender0107401 » 21 Nov 2008, 08:31

killasmurf86 wrote:as a FreeBSD desktop user, i experiment a lot. And backups saves my tons of time.


I am desktop user too, and I never experiment any terrible things (except several kernel panic for my mp3-player, but other usb device is normal).

Maybe the reason is the time that I use it is very short (since june) and I prefer the security_release branch. :e
Just be yourself.
User avatar
fender0107401
Member
 
Posts: 300
Joined: 17 Nov 2008, 03:57
Location: China, Beijing.

Postby blackjack » 21 Nov 2008, 12:14

This my script for dumpfilesystems. It run every day at 4:00 AM.
Code: Select all
cat /root/dumpfs.sh
#!/bin/sh
fl=`date "+%d-%m-%Y"`
path="/backup/dumpfs"

#root file system
/sbin/dump  -0 -L -f - /dev/ad4s1a > $path/rootfs/dump_ad4s1a_${fl}.img
tar cfz $path/rootfs/dump_ad4s1a_${fl}.tar.gz $path/rootfs/dump_ad4s1a_${fl}.img
rm -f $path/rootfs/dump_ad4s1a_${fl}.img
chmod 400 $path/rootfs/dump_ad4s1a_${fl}.tar.gz

#home
/sbin/dump  -0 -L -f - /dev/ad4s1f > $path/home/dump_ad4s1f_${fl}.img
tar cfz $path/home/dump_ad4s1f_${fl}.tar.gz $path/home/dump_ad4s1f_${fl}.img
rm -f $path/home/dump_ad4s1f_${fl}.img
chmod 400 $path/home/dump_ad4s1f_${fl}.tar.gz

#usr
/sbin/dump  -0 -L -f - /dev/ad4s1e > $path/usr/dump_ad4s1e_${fl}.img
tar cfz $path/usr/dump_ad4s1e_${fl}.tar.gz $path/usr/dump_ad4s1e_${fl}.img
rm -f $path/usr/dump_ad4s1e_${fl}.img
chmod 400 $path/usr/dump_ad4s1e_${fl}.tar.gz

#var
/sbin/dump  -0 -L -f - /dev/ad4s1d > $path/var/dump_ad4s1d_${fl}.img
tar cfz $path/var/dump_ad4s1d_${fl}.tar.gz $path/var/dump_ad4s1d_${fl}.img
rm -f $path/var/dump_ad4s1d_${fl}.img
chmod 400 $path/var/dump_ad4s1d_${fl}.tar.gz

And this script for backup MYSQL databses.
Code: Select all
cat /root/backup_db.sh
#!/bin/sh
passwd_root_mysql='password'
fl=`date "+%d-%m-%Y"`
#billing database
/usr/local/bin/mysqldump -Q --add-locks -u root --default-character-set=cp1251 --password=${passwd_root_mysql} bill > /backup/db/bill/bill_${fl}.sql
tar cfz /backup/db/bill/bill_${fl}.tar.gz /backup/db/bill/bill_${fl}.sql
rm -f /backup/db/bill/bill_${fl}.sql
chmod 400 /backup/db/bill/bill_${fl}.tar.gz
#all databases
/usr/local/bin/mysqldump --set-charset --all-databases -u root --password=${passwd_root_mysql} > /backup/db/all/all_databases_${fl}.sql
tar cfz /backup/db/all/all_databases_${fl}.tar.gz /backup/db/all/all_databases_${fl}.sql
rm -f /backup/db/all/all_databases_${fl}.sql
chmod 400 /backup/db/all/all_databases_${fl}.tar.gz
#old_base
/usr/local/bin/mysqldump -Q --add-locks -u root --default-character-set=cp1251 --password=${passwd_root_mysql} old_base > /backup/db/old_base/old_base_${fl}.sql
tar cfz /backup/db/old_base/old_base_${fl}.tar.gz /backup/db/old_base/old_base_${fl}.sql
rm -f /backup/db/old_base/old_base_${fl}.sql
chmod 400 /backup/db/old_base/old_base_${fl}.tar.gz
blackjack
Junior Member
 
Posts: 22
Joined: 18 Nov 2008, 08:04
Location: Ukraine

Postby Mel_Flynn » 21 Nov 2008, 13:04

killasmurf86 wrote:well, you used
#!/bin/sh
in your script, which means it MUST work everywhere the same, unless someone have replaced sh with something else.

and it doesn't matter under which shell you launch script, because it'll be run in SH


No. The shell redirect is on the target machine and passed on from ssh's command line parsing. All I remember is that it wouldn't work to a BSDi 4.1 host, nor an AIX host, but I can't for the life of me remember the error message.

echo foo|ssh host "cat - >/tmp/out"

works now, didn't work then.
Come to think of it, it's possible it was caused by a shell wrapper.
Mel_Flynn
Member
 
Posts: 379
Joined: 18 Nov 2008, 18:02
Location: Drachten, Netherlands

Postby graudeejs » 12 Dec 2008, 09:39

[color="Red"][SIZE="5"]UPDATE 2[/SIZE][/color]
as OpenBSD suggests using gzip instead of bzip2 will seed up compression at cost of larger (very little) archives

so now i suggest using gzip to compress and zcat to uncompress on fly

I've tested it, and i was amazed.
No more Bzip2 for me :D



P.S. can admin/moderator integrate this in original post (#1)
User avatar
graudeejs
Style(9) Addict
 
Posts: 4591
Joined: 16 Nov 2008, 23:23
Location: Riga, Latvia

Postby nakal » 14 Dec 2008, 22:38

I would not backup MBRs like you suggested, except you expect to restore things on the same drive again. MBR stores the drive geometry and partitioning information.

When you restore to a fresh drive, after a drive failure for example, it is a better idea to use fdisk, bsdlabel and eventually boot0cfg, in case you want a boot manager.

It is also possible to use gpart now. These is my preferred way to partition drives at the moment. For more information, how to use GPT partitions on i386 and amd64 and boot from them, read the article on my website: http://m8d.de/news/freebsd-on-gpt.php. It's a bit tricky, but you rather have to understand what I do there, not repeat the steps line by line.
User avatar
nakal
Member
 
Posts: 171
Joined: 16 Nov 2008, 16:31
Location: Germany

Postby graudeejs » 15 Dec 2008, 04:40

nakal wrote:I would not backup MBRs like you suggested, except you expect to restore things on the same drive again. MBR stores the drive geometry and partitioning information.

When you restore to a fresh drive, after a drive failure for example, it is a better idea to use fdisk, bsdlabel and eventually boot0cfg, in case you want a boot manager.

It is also possible to use gpart now. These is my preferred way to partition drives at the moment. For more information, how to use GPT partitions on i386 and amd64 and boot from them, read the article on my website: http://m8d.de/news/freebsd-on-gpt.php. It's a bit tricky, but you rather have to understand what I do there, not repeat the steps line by line.


ye, thank you for reminding me.... (i really forgot about this)
btw, i don't backup my MBR, if anything i use sysinstall to rebuild partitions on drive and then press "w" in fdisk editor.
It will write partition table to disk and ask for loader, pick MBR or FreeBSD loader, and exit sysinstall.
Then i just use bsdlabel to rebuild labels and that is it.
After that newfs and restore
User avatar
graudeejs
Style(9) Addict
 
Posts: 4591
Joined: 16 Nov 2008, 23:23
Location: Riga, Latvia

Postby sim » 20 Dec 2008, 01:18

thortos wrote:How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.


I backup my servers using rsnapshot from my archive server:

On each client server, a nightly cron makes a snapshot of each filesystem and mounts them on /snapped_fs (/snapped_fs/, /snapped_fs/usr/, /snapped_fs/var/ etc). So I always have yesterday's complete filetree, mounted and frozen in time. When my archive server connects for the nightly rsnapshot, it syncs the frozen tree, not the live tree. Filesystem snapshots are supposed to be consistent.

Just to be sure, another nightly cron also runs pg_dumpall. PostgreSQL dumps are point-in-time, consistent dumps which don't require the server to stop or lock any tables. I keep the last 15 dumps, and these are of course part of the filesystem snapshot so they get copied with rsnapshot.

It's getting late, I wonder if that makes sense lol!

/sim
User avatar
sim
Member
 
Posts: 157
Joined: 18 Nov 2008, 16:01
Location: The Spacetime Continuum

Next

Return to Howtos & FAQs (Moderated)

Who is online

Users browsing this forum: No registered users and 0 guests