1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

UFS Backup

Discussion in 'Howtos and FAQs (Moderated)' started by graudeejs, Nov 16, 2008.

  1. graudeejs

    graudeejs Member

    Messages:
    4,594
    Likes Received:
    0
    Ok, this is defiantly worth writing... especially for new users
    Here i will cover how to backup/restore (to file) FreeBSD using native utilities called dump and restore

    note: dump and restore works only for UFS (aka FFS)

    Backing up system
    To backup system you need to use dump utility

    backup:
    Code:
    $ dump -0Lauf /path/to/backups/ad0s1d.dump /dev/ad0s1d
    
    where /dev/ad0s1d is any of your partitions, slices or disks, formated with UFS

    Backup and compress on the fly
    Code:
    $ dump -0Lauf - /dev/ad0s1d | bzip2 > /path/to/backups/ad0s1d.dump.bz2
    
    -0 - means to backup entire filesystem
    -f name - output to file/device, or to stdout if you use -
    -a - you need this if you output to file. (tells dump not to worry about the backup media volume)
    -u - Means to update dumpdates file
    -L - needed if you backup mounted filesystem


    Restoring system
    to restore system restart in single user mode
    format filesystem that you want to restore
    in backup example, we backed up /dev/ad0s1d, so let's format it now
    Code:
    $ newfs -U /dev/ad0s1d
    
    now you need to mount it
    Code:
    $ mkdir /mnt/target
    $ mount /dev/ad0s1d /mnt/target
    
    Let's imagine you backed up files to usb stick (da0, in root directory)
    we need to mount it
    Code:
    $ mount -t msdosfs /dev/da0 /mnt/usb
    

    Important note: you need space in /tmp to be able to restore
    if you run out of space in /tmp, mount some filesystem somewhere and
    create symbolic links from /tmp to that mount point



    now to restore from backup you need to cd to dir where you mounted partition that you want to restore
    Code:
    $ cd /mnt/target
    
    to restore from uncompressed backup
    Code:
    $ restore -rf /mnt/usb/ad0s1d.dump
    
    to restore from compressed backup
    Code:
    $ bzcat /mnt/usb/ad0s1d.dump.bz2 | restore -rf -
    

    And that is it
    now you can delete file dumpdates (or something like that, check for weird file in target directory, in our case /mnt/target)

    now unmount filesystems and reboot

    Some notes
    you can do incremental backups - backup everything and then backup only files that have changed since (on current backup level) see manual for more info

    you can use dump/restore to clone your system to other PC's
    you will probably need to copy Master Boot Record (MBR) as well

    to backup MBR:
    Code:
    $ dd if=/dev/ad0 of=/path/to/mbr.img bs=512 count=1
    
    to restore MBR:
    Code:
    $ dd if=/path/to/mbr.img of=/dev/ad0 bs=512 count=1
    
    Tips
    * I prefer to compress backup, you can guess why

    * if you backup /usr you may delete content of ports directory
    this will speed up backup process, and reduce size of backup...
    It's good thing because by the time you will restore /usr from backups
    /usr/ports will be outdated, and you will need to update them anyway.
    And portsnap works very well (fast) in fetching ports

    * I prefer to do full backups, that way you can be 100% sure, there won't
    be any confusing situations

    * if you want to do backups while using filesystem, make sure you haven't
    deleted .snap directory, on partition that you want to backup

    * if you have backed up encrypted drive, you need to somehow encrypt backups
    because if someone gets these files, he can restore them to his pc, and read your files at will. (I used this method in FreeBSD + Geli guide, to encrypt drive, but process can be reversed)


    Resources
    dump(8)
    restore(8)


    Update 1
    Moving system
    You can move system from disk to disk on fly with
    Code:
    $ newfs -U /dev/ad2s1a
    $ mount /dev/ad2s1a.... /target
    $ cd /target
    $ dump -0Lauf - /dev/ad1s1a  | restore -rf -
    
    you can do the same using sudo
    Code:
    $ sudo echo
    $ sudo dump -0Lauf - /dev/ad1s1a  | sudo restore -rf -
    
    Update 2
    as OpenBSD suggests, using gzip instead of bzip2 will seed up compression at cost of larger (very little) archives

    so now i suggest using gzip to compress and zcat to uncompress on fly

    I've tested it, and i was amazed.
    No more Bzip2 for me

    Update 3
    Just wanted to remind, that you don't need to use -u flag if you're using dump from fixit...

    Update 4
    You can dump UFS filesystem and restore to any other FS. Dump is FS specific, while restore isn't.
    Note that you probably can't use linux restore to restore dumpfile created on FreeBSD and wise versa, because dumpfile formats ar probably different

    Update 5
    To increase compression ratio you can use xz(1) (which I personally prefer lately) instead of gzip(1)
     
  2. dave

    dave Member

    Messages:
    342
    Likes Received:
    0
    Super helpful!
     
  3. abarmot

    abarmot New Member

    Messages:
    22
    Likes Received:
    0
    yeah, thanks a lot for how-to!!
     
  4. thortos

    thortos New Member

    Messages:
    12
    Likes Received:
    0
    This strategy will probably fail for every server being used more than marginally. Especially dumping databases that are in use (such as Postgres or mySQL data directories) will yield inconsistent results and most likely result in non-working databases after recovery.

    While I am aware that important databases are to be replicated live onto backup servers, I want to illustrate that this dump-while-in-use strategy is best used for desktops or low-profile servers, not for heavily-used systems.

    How do you people handle the backups of your servers? I'm running a set of customized backup scripts per server that tar important directories and scp them to the backup server, starting and stopping daemons as needed, but obviously that's not for anyone with uptime requirements. I also have many servers running in VMware and use that to snapshot the VMs regularly and scp them to the backup server.
     
  5. graudeejs

    graudeejs Member

    Messages:
    4,594
    Likes Received:
    0
    Thanks for your reply
    I use FreeBSD as desktop, so this is more desktop-oriented guide
    You made some very good points....
     
  6. zszalbot

    zszalbot New Member

    Messages:
    1
    Likes Received:
    0
    I use a script called automysqlbackup. It works quite well and it suits my needs.

    http://sourceforge.net/projects/automysqlbackup/

    Yours,

    Zbigniew Szalbot
     
  7. soko1

    soko1 New Member

    Messages:
    14
    Likes Received:
    0
    Poor /sbin/dump that does not support uzip (geom_uzip.ko) = (
     
  8. graudeejs

    graudeejs Member

    Messages:
    4,594
    Likes Received:
    0
    read my 1st post again
    Code:
    $ bzcat /mnt/usb/ad0s1d.dump.bz2 | restore rf -
     
  9. Mel_Flynn

    Mel_Flynn New Member

    Messages:
    379
    Likes Received:
    0
    The attached script, runs a weekly full backup, and incrementals 1-6 for the other days. It can compress locally (the machine being backed up has faster CPU then the backup machine) or remotely.

    All this, from the daily periodic. Primarily useful for desktops that are on during the night or where the owner has chosen a different time for daily to run.

    The full back up can take a very long time, naturally depending on ammount of data, CPU speed for compression and network transfer speed.
     

    Attached Files:

  10. graudeejs

    graudeejs Member

    Messages:
    4,594
    Likes Received:
    0
    Mel_Flynn#
    if i understand you right, there's what i say about it:
    you can compress stdin to file (simplified)
    Code:
    dump -0Lauf - /dev/da0s1a | bzip2 > /path/to/backup.img.bz2
    
     
  11. fxp

    fxp New Member

    Messages:
    12
    Likes Received:
    0
    Mysql backup:
    Code:
    Stop mysql
    make snapshot
    Start mysql
    do dump
     
  12. abarmot

    abarmot New Member

    Messages:
    22
    Likes Received:
    0
    fxp, do not need to stop mysql.
    mysqldump lockes tables while dumping...
     
  13. Mel_Flynn

    Mel_Flynn New Member

    Messages:
    379
    Likes Received:
    0
    Yes, but this doesn't really work well with all shells. At least I had problems with it a few years back when i wrote it. Things may have improved since then, but I kept it to see the difference in transfer speed that dump and dd report:
    Code:
      DUMP: finished in 68 seconds, throughput 1796 KBytes/sec
      DUMP: level 3 dump on Wed Nov 19 03:22:39 2008
      DUMP: DUMP IS DONE
    53795+1 records in
    53795+1 records out
    27543432 bytes transferred in 269.804762 secs (102087 bytes/sec)
    
    
     
  14. graudeejs

    graudeejs Member

    Messages:
    4,594
    Likes Received:
    0
    well, you used
    #!/bin/sh
    in your script, which means it MUST work everywhere the same, unless someone have replaced sh with something else.

    and it doesn't matter under which shell you launch script, because it'll be run in SH
     
  15. gelraen

    gelraen New Member

    Messages:
    35
    Likes Received:
    0
    Only if launch it as binary. When you run it like "source ./myscript" it will be parsed by current shell ;)
     
  16. graudeejs

    graudeejs Member

    Messages:
    4,594
    Likes Received:
    0
    now, why would you like to do that?
     
  17. fender0107401

    fender0107401 Member

    Messages:
    303
    Likes Received:
    0
    Thank you for the post, I just need a system backup solution.

    I think backup is an important part of the system administration, though you may never need the backup data.
     
  18. graudeejs

    graudeejs Member

    Messages:
    4,594
    Likes Received:
    0
    as a FreeBSD desktop user, i experiment a lot. And backups saves my tons of time.
     
  19. fender0107401

    fender0107401 Member

    Messages:
    303
    Likes Received:
    0
    I am desktop user too, and I never experiment any terrible things (except several kernel panic for my mp3-player, but other usb device is normal).

    Maybe the reason is the time that I use it is very short (since june) and I prefer the security_release branch. :e
     
  20. blackjack

    blackjack New Member

    Messages:
    22
    Likes Received:
    0
    This my script for dumpfilesystems. It run every day at 4:00 AM.
    Code:
    cat /root/dumpfs.sh 
    #!/bin/sh
    fl=`date "+%d-%m-%Y"`
    path="/backup/dumpfs"
    
    #root file system
    /sbin/dump  -0 -L -f - /dev/ad4s1a > $path/rootfs/dump_ad4s1a_${fl}.img
    tar cfz $path/rootfs/dump_ad4s1a_${fl}.tar.gz $path/rootfs/dump_ad4s1a_${fl}.img
    rm -f $path/rootfs/dump_ad4s1a_${fl}.img
    chmod 400 $path/rootfs/dump_ad4s1a_${fl}.tar.gz
    
    #home
    /sbin/dump  -0 -L -f - /dev/ad4s1f > $path/home/dump_ad4s1f_${fl}.img
    tar cfz $path/home/dump_ad4s1f_${fl}.tar.gz $path/home/dump_ad4s1f_${fl}.img
    rm -f $path/home/dump_ad4s1f_${fl}.img
    chmod 400 $path/home/dump_ad4s1f_${fl}.tar.gz
    
    #usr
    /sbin/dump  -0 -L -f - /dev/ad4s1e > $path/usr/dump_ad4s1e_${fl}.img
    tar cfz $path/usr/dump_ad4s1e_${fl}.tar.gz $path/usr/dump_ad4s1e_${fl}.img
    rm -f $path/usr/dump_ad4s1e_${fl}.img
    chmod 400 $path/usr/dump_ad4s1e_${fl}.tar.gz
    
    #var
    /sbin/dump  -0 -L -f - /dev/ad4s1d > $path/var/dump_ad4s1d_${fl}.img
    tar cfz $path/var/dump_ad4s1d_${fl}.tar.gz $path/var/dump_ad4s1d_${fl}.img
    rm -f $path/var/dump_ad4s1d_${fl}.img
    chmod 400 $path/var/dump_ad4s1d_${fl}.tar.gz
    
    And this script for backup MYSQL databses.
    Code:
    cat /root/backup_db.sh 
    #!/bin/sh
    passwd_root_mysql='password'
    fl=`date "+%d-%m-%Y"`
    #billing database
    /usr/local/bin/mysqldump -Q --add-locks -u root --default-character-set=cp1251 --password=${passwd_root_mysql} bill > /backup/db/bill/bill_${fl}.sql
    tar cfz /backup/db/bill/bill_${fl}.tar.gz /backup/db/bill/bill_${fl}.sql
    rm -f /backup/db/bill/bill_${fl}.sql
    chmod 400 /backup/db/bill/bill_${fl}.tar.gz
    #all databases
    /usr/local/bin/mysqldump --set-charset --all-databases -u root --password=${passwd_root_mysql} > /backup/db/all/all_databases_${fl}.sql
    tar cfz /backup/db/all/all_databases_${fl}.tar.gz /backup/db/all/all_databases_${fl}.sql
    rm -f /backup/db/all/all_databases_${fl}.sql
    chmod 400 /backup/db/all/all_databases_${fl}.tar.gz
    #old_base
    /usr/local/bin/mysqldump -Q --add-locks -u root --default-character-set=cp1251 --password=${passwd_root_mysql} old_base > /backup/db/old_base/old_base_${fl}.sql
    tar cfz /backup/db/old_base/old_base_${fl}.tar.gz /backup/db/old_base/old_base_${fl}.sql
    rm -f /backup/db/old_base/old_base_${fl}.sql
    chmod 400 /backup/db/old_base/old_base_${fl}.tar.gz
     
  21. Mel_Flynn

    Mel_Flynn New Member

    Messages:
    379
    Likes Received:
    0
    No. The shell redirect is on the target machine and passed on from ssh's command line parsing. All I remember is that it wouldn't work to a BSDi 4.1 host, nor an AIX host, but I can't for the life of me remember the error message.

    echo foo|ssh host "cat - >/tmp/out"

    works now, didn't work then.
    Come to think of it, it's possible it was caused by a shell wrapper.
     
  22. graudeejs

    graudeejs Member

    Messages:
    4,594
    Likes Received:
    0
    UPDATE 2
    as OpenBSD suggests using gzip instead of bzip2 will seed up compression at cost of larger (very little) archives

    so now i suggest using gzip to compress and zcat to uncompress on fly

    I've tested it, and i was amazed.
    No more Bzip2 for me :D



    P.S. can admin/moderator integrate this in original post (#1)
     
  23. nakal

    nakal Member

    Messages:
    206
    Likes Received:
    0
    I would not backup MBRs like you suggested, except you expect to restore things on the same drive again. MBR stores the drive geometry and partitioning information.

    When you restore to a fresh drive, after a drive failure for example, it is a better idea to use fdisk, bsdlabel and eventually boot0cfg, in case you want a boot manager.

    It is also possible to use gpart now. These is my preferred way to partition drives at the moment. For more information, how to use GPT partitions on i386 and amd64 and boot from them, read the article on my website: http://m8d.de/news/freebsd-on-gpt.php. It's a bit tricky, but you rather have to understand what I do there, not repeat the steps line by line.
     
  24. graudeejs

    graudeejs Member

    Messages:
    4,594
    Likes Received:
    0
    ye, thank you for reminding me.... (i really forgot about this)
    btw, i don't backup my MBR, if anything i use sysinstall to rebuild partitions on drive and then press "w" in fdisk editor.
    It will write partition table to disk and ask for loader, pick MBR or FreeBSD loader, and exit sysinstall.
    Then i just use bsdlabel to rebuild labels and that is it.
    After that newfs and restore
     
  25. sim

    sim New Member

    Messages:
    157
    Likes Received:
    0
    I backup my servers using rsnapshot from my archive server:

    On each client server, a nightly cron makes a snapshot of each filesystem and mounts them on /snapped_fs (/snapped_fs/, /snapped_fs/usr/, /snapped_fs/var/ etc). So I always have yesterday's complete filetree, mounted and frozen in time. When my archive server connects for the nightly rsnapshot, it syncs the frozen tree, not the live tree. Filesystem snapshots are supposed to be consistent.

    Just to be sure, another nightly cron also runs pg_dumpall. PostgreSQL dumps are point-in-time, consistent dumps which don't require the server to stop or lock any tables. I keep the last 15 dumps, and these are of course part of the filesystem snapshot so they get copied with rsnapshot.

    It's getting late, I wonder if that makes sense lol!

    /sim