Cloning or duplicating a running system using dump/restore

fbsd1

Active Member

Reaction score: 52
Messages: 223

Cloning or duplicating a running system
using dump/restore.

Assumptions:

The dump command will be issued from a host running in multiple-user-mode. The host is a default configuration. IE: / as “a” partition, /var as “d” partition, /tmp as “e” partition and /usr as “f” partition. We will dump from the hard drive where the running system is installed to a second hard drive cabled to the motherboard or an external USB cabled hard drive or a USB memory stick of sufficient size. Whatever the target media is, all the data currently on it will be destroyed by this procedure. Even the MBR (Master Boot Record) will be recreated.

For this example /dev/ad0 is the hard drive where the live system is running.

The target is an USB flash stick /dev/da0.

The following is the sequence of commands used:

* Collect live file system sizes and save
df -h > liveFSsizes

Code:
cat liveFSsizes

Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/ad0s1a    496M    169M    287M    37%    /
devfs          1.0K    1.0K      0B   100%    /dev
/dev/ad0s1d    1.5G     26M    1.4G     2%    /var
/dev/ad0s1e    496M     10K    456M     0%    /tmp
/dev/ad0s1f     15G    410M     13G     3%    /usr
* Collect live Slice Partition sizes and save
Bsdlabel lists the live Slice Partition sizes to a file. This file will be used later to allocate the slice partitions to the target.

bsdlabel ad0s1 > liveSPsizes

Code:
cat liveSPsizes

# /dev/ad0s1:
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:  1048576        0    4.2BSD        0     0     0
  b:  2291584  1048576      swap
  c: 39862305        0    unused        0     0         
  d:  3241984  3340160    4.2BSD        0     0     0
  e:  1048576  6582144    4.2BSD        0     0     0
  f: 32231585  7630720    4.2BSD        0     0     0
If you want to increase or decrease the partition allocation space sizes edit the liveSPsizes.usb file. I have edited this changing partition sizes so it will fit on a 2GB USB stick. You can see from the df –h output that the default sizes sysinstall calculates leaves a lot of free space in the partitions.

Code:
# /dev/da0s1: 2GB USB flash drive stick
8 partitions:
#        size   offset    fstype   [fsize bsize bps/cpg]
  a:     300M        *    4.2BSD        0     0     0   
  b:     200M        *      swap                        
  c:        *        *    unused        0     0         
  d:      50M        *    4.2BSD        0     0     0   
  e:     100M        *    4.2BSD        0     0     0   
  f:        *        *    4.2BSD        0     0     0
* Zero out the target MBR destroying all the data on the target.

dd if=/dev/zero of=/dev/da0 count=2

* fdisk the target with a new MBR.

fdisk -BI /dev/da0

The –B means Reinitialize the boot code contained is sector 0 of the disk
Default from /boot/mbr
The ‘I’ means initialize sector 0 slice table for one slice covering the entire disk.

You will get 2 messages.
Code:
Fdisk: invalid fdisk partition table found 
Fdisk: Class not found
Disregard these messages, they are the result of zeroing out the old MBR.

* Label the target:

bsdlabel -B -w da0s1

The –B means bootstrap code will be read from /boot/boot & written to the disk
The –w means write a standard label

* Allocate the partitions
* Restore the partition sizes as defined in the file liveSPsizes

bsdlabel -R -w da0s1 liveSPsizes

* Format all the new empty file system on the target.

Code:
newfs –U /dev/da0s1a               # /   
newfs -U /dev/da0s1d               # /var
newfs -U /dev/da0s1e               # /tmp
newfs -U /dev/da0s1f                # /usr
* Mount target file system ‘a’ / and clone

Code:
mount /dev/da0s1a /mnt
cd /mnt
dump -0Lauf - /dev/ad1s1a  | restore -rf -
Flags:
-0 means do full dump,
-L means take snapshot because source is live file system,
-a means enforce writing until a end-of media is reached
-u means update the /etc/dumpdates file with the results
-f - means standard output is where the dumped data is to be written.

You can think of 'standard output' as a un-named virtual file or buffer. If there was no pipe | to the restore program the dumped data would roll across the terminal screen.

The restore program, on the other side of the | pipe, usually reads from the system's tape drive. But in this case, it reads from standard input as the -f – command line option indicates. It restores the data to where the working directory is currently positioned. In the example the current working directory is /mnt where the target file system. ‘a’ is mounted The –r flag means rebuild the file system. The restored file system will be of the same size as the one dumped including its free space it you did not change the content of the (liveSPsizes) file. In reality what we have with the above dump command is as dump writes a block of data to standard output its immediately handed to restores standard input and written to the target. The above snippet of code would have to be duplicated for each file system you wanted to dump.

The following (fbsd2clone) script employs the sample code snippet detailed above.

Code:
#!/bin/sh
# This script will use dump/restore to clone your running
# system to another motherboard cabled hard drive or
# USB cabled hard drive or USB flash stick.
# This is run as root. 
#
# Change these device unit pre-fixs as needed
#     ad0 is the live file system
#     da0 is the target


echo "Collect live file system sizes and save"
df -h  > liveFSsizes
cat liveFSsizes

echo "  "
echo "  "
echo "Collect live Slice Partition sizes and save"
bsdlabel ad0s1 > liveSPsizes
cat liveSPsizes
echo "  "
echo "  "
cat liveSPsizes.usb

# At this point you can edit the liveSPsizes files and 
# increase or decrease the file system partition sizes
# man bsdlabel for details

echo "  "
echo "  "
echo "Prepare the target"
dd if=/dev/zero of=/dev/da0 count=4
fdisk -BI /dev/da0
bsdlabel -B -w da0s1
bsdlabel -R da0s1 liveSPsizes.usb
newfs -U /dev/da0s1a
newfs -U /dev/da0s1d 
newfs -U /dev/da0s1e 
newfs -U /dev/da0s1f 

echo "  "
echo "  "
echo "Mount target file system 'a' / and clone"
mount /dev/da0s1a /mnt
cd /mnt
dump -0Lauf - /dev/ad0s1a  | restore -rf -

echo "  "
echo "  "
echo "Mount target file system 'd' /var and clone"
mount /dev/da0s1d /mnt/var
cd /mnt/var  
dump -0Lauf - /dev/ad0s1d  | restore -rf -

echo "  "
echo "  "
echo "Mount target file system 'e' /tmp and clone"
mount /dev/da0s1e /mnt/tmp
cd /mnt/tmp
dump -0Lauf - /dev/ad0s1e  | restore -rf -

echo "  "
echo "  "
echo "Mount target file system 'f'/usr  and clone"
mount /dev/da0s1f /mnt/usr
cd /mnt/usr
dump -0Lauf - /dev/ad0s1f  | restore -rf -

echo "  "
echo "  "
echo "Clean up"
sync
cd /root
umount /mnt/usr
umount /mnt/tmp
umount /mnt/var
umount /mnt

echo " Script completed"
 

philipan

New Member


Messages: 1

typo

* Restore the partition sizes as defined in the file liveSPsizes/liveSPsizes.usb

should be
Code:
bsdlabel -R da0s1 liveSPsizes
instead of
Code:
bsdlabel -R -w da0s1 liveSPsizes
?

However, it is correct in the script.
Thanks! Extremely useful!
 

synonymous

New Member


Messages: 4

Hi,
Does this method work with a LiveCD as well?
I booted from a FreeBSD variant LiveCD, and then at the shell prompt created a 700MB slice on the 20GB HDD, partitioned it, and mounted the first partition (/dev/ad0s1a) to /mnt/tmp. However, I am unable to "cd" to /mnt/tmp. Is it safe to presume that this method works only for pre-installed OS?
I am trying to dump the LiveCD filesystem footprint (that is loaded in RAM) to the slice, and then modify the grub menu to boot to that partition. Maybe I am misunderstanding something very basic - can you please enlighten?
-SR
 

Bashar

New Member


Messages: 3

No need todo anything to /dev/da0s1b. I believe it should be the swap partition, no?
 

J65nko

Well-Known Member

Reaction score: 127
Messages: 453

Indeed, you don't have to format/newfs a swap partition.
 

rustyx

New Member

Reaction score: 2
Messages: 11

This thread is a bit outdated, because:
1. The default partitioning scheme is no longer MBR but GPT
2. By default journaling soft-updates are enabled, which are incompatible with the dump -L flag
3. Partitions need to be aligned on SSDs

So here's my updated solution (tested on FreeBSD 10 and 11):

Suppose the source drive is da1 and the target is da0, and they are both of the same size.

List partitions:

gpart show da1
Code:
> gpart show da1
=>       34  104857532  da1  GPT  (50G)
         34    8388608    1  freebsd-swap  (4.0G)
    8388642       1024    2  freebsd-boot  (512K)
    8389666   96467900    3  freebsd-ufs  (46G)
mount
Code:
> mount
/dev/da1p3 on / (ufs, local, journaled soft-updates)
devfs on /dev (devfs, local, multilabel)
Clone the partition table:

If the drives are identical:

gpart backup da1 | gpart restore -F da0

Or create the partitions manually as needed:

Code:
gpart create -s gpt da0
gpart add -t freebsd-boot -s 512K da0
gpart add -t freebsd-swap -s 4G -a 1M da0
gpart add -t freebsd-ufs da0
Code:
> gpart show da0
=>       34  167772093  da0  GPT  (80G)
         34       1024    1  freebsd-boot  (512K)
       1058        990       - free -  (512K)
       2048    8388608    2  freebsd-swap  (4.0G)
    8390656  159381471    3  freebsd-ufs  (76G)
Notice how swap and ufs partitions are aligned on 1MB (-a 1M) - important for SSD's. Also note that a non-UEFI freebsd-boot should not be too large or it won't boot.

Install the boot code into the freebsd-boot partition:

gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 da0

Now for every freebsd-ufs partition:

Format the partition:

newfs -jt /dev/da0p3

-j is for journaling soft-updates
-t is for TRIM support

Clone the data:

When journaling is enabled, dump cannot make a snapshot, so stop as many processes as possible (especially database processes) and dump without making the snapshot (the -L flag):

Code:
mount -t ufs /dev/da0p3 /mnt
cd /mnt
dump -0af - /dev/da1p3 | restore -rf -
Note:
Dumping a live partition without a snapshot is unsafe (files that are written during the dump can become corrupted).

A safer alternative is to disable journaling (can't be done on a live FS) and dump offline (e.g. using a LiveCD) with the -L flag:

Code:
tunefs -j disable /dev/da0p3
mount -t ufs /dev/da0p3 /mnt
cd /mnt
dump -L0af - /dev/da1p3 | restore -rf -
tunefs -j enable /dev/da0p3
Perform some cleanup:

[CMD=""]rm /mnt/var/run/*[/CMD]
[CMD=""]rm /mnt/restoresymtable[/CMD]

Adjust /mnt/etc/fstab if the numbering of partitions changed. Remember that the kernel uses the information from /etc/fstab to determine the root partition (and can thus fail to boot).

Optionally TRIM the whole drive (can improve performance on SSDs and save space on VMs): [CMD=""]fsck_ufs -Ey /dev/da0p3[/CMD]
 
Last edited:

trev

Aspiring Daemon

Reaction score: 175
Messages: 806

Install the boot code into the freebsd-boot partition:

gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 da0
On my Mac Mini, to get it to boot the new drive I also needed:


gpart set -a bootme -i 2 da0


Whereas previously I would boot the OS X CDROM and "bless" the drive. This bootme attribute also enabled me to test the clone by booting the USB attached drive and not the internal drive before I disassembled the Mac Mini and replaced the old drive with the new one.

Clone the data:
[...]
Code:
dump -0afL - /dev/da1p3 | restore -rf -
Ah, not quite... you cannot use the -L flag after -f, so instead:

Code:
dump -L -0af - /dev/da1p3 | restore -rf -
Perform some cleanup:
[CMD=""]rm /mnt/var/run/*[/CMD]
Adjust /mnt/etc/fstab if the numbering of partitions changed. Remember that the kernel uses the information from /etc/fstab to determine the root partition (and can thus fail to boot).
I also had a restoresymtable left behind in every partition's root which needed cleaning up. More about that here.

Thanks for the post btw, it was invaluable.
 

horseflesh

Member

Reaction score: 4
Messages: 70

Thanks for this guide. It was just what I needed to back up my little firewall system to a bootable thumb drive.

If you cannot use dump -L on a journaled filesystem, how do people make copies of live, journaled filesystems? It seems like there must be a way to make a backup without rebooting off a Live CD and changing the journaling options.
 
Top