clone a local disk

fluca1978

fluca1978

Hi all,
I have to clone an entire disk into another locally attached disk, and I was thinking about dd(1) (I want to clone partitions at all). Any other suggestion and/or command to use?
 
fluca1978 said:
Hi all,
I have to clone an entire disk into another locally attached disk, and I was thinking about dd(1) (I want to clone partitions at all). Any other suggestion and/or command to use?
Click to expand...

Is the current disk partitioned with GPT?
(The backup GPT at the end of the disk will be in the wrong place on a copy, unless the new disk is exactly the same size as the old one.)

Is the new disk at least as large as the old one?
(dd(1) will copy the partition information exactly, the new disk cannot be smaller.)

Do you care if any extra space is wasted?
(See the question above. There are utilities to grow UFS filesystems, but they are not perfect.)
 
fluca1978 said:
Hi all,
I have to clone an entire disk into another locally attached disk, and I was thinking about dd(1) (I want to clone partitions at all). Any other suggestion and/or command to use?
Click to expand...

This one bugged me since I started to work seriously with FreeBSD at the end of 2010. Coming from Mac OS X, I was used to using ditto(1) or asr(8) for this kind of operation.

On FreeBSD dump(8)|restore(8) is for my setup very slow, about 10-15 MByte/s, so cloning my 3 TB disk which contains 2.3 TB of data takes about 50-70 hours. I still fail to understand how people come to keeping-on suggesting this.

dd(1) did a fairly well job - 70 MByte/s - 11 hours for 2.3 TB. As Warren pointed out, this is only recommendable for cloning of exactly identical drives.

dump|restore is not well suited for live file systems (there is an option) but the performance hit is totally unacceptable for huge file systems. dd is not suited for cloning live file systems. If you use it on a live FS, then you will certainly need to run fsck afterwards, and it will certainly find some issues.

cp(1) with the -pR options would be very good. It is quite fast, 67 MByte/s, and it can be used on live file systems (which means that the disk catalog will be OK, but of course, if you change a file during cloning, which has been copied already, then don't expect the changes magically making it into the clone). But, unfortunately it does not honour hard links, but copies them as separate files.

rsync(1) copies everything but it is so very much slower than cp, 20 MByte/s.

Anyhow, the situation did not satisfy me, and I started a project for a tool programmed in C named clone.

A week ago, I finished the work on the engine, and so far it is working to my satisfaction. I just released it under the 2 clause BSD license at Google Project Hosting: clone - File system cloning.

File tree cloning tool which runs 3 threads - a scheduler, a reader and a writer thread, and reading and writing occurs in parallel, which is most beneficial of course for transferring data from one disk to another.

For example cloning a whole file hierarchy of in total 2.3 TBytes of data from one disk to another took 7.5 h, so the average transfer rate was about 89 MByte/s -- for everything, i.e. directories, very small and very big files, symbolic links, hard links, and including file attributes, flags, ACLs and EAs.

This tool is useful for cloning (thus backing-up) live file systems. Works on FreeBSD and Mac OS X.
Click to expand...

For using it on FreeBSD 9.x:

$ svn checkout [URL]http://clone.googlecode.com/svn/trunk/[/URL] clone
$ cd clone
$ sudo make install clean
$ sudo clone /path/to/original /path/to/clone

Of course, you do not want to simply trust me, and so you do want to carefully examine the source code before using the tool. And by doing this, you passed by the license conditions and carefully paid attention to the risk dealing (CAPITAL LETTER) section.
 
Last edited by a moderator:
rolfheinrich said:
On FreeBSD dump(8)()|restore(8)() is for my setup very slow, about 10-15 MByte/s, so cloning my 3 TB disk which contains 2.3 TB of data takes about 50-70 hours. I still fail to understand how people come to keeping-on suggesting this.
Click to expand...

It's because dump(8) is the only thing that really understands the UFS filesystem. Note that using -b64 can double the speed over the default. Don't use more than 64, though. And yes, dump(8) should be faster than it is.

dd(1)() did a fairly well job - 70 MByte/s - 11 hours for 2.3 TB. As Warren pointed out, this is only recommendable for cloning of exactly identical drives.
Click to expand...

Or copying to a larger target drive. While dd(1) can get higher raw transfer speeds, it also wastes time copying empty blocks. Copying a half-full disk takes twice as long as it should.

cp(1)() with the -pR options would be very good. It is quite fast, 67 MByte/s, and it can be used on live file systems (which means that the disk catalog will be OK, but of course, if you change a file during cloning, which has been copied already, then don't expect the changes magically making it into the clone). But, unfortunately it does not honour hard links, but copies them as separate files.
Click to expand...

I confess that I rarely use cp(1) for anything but trivial copies, and use rsync(1) for anything else. Mostly that's because it doesn't recopy things that are already present.

rsync(1)() copies everything but it is so very much slower than cp, 20 MByte/s.
Click to expand...

There might be opportunities to improve the way rsync(1) works, also.

Anyhow, the situation did not satisfy me, and I started a project for a tool programmed in C named clone.

A week ago, I finished the work on the engine, and so far it is working to my satisfaction. I just released it under the 2 clause BSD license at Google Project Hosting: clone - File system cloning.
Click to expand...

Very cool. If you could write to an archive, it could be a better replacement for dump(8)/restore(8).
 
Thanks guys, of course I was talking about cloning two identical drives, otherwise some extra opreration would have been required (e.g., partition table editing). I will have a look at the porposed clone to see how it works.
 
wblock@ said:
... If you could write to an archive, it could be a better replacement for dump(8)/restore(8).
Click to expand...

May it be an option to wrap clone into a shell script, which would create and mount a disk-image of sufficient size, and finally unmount it. More or less like this script clone2img shown below does.

However, I do not know whether the extra time which is necessary for creating the disk image can be compensated by the speed of clone.

Code: 
#!/bin/sh
# clone2img -- this is a proof of the concept only.
# The script must be enhanced with error checking, before it
# may be used on production systems.
#
# Usage: clone2img originalDir imageFile
#
dd if=/dev/zero of=$2 bs=1k count=`echo "\`du -ksxP $1 | cut -f1\` + 512" | bc`
mdconfig -a -t vnode -f $2 -u 0
bsdlabel -w md0 auto
newfs md0a
mount /dev/md0a /mnt
rm -r /mnt/.snap
clone $1 /mnt
umount /mnt
mdconfig -d -u 0

For getting the data into the final clone, another shell script img2clone must be used:

Code: 
#!/bin/sh
# img2clone -- this is a proof of the concept only.
# The script must be enhanced with error checking, before it
# may be used on production systems.
#
# Usage: img2clone imageFile clonedDir
#
mdconfig -a -t vnode -f $1 -u 0
bsdlabel -w md0 auto
mount /dev/md0a /mnt
clone /mnt $2
umount /mnt
mdconfig -d -u 0

Please be careful, these scripts don't do error checking at all, as written in the respective comments, they are meant as proof of the concept only.

In the following examples, clone operates on the same disk for reading and writing, and a speed gain for parallel reading and writing cannot be expected.

# clone2img /usr/bin backup.img
Code: 
78504+0 records in
78504+0 records out
80388096 bytes transferred in 1.017963 secs (78969571 bytes/sec)
/dev/md0a: 76.7MB (156992 sectors) block size 32768, fragment size 4096
	using 4 cylinder groups of 19.19MB, 614 blks, 2560 inodes.
super-block backups (for fsck_ffs -b #) at:
 192, 39488, 78784, 118080
File system cloning by Dr. Rolf Jansen
Cyclaero Ltda. (c) 2013 - (r24)

clone /usr/bin/ /mnt/
.
70.8 MB in 2.62 s -- 27.0 MB/s

# img2clone backup.img test_usrbin
Code: 
File system cloning by Dr. Rolf Jansen
Cyclaero Ltda. (c) 2013 - (r24)

clone /mnt/ usrbin/
..
70.8 MB in 2.61 s -- 27.1 MB/s

# ls -lR /usr/bin > orig.txt; ls -lR test_usrbin > clone.txt; diff -y orig.txt clone.txt
Code: 
total 145516							total 145516
-r-xr-xr-x   3 root  wheel     200352 Dec 12 22:41 CC		-r-xr-xr-x   3 root  wheel     200352 Dec 12 22:41 CC
-r-xr-xr-x   3 root  wheel      92864 Dec 12 22:41 Mail		-r-xr-xr-x   3 root  wheel      92864 Dec 12 22:41 Mail
-r-xr-xr-x   1 root  wheel      28368 Dec 12 22:41 addftinfo	-r-xr-xr-x   1 root  wheel      28368 Dec 12 22:41 addftinfo
-r-xr-xr-x   1 root  wheel     634416 Dec 12 22:41 addr2line	-r-xr-xr-x   1 root  wheel     634416 Dec 12 22:41 addr2line
-r-xr-xr-x   1 root  wheel     162314 Jan  3  2012 afmtodit	-r-xr-xr-x   1 root  wheel     162314 Jan  3  2012 afmtodit
-r-xr-xr-x  15 root  wheel        164 Dec 12 22:41 alias	-r-xr-xr-x  15 root  wheel        164 Dec 12 22:41 alias
-r-xr-xr-x   1 root  wheel       9616 Dec 12 22:41 apply	-r-xr-xr-x   1 root  wheel       9616 Dec 12 22:41 apply
-r-xr-xr-x   4 root  wheel      21016 Dec 12 22:41 apropos	-r-xr-xr-x   4 root  wheel      21016 Dec 12 22:41 apropos
-r-xr-xr-x   2 root  wheel     863632 Dec 12 22:41 ar		-r-xr-xr-x   2 root  wheel     863632 Dec 12 22:41 ar
-r-xr-xr-x   1 root  wheel    1218648 Dec 12 22:41 as		-r-xr-xr-x   1 root  wheel    1218648 Dec 12 22:41 as
-r-xr-xr-x   1 root  wheel       7136 Dec 12 22:41 asa		-r-xr-xr-x   1 root  wheel       7136 Dec 12 22:41 asa
-r-sr-xr-x   4 root  wheel      29656 Dec 12 22:41 at		-r-sr-xr-x   4 root  wheel      29656 Dec 12 22:41 at
-r-sr-xr-x   4 root  wheel      29656 Dec 12 22:41 atq		-r-sr-xr-x   4 root  wheel      29656 Dec 12 22:41 atq
-r-sr-xr-x   4 root  wheel      29656 Dec 12 22:41 atrm		-r-sr-xr-x   4 root  wheel      29656 Dec 12 22:41 atrm
-r-xr-xr-x   2 root  wheel     137864 Dec 12 22:41 awk		-r-xr-xr-x   2 root  wheel     137864 Dec 12 22:41 awk
-r-xr-xr-x   2 root  wheel      12720 Dec 12 22:41 b64decode	-r-xr-xr-x   2 root  wheel      12720 Dec 12 22:41 b64decode
-r-xr-xr-x   2 root  wheel       8264 Dec 12 22:41 b64encode	-r-xr-xr-x   2 root  wheel       8264 Dec 12 22:41 b64encode
-r-xr-xr-x   1 root  wheel      19496 Dec 12 22:41 banner	-r-xr-xr-x   1 root  wheel      19496 Dec 12 22:41 banner
-r-xr-xr-x   1 root  wheel       7176 Dec 12 22:41 basename	-r-xr-xr-x   1 root  wheel       7176 Dec 12 22:41 basename
-r-sr-xr-x   4 root  wheel      29656 Dec 12 22:41 batch	-r-sr-xr-x   4 root  wheel      29656 Dec 12 22:41 batch
-r-xr-xr-x   1 root  wheel      40480 Dec 12 22:41 bc		-r-xr-xr-x   1 root  wheel      40480 Dec 12 22:41 bc
-r-xr-xr-x   1 root  wheel      16552 Dec 12 22:41 bdes		-r-xr-xr-x   1 root  wheel      16552 Dec 12 22:41 bdes
-r-xr-xr-x  15 root  wheel        164 Dec 12 22:41 bg		-r-xr-xr-x  15 root  wheel        164 Dec 12 22:41 bg
-r-xr-xr-x   1 root  wheel       5784 Dec 12 22:41 biff		-r-xr-xr-x   1 root  wheel       5784 Dec 12 22:41 biff
-r-xr-xr-x   1 root  wheel       8136 Dec 12 22:41 brandelf	-r-xr-xr-x   1 root  wheel       8136 Dec 12 22:41 brandelf
-r-xr-xr-x   1 root  wheel      37504 Dec 12 22:41 bsdcpio	-r-xr-xr-x   1 root  wheel      37504 Dec 12 22:41 bsdcpio
-r-xr-xr-x   7 root  wheel      52072 Dec 12 22:41 bsdgrep	-r-xr-xr-x   7 root  wheel      52072 Dec 12 22:41 bsdgrep
-r-xr-xr-x   1 root  wheel      11600 Dec 12 22:41 bsdiff	-r-xr-xr-x   1 root  wheel      11600 Dec 12 22:41 bsdiff
-r-xr-xr-x   1 root  wheel      68520 Dec 12 22:41 bsdtar	-r-xr-xr-x   1 root  wheel      68520 Dec 12 22:41 bsdtar
-r-xr-xr-x   3 root  wheel      24048 Dec 12 22:41 bsnmpget	-r-xr-xr-x   3 root  wheel      24048 Dec 12 22:41 bsnmpget
-r-xr-xr-x   3 root  wheel      24048 Dec 12 22:41 bsnmpset	-r-xr-xr-x   3 root  wheel      24048 Dec 12 22:41 bsnmpset
-r-xr-xr-x   3 root  wheel      24048 Dec 12 22:41 bsnmpwalk	-r-xr-xr-x   3 root  wheel      24048 Dec 12 22:41 bsnmpwalk
-r-xr-xr-x   1 root  wheel       8472 Dec 12 22:41 bspatch	-r-xr-xr-x   1 root  wheel       8472 Dec 12 22:41 bspatch
-r-xr-xr-x   1 root  wheel       7104 Dec 12 22:41 bthost	-r-xr-xr-x   1 root  wheel       7104 Dec 12 22:41 bthost
-r-xr-sr-x   1 root  kmem       12272 Dec 12 22:41 btsockstat	-r-xr-sr-x   1 root  kmem       12272 Dec 12 22:41 btsockstat
-r-xr-xr-x   3 root  wheel      32080 Dec 12 22:41 bunzip2	-r-xr-xr-x   3 root  wheel      32080 Dec 12 22:41 bunzip2
-r-xr-xr-x   2 root  wheel      78704 Dec 12 22:41 byacc	-r-xr-xr-x   2 root  wheel      78704 Dec 12 22:41 byacc
-r-xr-xr-x   3 root  wheel      32080 Dec 12 22:41 bzcat	-r-xr-xr-x   3 root  wheel      32080 Dec 12 22:41 bzcat
-r-xr-xr-x   9 root  wheel      89480 Dec 12 22:41 bzegrep	-r-xr-xr-x   9 root  wheel      89480 Dec 12 22:41 bzegrep
-r-xr-xr-x   9 root  wheel      89480 Dec 12 22:41 bzfgrep	-r-xr-xr-x   9 root  wheel      89480 Dec 12 22:41 bzfgrep
-r-xr-xr-x   9 root  wheel      89480 Dec 12 22:41 bzgrep	-r-xr-xr-x   9 root  wheel      89480 Dec 12 22:41 bzgrep
-r-xr-xr-x   3 root  wheel      32080 Dec 12 22:41 bzip2	-r-xr-xr-x   3 root  wheel      32080 Dec 12 22:41 bzip2
-r-xr-xr-x   1 root  wheel      11232 Dec 12 22:41 bzip2recov	-r-xr-xr-x   1 root  wheel      11232 Dec 12 22:41 bzip2recov
-r-xr-xr-x   4 root  wheel        164 Dec 12 22:41 bzless	-r-xr-xr-x   4 root  wheel        164 Dec 12 22:41 bzless
-r-xr-xr-x   3 root  wheel     200352 Dec 12 22:41 c++		-r-xr-xr-x   3 root  wheel     200352 Dec 12 22:41 c++
-r-xr-xr-x   1 root  wheel      36720 Dec 12 22:41 c++filt	-r-xr-xr-x   1 root  wheel      36720 Dec 12 22:41 c++filt
-r-xr-xr-x   1 root  wheel       5856 Dec 12 22:41 c89		-r-xr-xr-x   1 root  wheel       5856 Dec 12 22:41 c89
-r-xr-xr-x   1 root  wheel       6256 Dec 12 22:41 c99		-r-xr-xr-x   1 root  wheel       6256 Dec 12 22:41 c99
-r-xr-xr-x   2 root  wheel      23992 Dec 12 22:41 cal		-r-xr-xr-x   2 root  wheel      23992 Dec 12 22:41 cal
-r-xr-xr-x   1 root  wheel      50840 Dec 12 22:41 calendar	-r-xr-xr-x   1 root  wheel      50840 Dec 12 22:41 calendar
-r-xr-xr-x   1 root  wheel       7808 Dec 12 22:41 cap_mkdb	-r-xr-xr-x   1 root  wheel       7808 Dec 12 22:41 cap_mkdb
-r-xr-xr-x   1 root  wheel      19784 Dec 12 22:41 catman	-r-xr-xr-x   1 root  wheel      19784 Dec 12 22:41 catman
-r-xr-xr-x   2 root  wheel     450184 Dec 12 22:41 cc		-r-xr-xr-x   2 root  wheel     450184 Dec 12 22:41 cc
-r-xr-xr-x  15 root  wheel        164 Dec 12 22:41 cd		-r-xr-xr-x  15 root  wheel        164 Dec 12 22:41 cd
-r-xr-xr-x   1 root  wheel      23880 Dec 12 22:41 chat		-r-xr-xr-x   1 root  wheel      23880 Dec 12 22:41 chat
...
...
...

You may note, that clone (called by the user root) reproduces all the hard links, file modes and times in that directory. It reproduces also all the schg of a variety of files. So, before removing the created test_usrbin directory you need to call:
# chflags -R noschg test_usrbin
 
Please allow me a quick follow-up.

Since my last post, I added more features to my new cloning tool (exclude lists, incremental and synchronization mode, ...), and today it made it into the ports of FreeBSD:

sysutils/clone

# man clone
Code: 
clone(1)                    User's Reference Manual                   clone(1)

[B]NAME[/B]
     [B]clone[/B] -- A file tree cloning tool

[B]SYNOPSIS[/B]
     [B]clone[/B] [[B]-c[/B] [I]roff|woff|rwoff[/I]] [[B]-d[/B]|[B]-i[/B]|[B]-s[/B]] [[B]-x[/B] [I]exclude-list[/I]]
           [[B]-X[/B] [I]excl-list-file[/I]] [[B]-y[/B]] [[B]-h[/B]|[B]-?[/B]|[B]?[/B]] [I]source/[/I] [I]destination/[/I]

[B]DESCRIPTION[/B]
     [B]clone[/B] is a file tree cloning tool which runs 3 threads - a scheduler
     (main), a reader, and a writer thread. Reading and writing occurs in par-
     allel. While this is most beneficial for transferring data from one phys-
     ical disk to another, [B]clone[/B] is also very well suited for cloning a file
     tree to any place on the same disk.

     Cloning includes the whole directory hierarchy, i.e. sub-directories,
     files, hard links, symbolic links, attributes (modes, flags, times),
     extended attributes and access control lists.

     [B]clone[/B] is useful for cloning (thus backing-up) live file systems, and it
     can also be used in incremental and synchronization mode.

     [B]clone[/B] works on FreeBSD and Mac OS X.

     [B]clone[/B] is very fast, for example, cloning a whole UFS2 file hierarchy on
     FreeBSD 9.1 of in total 2.3 TBytes of data from one hard disk to another
     took 7.5 h, so the average transfer rate for all kind of files (very
     small up to very big ones) was about 89 MByte/s.

     [B]Optional flags:[/B]

     [B]-c[/B] [I]roff | woff | rwoff[/I]
              Selectively turn off the file system cache for reading or writ-
              ing or for reading and writing -- the caches are on by default.

     The options -d, -i, -s are mutually exclusive:

     [B]-d[/B]       Delete the contents of the destination before cloning, but do
              not remove the destination directory or mount point itself. Stop
              on error. Deletion requires user confirmation, unless it is con-
              firmed inherently by the -y flag. Be careful, deletion cannot be
              undone! NOTE: It may take its time to empty hugh file trees, and
              it may be faster to use newfs(8) for initializing big volumes.

     [B]-i[/B]       Incrementally add new content to or change content in the desti-
              nation, but do not touch content in destination that does not
              exist in source.

     [B]-s[/B]       Completely synchronize destination with source.

     [B]-x[/B] [I]exclude-list[/I]
              Colon separated list of entity names or full path names to be
              excluded from cloning. Use full path names to single out exactly
              one item. Use entity names, if all existing entities having that
              name should be excluded.

     For example:
              -x ".snap:/.sujournal:.DS_Store:/fullpath/to a/volatile cache"

     [B]-X[/B] [I]excl-list-file[/I]
              File containing a list of entity names or full path names to be
              excluded -- one item per line.

     [B]-y[/B]       Automatically answer with y(es) to y|n confirmation prompts.

     [B]-h[/B] | [B]-?[/B] | [B]?[/B]
              Show the usage instructions.

     [B]Required arguments:[/B]

     [I]source/[/I]  Path to the source directory or mount point to be cloned. The
              final '/' may be omitted.

     [I]destination/[/I]
              Path to the destination directory or mount point. If the desti-
              nation does not exist, then it will be created. The final '/'
              may be omitted.

[B]EXAMPLES[/B]
     Initial cloning of the root volume / to a volume mounted on /mnt:
              clone / /mnt

     Same as above, but empty /mnt before cloning:
              clone -d / /mnt

     Incrementally update the clone with respect to the master:
              clone -i / /mnt

     Keep the clone exactly synchronized to the master:
              clone -s / /mnt

     Create a non-versioned clone of a versioned source tree:
              clone -x .svn /path/to/my/project /path/to/non-versioned/project

     Clone a directory to a share and strip file system specific meta data:
              clone -x ".DS_Store:.AppleDouble:.AppleDesktop:.appdisk"
              /path/to/afpvol /path/to/the/share

[B]SEE ALSO[/B]
     cp(1), ditto(1), rsync(1), dd(1), newfs(8), dump(8), restore(8), asr(8)

[B]AUTHOR[/B]
     Dr. Rolf Jansen. Copyright (c) 2013 Cyclaero Ltda. All rights reserved.

[B]BUGS[/B]
     [B]clone[/B] has been carefully developed and tested. Anyway, [B]clone[/B] is provided
     without any expressed or implied warrantee on being 100 % bug free.

     In-proper use of [B]clone[/B] may delete or overwrite data. Take special care
     with the -d and the -s option. Also always be sure, that the last command
     line argument designates a path to a file tree which really is under dis-
     position.

     In NO event shall the author and/or copyright owner be liable for ANY
     damages resulting from ANY use of this software. Use [B]clone[/B] at your own
     risk!

FreeBSD, Darwin                February 8, 2013                FreeBSD, Darwin
 
old topic but interesting ,what are todays solutions for cloning freebsd to another hdd? (any size ,even small if possible).
 
Just today I had reason to investigate dd, gdd [coreutils], dcfldd, but settled on rsync because it can run in the background as the computer is used as usual, with the
bwlimit=800 parameter... . Takes a bit of practice and study first and the usual "wrongly placed parameter and data is lost" cautions...
 
You must log in or register to reply here.
Back
Top