The fastest HD cloning

I need to clone a running FreeBSD/386 with some applications inside it (the 300G HD was set up using GPT), to 20+ identical system and HD (well, it is 22 HD). I tried it using dd (although it worked, but it took almost 3 hours.).
gpart shows

Code:
ada0 GPT
1    freebsd-boot
2    freebsd-ufs
3    freebsd-swap

I read yesterday, someone has created a script for cloning in this forum, unfortunately it was created for MBR. Anyone has any idea, to clone as fast as possible?

Thanks.
 
If clone mean to simply copy the entire disk what better and safer way than dd. If there is anything faster how could you trust? If it skip one byte without your knowledge, you got problems. dd would never do that because it don't know how and you can write a very simple script to do all you need.

It's too important not to dedicate the time. It takes me about 68 minutes if I attach a blank/clean 500GB hard-drive to the machine and run a live CD to copy my working drive to the blank hard-drive. If I remember correctly, it takes half that time if I run dd from my working drive (ada0) and want to copy (ada1) to (ada2) ... which means I have three drives on the same machine. For me, for now I prefer the CD way. Forget the hype, you use the right tool for a given job. I use Arch Linux CD and on a partition to do this job but you can use a live FreeBSD CD if you like. A foreign OS has no knowledge to complain about anything. Also I think you're not using bs=64K (the true dd max); that's why it takes you three hours unless your machine was made before 2003. Btw: for cloning NEVER use conv=anything. Only do it this way and use (K) for Linux and small (k) for BSD and you will never go wrong.

copy a to b
Code:
dd if=/dev/sda of=/dev/sdb bs=64K
7631040 + 1 record in
7631040 + 1 record out
500107862016 bytes 500105.25 (500GB) 4096.69-sec  1-h  8-min  122MB/s

copy 0 to 1 - coping working
drive using CD in the middle
Code:
dd if=/dev/ada0 of=/dev/ada1 bs=64k

copy 1 to 2 - using 3
drive on one machine
Code:
dd if=/dev/ada1 of=/dev/ada2 bs=64k

This should work for you. I plan to test every kind of way using FreeBSD only when I find free time, but for now Arch has been doing a very fine job without any single issue when it comes to dd'ing.

btw: if GPT is a problem it might be better to use the CD thing in the middle running a non-BSD live OS to get the job done fuss-free.
 
I run dd to clone boot from live DVD. The reason I asked, because I have check that only 24% out of 300GB been used (so dd wasted a lot of 0z here), and it means, in a day I could only install three FreeBSD systems :(. Because I could only have one system for cloning with live DVD, the other 22 systems are being used by employees (In other words, one at a time). So for all 22 system, it would take more than 5 days works (a whole week just for cloning, set up users, set up IP, set up user Samba shares etc :( ).

Well after thinking again probably I got no choice, since all the old systems have old XP pre-installed already, so the HDs are not clean anyway.

BTW maybe I could increase bs to 128? Is it OK?

Thanks anyway.
 
Code:
so the HDs are not clean...

You could dd zero them clean but you don't have to.

I can't see three hours. Are you sure you add the bs=64k thing or maybe I'm just not understanding your issue.

24% out of 300GB been used (so dd wasted a lot of 0z here),

Windows scatters bytes all over the drive so you still you have to cover the entire disk I think or maybe test something like this:

defrag your original disk and do (100GB) as a test even though (75GB) would be 25%. Could save lots of time. I seen this type of command before but I never tried it beyond MBR size 32256.

dd if=/dev/ad1 of=/dev/ad2 bs=102400b count=1

If all else fail at lease 22 machines is better than a 100 :) Now I'm with you, I hope someone know of a remote solution but it don't seem possible.
 
binyo66 said:
I run dd to clones boot from live DVD. The reason I asked, because I have check that only 24% out of 300GB been used (so dd wasted a lot of 0z here), ...

Yes dd for cloning GPT partitioned disks, is only a viable option if you clone exactly the same partition sizes (this is your case), and the disk to clone is almost full, otherwise you would lose much more in total time than you gain in transfer speed (as you found out the hard way).

In order to avoid this, you need to do cloning on the file system level. There is a perl script in the ports sysutils/clonehdd which auto-partitions the new disk, and uses dump(8)()/restore(8)() for the actual copying.

Personally, I have not so good experiences with the performance of dump(8)()/restore(8)(), however, people keep recommending this because of its reliability.

Some weeks ago, I released my own tool for file system cloning, and it is also in the ports of FreeBSD - sysutils/clone. However, this does not partition the new disk, and this need to be done manually or by a script before cloning.

Other options are net/rsync, which is not the fastest in town, but would outperform dump/restore.

Given that you seem to have plenty of disk space left, you could also simply use cp(1)().

# cp -pxR /source /destination

This would not honour hard links, i.e. copy them as separate files, however this would not be a problem at all, if a GByte or so of extra space wouldn't matter. In my experience, cp is pretty fast.
 
binyo66 said:
I have check that only 24% out of 300GB been used (so dd wasted a lot of 0z here)
[...]
Well after thingking again probably I got no choice, since all the old systems have old xp preinstalled already, so the HDs are not clean anyway...
[...]
BTW may be i could increase bs to 128 ?
Yes you can increase bs to 128k, 256k, maybe even 512k, but not more as the limited disk speed will cancel the benefits.

As for saving space on disks that have been heavily used, you can do something like
# dd if=/dev/zero of=empty_file
on every partition. This will practically overwrite all the unused space that still contains old data with zeros. It will end with a "device is full" error. You can then delete the empty file. There surely are similar tools on Windows.
Now when you create a disk image, simply pipe the output of dd to some compression tool and you will save a lot of space.
 
My general strategy is:

1) dd the MBR, save it to a file
2) Use dump to make a dump of the filesystems
3) Go to the new machine, restore the MBR from the image created with dd earlier
4) Make filesystems with newfs
5) Now restore the filesystem images created with dump using restore

The advantage of dump is that it only dumps files, and not unused space. If you have a 250GB disk with only 10GB used, dd will also copy the unused 240GB, which is a waste of time.
You can also restore dump files to a filesystem of a different size.
You can also use tar or other tools, but dump has the advantage of preserving file flags etc. by default.

I'm not sure about GPT. But I would guess that just using dd to copy the first 150MB of /dev/ad0 should always be enough? This is still fairly fast (several seconds).

As an illustration, here are my notes for a Linux system, I boot the target system from parted magic over PXE, of course, connecting the destination drive to the source system and booting the source system is easier if that's possible.
Don't copy this mindlessly, use your brain, this is just to give you an idea of how this works.

dump:
Code:
mkdir /nfs
mount_nfs 192.168.1.37:/mnt/images /nfs
cd /nfs
dd if=/dev/sda of=image.mbr.20130206 bs=512 count=63
dump -0 -f image.boot.20130206.dump /dev/sda1
dump -0 -f image.root.20130206.dump /dev/sda2

restore:
Code:
mount_nfs 192.168.1.37:/mnt/images /nfs
cd /nfs
dd if=image.mbr.20130206 of=/dev/sda

# Nieuwe partitie: reboot opnieuw naar parted magic en remount nfs

mkfs.ext3 -L /boot /dev/sda1
mkfs.ext3 -L / /dev/sda2

mkdir /dest
mount /dev/sda2 /dest
cd /dest
/nfs/dump/restore -r -f /nfs/images.root.20130206.dump
mount /dev/sda1 /dest/boot
/nfs/dump/restore -r -f /nfs/images.boot.20130206.dump
 
Carpetsmoker said:
You can also use tar or other tools, but dump has the advantage of preserving file flags etc. by default.

net/rsync with the FLAGS option set will do that now, also. And while rsync copies to an empty filesystem can be slow, just syncing up a previously-copied drive is much faster than pretty much anything else. Again, I point to sysutils/rsnapshot, which will make local or remote rsync copies and uses hard links to conserve space.

I'm not sure about GPT. But I would guess that just using dd to copy the first 150MB of /dev/ad0 should always be enough? This is still fairly fast (several seconds).

No, that's not enough. GPT has a second partition table at the end of the disk for redundancy. gpart(8)'s backup and restore options can correctly copy partition tables from one drive to another, even if the second drive is larger (as far as I recall, anyway).

Another option is Clonezilla, which is nearly smart enough. Think of it as a cross-platform version of dump(8). The only problem I've seen in very limited tests with FreeBSD is that it does not recognize swap partitions and wants to copy them as binary data.

One final note about dump(8): thanks to snapshots with -L, it can run while the system is being used. Backup Options For FreeBSD shows examples of things like backup over SSH.
 
I might be wasting web-bytes but I was curious about the 128, 256 and 512 as size input for bs= and it don't matter at all. The only fact remain, if you don't use one of these values it can take up to 3-6-12 hours or more even for this tiny 32GB flash drive. I know this because that's what I use to do for years. I knew nothing of the important of adding a value. All though I now do it on my hard drive thank to what someone said to use bs=64 while googling (and he stood by it after questions of more), today for the first time using a value on my 32GB FLASH DRIVE and both took only about ½ hour to complete. Wow! It use to take all night long after all these years of using no size value. The things I finally learned just because of this OP question. Thank binyo66.

FreeBSD 8.2 say:
Code:
bash-4.1# dd if=/dev/zero of=/dev/da0 bs=64k
dd: /dev/da0: end of device
493569+0 records in
493568+0 records out
32346472448 bytes transferred in 1828.074743 secs (17694283 bytes/sec)
bash-4.1# 
bash-4.1# 
bash-4.1# dd if=/dev/zero of=/dev/da0 bs=512k
dd: /dev/da0: end of device
61697+0 records in
61696+0 records out
32346472448 bytes transferred in 1829.932301 secs (17676322 bytes/sec)
bash-4.1#

Moving forward, knowing that the OP is hard at work dd'ing (or will end-up doing maybe), i'm not going to let anyone else suffer what I been through since 2001. It make since that Windows needs a way to protect itself from the free-be world because it is commercial after all and none of us got the right to put down someone earning a living from his or her hard work. You don't go to school to learn computing to work for peanuts because of so called open-source.

Linux and open-source make money too behind your back. Simply by sueing for millions when some company make money off of using your code ... the one you gave to the open-source community. But open-source may never tell you the author about it, because they now own your code. FreeBSD is about the only open-source who don't respect that game. That's why they work hard to get rid of anything that anything that has to do with GPL or other trick-baby licenses. We might loss some good functions in the process but you still got to love the FreeBSD way.

Sorry about the talk, so back to the point .. You can't win with cp -prv, mv, dump or tar of Windows because it will relocate the ntldr file. If this file is not at the exact starting location on the hard drive partition as when you installed Windows, this OS has measures in place to detect that, and it will issue warning at every boot such as "registry has been change"or it will not start at all or it will leave a bug to never be found. That's why dd is the only function to use for such type partition and MS know it, that's why MS has or will take over the computer BIOS with the so called "safe boot". With this, all other OS have to only kiss-up to exist on that type system. MS friends are IBM, AMD, INTEL and they also have a seat on the ISO committee (front-seat). Anyway, test the ntldr on Windows C drive thing for yourself. You will find that even VirturalBox cannot save you unless you dd that partition. If any of this is incorrect please prove it with some real-life testing. A good hard-core week should do it. Once you get your main-drive setup and backed-up, there is nothing you can't try for yourself.
 
It's not 64, 128, and 512, it's 64K, 128K, or 512K or more. Trying to copy a drive with only the default 512-byte buffer size will take forever. A larger buffer cuts down the overhead and lets the hardware go as fast as it can. Once you get above 64K, it doesn't make much difference on a normal hard drive. Larger buffers could help on faster things like RAID arrays or SSDs.
 
You can't win with cp -prv, mv, dump or tar of Windows because it will relocate the ntldr file. If this file is not at the exact starting location on the hard drive partition as when you installed Windows

Huh? Who mentioned Windows? Certainly not the OP. And even if he is running Windows, how does that matter when cloning FreeBSD drives?

In addition, you can use the fixboot command to re-write the ntldr file (available in the recovery console). There's also a free Linux/UNIX tool which does the same, but I forgot the name ...
 
Carpetsmoker, your right. What happen was I was reading a blog about how lots of companies really use FreeBSD as firewalls, IDS systems and more, but with a bunch of Windows systems on the network. It kind of tripped me out to think big business's would use Windows on their networking when BSD or Linux can also be use for a more secure desktops and terminals. Anyway it brought back memories and I guest I took for granted that the OP had the same when he clearly stated difference. Anyway, it explains a very important Windows issue that many never known was the problem for given situation, and now, you just explained the solution to that problem, something even I been seeking all my life with Windows. Funny thing is I don't even use Windows anymore other than making room for it inside vBox so MS and crackers has a place to check their applications. I just hope the OP found what works best for him.
 
Back
Top