Solved dd disc clone advice

cpcnw

cpcnw

I need to clone an 8TB SATA disc that has a single UFS partition but contains 6TB of data onto an identical drive. So far I have tried;

Code: 
dd if=/dev/ada1 of=/dev/ada2 status=progress

I have SATA ports 0,1,2,3 - ports 0&1 are close together on the motherboard and 2&3 are together in a separate location on the motherboard.
The boot SSD is on 0, the first 8TB is on 1 and the second is on 3 - I am getting 12MB/s
This seems pretty slow? What could I do to speed things up? Should I have put the 8TB drives on ports next to each other?

Thanks.
 
dd if=/dev/ada1 of=/dev/ada2 bs=131072 status=progress

You can play with bs= option, but it is should be multiple to HDD's block size.
 
www.omnicalculator.com

Data Transfer Calculator

Use this data transfer calculator to find out how much time it's going to take to back up your computer to the cloud, and make sure you're getting what you pay for from your internet service provider.
www.omnicalculator.com www.omnicalculator.com

6TB at 12MB/s is going to take 1111 days!!! Eeek, better halt it now and test for better tx rate.
 
a sata disk should do around 80-150MB/s
if you can afford try it in single user mode or stop as many processes as you can
12MB/s is dog slow
 
You should have transfer speed at least 70 MB/s with dd.
I used dd many times for similar tasks.
12MB/s looks like extremely small value for 8TB HDDs.
Try to check raw read speed from source disk, use of=/dev/null for it
Also you can check write speed FOR NEW DISK, dd if=/dev/zero of=/new/disk bs=131072. Be carefull, it will erase all data on the destination disk!!!

In my opinion,
if you need to copy data, and do not require strongly to have a 100% clone, and if you do not have a lot of small files,
then try to copy files to new disk using net/rsync.
Rsync can sync file hierarchies, skipping already copied files.
You can run rsync as many times as needed until you have received a full copy.
 
cpcnw said:
Not far off - checked tonight and it is about 1TB / day however dd is going to do the additional 2TB even though there is no data. So 8 days.
Click to expand...
As others have observed, that's implausibly slow for a SATA III disk. The ballpark you should expect is about 100 MiB/sec, so less than a day to copy 8 TB.

If increasing the block size does not speed things up enough, you might consider buffering the output of dd with misc/mbuffer or misc/buffer. Just make sure that your block size stays an exact multiple of 4096.

Other options that have knowledge of the file system structures are likely speed thing up, a lot. Dump and restore is the obvious choice. Rsync also works well, but needs expert care in choosing the options (by default, it favours speed over correctness). However, all these options would require you to create the partition table, and empty file system.

A good reference for the options is Warren Block's Backup Options For FreeBSD. See the section titled Copying Filesystems.
 
covacat said:
a sata disk should do around 80-150MB/s
Click to expand...

Yep.

covacat said:
if you can afford try it in single user mode or stop as many processes as you can
Click to expand...

I don't think single-user will buy the OP much. Once it's running it'll be i/o bound, so lightweight multiuser usage shouldn't slow it much at all.

Running say systat -vm would soon show i/o rate, interrupt load etc.

covacat said:
12MB/s is dog slow
Click to expand...

About what you'd expect at the default 512 byte blocks.

Rather than muck around with testing I'd just go for bs=1m and get on with it :)
 
According to the output from the script shared from:
https://stackoverflow.com/questions/6161823/dd-how-to-calculate-optimal-blocksize

/bin/dd if=/dev/ada0 of=/dev/ada1 bs=1m status=progress

would transfer the raw data from one disk to the other at 22.1 MB/s.

$ ./dd_obs_test.sh
block size : transfer rate (speed multiplier) [device tx limit]
512 : 11.3 MB/s (1x)
1024 : 22.1 MB/s (2x)
2048 : 42.3 MB/s (4x) [USB2]
4096 : 75.2 MB/s (8x) [SATA]
8192 : 90.7 MB/s (16x)
16384 : 101 MB/s (32x)
32768 : 104 MB/s (64x)
65536 : 108 MB/s (128x)
131072 : 113 MB/s (256x)
262144 : 112 MB/s (512x)
524288 : 133 MB/s (1,024x)
1048576 : 125 MB/s (2,048x)
2097152 : 113 MB/s (4,096x)
4194304 : 106 MB/s (8,192x)
8388608 : 107 MB/s (16,394x)
16777216 : 110 MB/s (32,768x)
33554432 : 119 MB/s (65,546x)
67108864 : 134 MB/s (131,072x) [USB3]
Click to expand...
 
Usually I use "bs=4m" for things like this.
I currently do not know if dd is multithreaded, so you may speed up further by using two dd processes chained together.

dd if=/dev/source bs=4m | dd of=/dev/target bs=4m
 
alexseitsinger said:
According to the output from the script shared from:
https://stackoverflow.com/questions/6161823/dd-how-to-calculate-optimal-blocksize
Click to expand...

I'm lt not sure how much faith I have in that dataset, doesn't look like it would graph smoothly to me.

But anyway, and noticing your table didn't quote here:

alexseitsinger said:
/bin/dd if=/dev/ada0 of=/dev/ada1 bs=1m status=progress

would transfer the raw data from one disk to the other at 22.1 MB/s.
Click to expand...

No, that's the result for 1024 bytes (1k).

1m = 1048576 : 125 MB/s
 
The best I could get trying many different bs=x values was 22.2MB/s - ETA 4 Days 9 hours

After reading around a lot I just tried

Code: 
pv -s 8T < /dev/ada1 > /dev/ada2

and I am getting 216MiB/s with ETA of 10 hrs 46 mins
 
Came into the office this morning and the progress bar from pv showed 90% and no errors. That seemed odd? Was it reporting that it only copied out the actual data and partition info? In any case I mounted both source and destination discs at /mnt/ada1 and /mnt/ada2 - then issued an 'du -s' in both directories - both reported exactly the same. So I guess that is the result I needed :)
 
You must log in or register to reply here.
Back
Top