Optimum Block Size when dd from a hard drive to another

General questions about the FreeBSD operating system. Ask here if your question does not fit elsewhere.

Optimum Block Size when dd from a hard drive to another

Postby overmind » 23 Feb 2010, 11:40

Hello

If I want to move FreeBSD to a new hard drive with:

Code: Select all
dd if=/dev/ad0 of=/dev/ad1 bs=1M


What is the optimum size for "bs" ?

If I do not use bs, and leave it default then bs will be 512 bytes and it takes forever. I've used bs=20M but I want to know what is the optimum size for bs when dd-ing from a sata drive to another and also the optimum size when dd-ing from a usb stick to another.

(I'm not interested in other ways to move data to a new hard drive, I'm only interested in cloning.)

thanks and best regards,
overmind
Member
 
Posts: 318
Joined: 18 Nov 2008, 12:29

Postby gordon@ » 23 Feb 2010, 18:50

I think you'll need to experiment to find the ideal size. I suspect that any sufficient blocksize that takes advantage of a HDD buffer will be "big enough."

My guess would be to use something like 16M or 32M.
gordon@
Member
 
Posts: 404
Joined: 20 Feb 2008, 22:44
Location: San Diego, CA

Postby fallen_fbsd » 24 Feb 2010, 05:16

My guess is also 32M on modern disks. More can be degradation.
fallen_fbsd
Junior Member
 
Posts: 1
Joined: 22 Nov 2009, 11:13

Postby lme@ » 24 Feb 2010, 09:46

And what about USB-Sticks?
User avatar
lme@
 
Posts: 611
Joined: 08 Oct 2007, 19:42
Location: Düsseldorf, Germany

Postby overmind » 24 Feb 2010, 14:53

lme@ wrote:And what about USB-Sticks?


For USB Stick I've used bs=1M, still I don't know which is right value. I guess it depends on if the stick is detected as USB1 or USB2. Anyway, using bs=1M instead of leaving it default (512bytes) improves speed alot.
overmind
Member
 
Posts: 318
Joined: 18 Nov 2008, 12:29

Postby Ruler2112 » 24 Feb 2010, 17:49

Definitely stick with a power of 2 as a block size for optimal transfer rates. I'd guess a setting equal to the internal cache size of the drive, but that's just a gut feeling supported by nothing.

If you're serious about finding the optimal rate for a given machine, create an image of a few hundred meg from a CD or something on your drive, copy it a few times, reboot (an important step to eliminate drive caching from the equation), and then [cmd=]dd if=/path/to/filecreated.iso of=/second/drive bs=nM[/cmd] and vary n by factors of 2. Record the results and you'll soon know the optimal block size.

Note that the vfs.read_max kernel parameter can have a significant impact on disk performance as well, which means that you should find the ideal setting for this parameter before testing the block size. Remember to use fresh image files (that haven't been read or written in the present boot) or reboot for each test to ensure consistent results.

I've often wondered what I should use as a block size when cloning drives for best performance. Unfortunately, it's not a simple answer and varies greatly with the specific hardware involved.
User avatar
Ruler2112
Member
 
Posts: 419
Joined: 15 Sep 2009, 16:23
Location: Michigan, USA

Postby Carpetsmoker » 24 Feb 2010, 19:08

I've had good results with a bs of 16M, I get transfer speeds of about 100MB/s as measured by iostat.
UNIX was not designed to stop you from doing stupid things, because that would also stop you from doing clever things.
User avatar
Carpetsmoker
Member
 
Posts: 905
Joined: 16 Nov 2008, 19:07
Location: Eindhoven, Netherlands

Postby mamalos » 25 Feb 2010, 15:32

The fastest results I ever had were always with bs=512k. You should try all of them by letting dd run for a few seconds and then compare the outcome :)
mamalos
Member
 
Posts: 427
Joined: 18 Mar 2009, 12:17
Location: Greece

Postby overmind » 25 Feb 2010, 15:37

and default bs size is 512 bytes and not 512k right?
overmind
Member
 
Posts: 318
Joined: 18 Nov 2008, 12:29

Postby mamalos » 25 Feb 2010, 16:56

so says the man page.
mamalos
Member
 
Posts: 427
Joined: 18 Mar 2009, 12:17
Location: Greece

Postby sampablokuper » 18 Apr 2012, 23:45

I wrote dd-opt, a small utility with the sole function of helping to find the optimal setting for the bs parameter.

Since I'm not a FreeBSD user myself, I'd welcome people trying it out on FreeBSD and submitting patches/refinements/suggestions/etc.
sampablokuper
Junior Member
 
Posts: 1
Joined: 18 Apr 2012, 23:39

Postby rfg » 28 Mar 2013, 02:43

Where is the sources tarball for that?

I looked but could not find it.
rfg
Junior Member
 
Posts: 12
Joined: 10 Oct 2012, 09:25

Postby wblock@ » 28 Mar 2013, 03:12

Click on the "Zip" button on the github page.

Notes:

1. This thread is over a year old.
2. http://forums.freebsd.org/showpost.php?p=115526&postcount=9
User avatar
wblock@
Old Fart
 
Posts: 10227
Joined: 07 Sep 2009, 23:23
Location: Milky Way galaxy

Postby rfg » 29 Mar 2013, 01:32

1. This thread is over a year old.


I am in the habit of arriving fashionably late to every party.

Sorry.
rfg
Junior Member
 
Posts: 12
Joined: 10 Oct 2012, 09:25


Return to General

Who is online

Users browsing this forum: No registered users and 1 guest