ZFS benchmarks

traustitj

traustitj

I have a Core2Duo Quad core, 6 GB memory machine. Rocket RAID 2340 controller, all disks set up as single disks JBODs with 512 b sectors. This pretty much sums things up.

I was absolutely fed up with the speed I was getting with my zpool, raidz. I had 8 disks in 2 RAID sets of 4 disks each.
And the speed was about 1/2 that of a Drobo, which is awful.

So I went ahead and played with the machine, with 8 disks in it. This is the script I ran, of course I changed the top line for every test and I know I could have used a loop and variables, but that is not the main issue. The results were startling.

The script:
Code: 
zpool create test raidz da0 da1 da2 da3 da4 da5 da6 da7
zfs create test/test
cd /test/test
dd if=/dev/zero of=bigfile bs=1g count=1
dd if=/dev/zero of=bigfile2 bs=1g count=1
cd /
zpool destroy test

As you see I ran these tests 2 times, I also ran it 3 times but the numbers never varied out of the results.

Code: 
nr disks in raidz	test 1	test 2	diff
2	121,183,628.00	123,502,780.00	-2,319,152
3	201,231,672.00	194,088,892.00	7,142,780
4	19,464,276.00	19,019,305.00	444,971
5	380,296,687.00	314,630,723.00	65,665,964
6	374,334,902.00	302,374,071.00	71,960,831
7	50,007,819.00	50,601,162.00	-593,343
8	148,222,800.00	34,818,756.00	113,404,044
			
nr disks in raidz2	test 1	test 2	diff
3	98,266,724.00	98,196,932.00	69,792
4	107,689,118.00	80,928,521.00	26,760,597
5	22,243,224.00	21,693,076.00	550,148
6	280,035,350.00	271,791,816.00	8,243,534
7	40,604,456.00	43,800,049.00	-3,195,593
8	376,802,816.00	262,737,139.00	114,065,677

What totally blew me away was the fact that 5 disks, not 4 disks as I always had thought was the ticket. I always assumed that 4 disks were the trick and 1 disk used in parity, like RAID 5. Boy was I wrong on that one.

What are your ideas?
 
If you want performance with 8 disks, I'd set up a pool of 4 sets of 2 disk mirror VDEVs.

i.e.
Code: 
zpool create test mirror da0 da1 mirror da2 da3 mirror da4 da5 mirror da6 da7

This will essentially give you a RAID10 with 4 mirrors (at a cost of 50% of your space - but disk space is cheap, right ?- IOPs aren't.).

Should give you 2x the throughput of what you had, at least - and up to 4 disks can fail, if they're the "right ones".


Btw, /dev/zero isn't really a very realistic test...
 
Thanks. Unfortunately as cheap disks are, controllers that support 8 or more disks are not and cases with easy access to disks are nowhere to be found. I found one lian li box that can take 18 disks, although only 12 easily.

But when I am after more speed and disks are not the issue, I will definitely use your trick. If I hadn't already started the data transfer to the server, I would have tested that, I should have.

My main "discovery" was the 4 disk issue. I always went with 4 disks in the past and got awful speeds, although I am sure I got better speeds when I used smaller disks, you see in my example above that 4 disks only gave me 18 MB/sec (approx) instead of using 5 disks and get 380 MB/sec (approx).

I just wanted to be able to get more out of them and have the network be the bottleneck, which I now have succeeded. I went with Raidz2 with 8 disks for a total of 10 TB space, and the speed is good and I can loose any 2 disks at the same time, pretty sweet deal.

Before I decided to test different setup, I had decided to give up FreeBSD in favor of Linux, as I have been working with Linux mostly for the last 18 years or so, but it did not work with my sata controller, so I decided to test different setup and see if I got any better speeds, as 18 MB/sec barely can transfer 1 movie to my video player and if I take backups at the same time, everything stops.
Now everything should just be smooth sailings while taking backups, media scans and media plays, even on more than 1 device at the same time. I went from useless system with good data safety to excellent data safety and very good speed. I would never have found this out had Linux supported my controller. That was a close call
 
Hi

The number of disks and the configuration of your array will play a major role (e.g. number of vdevs, number of disks per vdev, physical sector size of the disks, etc.). I learned a lot from reading this thread and this blog.

That is the guidance I used in setting up my raid array and you can check the benchmarks here.

Regards
Malan
 
hi traustitj

Can you test this with a benchmarking program like bonnie++?

I am never too sure about using /dev/zero as a benchmark since it does not represent anything from the real world?

regards
Malan
 
Small test over Netatalk

A picture says more than a 1000 words, right?

http://d.pr/Vt7s

I am now getting more than 2x the speed of a Drobo connected with FW800 disk, all over Apple Talk. I could not be happier. This looks like pretty good usage of 1 Gbit ethernet.

I will try Bonnie++ probably tomorrow.

And thanks guys, I did not think ssh and CPU usage would be a bottle neck, now I transferred a few terabytes between machines, and I did notice the CPU usage, and before I did not think too much about it. But I decided to stop since one of the machine was completely used up, and I used NFS, mounted the right directories, started the sync, and it went over in very short time. Doing ssh rsync would probably have taken 4x the time.

But please look at the picture above. I have 8 disks, Hitachi 2 TB, all in 1 pool of raidz2. Any 2 disks can go haywire now and I am getting awesome speed, especially compared to 18 MB/sec earlier with 4 disks.

But with the critique of using /dev/zero. I am just doing writes, nothing else. If I would be doing reads, then that might be the limiting factor. To get the fastest possible write speeds, I think /dev/zero is it.
This is also not quite a benchmark test, more of a comparative test. Get a result, then compare it to other setups.
So far, this seems to be totally spot on.
 
I did try bonnie++, but it just gave me some CSV output, completely useless.

Anyone know what to do and what to aim for and how to understand this output?
 
You must log in or register to reply here.
Back
Top