RAID-Z Speed

X

xjohn

Hi Everyone,

I recently made an Installation of FreeBSD 8.2 amd64 for a RAID-Z iSCSI storage.

Everything works Ok, but I'm not sure that the throughput its OK.
I'm getting 41MBps for read and 46MBps for write of large files (more than 2GB).

My system is:
Core 2 Duo E6300 1.86Ghz
4GB DDR2 800
Intel DG33FB board
OS in 2 IDE 30GB WD (with gmirror)
4 Seagate LP 2TB (in raid-z shared with istgt)
Intel PRo1000/GT NIC (mtu 9014)

Thanks.
 
My /boot/loader.conf is this:
Code: 
# ZFS Tunning
#vfs.zfs.prefetch_disable="1"
vfs.zfs.txg.timeout="1"
geom_mirror_load="YES"
and /etc/sysctl.conf:
Code: 
# ZFS Tunning
kern.maxvnodes=250000
vfs.zfs.txg.write_limit_override=1073741824
 
The settings look ok to me but the disks are a bit old (30 gigs WD ? .. wow, that's ancient) and I'm not sure they can do more than what you got.
Btw, how did you test the speed ? You can also try bonnie and/or bonnie++.
 
The two 30GB WD are for the root only, and they are formated in UFS.

My problem is the RAID-Z pool made of 4 Seageate LP 2TB, it's shared using iSCSI to a W2008R2 and then formated with NTFS (5.31TB available).

I made the test by just copying files to the iSCSI drive (in Windows) from a 1TB WD Blue.
 
Did you set the blocksize property for the zvol to 8 KB before exporting it via iSCSI and formatting it with NTFS? You need to make sure that the underlying zvol's blocksize is set to the same size as the filesystem on top uses. Otherwise you end up with unaligned writes, and performance tanks.

I believe NTFS uses 8 KB blocks. You may need to do some research to find out what the correct blocksize is.

Blocksize on a zvol can only be set when you create it.
 
Well, I'd transferred all my data to an external drive , then I destroyed the zpool.
After various test, I'd get the best performance with a block size of 64k (in the ZFS and in Windows NTFS format), but the throughput of copying files only increased to 56MBps for read and 65MBps for write (an increase of 15MBps for read and 19MBps for write).

So, I decided to make a test using
Code: 
dd if=/dev/zero of=testfile bs=64k count=80000
and wrote the file at 157MBps.
Then I tested sending a file using WinSCP, and the transfer speed was 3.100KBps.

I think that the problem is actually the network, anyways the ZFS volume wasn't aligned with the NTFS format.
 
xjohn said:
Code: 
dd if=/dev/zero of=testfile bs=64k count=80000
and wrote the file at 157MBps.
Click to expand...

You can't test with dd that way. You must disable write cache somehow. The above test would only have low enough error if the file system cache was very small. Making the count large lessens the problem, but doesn't eliminate it.

Please see http://romanrm.ru/en/dd-benchmark

If you install the sysutils/coreutils port, you can use gdd (the dd command that comes with Linux) with conv=fdatasync:

[cmd=]gdd if=/dev/zero of=testfile bs=64k count=80000 conv=fdatasync[/cmd]
 
Oh and something nobody else seemed to mention... you could simply have a bad disk.

To find out if you have a bad disk, first find a test that is reliably slow. If only a part of the disk is bad, not all tests will show the same thing, so make sure you have an idea of how to cause it.

Then run gstat.

# gstat

or instead of seeing duplicates of everything (da0, da0s1, gpt/disk1, etc. all being the same), just select one:

# gstat -f "gpt/"

Then start your test. There are at least two things to look at. Even load, and ms/*.

During the test, watch the load% on the disks. If one disk is consistently high load all the time, hypothesize that it is bad, and test it. (eg. switch cables, switch disk bays, run some smartctl tests, etc.). If you determine it is a bad disk (not bad cables, etc.), replace it.

And second thing is check ms/r or ms/w during the test. If one disk is consistently higher than the rest, then it is slower for some reason. When you get lots of almost-bad sectors, this happens. A consumer grade disk won't throw an error, but just try really hard over and over to read the sector, wasting loads of time, without alerting you.

Or you could just run some short smart tests on all disks and check the errors without a write test.

# smartctl -t short /dev/da0
(wait however long it tells you to)
# smartctl -a /dev/da0
 
You must log in or register to reply here.
Back
Top