ZFS Apparent overall slow speed

D

David Wasson

Hey,

I have 2 machines that are serving as NAS boxes.

Machine 1 has 8x4TB WD Greens with idle turned off as a RaidZ2 array, 8GB non-ECC, an i5 4970S and a Gigabyte Z97 motherboard. I also have 2x1TB WD Blues as the OS drive. All disks are plugged into the motherboard, with the exception of the 1TB drives, which are on a raid card.

Machine 2 has 6x3TB WD Greens with idle turned off, 16GB ECC, an Intel E3 1240v3, and a supermicro motherboard. I have the OS on a thumbdrive plugged a the USB port in the back.

Machine 1 was built first when money was a little tighter hence the difference in parts.

My question is:

I'm generating a file with the following line in bash:
dd if=/dev/random of=/randomfile count=1024 bs=1M
I then run
rsync -Pav randomfile randomfile2

On Machine 2, I get approximately 350 MB/s on the ZFS array. I'm very happy with this and I think it's what I should be seeing.

On Machine 1, I seem to get approximately 70 MB/s no matter what, whether all operations are on the 1TB OS disk, are all on the ZFS array, or from one to the other. Changing from RaidZ2 to just a simple stripe also doesn't seem to make a difference.

So I'm hoping that someone can help me figure out where the problem is and whether it's something hardware related, or if it's simply an issue of ZFS tuning.

Any thoughts or comments are appreciated.

Thanks!
 
These new drives tend to use 4K block sizes. It's possible the alignment on Machine1 is a little off and not on a 4K boundary. That could be a cause of the bad performance.
 
You are testing with a file size of 1GB with a system having more memory than that. Your test results will not be correct in any scenario as long as that is the case.
 
Ok, I can up the file size to match the installed ram size.

I will also check the block size on the disks and report back. How would I correct the alignment if it is off?

Thanks.
 
I checked with files similar to the installed ram size and got similar results.

I've also checked the sector size, and they're all reporting 512.

Any other thoughts?
 
Use zdb to check your ashift value.
Use zfs get all zroot to check if you have dedup or compression or etc enabled.
Cpu low? Disk access steady or surging with an average of 70 MB/s?
Use gstat to see what the individual partitions are doing. % busy? queue depth?
 
Hey, thanks for the information. I haven't been able to go back and take a look at everything since I posted this, but again, I appreciate the help!
 
Can you try and narrow it down by changing /dev/random -> /dev/zero (/dev/null) I'm sort of guessing that it may relate to random generation, I don't know how it is configured in both machines or that it's the same.

It that doesn't work, you might want to look into dtrace(1) to try and figure out if there are any errors, or what's different from both machines' runs.
 
Last edited by a moderator:
You must log in or register to reply here.
Back
Top