Poor ZFS perfomance

[ilemur]

I have a strange setup. Running a VM FreeBSD which is serving as a data storage for ESXi. There is an LSi2308 passed through to FreeBSD.
Inside I have created a RAIDZ out of four 2Tb Hitachi HDDs.
The ZIL resides on a separate SDD. 16Gb of RAM and 4 Xeon cores are available. So performance wise everything should be ok. But...
What can be done to improve this?

root:/istorage/storage # bonnie++ -u root -d /istorage/storage -s 34000M -n 10:102400:1024:1024
Using uid:0, gid:0.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
xxx.xxx.xxx   34000M    86  99 115284  41 171625  51   282  99 735447  91 624.5  20
Latency               136ms   38152ms   18050ms   68444us     177ms     207ms
Version  1.97       ------Sequential Create------ --------Random Create--------
xxx.xxx.xxx     -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files:max:min        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
10:102400:1024/1024  1262  68 14471  99  7936  88  1208  64 10569  83  6169  99
Latency               159ms     332us   42065us   50637us     123ms   16906us
1.97,1.97,xxx.xxx.xxx,1,1469382702,34000M,,86,99,115284,41,171625,51,282,99,735447,91,624.5,20,10,102400,1024,,1024,1262,68,14471                        ,99,7936,88,1208,64,10569,83,6169,99,136ms,38152ms,18050ms,68444us,177ms,207ms,159ms,332us,42065us,50637us,123ms,16906us

 

[fnj]

I don't have anything specific to suggest, except to note that RAID-Z with only 4 drives is a very poor idea. For RAID-Z2, 6 drives is good, and for RAID-Z3, 7 drives. I don't even think RAID-Z1 is worth using at all, with ANY size array.