We're attempting a test ZFS+HAST+CARP+iSCSI build, and before we get too far down the pipeline we wanted to just reel back a little and start with ZFS and some base benchmarks with a solo (single head) HAST configuration (not replicating YET!).
Some details:
2 Head nodes (6 disks) - FreeBSD 9.1-RELEASE
2 JBODS (45 disks)
Disk layout:
Head: 2xSATA (zfs root)
Head: 4xSSD (cache)
JBOD: 4xSSD 6gbps (zil/log)
JBOD: 41xSAS 6gbps
For the SAS disks I trimmed a little off, started at -b 2048, and aligned the sectors for 4K (even though the disks are 512 sector, doesn't hurt that much and future disks will likely be 4K aligned). The SSDs are also started at -b 2048, and provisioned to ~180G (80%) - all of them.
During some cursory testing we're finding pretty poor performance (considering the hardware involved) reported from iozone, but the alarming thing discovered was the lack of ZIL usage. I see NOTHING. The zpool I built is 5x8dev raidz2 .. so this should pretty much scream on IO.
I wanted to see what might be going on underneeth. I've built similar setups in Linux, and this is a first attempt to FreeBSD build of this kind.
The hardware is certainly highly capable so I'm trying to understand what is going on underneath, and what potential settings apply to ZIL usage or how to figure out why it isn't being used.
I do see CACHE usage, just not ZIL usage.
Here are some current tuning settings:
Certainly up to providing more information per request.
Some details:
2 Head nodes (6 disks) - FreeBSD 9.1-RELEASE
2 JBODS (45 disks)
Disk layout:
Head: 2xSATA (zfs root)
Head: 4xSSD (cache)
JBOD: 4xSSD 6gbps (zil/log)
JBOD: 41xSAS 6gbps
For the SAS disks I trimmed a little off, started at -b 2048, and aligned the sectors for 4K (even though the disks are 512 sector, doesn't hurt that much and future disks will likely be 4K aligned). The SSDs are also started at -b 2048, and provisioned to ~180G (80%) - all of them.
During some cursory testing we're finding pretty poor performance (considering the hardware involved) reported from iozone, but the alarming thing discovered was the lack of ZIL usage. I see NOTHING. The zpool I built is 5x8dev raidz2 .. so this should pretty much scream on IO.
I wanted to see what might be going on underneeth. I've built similar setups in Linux, and this is a first attempt to FreeBSD build of this kind.
The hardware is certainly highly capable so I'm trying to understand what is going on underneath, and what potential settings apply to ZIL usage or how to figure out why it isn't being used.
I do see CACHE usage, just not ZIL usage.
Here are some current tuning settings:
Code:
sysctl ---
vfs.zfs.prefetch_disable: 1
vfs.zfs.txg.timeout: 5
kern.maxvnodes: 1605403
vfs.zfs.write_limit_override: 0
vfs.zfs.l2arc_headroom: 8
vfs.zfs.l2arc_write_max: 597688320
vfs.zfs.l2arc_write_boost: 597688320
vfs.zfs.l2arc_noprefetch: 0
zfs properties ---
NAME PROPERTY VALUE SOURCE
zhast1 sync standard default
NAME PROPERTY VALUE SOURCE
zhast1 atime off local
NAME PROPERTY VALUE SOURCE
zhast1 zfs:zfs_nocacheflush 1 local
Certainly up to providing more information per request.