ZFS ZFS + ISCSI , Low sequential read performance

Hi there! I've been playing for 4-6 weeks.

The system is:
ESXi 5.5 u2 + 4 NICS , round robin,
FreeBSD 10.1

I tried lot's of thing, read tons of http pages, however the sequential read speed is 50-60 MB\s from 6 HDD 7200rpm ( stripe of 3 mirrors ).

zvol chunk = 4k
NTFS allocation unit 64k

I see exactly the same on esxtop and windows in iomter. In iometer I use pattern as such: bs=512k , outstandoing i\os=1,2,3,4,5,6,7,8. I can get pretty cool number when we speak about write speed ( random or sequential) or random read speed. However sequential read speed is aweful.

primary cache = metadata
secondary cache = metadata

I don't use ssd arc as I want to get good read speed just from the pool and only after that I'm going to attach ssd arc. Well, I would like to know - is there any one who can get good sequential read numbers? Honestly - I've read and spent just tons of nights and days trying to figure out - failed. I tried FreeBSD, FreeNAS, OpenIndiana. the result the same.

When I use zvol chunk = 64k or zvol chunk = 128 k the speed is better, however I do not want to use those zvol chunks. There is a fundamental reason for that, as you know ( or maybe do not know ) ----- when you use big zvol chunks ( pretty much any bigger than 4k ) --- the result is horrible when we speak about writing performance. I mean - when you start writing on zvol (8,16,32,64,128k) with blocks lower than zvol chunk ----> the huge amount of readings (parasite readings) happens on pool. Can you believe it? You just want to write , instead you start massive reading. --- as a result write speed horribly goes down. it's because writing block is lower than zvol chunk , so to write 1 iops from the application zfs need to read block, change and write somewhere else. Maybe I understanding is not so correct - the fact is that writings just start provide massive reading pressure on the pool - and this is horrible. The only thing that help - is to use zvol =4k. This mode reduces such negative effect significantly. Of course if esxi would support 4k storage disks -- we could feel much better. Unfortunately right now esxi 5.5 -6 does not support 4k drives so we have to set zvol as if it is 512b drive. It means Windows thinks that 512b drive is here. This definetley cause side effects.


To make the long story short --- I did a profound searching and as I feel - all people just can't have good reading performance form zfs via iscsi when we READ sequentially.

Am I right? And what do you think about that? Right now I with great sorrow going to use Linux + esxt3 + raid + scsi + bcache . but that's another story. I will give you lots of useful information that I've gotten for this 6 weeks if you wish. It's just alot.

Sincerely yours Alex. K.
Last edited by a moderator:
Hi there, the details about configuration:

server with esxi:
  • proliant 180 G6, RAM 64GB
  • NIC = BCM95719A (4 ports)
  • I use round robin NICs with esxi 5.5 u2.
  • The number of iops to switch to another port nic = 1 iops.
  • server directly (without switches) connects to server with zfs

zfs server:
  • Proliant 180 G6, RAM 82GB
  • NIC = BCM95719A (4 ports)
  • FreeBSD 10.1
my zfs pool is about :

6 x hdd seagate ST2000DM002 ( stripe of 3 mirrors)
zil = 120 GB kingston

(however zil doesn't play role when reading happens as we know)

zvol volblocksize = 4k

ashift=12 for seagates
ashift=13 for kingston

primary cache=metadata
secondary cache = metadata

I don't use arc for testing purposes as I want to get good results without arc, later I will add it to the pool. I installed windows server 2012 + iometer. The testing pattern = sequential read bs=512k and number of queues=5-8

I expect to get 250-270 MB\s (at least 150 MB\s for christ sake) about sequential reading, however i get just only 60 MB\s. I want to underline that i get very good results in sequential writing. seq write 512k(iometer align=4k) , 8 threads ,throughput =226МБ\с lat=17.7 also i get good result about random reading read 4k(, 8 threads, iops= 614 lat= 13.0

So the system kind of work. It's just about sequential reading problem. I'm puzzled and frustrated, I want use zfs on iscsi but can't, I have to switch to iscsi + ext3.

What can you suggest? Is anyone struggling with bad reading?
Last edited by a moderator: