Slow iSCSI initiator performance


New Member

Reaction score: 2
Messages: 10

On an ESXi host I created a Centos 6.2 Linux virtual machine running ZFS (the new kernel-mode module) and a FreeBSD 9.0 virtual machine running ZFS. This ESXi host has a gigabit Ethernet connection dedicated to an iSCSI connection to two iSCSI storage appliances with a combined 144 terabytes of storage, which is used for its virtual machine storage as well as data shares exported for filesystem use to virtual machines.

On the iSCSI storage appliances I created a 2 terabyte share on each, and gave the FreeBSD initiator permission to talk to it. I then created a mirrored zpool across the pair using those two shares, created a couple of filesystems, and set to writing a stream of data that simulates video data... and obtained a whole 5 megabytes per second throughput. Tweaking network buffers, iSCSI tag depth, and zfs parameters could result in a temporary boost to as much as 30mb/sec, but within minutes it settled right back down to 5 megabytes per second.

I then exported the zpool, unlinked the initiators on FreeBSD, gave the Linux VM permissions on the iSCSI appliances, and imported the zpool onto Linux. I again ran my simulated video stream writing to the ZFS pool and it went at the full 100 megabytes per second that are predicted by the fact that I'm limited by that 1Gb Ethernet connection (the iSCSI appliances are capable of far more performance, they can saturate four 1Gb Etherchannel-trunked connections when simulating video data, but this ESXi box wasn't set up for that).

So it seems that there's no issue with ZFS (or zfs parameters, since they came in with the zpool) causing my poor write performance with FreeBSD iSCSI. ZFS plugs along just fine -- on other operating systems, and on FreeBSD talking to DASD's for that matter. So then I turned my eye on the iSCSI initiator. Looking at the source code, it appears that the performance problem is caused by lack of immediateData mode in the iSCSI initiator. The Linux OpenISCSI initiator of course implements that mode, as does the Windows 2008R2 initiator (I performed this same test with that OS too, though with NTFS of course, and obtained the same results as with Linux ZFS).

So -- is there a better iSCSI initiator for FreeBSD? Or some parameter I'm ignoring? Or am I stuck with Linux or OpenIndiana if I want to use ZFS with iSCSI?