Horrible iSCSI (istgt) performance

I get on average around 1MB/s on a gigabit network.

I've created a zfs volume which I've exported as a block device with istgt
[CMD=""]zfs create -V 330G -b 512 tank/vol[/CMD]

Jumbo frames are enabled. MTU is set to 7000. It also runs on a seperate network/vlan.
I've tried two different switches, both has same performance.

I guess it's a configuration issue?

/usr/local/etc/istgt/istgt.conf
Code:
 17 [Global]
 18   Comment "Global section"
 19   # node name (not include optional part)
 20   NodeBase "iqn.backupbay.com"
 21
 22   # files
 23   PidFile /var/run/istgt.pid
 24   AuthFile /usr/local/etc/istgt/auth.conf
 25
 26   # directories
 27   # for removable media (virtual DVD/virtual Tape)
 28   MediaDirectory /var/istgt
 29
 30   # syslog facility
 31   LogFacility "local7"
 32
 33   # socket I/O timeout sec. (polling is infinity)
 34   Timeout 30
 35   # NOPIN sending interval sec.
 36   NopInInterval 20
 37
 38   # authentication information for discovery session
 39   DiscoveryAuthMethod Auto
 40   #DiscoveryAuthGroup AuthGroup9999
 41
 42   # reserved maximum connections and sessions
 43   # NOTE: iSCSI boot is 2 or more sessions required
 44   MaxSessions 32
 45   MaxConnections 8
 46
 47   # maximum number of sending R2T in each connection
 48   # actual number is limited to QueueDepth and MaxCmdSN and ExpCmdSN
 49   # 0=disabled, 1-256=improves large writing
 50   MaxR2T 128
 51
 52   # iSCSI initial parameters negotiate with initiators
 53   # NOTE: incorrect values might crash
 54   MaxOutstandingR2T 16
 55   DefaultTime2Wait 2
 56   DefaultTime2Retain 60
 57   FirstBurstLength 262144
 58   MaxBurstLength 1048576
 59   MaxRecvDataSegmentLength 262144
 60
 61   # NOTE: not supported
 62   InitialR2T Yes
 63   ImmediateData Yes
 64   DataPDUInOrder Yes
 65   DataSequenceInOrder Yes
 66   ErrorRecoveryLevel 0
 67
 68 [UnitControl]
 69   Comment "Internal Logical Unit Controller"
 70   #AuthMethod Auto
 71   AuthMethod CHAP Mutual
 72   AuthGroup AuthGroup1
 73   # this portal is only used as controller (by istgtcontrol)
 74   # if it's not necessary, no portal is valid
 75   #Portal UC1 [::1]:3261
 76   Portal UC1 127.0.0.1:3261
 77   # accept IP netmask
 78   #Netmask [::1]
 79   Netmask 127.0.0.1
 80
 81 # You should set IPs in /etc/rc.conf for physical I/F
 82 [PortalGroup1]
 83   Comment "SINGLE PORT TEST"
 84   # Portal Label(not used) IP(IPv6 or IPv4):Port
 85   #Portal DA1 [2001:03e0:06cf:0003:021b:21ff:fe04:f405]:3260
 86   Portal DA1 192.168.3.36:3260
 87
 96 [InitiatorGroup1]
 97   Comment "Initiator Group1"
 98   #InitiatorName "iqn.1991-05.com.microsoft:saturn"
 99   # special word "ALL" match all of initiators
100   InitiatorName "ALL"
101   Netmask 192.168.3.0/24
102
103 # TargetName, Mapping, UnitType, LUN0 are minimum required
104 [LogicalUnit1]
105   Comment "Hard Disk Sample"
106   # full specified iqn (same as below)
107   #TargetName iqn.2007-09.jp.ne.peach.istgt:disk1
108   # short specified non iqn (will add NodeBase)
109   TargetName disk1
110   TargetAlias "Data Disk1"
111   # use initiators in tag1 via portals in tag1
112   Mapping PortalGroup1 InitiatorGroup1
113   # accept both CHAP and None
114   AuthMethod Auto
115   AuthGroup AuthGroup2
116   #UseDigest Header Data
117   UseDigest Auto
118   UnitType Disk
119   # SCSI INQUIRY - Vendor(8) Product(16) Revision(4) Serial(16)
120   UnitInquiry "FreeBSD" "iSCSI Disk" "0123" "10000001"
121   # Queuing 0=disabled, 1-255=enabled with specified depth.
122   #QueueDepth 32
123
124   # override global setting if need
125   #MaxOutstandingR2T 16
126   #DefaultTime2Wait 2
127   #DefaultTime2Retain 60
128   #FirstBurstLength 262144
129   #MaxBurstLength 1048576
130   #MaxRecvDataSegmentLength 262144
131   #InitialR2T Yes
132   #ImmediateData Yes
133   #DataPDUInOrder Yes
134   #DataSequenceInOrder Yes
135   #ErrorRecoveryLevel 0
136
137   # LogicalVolume for this unit on LUN0
138   # for file extent
139   #LUN0 Storage /tank/iscsi/istgt-disk1 10GB
140   # for raw device extent
141   #LUN0 Storage /dev/ad4 Auto
142   # for ZFS volume extent
143   LUN0 Storage /dev/zvol/tank/vol Auto

I was hoping to get something around at least 50MB/s with iSCSI. The same zfs pool can easily read and write over 70-80MB/s with Samba (and no jumbo frames).
 
Try to isolate the problem. Run your experiment without the use of vlans or other virtual devices. Run vmstat and/or iostat on both machines and see if something peculiar is happening. Test it without jumbo frame support using the normal MTU (1500). Try exporting a ufs filesystem of the same physical disk. Crosscheck performance to see where the bottleneck comes from. Isolate your problem.

Check your logs!

By the way, have you checked your dmesg in case it shows something about sysctl tuning?
 
I've not tuned sysctl, I use default settings. None of the servers seems to have any io/cpu issues. I've tested without jumbo frames and got same result.

I'm gonna try to export an empty file as a volume from the ufs partition
Can't find anything in the logs either.

Interesting though, gstat says the iscsi device is working 100% when I use dd
 
Something is really wrong. After I rebooted the initiator now, I can't seem to connect to the iscsi target at all.

I find this error message on the target
Code:
Mar 20 11:54:53 zpool istgt[3262]: istgt_iscsi.c:4328:istgt_iscsi_send_nopin: ***ERROR*** before Full Feature
Mar 20 11:54:53 zpool istgt[3262]: istgt_iscsi.c:4817:worker: ***ERROR*** iscsi_send_nopin() failed

The initiator says
Code:
recvpdu: Socket is not connected
recvpdu failed
 
The extremely slow iSCSI was because of a broken network cable. I replaced it and now it runs very good! However I get maximum 50MB/s, the target is far from stressed. Are there other things I can do improve performance?
 
There are many ways of tuning ZFS. There isn't a best method as setups vary. Try searching the mailing lists since many people have posted tuning stuff (especially Jeremy Chadwick).
 
I think the first thing to do would be try without ZFS. I'd be interested to see what performance you get using a disk device directly, i.e. /dev/ada1 or what ever.

I also get terrible istgt performance with ZFS, but don't currently have the kit to do the testing I propose above as my istgt system is already in production.

Thanks, Andy.
 
I get 90-80MB/s with the Ubuntu initiator. I don't have a (fast) raw harddrive at the moment to test with.

I do have problems with my switch though, it doesn't seem to like jumbo frames. I need to test it more. A direct cable to each network card works fine with jumbo frames (but I don't get any performance benefits). But if anyone else is thinking about buying the NetGear GS724T, don't do it.
 
olav said:
I don't have a (fast) raw harddrive at the moment to test with.

I think any disk would do to test, if the problem is just ZFS related you will see a huge jump in performance even with a slow disk.
 
iscsi performance

Hi.

I've installed a server with zfs version 28 and FreeBSD 8.2 prerelease. The pool consists of 11 disks in raidz2. The disks are 2 TB sata disks on an areca-controller. The server has 12 GB of RAM and a xeon E5620 @ 2.4 Ghz. I installed net/istgt from ports. The only thing I tuned was 'QueueDepth 64'.

I have mounted the iscsi-disk from Windows 7/2008r2 and vmware 4.1 and have performance peaking at 118 MB/s which is very nice on plain GB nics. :)

Code:
[LogicalUnit99]
  Comment "disk99"
  TargetName disk99
  TargetAlias "Data Disk99"
  Mapping PortalGroup1 InitiatorGroup1
  AuthMethod CHAP
  AuthGroup AuthGroup1
  UseDigest Auto
  UnitType Disk
  QueueDepth 64
  LUN0 Storage /dev/zvol/data/iscsi/disk99 Auto

Regards
Claus
 
iscsi-performance

Hi.

The only setting I changed was queue depth, which was commented out in the example iscsi.conf-file. I did not try zfs ver. 15 which is default in 8.2 but patched it to ver. 28 before creating the pool.

regards
Claus
 
Hi Claus,

I don't see any noticeable difference when varying the size of queue depth. Do you actually see those speeds when moving real data around or are you seeing that from a windows benchmark utility? I see very good performance using benchmarking utils, but when actually moving real data the performance is very low...
If you really have got good performance then the only obvious explanation would be ZFS v28, I am using v15 currently,

thanks Andy.
 
iscsi-performance

Hi.

I copied my Download-folder on Win 7 using both samba and iscsi, the smaller files don't utilize the network completely (of course) but the larger iso-files have no problem moving approx. to 100 MB/s, sometimes more, sometimes less.

The folder has 43.3 GB of data.

regards
Claus
 
The solution, as stated by a few, is to set QueueDepth (I found 64 works well, but only tested the VirtualBox initiator).

Why is this thread not marked solved? Is it because you wanted to know why the different initiators will perform well on the same default unset QueueDepth? Or did different initiators not benefit from QueueDepth? (I ask for my own personal knowledge, as I am starting to use istgt)
 
I too can get no faster than about 50MB/sec (in this case, a Windows host with iSCSI initiator, running Veeam backup software for VMware, that's writing to the disk that is connected via a 1Gbit link (dark fiber) to the FreeBSD host running istgt).

Iperf gets more traffic over the link (can max out the 1Gbit link), but istgt seems to get no faster than about half the 1Gbit link.

Probably because it's not running in the kernel but in user space?
 
I tested istgt with a file on zfs; it went only 4.5 MB/s! A zvol went over 100. Is this what you did? (and due to zvol hangs, I'm reverting to NFS for now.)
 
As an aside, have you found any decent documentation for istgt?

I gave up with it because I couldn't make sense of the documentation. I'm by no means a storage guru, but I had run an iSCSI SAN for 2-3 years at that point...
 
One online howto made istgt segfault. :D But editing the sample config files worked fine, but no I found no documentation whatsoever.... just examples by 3rd parties. I wanted documentation so I would know what every setting meant, but found none.

In addition to the no documentation, also, if you typo in the istgt.conf file and try to reload/refresh the config, istgt can hang. So I don't think I want to use istgt at all. But there seems to be no good alternative. But if there is, I would love to hear about it.
 
Something to keep in mind when using NTFS-on-iSCSI-on-ZFS: when you create the zvol, make sure that you match the recordsize to the NTFS block size, and (if possibile) set the MTU of the link to be the same. If you don't, then you will be doing a lot of extra roundtrips for all reads and writes to the zvol.
 
peetaur said:
I tested istgt with a file on zfs; it went only 4.5 MB/s! A zvol went over 100.

I tested this again, with VERY different results.

Previously the initiator machine was using FreeBSD with VirtualBox's built in initiator, and 1Gbps. And as I said before, it was only around 4.5 MB/s (writing).

This time, I used a Linux machine, and it runs as fast as zvols (95-110 MB/s over 1Gbps, and 210-230 over 10 Gbps). And for comparison, netcat over the 10Gbps goes about 550 MB/s, and 2 threads goes 950 MB/s; I'm not sure what to do to speed it up for a single thread or iSCSI.

But this doesn't mean there is anything wrong with FreeBSD. It might simply be keeping better data integrity but Linux vbox doesn't. If I test this with Linux KVM, which I know does not have data integrity issues like vbox does, and get the slow result, I would blame zfs. If I get the fast result, I would blame FreeBSD or their changes to vbox.
 
Don't forget to add your NICs into the equation. There are good NICs and bad. Just because it says gigabit doesn't mean you will get the same from NIC to NIC. Some rely on the CPU, some don't.
 
dave said:
Don't forget to add your NICs into the equation. There are good NICs and bad. Just because it says gigabit doesn't mean you will get the same from NIC to NIC. Some rely on the CPU, some don't.

If you are referring to my tests:

I doubt that a CPU or a cheap network card would be the difference between 4.5 MB/s and 40-80 MB/s. And the test clearly showed the zil at 100% load with the FreeBSD + vbox machine. And other tests (eg. scp) always show fast enough results.
 
peetaur said:
If you are referring to my tests:

I doubt that a CPU or a cheap network card would be the difference between 4.5 MB/s and 40-80 MB/s. And the test clearly showed the zil at 100% load with the FreeBSD + vbox machine. And other tests (eg. scp) always show fast enough results.

I meant that more as a general reply because I noted that noone had mentioned it yet. I was not referring specifically to your tests.
 
peetaur said:
If you are referring to my tests:

I doubt that a CPU or a cheap network card would be the difference between 4.5 MB/s and 40-80 MB/s. And the test clearly showed the zil at 100% load with the FreeBSD + vbox machine. And other tests (eg. scp) always show fast enough results.

Actually, when it comes to iSCSI there is a HUGE difference between a good and a BAD NIC. I tested first with a Realtek NIC and then tried an Intel. I went from 5MB/s to 125MB/s.
 
Back
Top