ZFS Slow performance in zvol block device, but normal performance with normal files

G

gnoma

Hello,

I am running storage system on ZFS, what I needed was iSCSI target for my Vmware systems. I have four disks in RAID10 used only for iSCSI and FreeBSD OS needs and 6 GB RAM, no other services running on that system. iSCSI is usually working with block devices, but ZFS doesn't use them. So I created a block device this way: zfs create -b 128k -o sync=always -o compress=on -s -V 600G zpool/DISK0
I used blocksize 128k because above it is going to be a vmfs filesystem and it deals with even larger block sizes. Still this is the maximum block size you can get from ZFS.

After many performance issues, I start tracing the problem trough the network, iSCSI protocol, multipathing, switches flow control. I ended up with making normal UFS filesystem at /dev/zvol/zpool/DISK0, mounting it locally and transferring file with SCP trough localhost, so I can see the transfer speed.

I got transfer speed 2-5 MB/s. But if I go to some local directory, fill up a file with zeros, create a UFS filesystem on it and mount it, I got about 30 MB/s transfer speed to it.

Any idea, is this normal for ZFS block devices? And would it be a problem if I use files instead of block devices for iSCSI? And also any idea if the HAST can work with local files instead of block devices? Because I am interested in running a redundant storage system :)

Thank you.
 
All writes to your ZVOL are synced immediately. This implies the equivalent of an fsync() after every write(). If this is required get very fast ZIL. The ZIL is where ZFS keeps its journal. If your use a dedicated ZIL VDEV sync writes are converted in a sync write to the ZIL and an async write to the data VDEVs later while maintaining the semantics of stable storage. The ZIL throughput is limited by the bandwidth-delay product of your ZIL devices with a queue depth of 1 per zpool. Multiple VDEVs are only useful to if your writes are large enough to saturate your ZIL devices. With two mirrored SATA SLC SSDs your sync write performance will increase by a lot. The rule of thumb is to have at least 1 GiB of ZIL VDEV per 100 MiB/s of sustained writes.
 
I ran a test on a system of mine:
Code: 
zfs create -V100G data/test
zfs set sync=alwaays data/test
dd bs=128k if=/dev/zero of=/dev/zvol/data/test # stopped after ca. 10 GiB, avg write speed 133.088 MiB/s
The pool is configured as four RAID-Z2 VDEVs, a mirrored ZIL and two cache devices. The ZIL is on Intel 520 SSDs (MLC consumer disks). The their latency limits them to ca. 4800 IOPS. A lower latency (SLC) SSD could improve on this.
 
Hello @Crest,

Thank you for your response, that makes it all clear now.

Unfortunately, I can't put in there SSD disks, because I don't have more SATA slots and even though, the server is DELL Poweredge 2950 with SATA1. I think it will be a waste if I put SSD disks in a SATA1 controller. Is there any way to disable this immediately sync on zvol?

Or if not, I have two possible workarounds:
  1. I wonder if a 8 GB or 16 GB flash drive at USB 2.0 will be enough to get normal performance. I will probably get a flash pen USB 3.0 because of the higher read/write speed of the chip, but as it is compatible, I'll plug it in to the USB 2.0 port of the server.
  2. If not a flash drive, will it be possible to run HAST with files instead of block devices?

Thank you.
 
Last edited by a moderator:
It would be less of a waste than you expect because most SSDs won't exceed 1.5 Gb/s under this workload. Avoid USB sticks or you will hate the day you were born so point 1 is out of the question.

If you have spare SATA ports and don't mind tinkering with hardware you could rip out the CD drive and use the space for two SSDs. Maybe there is enough free space next to the CD drive for one SSD. In that case you could "replace" the CD drive with a 5.25" slim -> 2.5" adapter for the other SSD.
 
You must log in or register to reply here.
Back
Top