Slow ZVOL read for mirrored configuration

A

ApoC

Hi,

I am experiencing very slow raw read performance from mirrored ZVOL (about 60 MiB/s).

Used system: FreeBSD 10-BETA2
Used disks: 2x ST2000DM001-1CH164 CC26 (2TB 4k sectors)

I created a GPT for each disk as follows:
Code: 
gpart create -s gpt ada0
gpart create -s gpt ada1
gpart add -t freebsd-zfs -a 4k -b 1M -l bay3 ada0
gpart add -t freebsd-zfs -a 4k -b 1M -l bay4 ada1

Than I created zfs pool tank as mirror:
Code: 
zpool create tank mirror gpt/bay3 gpt/bay4
zfs create -V 1770G tank/iscsi

Than I exported this ZVOL to my Windows 8 client via iSCSI over a direct 1Gbit connection.
Code: 
portal-group pg0 {
        discovery-auth-group no-authentication
        listen 192.168.0.1
}

target iqn.xxx.server:target0 {
        auth-group no-authentication
        portal-group pg0

        lun 0 {
                path /dev/zvol/tank/iscsi
        }
}

I created an NTFS file system over an iSCSI exported volume with 8k block size. I did some tests and Read/Write is clipped around 50-60 MiB/s. When I added the ZIL to tank/iscsi I can reach circa 115 MiB/s write speed (so the network shouldn't be the problem), which is great, but read speed is still only 50-60 MiB/s. Also dd from /dev/zvol/tank/iscsi to /dev/null is clipped at 60 MiB/s.

I am using the default ZFS sysctl setting.

Output of gstat for devices under read
Code: 
dT: 1.002s  w: 1.000s
 L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
   11    410    410  52503   10.2      0      0    0.0   75.5| zvol/tank/iscsi
    2    959    959  28136    1.7      0      0    0.0   48.0| gpt/bay4
    4    945    945  28239    1.8      0      0    0.0   53.6| gpt/bay3

I would be grateful for any advice how to improve the read performance.
 
One update:
I just tried to dd from both underlying block devices (gpt/bay3, gpt/bay4) and both disk provided transfer rates around 170 MiB/s so I would expect zmirror transfer rate should be somewhere around 200 MiB/s at least.
 
ashift is set to 12:
Code: 
MOS Configuration:
        version: 5000
        name: 'tank'
        state: 0
        txg: 30291
        pool_guid: 185994646100184535
        hostid: 1342477373
        hostname: 'mail'
        vdev_children: 2
        vdev_tree:
            type: 'root'
            id: 0
            guid: 185994646100184535
            create_txg: 4
            children[0]:
                type: 'mirror'
                id: 0
                guid: 10156552873793111708
                metaslab_array: 33
                metaslab_shift: 34
                ashift: 12
                asize: 2000393076736
                is_log: 0
                create_txg: 4
                children[0]:
                    type: 'disk'
                    id: 0
                    guid: 18034684051788006846
                    path: '/dev/gpt/bay3'
                    phys_path: '/dev/gpt/bay3'
                    whole_disk: 1
                    DTL: 90
                    create_txg: 4
                children[1]:
                    type: 'disk'
                    id: 1
                    guid: 2499443007566452254
                    path: '/dev/gpt/bay4'
                    phys_path: '/dev/gpt/bay4'
                    whole_disk: 1
                    DTL: 89
                    create_txg: 4

I think it is ok for a 4k sector drive, isn't it?
 
I did another performance test. I take gpt/bay3 and gpt/bay4 and created gmirror(8) volume and exported it as iSCSI target. Now I can reach circa 110 MiB/s read/write to this target.

Is there any way how to do performance profiling of ZFS?
 
ZVOL has some known performance issues due to the fact that it executes requests synchronously and one at a time. Partially it was fixed in FreeBSD head branch recently, but change was not merged down yet. Otherwise you may just use single file on ZFS for iSCSI storage.
 
Thanks for the information I tested proposed solution and it works as expected. But I think I will stick to gmirror(8) because it seems to me "less hacky".
 
It depends on your needs. gmirror will likely be faster, but it won't give you snapshots, extensive checksumming, compression and other features of ZFS.
 
ZVOL has some known performance issues due to the fact that it executes requests synchronously and one at a time. Partially it was fixed in FreeBSD head branch recently, but change was not merged down yet. Otherwise you may just use single file on ZFS for iSCSI storage.
Click to expand...

Hi, is the fix merged into stable-10 ?
 
You must log in or register to reply here.
Back
Top