ZFS Can ZFS prioritize reads from higher preference drives?

[rowan194]

With gmirror, you can set the read priority for individual drives, which means you can do useful things like create a mirror with an SSD for fast reads, backed by cheaper HDD storage.

I can't find any mention of priority with ZFS. Can it be done?

 

[ralphbsz]

You need to find the ZFS source code, or find someone who knows it. The question here is the following: When you have a 2-way mirror, you can read from either of the two disks. Which one will ZFS pick? I have never looked at the ZFS source code, but there are many possibilities of implementing this. They include fixed choice (like always pick the first disk in alphabetical order that is not down), round-robin (alternate between the disks), round-robin with priorities (do 5 IOs on disk one for every 2 IOs on disk two), random choice with or without priorities, based on current queue depth (which works good on a loaded system, not so good on an idle system), expected latency based on static latency estimation (tends to break on busy systems), and dynamic latency estimation from queue depth. I know that ZFS has queue depth adjustment for different IO classes, but those are not adjustable per device as sysctls. If ZFS uses intelligent IO scheduling, there should be no need to adjust them.

 

[Argentum]

You need to find the ZFS source code, or find someone who knows it. The question here is the following: When you have a 2-way mirror, you can read from either of the two disks. Which one will ZFS pick? I have never looked at the ZFS source code, but there are many possibilities of implementing this. They include fixed choice (like always pick the first disk in alphabetical order that is not down), round-robin (alternate between the disks), round-robin with priorities (do 5 IOs on disk one for every 2 IOs on disk two), random choice with or without priorities, based on current queue depth (which works good on a loaded system, not so good on an idle system), expected latency based on static latency estimation (tends to break on busy systems), and dynamic latency estimation from queue depth. I know that ZFS has queue depth adjustment for different IO classes, but those are not adjustable per device as sysctls. If ZFS uses intelligent IO scheduling, there should be no need to adjust them.

With OpenZFS, maybe a good point to start is /usr/ports/sysutils/openzfs/work/zfs-aa2778d10/module/zfs/vdev_mirror.c

It reads in comments:

/*
 * The load configuration settings below are tuned by default for
 * the case where all devices are of the same rotational type.
 *
 * If there is a mixture of rotating and non-rotating media, setting
 * zfs_vdev_mirror_non_rotating_seek_inc to 0 may well provide better results
 * as it will direct more reads to the non-rotating vdevs which are more likely
 * to have a higher performance.
 */