Understanding zpool iostat output

tdb@

tdb@

Developer
I've recently built a new server which has a mirrored ZFS pool for its data. Whilst trying to diagnose some performance issues I've come across the following unusual output from zpool iostat:

Code: 
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
pool0       93.4G  40.6G    221      0   727K  19.7K
  mirror    93.4G  40.6G    221      0   727K  19.7K
    ad4s3       -      -     61      0  3.70M  19.7K
    ad6s3       -      -     58      0  3.51M  19.7K
----------  -----  -----  -----  -----  -----  -----

Code: 
               capacity     operations    bandwidth
pool         used  avail   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
pool0       91.6G  42.4G     49      0  6.10M      0
  mirror    91.6G  42.4G     49      0  6.10M      0
    ad4s3       -      -     23      0  2.98M      0
    ad6s3       -      -     25      0  3.14M      0
----------  -----  -----  -----  -----  -----  -----

The first instances shows more bandwidth being used on the disks than on the pool itself. I'm not sure how to interpret this. Does it mean that ZFS is internally doing a lot of activity? The second instance is what I consider to be more normal.

I'd be interested to know what others have seen.

As an aside, in the first instance the file system is pretty slow to respond. Mutt with large maildir folders becomes a nightmare to use :-(
 
If that is the output of just zpool iostat -v then it is showing average values since the system was turned on. If you want to see current usage stats, you need to add an interval value to the command:# zpool iostat -v 30

That will run iostat every 30 seconds and show you the activity over that period (the very first set of output will still be an average since the system was turned on).

Also check if there is a scrub/resilver happening. That will artificially inflate the numbers for the individual drives.
 
They were the output of:

# zpool iostat -v pool0 5

Not the first output either.

And no, unless I missed something, there was no scrub or resilver running.
 
Ah, ok. Yeah, in that case, it does seem strange that there's more read bandwidth reported for the individual disks than for the pool. I don't think I've ever noticed it personally. But I'll keep an eye out for it on our boxes.

Perhaps it's something related to prefetch?
 
I've only got 3GB of RAM, so prefetch was disabled (and I haven't enabled it). I'm also currently running with the default settings on amd64.

I think when this was happening mutt was polling a lot of Maildir folders by doing many getdirentries calls. Maybe this was creating a lot of work for ZFS which resulted in little data being generated? Sounds quite likely now I've typed it... :)
 
In that case, it might be ZFS reading metadata, which it might not count towards the 'bandwidth' which is actual file data.

But, I am just guessing
 
You must log in or register to reply here.
Back
Top