Just a heads up.
I recently analysed a lot of servers using zfs-stats and the 'kstat.zfs.misc.arcstats' sysctl's. I observed that the vast majority of hits on the cache was metadata, MRU cache in my case on every single server was almost useless, whilst MFU was working brilliant. There was some data caching going on but metadata was the priority.
The servers previously didn't have a lot of tuning. It was basically prefetch been disabled (as wanted maximum random i/o at expense of some sequential i/o), ARC capped but still set to use more than half of ram, and metadata limit increased to around 50% of ARC size from its default 25%.
After analysing the stats I determined my ARC cap was way too high, it still filled up but was with lots of MRU data that had bad hit rates. So just wasting ram, as an example on one server MFU was about 7.6 gig, and the ARC was capped at 17 gig. 80% of cache hits were metadata, and metadata was limited to 8 gig with 6.5 gig been actively used. So only 1.1 gig of the MFU was from data.
The tuning I applied was reducing ARC to 10 gig (so still some wiggle room over current MFU usage), and increased metadata limit to 80% of ARC.
I feel the metadata 25% stock is way too low based on what I am observing on every single server, there is no performance loss based on the changes, so essentially added 7 gig of ram to the pool on the server that can be used for larger innodb buffer or something else. The overall ARC hit rate is the same as before which is is around 99.7%.
I found other discussion's on the mailing lists with similar findings where people were actually advised to raise metadata limit to the full ARC limit in those cases.
I plan to enable prefetch again just to see if it helps any of the server's so bear in mind these findings are with that off, and different workloads can also have different caching patterns.
I recently analysed a lot of servers using zfs-stats and the 'kstat.zfs.misc.arcstats' sysctl's. I observed that the vast majority of hits on the cache was metadata, MRU cache in my case on every single server was almost useless, whilst MFU was working brilliant. There was some data caching going on but metadata was the priority.
The servers previously didn't have a lot of tuning. It was basically prefetch been disabled (as wanted maximum random i/o at expense of some sequential i/o), ARC capped but still set to use more than half of ram, and metadata limit increased to around 50% of ARC size from its default 25%.
After analysing the stats I determined my ARC cap was way too high, it still filled up but was with lots of MRU data that had bad hit rates. So just wasting ram, as an example on one server MFU was about 7.6 gig, and the ARC was capped at 17 gig. 80% of cache hits were metadata, and metadata was limited to 8 gig with 6.5 gig been actively used. So only 1.1 gig of the MFU was from data.
The tuning I applied was reducing ARC to 10 gig (so still some wiggle room over current MFU usage), and increased metadata limit to 80% of ARC.
I feel the metadata 25% stock is way too low based on what I am observing on every single server, there is no performance loss based on the changes, so essentially added 7 gig of ram to the pool on the server that can be used for larger innodb buffer or something else. The overall ARC hit rate is the same as before which is is around 99.7%.
I found other discussion's on the mailing lists with similar findings where people were actually advised to raise metadata limit to the full ARC limit in those cases.
I plan to enable prefetch again just to see if it helps any of the server's so bear in mind these findings are with that off, and different workloads can also have different caching patterns.
