ZFS L1 vs L2 ARC for many files/metadata cache

A

AndyUKG

Hi,

just wanted to ask if anyone has any first hand experience with improving slow ZFS performance with large pools with directories with many thousands of files. Specifically I have a 30TB pool on which I'd like to improve the general performance for directories containing large numbers of files. Ie ls is slow etc etc. The system currently has 8GB RAM, I believe the max possible config would be 32GB.
A smallish SSD is alreaday going to be several times greater in capacity than my theoretical max RAM and would be much cheaper. Is an SSD L2 ARC cache going to cache the pool metadata and give me decent performance? I'm guessing it should be fine, but would appreciate any feed back,

thanks, Andy.
 
I think the general rule is 1 GB of RAM per 1 TB of storage. So your 8 GB wouldn't be sufficient. Adding a L2ARC may improve things but not always. On my 8 GB home server I have about 12 TB of storage. I added a small SSD as L2ARC but it's cache was less than 3% used. Needless to say it barely made a dent in the performance.

In your case I'd max out the machine with RAM. I think that would improve performance a lot more.
 
Hi SirDice,

I'm almost not bothered at all about caching data, although I haven't set metadata caching only I have allowed the metadata max to be equal to ARC max. So on that basis my RAM requirements are going to be more related to number of files than quantity of storage. Not sure if there is any rule of thumb rules for that type of scenario.

With respect to your experience of L2 cache being under used, my understanding is that L2 cache should be used exactly as L1 cache, or to put it another way if L1+L2 is only using 10GB then if you took away the L2 and stuck in more RAM then you should see the same 10GB utilization. If I remember right L2 is basically an overflow for L1 where stuff goes when it would otherwise be evicted completely from the ARC, hence my belief that overall usage profile should be the same for XGB of L1 ARC vs XGB L1+L2 ARC,

thanks, Andy.
 
16GB RAM (2x 80GB L2ARC SSD) Server filled with 9TB (max. 18TB) mixed data (lots of small files, some big files)
ZFS primary and secondary cache are set to metadata.

After using 2 SSDs as L2ARC the server do his jobs (backup via rsync) in nearly half of the time as before.
 
Thanks for the info user23, your setup is pretty close to mine so interesting to hear.

So what utilisation are you seeing on your 2x 80GB SSDs? Does that pretty much all get used despite being set to metadata only?

thanks, Andy.
 
You definitely want to increase the RAM in that system. 8 GB is not enough for a 30 TB pool. And you need ARC to track objects in the L2ARC, meaning you need more RAM if you want to add a cache vdev (L2ARC).

If you only care about metadata, then you don't need an L2ARC. Increase the RAM to 16 GB and set primarycache=metadata on the pool, and increases the metadata limit to the ARC max limit.
 
Hi Freddie,

given the info from user23 and also given the fact that it is possible to set L2ARC to metadata only that would seem to say to me that an SSD is a valid solution for speeding up metadata related actions on a large pool, ie ls, checking file dates etc. The reason I posted this question was to verify this idea in the real world from anyone with experience, both you and SirDice say that SSD is a wrong choice. I believe it to be a good choice in theory and this is backed up by user23 so we have divided opinion.
I am also aware that you need extra memory in L1 to support L2, however if very high speed isn't my cocern, rather avoiding very poor performance is the objective I think this may be solved by running an SSD L2 cache even with only 8GB RAM (obvouisly more is more when it comes to RAM),

thanks, Andy.
 
As phoenix said: "And you need ARC to track objects in the L2ARC"
All the more RAM ... needless to say anything more :)

The only reason why our server don't have more RAM than 16GB is because it is expensive to buy old DDR2 reg. ECC.
One 4GB Module would cost nearly the same as 16GB DDR3 ECC (non reg.) :(

For our server i can say, after 4 days uptime the SSDs are completely filled up.
And while rsync is running the server is reading 100+ MB/s on each SSD.
Perhaps its because of a lot of tiny emails and other small files.
 
You must log in or register to reply here.
Back
Top