kmem and zfs arc value on 8.1 64bit systems

[chrcol]

lots of previous discussions on what people use on i386 and on freebsd7, does anyone tune these now on amd64/freebsd8?

I see the default on a recent 8 gig system I setup is 2.7gig of kmem and 1.7gig for zfs, they both seem conservative values but is the zfs value going to constrain zfs?

 

[AndyUKG]

In many places I have read that in 8.x that you dont have to tune memory for ZFS.
But from digging a bit deeper that seems true with the exception of "vm.kmem_size".
My understanding is that if you have a system with a lot of RAM and you want to use that RAM for ZFS then you should set "vm.kmem_size" to some value slightly higher than you want your ARC cache to be.
Ie you have 8GB RAM and only need 2GB for user land applications, then set vm.kmem_size to 6GB.

Glad to hear others interpretations...

Andy.

 

[AndyUKG]

Actually I think you have a point, seems like its still worth tuning vfs.zfs.arc_max.
I just tested setting vm.kmem_size to 2621440000 and this gives me a default value of 1638400000 for vfs.zfs.arc_max. Thats only 62% of kmem, and leaves 1GB of memory free after the ARC cache which seems unnecessarily high...

Andy.

 

[chrcol]

So if kmem is increased and ARC left to auto, does the system set a bigger ARC but always 1 gig less then kmem?

 

[AndyUKG]

so if kmem is increased and ARC left to auto, does the system set a bigger ARC but always 1 gig less then kmem?

Not sure, as I dont have any systems with more than 4GB to fully test this. But certainly seems you can push up the max ARC cache over the default value. Not sure if anyone will stick their neck out and tell you how high you can set it, as too high may lead to crashes (I suppose)....

Anyone else???

 

[User23]

32GB RAM + ZFS v14, without any ZFS tuning:

vm.kmem_size_scale: 3
vm.kmem_size_max: 329853485875
vm.kmem_size_min: 0
vm.kmem_size: 11108761600

kstat.zfs.misc.arcstats.c_min: 1254377472
kstat.zfs.misc.arcstats.c_max: 10035019776
kstat.zfs.misc.arcstats.size: 8756924392

 

[phoenix]

In many places I have read that in 8.x that you dont have to tune memory for ZFS.
But from digging a bit deeper that seems true with the exception of "vm.kmem_size".
My understanding is that if you have a system with a lot of RAM and you want to use that RAM for ZFS then you should set "vm.kmem_size" to some value slightly higher than you want your ARC cache to be.
Ie you have 8GB RAM and only need 2GB for user land applications, then set vm.kmem_size to 6GB.

On 64-bit systems, kmem_size should be larger than your total RAM, in order to avoid memory fragmentation leading to "kmem_map too small" errors even when you have tonnes of free mem. One of the leading recommendations on the mailing lists is 1.5x RAM.

On 32-bit systems, you want to keep kmem_size under the RAM size, with arc_max limited to 1/2 kmem_size.

 

[chrcol]

On 64-bit systems, kmem_size should be larger than your total RAM, in order to avoid memory fragmentation leading to "kmem_map too small" errors even when you have tonnes of free mem. One of the leading recommendations on the mailing lists is 1.5x RAM.

On 32-bit systems, you want to keep kmem_size under the RAM size, with arc_max limited to 1/2 kmem_size.

Are you suggesting the autotuning is bad then?

As 2.7gig is what? Maybe a 3rd of the ram rather than 1.5x the ram.

 

[AndyUKG]

If setting kmem_size above actual RAM size is recommended it begs the question why this setting exists in 64bit versions of FreeBSD. Perhaps there are still some downsides to setting this high?
Anyway, I think one thing is clear. If you want ZFS to make use of large amounts of RAM you have to tune this even on amd64 systems...

 

[phoenix]

It's not that auto-tuning is bad ... just that it's not perfect.

For many people, with many workloads, the defaults work fine.

For others, manual tuning is needed to get the best stability and the best performance.

No one can say (definitely) that value X will always work best for everyone, nor even that value Y is best for you. You have to do the testing yourself, to find the correct values for your workloads.