• This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn more.

ZFS Recommended way to restrict ZFS memory usage to reasonable limits

Snurg

Aspiring Daemon

Thanks: 241
Messages: 698

#1
I want to limit ZFS memory usage because ZFS' normal behavior to suck up all memory just sucks balls.
Because, this makes other applications drop temporary memory, leading to reloading/rebuilding all possible stuff when needed, bogging down performance and introducing unwanted latencies and annoying screen redraws, only for stupid oversized useless zfs disk caching.

I noticed this after copying two disks. All memory was sucked up by ZFS cache, leaving only the bare minimum for other apps. After rebooting it was fine again. I want not to have to reboot after large copy actions, so I am looking to fix that issue.

Reading the FreeBSD ZFS tuning page I wonder whether the vfs.zfs.arc_max and vfs.zfs.vdev.cache.size parameters are mentioned only for i386.
Are these still being honored on amd64?

I am thinking of settings like, say,
Code:
vm.kmem_size="5G"
vm.kmem_size_max="5G"
vfs.zfs.arc_max="5G"
vfs.zfs.vdev.cache.size="5G"
Questions:
Is there a page detailing what exactly these settings do?
Would it be a good idea not to set vm.kmem_size, just vm.kmem_size_max to limit kernel memory size bloating up? Or would this be bad, for example, fragmenting memory?
Or, would it be better to set only the cache size limits?
And, what ratio should arc and vdev caches have?

Suggestions? Advices?

(An alternative would be the user_reserve_hint_pct variable - but this seems not available on FreeBSD. Having this would save one from reboot...)
 

lebarondemerde

Aspiring Daemon

Thanks: 368
Messages: 975

#2
Hi!

If you do not set those numbers are dynamic, in a meaning of when something need memory ZFS give it away. But, it seems to not work very well for some people too.

Anyway, I think just setting vfs.zfs.arc_max should be enough.
 

Eric A. Borisch

Well-Known Member

Thanks: 160
Messages: 259

#3
Surprise, it’s hard to tune a priori for every workload. That said, vfs.zfs.arc_max is certainly the place to start.

You can also use
vm.v_free_target > vm.pageout_wakeup_thread (both in #of 4k pages) to adjust the system that reclaims / frees free-able memory from the system.
 

Snurg

Aspiring Daemon

Thanks: 241
Messages: 698

#4
Eric A. Borisch
This is for desktop PC usage case...

The problem was that after the copying action (200GB from laptop HD plugged into PC for copying and pulled out after finishing) practically all memory was wired for zfs cache.
All applications I had running had to drop their memory usage to bare minimum.

For example Firefox, which usually takes a few gigabytes with many tabs open, had shrunk to 57 MB. When I switched tabs, they had to reload/rebuild. Quite annoying. It was the same with other apps, too.
This is the reason why I am thinking about how to restrict ZFS memory usage to a fixed upper limit...

There was no real swapping, about 80MB are used of the swap partition. (But that some little bit seems always be used of the swap)
So I am not really sure whether it is related to paging out...
 

Eric A. Borisch

Well-Known Member

Thanks: 160
Messages: 259

#5
This is for desktop PC usage case...
...
There was no real swapping, about 80MB are used of the swap partition. (But that some little bit seems always be used of the swap)
So I am not really sure whether it is related to paging out...
I'd argue FreeBSD ("The power to serve") should be tuned by default for servers. That said, an option in the installer, or even an rc.d/desktop_tune script that would tweak some sysctls appropriately might be interesting.. That might be a better route, as it could continue to evolve over time.

From my recollection, the pageout_wakeup_thread can help make sure that any pages that can be flushed are proactively (in the background) cleaned up sooner than when you’re actually running out of memory, and more likely to get bogged down by contention.
 

Snurg

Aspiring Daemon

Thanks: 241
Messages: 698

#6
I agree that the default tuning should stay as it is.
It's hard to find information about these tunings you suggested. I found little information using google.
Seems there is no complete listing of currently available tuning knobs...

What you suggested could improve responsiveness in scarce memory situations, good for desktop usage.
In my case it was not scarcity of memory, just the problem that zfs took it all, without actual need - not optimum for desktop use.

A desktop tuning script included in the distribution, to be activated optionally, could be a great idea.
It would cost little and help much making dual-use easier by providing some standardization for this.
Otherwise one could argue, why even include X, DEs etc... as this implies desktop usage. And FreeBSD can shine on the desktop also!
 

Maxnix

Active Member

Thanks: 138
Messages: 245

#8
AFAICT, kernel memory limit should be higher than ARC's one (2 x arc_max suggested, IIRC). Personally, I don't limit kernel memory, I just set vfs.zfs.arc_max (and vfs.zfs.arc_min to have a minimum cache) this way on my laptop with 3GB of RAM:
Code:
vfs.zfs.arc_max="500M"
vfs.zfs.arc_min="100M"
I could set a higher limit, but I could need memory for one or more VMs, or for use with tmpfs(5).