ZFS: How To Calulate for Tuning?

dave

dave

I have looked at the tuning guide, and I am still pretty confused. I guess you have to know a lot about FreeBSD internals to be able to know how much kmem, arc, etc to configure? I am pretty good with FreeBSD, but this is totally confusing to me.

For instance, consider the following: "The following script will summarize kernel memory utilization and assist in tuning arc_max and VDEV cache size." I run the script, but what is it supposed to be telling me about how to configure /boot/loader.conf?

I have 2GB of RAM

uname -a
Code: 
[dave@tank ~]$ uname -a
FreeBSD tank.xxx.xxx 8.0-RELEASE FreeBSD 8.0-RELEASE #1: Fri Nov 27 13:26:50 PST 2009     root@tank.xxx.xxx:/usr/obj/usr/src/sys/TANKZFS  i386

memory monitor script ouput
Code: 
[dave@tank ~]$ zmem.sh 
TEXT=13189088, 12.5781 MB
DATA=47984640, 45.7617 MB
TOTAL=61173728, 58.3398 MB

cpu:
Code: 
AMD Athlon(tm) XP 2800+ (2083.11-MHz 686-class CPU)

kernel conf:
Code: 
#
# TANKZFS -- Kernel conf file for Tank with ZFS Tuning by DT on 20091113
#
include		GENERIC
ident		TANKZFS

options		KVA_PAGES=512

/boot/loader.conf
Code: 
vm.kmem_size="1024M"
vm.kmem_size_max="1024M"
vfs.zfs.arc_max="320M"
vfs.zfs.vdev.cache.size="5M"
vfs.zfs.prefetch_disable=1

I have 3 750GB drives configured in a ZRAID.

I have tried many different combinations in the /boot/loader.conf file.

Whenever my system does anything on more than few files on my zfs, the system just reboots. For instance, the daily cron check setiud causes the machine to reboot. So, this is not stable at all.

Am I missing something? Is this machine not good enough for ZFS? Is there a way to calculate how to configure /boot/loader.conf?
 
I'd say remove vfs.zfs.vdev.cache.size="5M" and vfs.zfs.prefetch_disable=1
and set vfs.zfs.arc_max="512M"

prefetch is disabled by default on i386

here's my settings for 2.5G ram
Code: 
vm.kmem_size="1100M"
vm.kmem_size_max="1100M"
vfs.zfs.arc_max="600M"


Note that you may need to create custom kernel.
In my custom kernel I have
Code: 
options     KVA_PAGES=384
you may need to adjust this number... with many reboots and panics...
I would like to set it higher, but I can't..... because reboots...

If I remember correctly you must be able to divide KVA_PAGES by 4

Hope this will help you
 
I'm FAIRLY sure freebsd 8.0 can run fine without tuning....i know for SURE it can on amd64....not 100% sure about 32bit though.


what i would do is this:

install freebsd and set up raidz like you want...then install something like iozone and run a few tests on it.

i would imagine your system should be fine for ZFS, though it's kinda on the low end....i know people who have run it with a p4 and only 1 GB ram, though i haven't PERSONALLY done this.
 
yah, i see that now after reading up on it (this post got me interested in it)

I remember checking up on it when i set my system but but i must not have read it as well as i should have....I wasn't 100% sure...thanks for the info.
 
What is the script in the tuning guide supposed to tell me about how to tune for ZFS? Here is my output:

Code: 
TEXT=13189088, 12.5781 MB
DATA=109240320, 104.18 MB
TOTAL=122429408, 116.758 MB

I have seen it get to maybe 250MB TOTAL before a panic. Does this somehow help me know what numbers to put into the /boot/loader.conf file? What is the relationship?
 
My rule of thumb has been to set vm.kmem_max_size to half of RAM, and set vfs.zfs.arc_max to half of vm.kmem_max_size.

With the caveat that 32-bit FreeBSD up to 7.1 has a kmem_max_size limit of 1596 MB, and 32-bit FreeBSD 7.2 has a kmem_max_size_limit of about 2.5 GB.

That's a good starting place. And then you just play with (reduce) the arc_max setting whenever the system locks up. :)

If possible, even if you have less than 4 GB of RAM, run 64-bit FreeBSD. You'll get fewer memory fragmentation and exhaustion issues.
 
Yeah, each time it crashes, you reduce the ARC max size a bit. Eventually, you'll find a setting that works "perfectly".

On my home server, which is 32-bit, I have arc_max set to 128M, for example.
 
hmmm, my curent acr_max is 600M.... I have 2.5G ram on i686 arch
I managed to compile all OOO packages with this {I think it's good stress test. lol}

So, would you then suggest me to increase it untill I find point, when it starts to crash, and then go 1 step back?

This is interesting....


P.S.
Code: 
vm.kmem_size="1100M"
vm.kmem_size_max="1100M"
vfs.zfs.arc_max="600M"
 
That's pretty much the process on 32-bit systems. Change a setting, run until it crashes. Change a setting, run until it crashes. Repeat until it takes several weeks/months to crash. :) Each workload will have its own "sweetspot".
 
I should note that while ZFS tuning was a bit troublesome, I later found out that there was another problem which was exacerbating the situation. I had a corrupt folder on the ZFS pool that was crashing the system every time clamav tried to scan it. I thought that the machine was rebooting due to performance/memory issues with ZFS, and in some cases it was. But I was closer than I thought to a stable config.
 
You must log in or register to reply here.
Back
Top