1f43e [ZFS] ZFS and Memory - The FreeBSD Forums
The FreeBSD Forums  

Go Back   The FreeBSD Forums > Base System > Storage
Reload this Page [ZFS] ZFS and Memory
Register FAQ Calendar Search Today's Posts Mark Forums Read

Storage Place to ask questions about partitioning, labelling, filesystems, encryption or anything else related to storage area.

Reply
 
Thread Tools Display Modes
  #1  
Old May 1st, 2012, 13:05
Leander Leander is offline
Junior Member
  
Join Date: Apr 2010
Location: Stuttgart
Posts: 58
Thanks: 4
Thanked 0 Times in 0 Posts
Default ZFS and Memory
Hi,

I also tried my luck with ZFS on

FreeBSD - 9.0-RELEASE
Intel® Core™ i5 CPU 760 @ 2.80GHz (4xCore)
64 bit
8GB RAM
4GB file based swap

Now, obviously one can read lots of things, tips and tweaks for FreeBSD System in order to use ZFS "proper". Now, some of those things you'll fall over the most when reading are basically the following four sysctl nodes in order to get a stable system without kernel panics:
  • vm.kmem_size Size of kernel memory
  • vm.kmem_size_max Maximum size of kernel memory
  • vfs.zfs.arc_min Minimum ARC size
  • vfs.zfs.arc_max Maximum ARC size
I left those variables on its default values for my first try:
  • vm.kmem_size 7921.0 MB
  • vm.kmem_size_max 314572.8 MB
  • vfs.zfs.arc_min 862.1 MB
  • vfs.zfs.arc_max 6897.0 MB
... and my system got kernel panic after a 72G of heavy copying. Note that I do not have L2ARC.

Please correct me if I'm wrong here but as far as I understood those variables is that vfs.zfs.arc_* sets the frame/boundaries for the Layer 1 cache of ZFS. vfs.zfs.arc_min is what system should always keep full while vfs.zfs.arc_max represents buffer for higher loads. Also this Layer 1 cache depends and acts within the given frame of vm.kmem_size.
Basically to be on the same page here
  • Layer 1 cache ==> slice of system RAM
  • Layer 2 cache ==> Additional Storage Medium (i.e. SSD)

Now some people advice you to give vm.kmem_size upto 16GB on a 8GB RAM system. This sounds absurd to me since system can not even make use of 16GB since there is only 8GB RAM installed?! Or would this mean you also need a 8GB swap in order to not run out of pages here?

Basically my intuition tells me to let vm.kmem_size = Installed RAM. In fact I figured FreeBSD automatically defaults vm.kmem_size to 94% of hw.realmem - which sounds healthy - while the default vm.kmem_size_max = 314572.8 MB just doesn't sound healthy to me?! Why such a big number? Where should all this be stored in such a "worst case scenario"? Wouldn't it sound more reasonalbe to let vm.kmem_size_max = 94% of hw.realmem PLUS the amount of currently setup swap file's/drive's size?

Is there any reasons against my theory?

Anyway, after some tests I figured to get kernel panics under control by decreasing vfs.zfs.arc_max to 4G (50% of my RAM) and at the same time increase vfs.zfs.arc_min to 1G (12.5% of my RAM).

Now fine, I don't have kernel Panics anymore (right now) - but note that at the same time there is a samba DC, apache, and some other services running on the system as well - and frankly I'm afraid that the system may run into a kernel panic again as soon as those service get some higher load ... ;/ - which is totally unacceptable for a productive system!

This doesn't give me butterflies in my stomach - in fact it makes me feel like FreeBSD's memory management is not well enough off to deal with high load, since I expect it to let a lower prio app wait until the higher prio one finished its work?! But I'm probably just missing out on the right sysctl variables to restrict/prevent kernel panics at all. Is there any suggestions for this based on facts? Or how Can I make sure other services don't disturb ZFS and the other way around and at the same time make full use of given RAM resources?

So how do we milk this cow?

Thanks
Last edited by phoenix; May 1st, 2012 at 17:51. Reason: Minor spelling nits and typos.
Reply With Quote
  #2  
Old May 1st, 2012, 15:41
Leander Leander is offline
Junior Member
  
Join Date: Apr 2010
Location: Stuttgart
Posts: 58
Thanks: 4
Thanked 0 Times in 0 Posts
Default
Quote:
## KERNEL MEMORY
#
# Kernel memory is what ZFS uses and thus tuning this parameter may be useful.
# General recommendation is to set this variable at 1.5 times your RAM size,
# leaving the _max variable untouched.
# Example: if you have 4GiB RAM, then set the kmem_size to "6g" instead.
#
# if you exceed the limits of kernel memory, you may get a panic like this:
# panic: kmem_malloc(131072): kmem_map too small: <totalkmem> total allocated
# this would be a sign you have memory tuning to do, or add physical RAM instead
found on: http://zfsguru.com/forum/zfsgurusupport/82

Why more than physical RAM? I don't get it? Isn't that provoking a panic?
Last edited by Leander; May 1st, 2012 at 20:35.
Reply With Quote
  #3  
Old May 1st, 2012, 16:02
kpa kpa is offline
Giant Locked
  
Join Date: Jul 2010
Location: People's Technocratic Republic of Finland
Posts: 2,141
Thanks: 46
Thanked 512 Times in 433 Posts
Default
The kmem* parameters are not actual amounts of memory but size parameters for a virtual address space where pages of memory (4KB units in i386 and amd64) can be mapped from the physical memory anywhere within the address range. This virtual address space will mostly consist of unused large gaps between small areas of used memory, that's why the address space can be just about as large you want even if you have small amount of physical memory (actually the amount of physical memory does limit the maximum size of this virtual address space because there has to be bookkeeping about which pages are in use and which ones are not).

The idea is that pages of memory can be moved around and merged to larger continuous units in that virtual address space when memory is deallocated to avoid memory fragmentation.

Tweaking the kmem* tunables in generally not recommended, starting with FreeBSD 8.2 the kernel can autotune them quite well.
Last edited by kpa; May 1st, 2012 at 16:43.
Reply With Quote
  #4  
Old May 1st, 2012, 20:22
Leander Leander is offline
Junior Member
  
Join Date: Apr 2010
Location: Stuttgart
Posts: 58
Thanks: 4
Thanked 0 Times in 0 Posts
Default
Code:
Tweaking the kmem* tunables in generally not recommended, starting with FreeBSD 8.2 the kernel can autotune them quite well.
Thanks, this makes me feel more on the safe side too, since I've noticed kernel autotunes them according to the detected RAM amount.

Anyway, I've been testing the entire day under higher system load without any kernel panic thus far:
Code:
vm.kmem_size:        7,735 GB
vm.kmem_size_max:  307,2 GB
vfs.zfs.arc_min:     1,0 GB
vfs.zfs.arc_max:     5,0 GB
Code:
 vfs.numvnodes: 56385
kern.maxvnodes: 206627
Since I consider myself a ZFS beginner it would be lovely if someone could give me hints on where to adjust more to the safe but still performant side. I found the arc_summary.pl script which should summarize the most important information. The script can be found at FreeBSD Wiki - ZFS Tuning Guide

Here is my output after a fresh reboot and before any read/write action on the ZFS:
Code:
arc_summary.pl 

------------------------------------------------------------------------
ZFS Subsystem Report				Tue May  1 16:27:56 2012
------------------------------------------------------------------------

System Memory:

	0.97%	77.10	MiB Active,	0.57%	44.99	MiB Inact
	2.52%	199.87	MiB Wired,	0.00%	280.00	KiB Cache
	95.93%	7.42	GiB Free,	0.01%	544.00	KiB Gap

	Real Installed:				8.00	GiB
	Real Available:			99.72%	7.98	GiB
	Real Managed:			96.96%	7.74	GiB

	Logical Total:				8.00	GiB
	Logical Used:			6.70%	548.53	MiB
	Logical Free:			93.30%	7.46	GiB

Kernel Memory:					98.66	MiB
	Data:				79.50%	78.44	MiB
	Text:				20.50%	20.22	MiB

Kernel Memory Map:				7.74	GiB
	Size:				0.37%	29.06	MiB
	Free:				99.63%	7.71	GiB
								Page:  1
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
	Storage pool Version:			28
	Filesystem Version:			5
	Memory Throttle Count:			0

ARC Misc:
	Deleted:				7
	Recycle Misses:				0
	Mutex Misses:				0
	Evict Skips:				0

ARC Size:				0.03%	1.39	MiB
	Target Size: (Adaptive)		100.00%	5.00	GiB
	Min Size (Hard Limit):		20.00%	1.00	GiB
	Max Size (High Water):		5:1	5.00	GiB

ARC Size Breakdown:
	Recently Used Cache Size:	50.00%	2.50	GiB
	Frequently Used Cache Size:	50.00%	2.50	GiB

ARC Hash Breakdown:
	Elements Max:				166
	Elements Current:		100.00%	166
	Collisions:				0
	Chain Max:				0
	Chains:					0
								Page:  2
------------------------------------------------------------------------

ARC Efficiency:					5.65k
	Cache Hit Ratio:		96.92%	5.48k
	Cache Miss Ratio:		3.08%	174
	Actual Hit Ratio:		95.47%	5.40k

	Data Demand Efficiency:		100.00%	0

	CACHE HITS BY CACHE LIST:
	  Anonymously Used:		1.50%	82
	  Most Recently Used:		14.29%	783
	  Most Frequently Used:		84.22%	4.62k
	  Most Recently Used Ghost:	0.00%	0
	  Most Frequently Used Ghost:	0.00%	0

	CACHE HITS BY DATA TYPE:
	  Demand Data:			0.00%	0
	  Prefetch Data:		0.00%	0
	  Demand Metadata:		98.39%	5.39k
	  Prefetch Metadata:		1.61%	88

	CACHE MISSES BY DATA TYPE:
	  Demand Data:			0.00%	0
	  Prefetch Data:		0.00%	0
	  Demand Metadata:		51.72%	90
	  Prefetch Metadata:		48.28%	84
								Page:  3
------------------------------------------------------------------------

File-Level Prefetch: (HEALTHY)

DMU Efficiency:					6.25k
	Hit Ratio:			99.81%	6.24k
	Miss Ratio:			0.19%	12

	Colinear:				12
	  Hit Ratio:			0.00%	0
	  Miss Ratio:			100.00%	12

	Stride:					6.16k
	  Hit Ratio:			100.00%	6.16k
	  Miss Ratio:			0.00%	0

DMU Misc:
	Reclaim:				12
	  Successes:			0.00%	0
	  Failures:			100.00%	12

	Streams:				82
	  +Resets:			0.00%	0
	  -Resets:			100.00%	82
	  Bogus:				0
								Page:  5
------------------------------------------------------------------------

ZFS Tunable (sysctl):
	kern.maxusers                           384
	vm.kmem_size                            8305745920
	vm.kmem_size_scale                      1
	vm.kmem_size_min                        0
	vm.kmem_size_max                        329853485875
	vfs.zfs.l2c_only_size                   0
	vfs.zfs.mfu_ghost_data_lsize            0
	vfs.zfs.mfu_ghost_metadata_lsize        0
	vfs.zfs.mfu_ghost_size                  0
	vfs.zfs.mfu_data_lsize                  0
	vfs.zfs.mfu_metadata_lsize              168960
	vfs.zfs.mfu_size                        175104
	vfs.zfs.mru_ghost_data_lsize            0
	vfs.zfs.mru_ghost_metadata_lsize        4096
	vfs.zfs.mru_ghost_size                  4096
	vfs.zfs.mru_data_lsize                  0
	vfs.zfs.mru_metadata_lsize              535040
	vfs.zfs.mru_size                        1063424
	vfs.zfs.anon_data_lsize                 0
	vfs.zfs.anon_metadata_lsize             0
	vfs.zfs.anon_size                       0
	vfs.zfs.l2arc_norw                      1
	vfs.zfs.l2arc_feed_again                1
	vfs.zfs.l2arc_noprefetch                1
	vfs.zfs.l2arc_feed_min_ms               200
	vfs.zfs.l2arc_feed_secs                 1
	vfs.zfs.l2arc_headroom                  2
	vfs.zfs.l2arc_write_boost               8388608
	vfs.zfs.l2arc_write_max                 8388608
	vfs.zfs.arc_meta_limit                  1342177280
	vfs.zfs.arc_meta_used                   1452568
	vfs.zfs.arc_min                         1073741824
	vfs.zfs.arc_max                         5368709120
	vfs.zfs.dedup.prefetch                  1
	vfs.zfs.mdcomp_disable                  0
	vfs.zfs.write_limit_override            0
	vfs.zfs.write_limit_inflated            25698410496
	vfs.zfs.write_limit_max                 1070767104
	vfs.zfs.write_limit_min                 33554432
	vfs.zfs.write_limit_shift               3
	vfs.zfs.no_write_throttle               0
	vfs.zfs.zfetch.array_rd_sz              1048576
	vfs.zfs.zfetch.block_cap                256
	vfs.zfs.zfetch.min_sec_reap             2
	vfs.zfs.zfetch.max_streams              8
	vfs.zfs.prefetch_disable                0
	vfs.zfs.mg_alloc_failures               8
	vfs.zfs.check_hostid                    1
	vfs.zfs.recover                         0
	vfs.zfs.txg.synctime_ms                 1000
	vfs.zfs.txg.timeout                     5
	vfs.zfs.scrub_limit                     10
	vfs.zfs.vdev.cache.bshift               16
	vfs.zfs.vdev.cache.size                 0
	vfs.zfs.vdev.cache.max                  16384
	vfs.zfs.vdev.write_gap_limit            4096
	vfs.zfs.vdev.read_gap_limit             32768
	vfs.zfs.vdev.aggregation_limit          131072
	vfs.zfs.vdev.ramp_rate                  2
	vfs.zfs.vdev.time_shift                 6
	vfs.zfs.vdev.min_pending                4
	vfs.zfs.vdev.max_pending                10
	vfs.zfs.vdev.bio_flush_disable          0
	vfs.zfs.cache_flush_disable             0
	vfs.zfs.zil_replay_disable              0
	vfs.zfs.zio.use_uma                     0
	vfs.zfs.version.zpl                     5
	vfs.zfs.version.spa                     28
	vfs.zfs.version.acl                     1
	vfs.zfs.debug                           0
	vfs.zfs.super_owner                     0
								Page:  7
------------------------------------------------------------------------
Last edited by DutchDaemon; May 2nd, 2012 at 00:27.
Reply With Quote
  #5  
Old May 1st, 2012, 20:25
Leander Leander is offline
Junior Member
  
Join Date: Apr 2010
Location: Stuttgart
Posts: 58
Thanks: 4
Thanked 0 Times in 0 Posts
Default
... Output of arc_summary.pl right after I started cp to copy the content of a 2TB HDD over to my new ZFS:

Code:
------------------------------------------------------------------------
ZFS Subsystem Report                            Tue May  1 18:08:44 2012
------------------------------------------------------------------------

System Memory:

        1.67%   132.02  MiB Active,     38.03%  2.94    GiB Inact
        36.19%  2.80    GiB Wired,      0.00%   296.00  KiB Cache
        24.10%  1.86    GiB Free,       0.01%   644.00  KiB Gap

        Real Installed:                         8.00    GiB
        Real Available:                 99.72%  7.98    GiB
        Real Managed:                   96.96%  7.74    GiB

        Logical Total:                          8.00    GiB
        Logical Used:                   39.92%  3.19    GiB
        Logical Free:                   60.08%  4.81    GiB

Kernel Memory:                                  1.86    GiB
        Data:                           98.94%  1.84    GiB
        Text:                           1.06%   20.22   MiB

Kernel Memory Map:                              7.73    GiB
        Size:                           23.01%  1.78    GiB
        Free:                           76.99%  5.95    GiB
                                                                Page:  1
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Storage pool Version:                   28
        Filesystem Version:                     5
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                7
        Recycle Misses:                         0
        Mutex Misses:                           0
        Evict Skips:                            0

ARC Size:                               35.18%  1.76    GiB
        Target Size: (Adaptive)         100.00% 5.00    GiB
        Min Size (Hard Limit):          20.00%  1.00    GiB
        Max Size (High Water):          5:1     5.00    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       50.00%  2.50    GiB
        Frequently Used Cache Size:     50.00%  2.50    GiB

ARC Hash Breakdown:
        Elements Max:                           18.11k
        Elements Current:               100.00% 18.11k
        Collisions:                             3.89k
        Chain Max:                              3
        Chains:                                 1.14k
                                                                Page:  2
------------------------------------------------------------------------

ARC Efficiency:                                 998.22k
        Cache Hit Ratio:                86.90%  867.43k
        Cache Miss Ratio:               13.10%  130.79k
        Actual Hit Ratio:               62.03%  619.25k

        Data Demand Efficiency:         99.97%  6.71k

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             28.61%  248.19k
          Most Recently Used:           54.80%  475.31k
          Most Frequently Used:         16.59%  143.93k
          Most Recently Used Ghost:     0.00%   0
          Most Frequently Used Ghost:   0.00%   0

        CACHE HITS BY DATA TYPE:
          Demand Data:                  0.77%   6.70k
          Prefetch Data:                0.00%   0
          Demand Metadata:              70.61%  612.53k
          Prefetch Metadata:            28.61%  248.20k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  0.00%   2
          Prefetch Data:                0.00%   0
          Demand Metadata:              95.87%  125.39k
          Prefetch Metadata:            4.12%   5.39k
                                                                Page:  3
------------------------------------------------------------------------

File-Level Prefetch: (HEALTHY)

DMU Efficiency:                                 3.62m
        Hit Ratio:                      81.64%  2.95m
        Miss Ratio:                     18.36%  664.16k

        Colinear:                               664.16k
          Hit Ratio:                    0.03%   195
          Miss Ratio:                   99.97%  663.97k

        Stride:                                 2.93m
          Hit Ratio:                    100.00% 2.93m
          Miss Ratio:                   0.00%   0

DMU Misc:
        Reclaim:                                663.97k
          Successes:                    0.15%   1.00k
          Failures:                     99.85%  662.97k

        Streams:                                22.89k
          +Resets:                      0.00%   0
          -Resets:                      100.00% 22.89k
          Bogus:                                0
                                                                Page:  5
------------------------------------------------------------------------

ZFS Tunable (sysctl):
        kern.maxusers                           384
        vm.kmem_size                            8305745920
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        329853485875
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_lsize            0
        vfs.zfs.mfu_ghost_metadata_lsize        0
        vfs.zfs.mfu_ghost_size                  0
        vfs.zfs.mfu_data_lsize                  59343360
        vfs.zfs.mfu_metadata_lsize              173056
        vfs.zfs.mfu_size                        59522560
        vfs.zfs.mru_ghost_data_lsize            0
        vfs.zfs.mru_ghost_metadata_lsize        2449408
        vfs.zfs.mru_ghost_size                  2449408
        vfs.zfs.mru_data_lsize                  1709782528
        vfs.zfs.mru_metadata_lsize              8559616
        vfs.zfs.mru_size                        1722849792
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       95635456
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  1342177280
        vfs.zfs.arc_meta_used                   27272600
        vfs.zfs.arc_min                         1073741824
        vfs.zfs.arc_max                         5368709120
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.write_limit_override            0
        vfs.zfs.write_limit_inflated            25698410496
        vfs.zfs.write_limit_max                 1070767104
        vfs.zfs.write_limit_min                 33554432
        vfs.zfs.write_limit_shift               3
        vfs.zfs.no_write_throttle               0
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                0
        vfs.zfs.mg_alloc_failures               8
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.txg.synctime_ms                 1000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.scrub_limit                     10
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.ramp_rate                  2
        vfs.zfs.vdev.time_shift                 6
        vfs.zfs.vdev.min_pending                4
        vfs.zfs.vdev.max_pending                10
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     28
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0
                                                                Page:  7
------------------------------------------------------------------------
Last edited by Leander; May 1st, 2012 at 20:38.
Reply With Quote
  #6  
Old May 1st, 2012, 20:31
Leander Leander is offline
Junior Member
  
Join Date: Apr 2010
Location: Stuttgart
Posts: 58
Thanks: 4
Thanked 0 Times in 0 Posts
Default
And eventually my output of arc_summary.pl couple of hours later - still the same action - writing random files without - interruption of the copy process:
Code:
------------------------------------------------------------------------
ZFS Subsystem Report                            Tue May  1 21:05:33 2012
------------------------------------------------------------------------

System Memory:

        3.57%   282.61  MiB Active,     64.30%  4.97    GiB Inact
        27.30%  2.11    GiB Wired,      4.67%   370.14  MiB Cache
        0.15%   11.93   MiB Free,       0.01%   628.00  KiB Gap

        Real Installed:                         8.00    GiB
        Real Available:                 99.72%  7.98    GiB
        Real Managed:                   96.96%  7.74    GiB

        Logical Total:                          8.00    GiB
        Logical Used:                   33.17%  2.65    GiB
        Logical Free:                   66.83%  5.35    GiB

Kernel Memory:                                  1.08    GiB
        Data:                           98.17%  1.06    GiB
        Text:                           1.83%   20.32   MiB

Kernel Memory Map:                              7.19    GiB
        Size:                           13.76%  1013.16 MiB
        Free:                           86.24%  6.20    GiB
                                                                Page:  1
------------------------------------------------------------------------

ARC Summary: (THROTTLED)
        Storage pool Version:                   28
        Filesystem Version:                     5
        Memory Throttle Count:                  3.96k

ARC Misc:
        Deleted:                                1.68m
        Recycle Misses:                         72.33k
        Mutex Misses:                           7
        Evict Skips:                            7

ARC Size:                               20.00%  1.00    GiB
        Target Size: (Adaptive)         20.00%  1.00    GiB
        Min Size (Hard Limit):          20.00%  1.00    GiB
        Max Size (High Water):          5:1     5.00    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       93.75%  960.06  MiB
        Frequently Used Cache Size:     6.25%   63.97   MiB

ARC Hash Breakdown:
        Elements Max:                           32.70k
        Elements Current:               71.77%  23.46k
        Collisions:                             302.87k
        Chain Max:                              5
        Chains:                                 2.26k
                                                                Page:  2
------------------------------------------------------------------------

ARC Efficiency:                                 2.41m
        Cache Hit Ratio:                93.64%  2.25m
        Cache Miss Ratio:               6.36%   152.99k
        Actual Hit Ratio:               81.53%  1.96m

        Data Demand Efficiency:         99.98%  1.19m
        Data Prefetch Efficiency:       61.05%  42.68k

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             12.70%  286.05k
          Most Recently Used:           30.11%  678.21k
          Most Frequently Used:         56.96%  1.28m
          Most Recently Used Ghost:     0.09%   2.02k
          Most Frequently Used Ghost:   0.14%   3.13k

        CACHE HITS BY DATA TYPE:
          Demand Data:                  52.67%  1.19m
          Prefetch Data:                1.16%   26.06k
          Demand Metadata:              34.10%  768.23k
          Prefetch Metadata:            12.07%  271.92k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  0.12%   184
          Prefetch Data:                10.87%  16.62k
          Demand Metadata:              83.51%  127.77k
          Prefetch Metadata:            5.50%   8.42k
                                                                Page:  3
------------------------------------------------------------------------

File-Level Prefetch: (HEALTHY)

DMU Efficiency:                                 55.20m
        Hit Ratio:                      98.80%  54.53m
        Miss Ratio:                     1.20%   664.34k

        Colinear:                               664.34k
          Hit Ratio:                    0.03%   196
          Miss Ratio:                   99.97%  664.14k

        Stride:                                 54.49m
          Hit Ratio:                    100.00% 54.49m
          Miss Ratio:                   0.00%   114

DMU Misc:
        Reclaim:                                664.14k
          Successes:                    0.17%   1.12k
          Failures:                     99.83%  663.03k

        Streams:                                41.08k
          +Resets:                      0.10%   43
          -Resets:                      99.90%  41.04k
          Bogus:                                0
                                                                Page:  5
------------------------------------------------------------------------

ZFS Tunable (sysctl):
        kern.maxusers                           384
        vm.kmem_size                            8305745920
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        329853485875
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_lsize            981729280
        vfs.zfs.mfu_ghost_metadata_lsize        23986688
        vfs.zfs.mfu_ghost_size                  1005715968
        vfs.zfs.mfu_data_lsize                  0
        vfs.zfs.mfu_metadata_lsize              9241600
        vfs.zfs.mfu_size                        11002880
        vfs.zfs.mru_ghost_data_lsize            0
        vfs.zfs.mru_ghost_metadata_lsize        67137536
        vfs.zfs.mru_ghost_size                  67137536
        vfs.zfs.mru_data_lsize                  932244480
        vfs.zfs.mru_metadata_lsize              9625600
        vfs.zfs.mru_size                        965317632
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       41498112
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  1342177280
        vfs.zfs.arc_meta_used                   100359352
        vfs.zfs.arc_min                         1073741824
        vfs.zfs.arc_max                         5368709120
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.write_limit_override            0
        vfs.zfs.write_limit_inflated            25698410496
        vfs.zfs.write_limit_max                 1070767104
        vfs.zfs.write_limit_min                 33554432
        vfs.zfs.write_limit_shift               3
        vfs.zfs.no_write_throttle               0
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                0
        vfs.zfs.mg_alloc_failures               8
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.txg.synctime_ms                 1000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.scrub_limit                     10
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.ramp_rate                  2
        vfs.zfs.vdev.time_shift                 6
        vfs.zfs.vdev.min_pending                4
        vfs.zfs.vdev.max_pending                10
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     28
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0
                                                                Page:  7
------------------------------------------------------------------------

Note that
Code:
Kernel Memory:                                  1.08    GiB
        Data:                           98.17%  1.06    GiB
        Text:                           1.83%   20.32   MiB
Kept pretty much static on those values, it was only in the very beginning that I've had such high values
Code:
Kernel Memory:                                  1.86    GiB
        Data:                           98.94%  1.84    GiB
        Text:                           1.06%   20.22   MiB
Last edited by DutchDaemon; May 2nd, 2012 at 00:28.
Reply With Quote
  #7  
Old May 1st, 2012, 20:32
Leander Leander is offline
Junior Member
  
Join Date: Apr 2010
Location: Stuttgart
Posts: 58
Thanks: 4
Thanked 0 Times in 0 Posts
Default
Btw. I have only one RAIDz1 running (currently resilvering for testing purposes):
... The md[n] devices are only for temporary testing. But sill, each of them has its own source file on a seperate HDD.

Code:
zpool status -v
  pool: zStar
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since Tue May  1 20:54:40 2012
    68,0G scanned out of 214G at 33,5M/s, 1h14m to go
    7,56G resilvered, 31,75% done
config:

        NAME                       STATE     READ WRITE CKSUM
        zStar                      DEGRADED     0     0     0
          raidz1-0                 DEGRADED     0     0     0
            md3                    ONLINE       0     0     0
            md4                    ONLINE       0     0     0
            md5                    ONLINE       0     0     0
            md6                    ONLINE       0     0     0
            md7                    ONLINE       0     0     0
            md8                    ONLINE       0     0     0
            replacing-5            UNAVAIL      0     0     0
              6048967463963296996  UNAVAIL      0     0     0  was /dev/md8/old
              md8                  ONLINE       0     0     0  (resilvering)
            md9                    ONLINE       0     0     0
            ada3p1                 ONLINE       0     0     0
            ada8p1                 ONLINE       0     0     0

errors: No known data errors

Thanks for hints
Last edited by DutchDaemon; May 2nd, 2012 at 00:29.
Reply With Quote
  #8  
Old May 1st, 2012, 20:43
kpa kpa is offline
Giant Locked
  
Join Date: Jul 2010
Location: People's Technocratic Republic of Finland
Posts: 2,141
Thanks: 46
Thanked 512 Times in 433 Posts
Default
Start with a very conservative setting of vfs.zfs.arc_max, let's say 1GB and up it in 1GB increments. Leave other settings to their defaults. This way you can very quickly tell if the crash is really from memory exhaustion caused by the ARC cache or something else.
Reply With Quote
  #9  
Old May 1st, 2012, 22:04
gkontos's Avatar
gkontos gkontos is offline
Senior Member
  
Join Date: Dec 2009
Location: Polidendri, GR
Posts: 1,298
Thanks: 44
Thanked 230 Times in 172 Posts
Default
Quote:
Originally Posted by Leander View Post
Anyway, after some tests I figured to get kernel panics under control by decreasing vfs.zfs.arc_max to 4G (50% of my RAM) and at the same time increase vfs.zfs.arc_min to 1G (12.5% of my RAM).
Start by limiting the size of ARC_MAX and disabling PREFETCH. That should be enough.

Code:
vfs.zfs.prefetch_disable=1
vfs.zfs.arc_max="4096M"
Don't tweak any other values before you really know what you are doing.

Also please post the output of:

# zfs get all zStar
# zdb -DD zStar

Regards
__________________
Powered by BareBSD
Reply With Quote
  #10  
Old May 1st, 2012, 22:27
usdmatt usdmatt is offline
Member
  
Join Date: Mar 2009
Posts: 253
Thanks: 1
Thanked 67 Times in 56 Posts
Default
Not going to get involved in zfs tuning discussion as I have no expertise in it yet but just noting that there is a port of arc_summary.pl under sysutils/zfs-stats.
Last edited by phoenix; May 1st, 2012 at 22:38. Reason: Please format your post!
Reply With Quote
  #11  
Old May 2nd, 2012, 01:43
t1066 t1066 is offline
Member
  
Join Date: Jun 2010
Posts: 144
Thanks: 3
Thanked 26 Times in 25 Posts
Default
How do you set up your system? Where doea\s root reside and where do you put the swap file?
Last edited by DutchDaemon; May 2nd, 2012 at 16:21.
Reply With Quote
  #12  
Old May 2nd, 2012, 09:48
gkontos's Avatar
gkontos gkontos is offline
Senior Member
  
Join Date: Dec 2009
Location: Polidendri, GR
Posts: 1,298
Thanks: 44
Thanked 230 Times in 172 Posts
Default
Quote:
Originally Posted by usdmatt View Post
Not going to get involved in zfs tuning discussion as I have no expertise in it yet but just noting that there is a port of arc_summary.pl under sysutils/zfs-stats.
I think that the statistics he posted were coming from that script.
__________________
Powered by BareBSD
Reply With Quote
  #13  
Old May 3rd, 2012, 09:27
usdmatt usdmatt is offline
Member
  
Join Date: Mar 2009
Posts: 253
Thanks: 1
Thanked 67 Times in 56 Posts
Default
Quote:
Originally Posted by gkontos View Post
I think that the statistics he posted were coming from that script.
Yes, I know. He is using arc_summary.pl downloaded via the FreeBSD wiki, as mentioned in his post. I was just making people aware that there is a port of this now, so rather that finding it on the net you can just type:

pkg_add -r zfs-stats
Last edited by DutchDaemon; May 3rd, 2012 at 12:25.
Reply With Quote
  #14  
Old May 4th, 2012, 01:37
kpa kpa is offline
Giant Locked
  
Join Date: Jul 2010
Location: People's Technocratic Republic of Finland
Posts: 2,141
Thanks: 46
Thanked 512 Times in 433 Posts
Default
Please note that what I wrote above about the kmem tunables only applies to amd64, i386 is quite different because of limited address space and usually requires tuning of kmem settings for stable operation of ZFS.
Reply With Quote
Reply

« Previous Thread | Next Thread »
Thread Tools
Display Modes
Linear Mode Linear Mode

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
KDE 4.3 is great. But it uses too much memory. everypot KDE 10 July 25th, 2012 08:43
[Solved] posix memory based semaphore in shared memory renzor FreeBSD Development 2 January 24th, 2011 05:41
[Solved] managing memory in freebsd - inactive memory chrcol General 5 March 12th, 2010 01:33
[Solved] Inactive Memory vs. Free Memory dordal General 5 November 23rd, 2009 13:39
Memory check ccc System Hardware 9 July 14th, 2009 18:47


All times are GMT +1. The time now is 13:12.

Contact Us - FreeBSD - Archive - Top

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2013, vBulletin Solutions, Inc.
The mark FreeBSD is a registered trademark of The FreeBSD Foundation and is used by The FreeBSD Project with the permission of The FreeBSD Foundation.
Web protection and acceleration provided by CloudFlare
0