1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

ZFS and Memory

Discussion in 'Storage' started by Leander, May 1, 2012.

  1. Leander

    Leander Member

    Messages:
    180
    Thanks Received:
    2
    Hi,

    I also tried my luck with ZFS on

    FreeBSD - 9.0-RELEASE
    Intel® Core™ i5 CPU 760 @ 2.80GHz (4xCore)
    64 bit
    8GB RAM
    4GB file based swap


    Now, obviously one can read lots of things, tips and tweaks for FreeBSD System in order to use ZFS "proper". Now, some of those things you'll fall over the most when reading are basically the following four sysctl nodes in order to get a stable system without kernel panics:

    • vm.kmem_size Size of kernel memory
    • vm.kmem_size_max Maximum size of kernel memory
    • vfs.zfs.arc_min Minimum ARC size
    • vfs.zfs.arc_max Maximum ARC size
    I left those variables on its default values for my first try:

    • vm.kmem_size 7921.0 MB
    • vm.kmem_size_max 314572.8 MB
    • vfs.zfs.arc_min 862.1 MB
    • vfs.zfs.arc_max 6897.0 MB
    ... and my system got kernel panic after a 72G of heavy copying. Note that I do not have L2ARC.

    Please correct me if I'm wrong here but as far as I understood those variables is that vfs.zfs.arc_* sets the frame/boundaries for the Layer 1 cache of ZFS. vfs.zfs.arc_min is what system should always keep full while vfs.zfs.arc_max represents buffer for higher loads. Also this Layer 1 cache depends and acts within the given frame of vm.kmem_size.
    Basically to be on the same page here
    • Layer 1 cache ==> slice of system RAM
    • Layer 2 cache ==> Additional Storage Medium (i.e. SSD)

    Now some people advice you to give vm.kmem_size upto 16GB on a 8GB RAM system. This sounds absurd to me since system can not even make use of 16GB since there is only 8GB RAM installed?! Or would this mean you also need a 8GB swap in order to not run out of pages here?

    Basically my intuition tells me to let vm.kmem_size = Installed RAM. In fact I figured FreeBSD automatically defaults vm.kmem_size to 94% of hw.realmem - which sounds healthy - while the default vm.kmem_size_max = 314572.8 MB just doesn't sound healthy to me?! Why such a big number? Where should all this be stored in such a "worst case scenario"? Wouldn't it sound more reasonalbe to let vm.kmem_size_max = 94% of hw.realmem PLUS the amount of currently setup swap file's/drive's size?

    Is there any reasons against my theory?

    Anyway, after some tests I figured to get kernel panics under control by decreasing vfs.zfs.arc_max to 4G (50% of my RAM) and at the same time increase vfs.zfs.arc_min to 1G (12.5% of my RAM).

    Now fine, I don't have kernel Panics anymore (right now) - but note that at the same time there is a samba DC, apache, and some other services running on the system as well - and frankly I'm afraid that the system may run into a kernel panic again as soon as those service get some higher load ... ;/ - which is totally unacceptable for a productive system!

    This doesn't give me butterflies in my stomach - in fact it makes me feel like FreeBSD's memory management is not well enough off to deal with high load, since I expect it to let a lower prio app wait until the higher prio one finished its work?! But I'm probably just missing out on the right sysctl variables to restrict/prevent kernel panics at all. Is there any suggestions for this based on facts? Or how Can I make sure other services don't disturb ZFS and the other way around and at the same time make full use of given RAM resources?

    So how do we milk this cow? ;)

    Thanks
     
  2. Leander

    Leander Member

    Messages:
    180
    Thanks Received:
    2
    found on: http://zfsguru.com/forum/zfsgurusupport/82

    Why more than physical RAM? I don't get it? Isn't that provoking a panic?
     
  3. kpa

    kpa Well-Known Member

    Messages:
    4,106
    Thanks Received:
    812
    The kmem* parameters are not actual amounts of memory but size parameters for a virtual address space where pages of memory (4KB units in i386 and amd64) can be mapped from the physical memory anywhere within the address range. This virtual address space will mostly consist of unused large gaps between small areas of used memory, that's why the address space can be just about as large you want even if you have small amount of physical memory (actually the amount of physical memory does limit the maximum size of this virtual address space because there has to be bookkeeping about which pages are in use and which ones are not).

    The idea is that pages of memory can be moved around and merged to larger continuous units in that virtual address space when memory is deallocated to avoid memory fragmentation.

    Tweaking the kmem* tunables in generally not recommended, starting with FreeBSD 8.2 the kernel can autotune them quite well.
     
  4. Leander

    Leander Member

    Messages:
    180
    Thanks Received:
    2
    Code:
    Tweaking the kmem* tunables in generally not recommended, starting with FreeBSD 8.2 the kernel can autotune them quite well.

    Thanks, this makes me feel more on the safe side too, since I've noticed kernel autotunes them according to the detected RAM amount.

    Anyway, I've been testing the entire day under higher system load without any kernel panic thus far:
    Code:
    vm.kmem_size:        7,735 GB
    vm.kmem_size_max:  307,2 GB
    vfs.zfs.arc_min:     1,0 GB
    vfs.zfs.arc_max:     5,0 GB


    Code:
     vfs.numvnodes: 56385
    kern.maxvnodes: 206627


    Since I consider myself a ZFS beginner it would be lovely if someone could give me hints on where to adjust more to the safe but still performant side. I found the arc_summary.pl script which should summarize the most important information. The script can be found at FreeBSD Wiki - ZFS Tuning Guide

    Here is my output after a fresh reboot and before any read/write action on the ZFS:
    Code:
    arc_summary.pl 
    
    ------------------------------------------------------------------------
    ZFS Subsystem Report				Tue May  1 16:27:56 2012
    ------------------------------------------------------------------------
    
    System Memory:
    
    	0.97%	77.10	MiB Active,	0.57%	44.99	MiB Inact
    	2.52%	199.87	MiB Wired,	0.00%	280.00	KiB Cache
    	95.93%	7.42	GiB Free,	0.01%	544.00	KiB Gap
    
    	Real Installed:				8.00	GiB
    	Real Available:			99.72%	7.98	GiB
    	Real Managed:			96.96%	7.74	GiB
    
    	Logical Total:				8.00	GiB
    	Logical Used:			6.70%	548.53	MiB
    	Logical Free:			93.30%	7.46	GiB
    
    Kernel Memory:					98.66	MiB
    	Data:				79.50%	78.44	MiB
    	Text:				20.50%	20.22	MiB
    
    Kernel Memory Map:				7.74	GiB
    	Size:				0.37%	29.06	MiB
    	Free:				99.63%	7.71	GiB
    								Page:  1
    ------------------------------------------------------------------------
    
    ARC Summary: (HEALTHY)
    	Storage pool Version:			28
    	Filesystem Version:			5
    	Memory Throttle Count:			0
    
    ARC Misc:
    	Deleted:				7
    	Recycle Misses:				0
    	Mutex Misses:				0
    	Evict Skips:				0
    
    ARC Size:				0.03%	1.39	MiB
    	Target Size: (Adaptive)		100.00%	5.00	GiB
    	Min Size (Hard Limit):		20.00%	1.00	GiB
    	Max Size (High Water):		5:1	5.00	GiB
    
    ARC Size Breakdown:
    	Recently Used Cache Size:	50.00%	2.50	GiB
    	Frequently Used Cache Size:	50.00%	2.50	GiB
    
    ARC Hash Breakdown:
    	Elements Max:				166
    	Elements Current:		100.00%	166
    	Collisions:				0
    	Chain Max:				0
    	Chains:					0
    								Page:  2
    ------------------------------------------------------------------------
    
    ARC Efficiency:					5.65k
    	Cache Hit Ratio:		96.92%	5.48k
    	Cache Miss Ratio:		3.08%	174
    	Actual Hit Ratio:		95.47%	5.40k
    
    	Data Demand Efficiency:		100.00%	0
    
    	CACHE HITS BY CACHE LIST:
    	  Anonymously Used:		1.50%	82
    	  Most Recently Used:		14.29%	783
    	  Most Frequently Used:		84.22%	4.62k
    	  Most Recently Used Ghost:	0.00%	0
    	  Most Frequently Used Ghost:	0.00%	0
    
    	CACHE HITS BY DATA TYPE:
    	  Demand Data:			0.00%	0
    	  Prefetch Data:		0.00%	0
    	  Demand Metadata:		98.39%	5.39k
    	  Prefetch Metadata:		1.61%	88
    
    	CACHE MISSES BY DATA TYPE:
    	  Demand Data:			0.00%	0
    	  Prefetch Data:		0.00%	0
    	  Demand Metadata:		51.72%	90
    	  Prefetch Metadata:		48.28%	84
    								Page:  3
    ------------------------------------------------------------------------
    
    File-Level Prefetch: (HEALTHY)
    
    DMU Efficiency:					6.25k
    	Hit Ratio:			99.81%	6.24k
    	Miss Ratio:			0.19%	12
    
    	Colinear:				12
    	  Hit Ratio:			0.00%	0
    	  Miss Ratio:			100.00%	12
    
    	Stride:					6.16k
    	  Hit Ratio:			100.00%	6.16k
    	  Miss Ratio:			0.00%	0
    
    DMU Misc:
    	Reclaim:				12
    	  Successes:			0.00%	0
    	  Failures:			100.00%	12
    
    	Streams:				82
    	  +Resets:			0.00%	0
    	  -Resets:			100.00%	82
    	  Bogus:				0
    								Page:  5
    ------------------------------------------------------------------------
    
    ZFS Tunable (sysctl):
    	kern.maxusers                           384
    	vm.kmem_size                            8305745920
    	vm.kmem_size_scale                      1
    	vm.kmem_size_min                        0
    	vm.kmem_size_max                        329853485875
    	vfs.zfs.l2c_only_size                   0
    	vfs.zfs.mfu_ghost_data_lsize            0
    	vfs.zfs.mfu_ghost_metadata_lsize        0
    	vfs.zfs.mfu_ghost_size                  0
    	vfs.zfs.mfu_data_lsize                  0
    	vfs.zfs.mfu_metadata_lsize              168960
    	vfs.zfs.mfu_size                        175104
    	vfs.zfs.mru_ghost_data_lsize            0
    	vfs.zfs.mru_ghost_metadata_lsize        4096
    	vfs.zfs.mru_ghost_size                  4096
    	vfs.zfs.mru_data_lsize                  0
    	vfs.zfs.mru_metadata_lsize              535040
    	vfs.zfs.mru_size                        1063424
    	vfs.zfs.anon_data_lsize                 0
    	vfs.zfs.anon_metadata_lsize             0
    	vfs.zfs.anon_size                       0
    	vfs.zfs.l2arc_norw                      1
    	vfs.zfs.l2arc_feed_again                1
    	vfs.zfs.l2arc_noprefetch                1
    	vfs.zfs.l2arc_feed_min_ms               200
    	vfs.zfs.l2arc_feed_secs                 1
    	vfs.zfs.l2arc_headroom                  2
    	vfs.zfs.l2arc_write_boost               8388608
    	vfs.zfs.l2arc_write_max                 8388608
    	vfs.zfs.arc_meta_limit                  1342177280
    	vfs.zfs.arc_meta_used                   1452568
    	vfs.zfs.arc_min                         1073741824
    	vfs.zfs.arc_max                         5368709120
    	vfs.zfs.dedup.prefetch                  1
    	vfs.zfs.mdcomp_disable                  0
    	vfs.zfs.write_limit_override            0
    	vfs.zfs.write_limit_inflated            25698410496
    	vfs.zfs.write_limit_max                 1070767104
    	vfs.zfs.write_limit_min                 33554432
    	vfs.zfs.write_limit_shift               3
    	vfs.zfs.no_write_throttle               0
    	vfs.zfs.zfetch.array_rd_sz              1048576
    	vfs.zfs.zfetch.block_cap                256
    	vfs.zfs.zfetch.min_sec_reap             2
    	vfs.zfs.zfetch.max_streams              8
    	vfs.zfs.prefetch_disable                0
    	vfs.zfs.mg_alloc_failures               8
    	vfs.zfs.check_hostid                    1
    	vfs.zfs.recover                         0
    	vfs.zfs.txg.synctime_ms                 1000
    	vfs.zfs.txg.timeout                     5
    	vfs.zfs.scrub_limit                     10
    	vfs.zfs.vdev.cache.bshift               16
    	vfs.zfs.vdev.cache.size                 0
    	vfs.zfs.vdev.cache.max                  16384
    	vfs.zfs.vdev.write_gap_limit            4096
    	vfs.zfs.vdev.read_gap_limit             32768
    	vfs.zfs.vdev.aggregation_limit          131072
    	vfs.zfs.vdev.ramp_rate                  2
    	vfs.zfs.vdev.time_shift                 6
    	vfs.zfs.vdev.min_pending                4
    	vfs.zfs.vdev.max_pending                10
    	vfs.zfs.vdev.bio_flush_disable          0
    	vfs.zfs.cache_flush_disable             0
    	vfs.zfs.zil_replay_disable              0
    	vfs.zfs.zio.use_uma                     0
    	vfs.zfs.version.zpl                     5
    	vfs.zfs.version.spa                     28
    	vfs.zfs.version.acl                     1
    	vfs.zfs.debug                           0
    	vfs.zfs.super_owner                     0
    								Page:  7
    ------------------------------------------------------------------------
    
     
  5. Leander

    Leander Member

    Messages:
    180
    Thanks Received:
    2
    ... Output of arc_summary.pl right after I started cp to copy the content of a 2TB HDD over to my new ZFS:

    Code:
    ------------------------------------------------------------------------
    ZFS Subsystem Report                            Tue May  1 18:08:44 2012
    ------------------------------------------------------------------------
    
    System Memory:
    
            1.67%   132.02  MiB Active,     38.03%  2.94    GiB Inact
            36.19%  2.80    GiB Wired,      0.00%   296.00  KiB Cache
            24.10%  1.86    GiB Free,       0.01%   644.00  KiB Gap
    
            Real Installed:                         8.00    GiB
            Real Available:                 99.72%  7.98    GiB
            Real Managed:                   96.96%  7.74    GiB
    
            Logical Total:                          8.00    GiB
            Logical Used:                   39.92%  3.19    GiB
            Logical Free:                   60.08%  4.81    GiB
    
    Kernel Memory:                                  1.86    GiB
            Data:                           98.94%  1.84    GiB
            Text:                           1.06%   20.22   MiB
    
    Kernel Memory Map:                              7.73    GiB
            Size:                           23.01%  1.78    GiB
            Free:                           76.99%  5.95    GiB
                                                                    Page:  1
    ------------------------------------------------------------------------
    
    ARC Summary: (HEALTHY)
            Storage pool Version:                   28
            Filesystem Version:                     5
            Memory Throttle Count:                  0
    
    ARC Misc:
            Deleted:                                7
            Recycle Misses:                         0
            Mutex Misses:                           0
            Evict Skips:                            0
    
    ARC Size:                               35.18%  1.76    GiB
            Target Size: (Adaptive)         100.00% 5.00    GiB
            Min Size (Hard Limit):          20.00%  1.00    GiB
            Max Size (High Water):          5:1     5.00    GiB
    
    ARC Size Breakdown:
            Recently Used Cache Size:       50.00%  2.50    GiB
            Frequently Used Cache Size:     50.00%  2.50    GiB
    
    ARC Hash Breakdown:
            Elements Max:                           18.11k
            Elements Current:               100.00% 18.11k
            Collisions:                             3.89k
            Chain Max:                              3
            Chains:                                 1.14k
                                                                    Page:  2
    ------------------------------------------------------------------------
    
    ARC Efficiency:                                 998.22k
            Cache Hit Ratio:                86.90%  867.43k
            Cache Miss Ratio:               13.10%  130.79k
            Actual Hit Ratio:               62.03%  619.25k
    
            Data Demand Efficiency:         99.97%  6.71k
    
            CACHE HITS BY CACHE LIST:
              Anonymously Used:             28.61%  248.19k
              Most Recently Used:           54.80%  475.31k
              Most Frequently Used:         16.59%  143.93k
              Most Recently Used Ghost:     0.00%   0
              Most Frequently Used Ghost:   0.00%   0
    
            CACHE HITS BY DATA TYPE:
              Demand Data:                  0.77%   6.70k
              Prefetch Data:                0.00%   0
              Demand Metadata:              70.61%  612.53k
              Prefetch Metadata:            28.61%  248.20k
    
            CACHE MISSES BY DATA TYPE:
              Demand Data:                  0.00%   2
              Prefetch Data:                0.00%   0
              Demand Metadata:              95.87%  125.39k
              Prefetch Metadata:            4.12%   5.39k
                                                                    Page:  3
    ------------------------------------------------------------------------
    
    File-Level Prefetch: (HEALTHY)
    
    DMU Efficiency:                                 3.62m
            Hit Ratio:                      81.64%  2.95m
            Miss Ratio:                     18.36%  664.16k
    
            Colinear:                               664.16k
              Hit Ratio:                    0.03%   195
              Miss Ratio:                   99.97%  663.97k
    
            Stride:                                 2.93m
              Hit Ratio:                    100.00% 2.93m
              Miss Ratio:                   0.00%   0
    
    DMU Misc:
            Reclaim:                                663.97k
              Successes:                    0.15%   1.00k
              Failures:                     99.85%  662.97k
    
            Streams:                                22.89k
              +Resets:                      0.00%   0
              -Resets:                      100.00% 22.89k
              Bogus:                                0
                                                                    Page:  5
    ------------------------------------------------------------------------
    
    ZFS Tunable (sysctl):
            kern.maxusers                           384
            vm.kmem_size                            8305745920
            vm.kmem_size_scale                      1
            vm.kmem_size_min                        0
            vm.kmem_size_max                        329853485875
            vfs.zfs.l2c_only_size                   0
            vfs.zfs.mfu_ghost_data_lsize            0
            vfs.zfs.mfu_ghost_metadata_lsize        0
            vfs.zfs.mfu_ghost_size                  0
            vfs.zfs.mfu_data_lsize                  59343360
            vfs.zfs.mfu_metadata_lsize              173056
            vfs.zfs.mfu_size                        59522560
            vfs.zfs.mru_ghost_data_lsize            0
            vfs.zfs.mru_ghost_metadata_lsize        2449408
            vfs.zfs.mru_ghost_size                  2449408
            vfs.zfs.mru_data_lsize                  1709782528
            vfs.zfs.mru_metadata_lsize              8559616
            vfs.zfs.mru_size                        1722849792
            vfs.zfs.anon_data_lsize                 0
            vfs.zfs.anon_metadata_lsize             0
            vfs.zfs.anon_size                       95635456
            vfs.zfs.l2arc_norw                      1
            vfs.zfs.l2arc_feed_again                1
            vfs.zfs.l2arc_noprefetch                1
            vfs.zfs.l2arc_feed_min_ms               200
            vfs.zfs.l2arc_feed_secs                 1
            vfs.zfs.l2arc_headroom                  2
            vfs.zfs.l2arc_write_boost               8388608
            vfs.zfs.l2arc_write_max                 8388608
            vfs.zfs.arc_meta_limit                  1342177280
            vfs.zfs.arc_meta_used                   27272600
            vfs.zfs.arc_min                         1073741824
            vfs.zfs.arc_max                         5368709120
            vfs.zfs.dedup.prefetch                  1
            vfs.zfs.mdcomp_disable                  0
            vfs.zfs.write_limit_override            0
            vfs.zfs.write_limit_inflated            25698410496
            vfs.zfs.write_limit_max                 1070767104
            vfs.zfs.write_limit_min                 33554432
            vfs.zfs.write_limit_shift               3
            vfs.zfs.no_write_throttle               0
            vfs.zfs.zfetch.array_rd_sz              1048576
            vfs.zfs.zfetch.block_cap                256
            vfs.zfs.zfetch.min_sec_reap             2
            vfs.zfs.zfetch.max_streams              8
            vfs.zfs.prefetch_disable                0
            vfs.zfs.mg_alloc_failures               8
            vfs.zfs.check_hostid                    1
            vfs.zfs.recover                         0
            vfs.zfs.txg.synctime_ms                 1000
            vfs.zfs.txg.timeout                     5
            vfs.zfs.scrub_limit                     10
            vfs.zfs.vdev.cache.bshift               16
            vfs.zfs.vdev.cache.size                 0
            vfs.zfs.vdev.cache.max                  16384
            vfs.zfs.vdev.write_gap_limit            4096
            vfs.zfs.vdev.read_gap_limit             32768
            vfs.zfs.vdev.aggregation_limit          131072
            vfs.zfs.vdev.ramp_rate                  2
            vfs.zfs.vdev.time_shift                 6
            vfs.zfs.vdev.min_pending                4
            vfs.zfs.vdev.max_pending                10
            vfs.zfs.vdev.bio_flush_disable          0
            vfs.zfs.cache_flush_disable             0
            vfs.zfs.zil_replay_disable              0
            vfs.zfs.zio.use_uma                     0
            vfs.zfs.version.zpl                     5
            vfs.zfs.version.spa                     28
            vfs.zfs.version.acl                     1
            vfs.zfs.debug                           0
            vfs.zfs.super_owner                     0
                                                                    Page:  7
    ------------------------------------------------------------------------
     
  6. Leander

    Leander Member

    Messages:
    180
    Thanks Received:
    2
    And eventually my output of arc_summary.pl couple of hours later - still the same action - writing random files without - interruption of the copy process:
    Code:
    ------------------------------------------------------------------------
    ZFS Subsystem Report                            Tue May  1 21:05:33 2012
    ------------------------------------------------------------------------
    
    System Memory:
    
            3.57%   282.61  MiB Active,     64.30%  4.97    GiB Inact
            27.30%  2.11    GiB Wired,      4.67%   370.14  MiB Cache
            0.15%   11.93   MiB Free,       0.01%   628.00  KiB Gap
    
            Real Installed:                         8.00    GiB
            Real Available:                 99.72%  7.98    GiB
            Real Managed:                   96.96%  7.74    GiB
    
            Logical Total:                          8.00    GiB
            Logical Used:                   33.17%  2.65    GiB
            Logical Free:                   66.83%  5.35    GiB
    
    Kernel Memory:                                  1.08    GiB
            Data:                           98.17%  1.06    GiB
            Text:                           1.83%   20.32   MiB
    
    Kernel Memory Map:                              7.19    GiB
            Size:                           13.76%  1013.16 MiB
            Free:                           86.24%  6.20    GiB
                                                                    Page:  1
    ------------------------------------------------------------------------
    
    ARC Summary: (THROTTLED)
            Storage pool Version:                   28
            Filesystem Version:                     5
            Memory Throttle Count:                  3.96k
    
    ARC Misc:
            Deleted:                                1.68m
            Recycle Misses:                         72.33k
            Mutex Misses:                           7
            Evict Skips:                            7
    
    ARC Size:                               20.00%  1.00    GiB
            Target Size: (Adaptive)         20.00%  1.00    GiB
            Min Size (Hard Limit):          20.00%  1.00    GiB
            Max Size (High Water):          5:1     5.00    GiB
    
    ARC Size Breakdown:
            Recently Used Cache Size:       93.75%  960.06  MiB
            Frequently Used Cache Size:     6.25%   63.97   MiB
    
    ARC Hash Breakdown:
            Elements Max:                           32.70k
            Elements Current:               71.77%  23.46k
            Collisions:                             302.87k
            Chain Max:                              5
            Chains:                                 2.26k
                                                                    Page:  2
    ------------------------------------------------------------------------
    
    ARC Efficiency:                                 2.41m
            Cache Hit Ratio:                93.64%  2.25m
            Cache Miss Ratio:               6.36%   152.99k
            Actual Hit Ratio:               81.53%  1.96m
    
            Data Demand Efficiency:         99.98%  1.19m
            Data Prefetch Efficiency:       61.05%  42.68k
    
            CACHE HITS BY CACHE LIST:
              Anonymously Used:             12.70%  286.05k
              Most Recently Used:           30.11%  678.21k
              Most Frequently Used:         56.96%  1.28m
              Most Recently Used Ghost:     0.09%   2.02k
              Most Frequently Used Ghost:   0.14%   3.13k
    
            CACHE HITS BY DATA TYPE:
              Demand Data:                  52.67%  1.19m
              Prefetch Data:                1.16%   26.06k
              Demand Metadata:              34.10%  768.23k
              Prefetch Metadata:            12.07%  271.92k
    
            CACHE MISSES BY DATA TYPE:
              Demand Data:                  0.12%   184
              Prefetch Data:                10.87%  16.62k
              Demand Metadata:              83.51%  127.77k
              Prefetch Metadata:            5.50%   8.42k
                                                                    Page:  3
    ------------------------------------------------------------------------
    
    File-Level Prefetch: (HEALTHY)
    
    DMU Efficiency:                                 55.20m
            Hit Ratio:                      98.80%  54.53m
            Miss Ratio:                     1.20%   664.34k
    
            Colinear:                               664.34k
              Hit Ratio:                    0.03%   196
              Miss Ratio:                   99.97%  664.14k
    
            Stride:                                 54.49m
              Hit Ratio:                    100.00% 54.49m
              Miss Ratio:                   0.00%   114
    
    DMU Misc:
            Reclaim:                                664.14k
              Successes:                    0.17%   1.12k
              Failures:                     99.83%  663.03k
    
            Streams:                                41.08k
              +Resets:                      0.10%   43
              -Resets:                      99.90%  41.04k
              Bogus:                                0
                                                                    Page:  5
    ------------------------------------------------------------------------
    
    ZFS Tunable (sysctl):
            kern.maxusers                           384
            vm.kmem_size                            8305745920
            vm.kmem_size_scale                      1
            vm.kmem_size_min                        0
            vm.kmem_size_max                        329853485875
            vfs.zfs.l2c_only_size                   0
            vfs.zfs.mfu_ghost_data_lsize            981729280
            vfs.zfs.mfu_ghost_metadata_lsize        23986688
            vfs.zfs.mfu_ghost_size                  1005715968
            vfs.zfs.mfu_data_lsize                  0
            vfs.zfs.mfu_metadata_lsize              9241600
            vfs.zfs.mfu_size                        11002880
            vfs.zfs.mru_ghost_data_lsize            0
            vfs.zfs.mru_ghost_metadata_lsize        67137536
            vfs.zfs.mru_ghost_size                  67137536
            vfs.zfs.mru_data_lsize                  932244480
            vfs.zfs.mru_metadata_lsize              9625600
            vfs.zfs.mru_size                        965317632
            vfs.zfs.anon_data_lsize                 0
            vfs.zfs.anon_metadata_lsize             0
            vfs.zfs.anon_size                       41498112
            vfs.zfs.l2arc_norw                      1
            vfs.zfs.l2arc_feed_again                1
            vfs.zfs.l2arc_noprefetch                1
            vfs.zfs.l2arc_feed_min_ms               200
            vfs.zfs.l2arc_feed_secs                 1
            vfs.zfs.l2arc_headroom                  2
            vfs.zfs.l2arc_write_boost               8388608
            vfs.zfs.l2arc_write_max                 8388608
            vfs.zfs.arc_meta_limit                  1342177280
            vfs.zfs.arc_meta_used                   100359352
            vfs.zfs.arc_min                         1073741824
            vfs.zfs.arc_max                         5368709120
            vfs.zfs.dedup.prefetch                  1
            vfs.zfs.mdcomp_disable                  0
            vfs.zfs.write_limit_override            0
            vfs.zfs.write_limit_inflated            25698410496
            vfs.zfs.write_limit_max                 1070767104
            vfs.zfs.write_limit_min                 33554432
            vfs.zfs.write_limit_shift               3
            vfs.zfs.no_write_throttle               0
            vfs.zfs.zfetch.array_rd_sz              1048576
            vfs.zfs.zfetch.block_cap                256
            vfs.zfs.zfetch.min_sec_reap             2
            vfs.zfs.zfetch.max_streams              8
            vfs.zfs.prefetch_disable                0
            vfs.zfs.mg_alloc_failures               8
            vfs.zfs.check_hostid                    1
            vfs.zfs.recover                         0
            vfs.zfs.txg.synctime_ms                 1000
            vfs.zfs.txg.timeout                     5
            vfs.zfs.scrub_limit                     10
            vfs.zfs.vdev.cache.bshift               16
            vfs.zfs.vdev.cache.size                 0
            vfs.zfs.vdev.cache.max                  16384
            vfs.zfs.vdev.write_gap_limit            4096
            vfs.zfs.vdev.read_gap_limit             32768
            vfs.zfs.vdev.aggregation_limit          131072
            vfs.zfs.vdev.ramp_rate                  2
            vfs.zfs.vdev.time_shift                 6
            vfs.zfs.vdev.min_pending                4
            vfs.zfs.vdev.max_pending                10
            vfs.zfs.vdev.bio_flush_disable          0
            vfs.zfs.cache_flush_disable             0
            vfs.zfs.zil_replay_disable              0
            vfs.zfs.zio.use_uma                     0
            vfs.zfs.version.zpl                     5
            vfs.zfs.version.spa                     28
            vfs.zfs.version.acl                     1
            vfs.zfs.debug                           0
            vfs.zfs.super_owner                     0
                                                                    Page:  7
    ------------------------------------------------------------------------



    Note that
    Code:
    Kernel Memory:                                  1.08    GiB
            Data:                           98.17%  1.06    GiB
            Text:                           1.83%   20.32   MiB
    


    Kept pretty much static on those values, it was only in the very beginning that I've had such high values
    Code:
    Kernel Memory:                                  1.86    GiB
            Data:                           98.94%  1.84    GiB
            Text:                           1.06%   20.22   MiB
    
     
  7. Leander

    Leander Member

    Messages:
    180
    Thanks Received:
    2
    Btw. I have only one RAIDz1 running (currently resilvering for testing purposes):
    ... The md[n] devices are only for temporary testing. But sill, each of them has its own source file on a seperate HDD.

    Code:
    zpool status -v
      pool: zStar
     state: DEGRADED
    status: One or more devices is currently being resilvered.  The pool will
            continue to function, possibly in a degraded state.
    action: Wait for the resilver to complete.
     scan: resilver in progress since Tue May  1 20:54:40 2012
        68,0G scanned out of 214G at 33,5M/s, 1h14m to go
        7,56G resilvered, 31,75% done
    config:
    
            NAME                       STATE     READ WRITE CKSUM
            zStar                      DEGRADED     0     0     0
              raidz1-0                 DEGRADED     0     0     0
                md3                    ONLINE       0     0     0
                md4                    ONLINE       0     0     0
                md5                    ONLINE       0     0     0
                md6                    ONLINE       0     0     0
                md7                    ONLINE       0     0     0
                md8                    ONLINE       0     0     0
                replacing-5            UNAVAIL      0     0     0
                  6048967463963296996  UNAVAIL      0     0     0  was /dev/md8/old
                  md8                  ONLINE       0     0     0  (resilvering)
                md9                    ONLINE       0     0     0
                ada3p1                 ONLINE       0     0     0
                ada8p1                 ONLINE       0     0     0
    
    errors: No known data errors
    



    Thanks for hints
     
  8. kpa

    kpa Well-Known Member

    Messages:
    4,106
    Thanks Received:
    812
    Start with a very conservative setting of vfs.zfs.arc_max, let's say 1GB and up it in 1GB increments. Leave other settings to their defaults. This way you can very quickly tell if the crash is really from memory exhaustion caused by the ARC cache or something else.
     
  9. gkontos

    gkontos Active Member

    Messages:
    1,395
    Thanks Received:
    246
    Start by limiting the size of ARC_MAX and disabling PREFETCH. That should be enough.

    Code:
    vfs.zfs.prefetch_disable=1
    vfs.zfs.arc_max="4096M"


    Don't tweak any other values before you really know what you are doing.

    Also please post the output of:

    # zfs get all zStar
    # zdb -DD zStar

    Regards
     
  10. usdmatt

    usdmatt Member

    Messages:
    573
    Thanks Received:
    94
    Not going to get involved in zfs tuning discussion as I have no expertise in it yet but just noting that there is a port of arc_summary.pl under sysutils/zfs-stats.
     
  11. t1066

    t1066 Member

    Messages:
    174
    Thanks Received:
    34
    How do you set up your system? Where doea\s root reside and where do you put the swap file?
     
  12. gkontos

    gkontos Active Member

    Messages:
    1,395
    Thanks Received:
    246
    I think that the statistics he posted were coming from that script.
     
  13. usdmatt

    usdmatt Member

    Messages:
    573
    Thanks Received:
    94
    Yes, I know. He is using arc_summary.pl downloaded via the FreeBSD wiki, as mentioned in his post. I was just making people aware that there is a port of this now, so rather that finding it on the net you can just type:

    pkg_add -r zfs-stats
     
  14. kpa

    kpa Well-Known Member

    Messages:
    4,106
    Thanks Received:
    812
    Please note that what I wrote above about the kmem tunables only applies to amd64, i386 is quite different because of limited address space and usually requires tuning of kmem settings for stable operation of ZFS.