vm.kmem_size_max: 329853485875

Y

yayix

On a server with 512GB RAM it appears that vm.kmem_size_max is not being auto-tuned to use >329853485875 (~307GB).

On this machine vm.kmem_size is equal to vm.kmem_size_max
Code: 
vm.kmem_size_max: 329853485875
vm.kmem_size: 329853485875

On a machine with 1GB of RAM, I have successfully set vm.kmem_size_max to 330GB and vm.kmem_size automatically adjusts to 1GB even if I manually set it in /boot/loader.conf.

But on the machine with 512GB of RAM it just resets. For the machine to boot, we need to go to the loader prompt and issue:

Code: 
OK set vm.kmem_size_max="300G"
OK boot

On all PCBSD (8,9) or FreeBSD (8.1,8.2,9) machines we have vm.kmem_size_max is always set to 329853485875.

How can I increase vm.kmem_size_max to use at least 500GB? And how is 329853485875 determined (formula)? I need to increase vm.kmem_size_max and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say 490GB.

TIA!
 
yayix said:
But on the machine with 512GB of RAM it just resets. For the machine to boot, we need to go to the loader prompt and issue:

Code: 
OK set vm.kmem_size_max="300G"
OK boot
Click to expand...

Sorry, forgot to state that the machine resets during boot because I have set this:

/boot/loader.conf
Code: 
vm.kmem_size_max="500G"
 
zfs-stats -a (part 1)

[CMD=""]zfs-stats -a[/CMD]
Code: 
------------------------------------------------------------------------
ZFS Subsystem Report                            Wed Aug 15 09:08:28 2012
------------------------------------------------------------------------

System Information:

        Kernel Version:                         900044 (osreldate)
        Hardware Platform:                      amd64
        Processor Architecture:                 amd64

        ZFS Storage pool Version:               28
        ZFS Filesystem Version:                 5

FreeBSD 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011 root
 9:08AM  up 15:29, 2 users, load averages: 1.89, 1.75, 1.67

------------------------------------------------------------------------

System Memory:

        0.06%   292.86  MiB Active,     0.02%   111.19  MiB Inact
        47.21%  234.50  GiB Wired,      0.00%   896.00  KiB Cache
        51.07%  253.66  GiB Free,       1.65%   8.18    GiB Gap

        Real Installed:                         512.00  GiB
        Real Available:                 99.96%  511.82  GiB
        Real Managed:                   97.05%  496.73  GiB

        Logical Total:                          512.00  GiB
        Logical Used:                   50.43%  258.23  GiB
        Logical Free:                   49.57%  253.77  GiB

Kernel Memory:                                  227.74  GiB
        Data:                           99.99%  227.72  GiB
        Text:                           0.01%   21.64   MiB

Kernel Memory Map:                              304.23  GiB
        Size:                           74.81%  227.60  GiB
        Free:                           25.19%  76.63   GiB

------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                28.21m
        Recycle Misses:                         39.49k
        Mutex Misses:                           366.36k
        Evict Skips:                            677.56k

ARC Size:                               74.47%  228.03  GiB
        Target Size: (Adaptive)         74.47%  228.03  GiB
        Min Size (Hard Limit):          50.00%  153.10  GiB
        Max Size (High Water):          2:1     306.20  GiB

ARC Size Breakdown:
        Recently Used Cache Size:       93.48%  213.17  GiB
        Frequently Used Cache Size:     6.52%   14.86   GiB

ARC Hash Breakdown:
        Elements Max:                           5.64m
        Elements Current:               100.00% 5.64m
        Collisions:                             14.49m
        Chain Max:                              8
        Chains:                                 1.24m

------------------------------------------------------------------------

ARC Efficiency:                                 259.31m
        Cache Hit Ratio:                92.53%  239.93m
        Cache Miss Ratio:               7.47%   19.37m
        Actual Hit Ratio:               92.40%  239.59m

        Data Demand Efficiency:         99.97%  220.93m
        Data Prefetch Efficiency:       1.45%   19.40m

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           28.30%  67.90m
          Most Frequently Used:         71.55%  171.68m
          Most Recently Used Ghost:     0.13%   315.89k
          Most Frequently Used Ghost:   0.28%   672.06k

        CACHE HITS BY DATA TYPE:
          Demand Data:                  92.05%  220.85m
          Prefetch Data:                0.12%   282.17k
          Demand Metadata:              7.81%   18.74m
          Prefetch Metadata:            0.03%   62.46k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  0.37%   71.62k
          Prefetch Data:                98.70%  19.12m
          Demand Metadata:              0.88%   170.76k
          Prefetch Metadata:            0.04%   8.61k

------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        1.90m
        Tried Lock Failures:                    272.32k
        IO In Progress:                         5
        Low Memory Aborts:                      0
        Free on Write:                          37.71k
        Writes While Full:                      30.41k
        R/W Clashes:                            140
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           461.17m

L2 ARC Size: (Adaptive)                         231.94  GiB
        Header Size:                    0.18%   416.63  MiB

L2 ARC Breakdown:                               19.37m
        Hit Ratio:                      0.40%   77.99k
        Miss Ratio:                     99.60%  19.29m
        Feeds:                                  80.72k

L2 ARC Buffer:
        Bytes Scanned:                          36.73   TiB
        Buffer Iterations:                      80.72k
        List Iterations:                        4.56m
        NULL List Iterations:                   391.60k

L2 ARC Writes:
        Writes Sent:                    100.00% 45.50k

------------------------------------------------------------------------
 
zfs-stats -a (part 2)

[CMD=""]zfs-stats -a[/CMD]

Code: 
File-Level Prefetch: (HEALTHY)

DMU Efficiency:                                 952.47m
        Hit Ratio:                      99.82%  950.71m
        Miss Ratio:                     0.18%   1.76m

        Colinear:                               1.76m
          Hit Ratio:                    0.01%   241
          Miss Ratio:                   99.99%  1.76m

        Stride:                                 929.86m
          Hit Ratio:                    100.00% 929.85m
          Miss Ratio:                   0.00%   2.24k

DMU Misc:
        Reclaim:                                1.76m
          Successes:                    0.63%   11.08k
          Failures:                     99.37%  1.75m

        Streams:                                20.86m
          +Resets:                      0.00%   440
          -Resets:                      100.00% 20.86m
          Bogus:                                0

------------------------------------------------------------------------

VDEV cache is disabled

------------------------------------------------------------------------

ZFS Tunables (sysctl):
        kern.maxusers                           384
        vm.kmem_size                            329853485875
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        329853485875
        vfs.zfs.l2c_only_size                   242597267456
        vfs.zfs.mfu_ghost_data_lsize            64166322176
        vfs.zfs.mfu_ghost_metadata_lsize        645829632
        vfs.zfs.mfu_ghost_size                  64812151808
        vfs.zfs.mfu_data_lsize                  176958659584
        vfs.zfs.mfu_metadata_lsize              31165952
        vfs.zfs.mfu_size                        177167786496
        vfs.zfs.mru_ghost_data_lsize            174980341760
        vfs.zfs.mru_ghost_metadata_lsize        336025600
        vfs.zfs.mru_ghost_size                  175316367360
        vfs.zfs.mru_data_lsize                  65570037760
        vfs.zfs.mru_metadata_lsize              101715968
        vfs.zfs.mru_size                        65818363904
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       24707072
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  82194936012
        vfs.zfs.arc_meta_used                   2292602848
        vfs.zfs.arc_min                         164389872024
        vfs.zfs.arc_max                         328779744048
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.write_limit_override            0
        vfs.zfs.write_limit_inflated            1648684572672
        vfs.zfs.write_limit_max                 68695190528
        vfs.zfs.write_limit_min                 33554432
        vfs.zfs.write_limit_shift               3
        vfs.zfs.no_write_throttle               0
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                0
        vfs.zfs.mg_alloc_failures               96
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.txg.synctime_ms                 1000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.scrub_limit                     10
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.ramp_rate                  2
        vfs.zfs.vdev.time_shift                 6
        vfs.zfs.vdev.min_pending                4
        vfs.zfs.vdev.max_pending                10
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     28
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0

------------------------------------------------------------------------
 
yayix said:
How can I increase vm.kmem_size_max to use at least 500GB? And how is 329853485875 determined (formula)? I need to increase vm.kmem_size_max and vm.kmem_size so I can set vfs.zfs.arc_max (ZFS ARC) to use say 490GB.

TIA!
Click to expand...

Why would you want to do that?
 
gkontos said:
Why would you want to do that?
Click to expand...

This machine will be mainly used for postgreSQL. We have a huge db (3TB and growing) and we need to have as much of it as we can on zfs' ARC. I don't think there's any reason to not want to utilize all RAM and leave just enough for other stuff. Moreso, on our 8.2 and 9 machines vm.kmem_size is always auto-tuned to almost the same size as our installed RAM. What I've tuned on those machines is lower vfs.zfs.arc_max to ~75% of vm.kmem_size and that have worked well for us and the machines does not swap out.

By the way, we were able to successfully bump vm.kmem_size_max and vm.kmem_size to 400GB. Tried 450GB and 500GB, server boots but it spewed out garbage during the boot up process.

[CMD=""]zfs-stats -a[/CMD]

Code: 
------------------------------------------------------------------------
ZFS Subsystem Report                            Wed Aug 15 12:46:22 2012
------------------------------------------------------------------------

System Information:

        Kernel Version:                         900044 (osreldate)
        Hardware Platform:                      amd64
        Processor Architecture:                 amd64

        ZFS Storage pool Version:               28
        ZFS Filesystem Version:                 5

FreeBSD 9.0-RELEASE #3: Tue Dec 27 14:14:29 PST 2011 root
12:46PM  up 25 mins, 2 users, load averages: 0.07, 0.03, 0.12

------------------------------------------------------------------------

System Memory:

        0.02%   94.02   MiB Active,     0.01%   56.74   MiB Inact
        1.00%   4.98    GiB Wired,      0.00%   792.00  KiB Cache
        98.85%  491.03  GiB Free,       0.12%   589.25  MiB Gap

        Real Installed:                         512.00  GiB
        Real Available:                 99.96%  511.82  GiB
        Real Managed:                   97.05%  496.73  GiB

        Logical Total:                          512.00  GiB
        Logical Used:                   4.08%   20.91   GiB
        Logical Free:                   95.92%  491.09  GiB

Kernel Memory:                                  1.72    GiB
        Data:                           98.77%  1.70    GiB
        Text:                           1.23%   21.64   MiB

Kernel Memory Map:                              400.00  GiB
        Size:                           0.42%   1.69    GiB
        Free:                           99.58%  398.31  GiB

------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                27
        Recycle Misses:                         0
        Mutex Misses:                           0
        Evict Skips:                            275

ARC Size:                               0.10%   427.15  MiB
        Target Size: (Adaptive)         100.00% 399.00  GiB
        Min Size (Hard Limit):          12.50%  49.88   GiB
        Max Size (High Water):          8:1     399.00  GiB

ARC Size Breakdown:
        Recently Used Cache Size:       50.00%  199.50  GiB
        Frequently Used Cache Size:     50.00%  199.50  GiB

ARC Hash Breakdown:
        Elements Max:                           4.05k
        Elements Current:               99.95%  4.05k
        Collisions:                             0
        Chain Max:                              0
        Chains:                                 0

------------------------------------------------------------------------

ARC Efficiency:                                 69.58k
        Cache Hit Ratio:                94.18%  65.53k
        Cache Miss Ratio:               5.82%   4.05k
        Actual Hit Ratio:               94.02%  65.42k

        Data Demand Efficiency:         99.87%  52.76k
        Data Prefetch Efficiency:       0.00%   3.29k

        CACHE HITS BY CACHE LIST:
          Anonymously Used:             0.17%   113
          Most Recently Used:           15.93%  10.44k
          Most Frequently Used:         83.89%  54.98k
          Most Recently Used Ghost:     0.00%   0
          Most Frequently Used Ghost:   0.00%   0

        CACHE HITS BY DATA TYPE:
          Demand Data:                  80.41%  52.69k
          Prefetch Data:                0.00%   0
          Demand Metadata:              19.41%  12.72k
          Prefetch Metadata:            0.18%   116

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  1.63%   66
          Prefetch Data:                81.20%  3.29k
          Demand Metadata:              12.23%  495
          Prefetch Metadata:            4.94%   200

------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        0
        Tried Lock Failures:                    11
        IO In Progress:                         0
        Low Memory Aborts:                      0
        Free on Write:                          0
        Writes While Full:                      25
        R/W Clashes:                            0
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           327.45k

L2 ARC Size: (Adaptive)                         417.07  MiB
        Header Size:                    0.00%   0

L2 ARC Breakdown:                               4.03k
        Hit Ratio:                      0.00%   0
        Miss Ratio:                     100.00% 4.03k
        Feeds:                                  1.48k

L2 ARC Buffer:
        Bytes Scanned:                          29.98   GiB
        Buffer Iterations:                      1.48k
        List Iterations:                        94.33k
        NULL List Iterations:                   44.80k

L2 ARC Writes:
        Writes Sent:                    100.00% 47

continued on next post..
 
continuation zfs-stats -a with vm.kmem_size_max and vm.kmem_size set to "400G"

Continued from previous post. With vm.kmem_size_max and vm.kmem_size set to "400G".

Code: 
------------------------------------------------------------------------

File-Level Prefetch: (HEALTHY)

DMU Efficiency:                                 121.02k
        Hit Ratio:                      99.67%  120.61k
        Miss Ratio:                     0.33%   404

        Colinear:                               404
          Hit Ratio:                    0.25%   1
          Miss Ratio:                   99.75%  403

        Stride:                                 117.26k
          Hit Ratio:                    100.00% 117.26k
          Miss Ratio:                   0.00%   0

DMU Misc:
        Reclaim:                                403
          Successes:                    4.71%   19
          Failures:                     95.29%  384

        Streams:                                3.36k
          +Resets:                      0.09%   3
          -Resets:                      99.91%  3.36k
          Bogus:                                0

------------------------------------------------------------------------

VDEV cache is disabled

------------------------------------------------------------------------

ZFS Tunables (sysctl):
        kern.maxusers                           384
        vm.kmem_size                            429496729600
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        429496729600
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_lsize            9216
        vfs.zfs.mfu_ghost_metadata_lsize        0
        vfs.zfs.mfu_ghost_size                  9216
        vfs.zfs.mfu_data_lsize                  424943104
        vfs.zfs.mfu_metadata_lsize              365568
        vfs.zfs.mfu_size                        425761280
        vfs.zfs.mru_ghost_data_lsize            0
        vfs.zfs.mru_ghost_metadata_lsize        114688
        vfs.zfs.mru_ghost_size                  114688
        vfs.zfs.mru_data_lsize                  11144704
        vfs.zfs.mru_metadata_lsize              4005376
        vfs.zfs.mru_size                        18676736
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       101888
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  107105746944
        vfs.zfs.arc_meta_used                   11810416
        vfs.zfs.arc_min                         53552873472
        vfs.zfs.arc_max                         428422987776
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.write_limit_override            0
        vfs.zfs.write_limit_inflated            1648684572672
        vfs.zfs.write_limit_max                 68695190528
        vfs.zfs.write_limit_min                 33554432
        vfs.zfs.write_limit_shift               3
        vfs.zfs.no_write_throttle               0
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                0
        vfs.zfs.mg_alloc_failures               96
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.txg.synctime_ms                 1000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.scrub_limit                     10
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.ramp_rate                  2
        vfs.zfs.vdev.time_shift                 6
        vfs.zfs.vdev.min_pending                4
        vfs.zfs.vdev.max_pending                10
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     28
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0

------------------------------------------------------------------------
 
I don't think that at this stage you need to change any of those values.
Your memory capacity is much higher for any type of ZFS tuning regarding arc and vm.kmem_size.
 
gkontos said:
I don't think that at this stage you need to change any of those values.
Your memory capacity is much higher for any type of ZFS tuning regarding arc and vm.kmem_size.
Click to expand...

Thanks!

Exactly why it needs to be tuned for our needs. Since FreeBSD imposes a limit on vm.kmem_size_max and have probably hard-coded it to ~307GB, we are blindly doing a trial and error on this setting. The machine is not yet in production but will be soon and once it is we won't have the luxury of doing a reboot for tuning as rebooting takes a while (~20mins).

Again bringing up the question of how 329853485875 (307GB) is computed? And what more if the installed RAM is 1TB?

From: Frequently Asked Questions for FreeBSD 7.X, 8.X, and 9.X [4.2 Memory]
4.2.1. Does FreeBSD support more than 4 GB of memory (RAM)? More than 16 GB? More than 48 GB?

Yes. FreeBSD as an operating system generally supports as much physical memory (RAM) as the platform it is running on does. Keep in mind that different platforms have different limits for memory; for example i386â„¢ without PAE supports at most 4 GB of memory (and usually less than that because of PCI address space) and i386 with PAE supports at most 64 GB memory. AMD64 platforms currently deployed support up to 1 TB of physical memory.
Click to expand...

Hope there are others who can shed some light on this?
 
I have this problem too, but if
Code: 
vfs.zfs.vdev.cache.size=16M
or more M - freebsd FreeBSD doesn't panic any more. (FreeBSD 9.2-STABLE + ZFS zpool)
 
You must log in or register to reply here.
Back
Top