FreeBSD 9 64bit with ZFS crashes every 3 days

belon_cfy · Mar 9, 2012

phoenix said:
Uhm, 32 MB of swap use is not "heavy swap usage".

Hi Phoenix,
Not "heavy usage", but it keeps swapping in and out continuously until server crashed.

Possible caused by the nfsd because I have increased the number of server to 128 instead of 4.

Code:

nfs_server_flags="-u -t -n 128"

But stopping nfsd won't reclaim back my memory

belon_cfy · Mar 13, 2012

Suspecting it might be due to insufficient memory for holding more than 3TB data with a storage contains 4GB Ram. According to the system requirement of Nexenta, they recommend at least 4-8GB ram for production server and additional 1GB ram is needed for every 1TB space added.

I have upgraded 1 of the box to 8GB ram and it survives for more than 4 days. Don't know whether will it crashes again in the next few days or weeks.

DutchDaemon · Mar 13, 2012

Let us know, so we can mark this [Solved] (or mark it as such yourself).

belon_cfy · Mar 14, 2012

DutchDaemon said:
Let us know, so we can mark this [Solved] (or mark it as such yourself).

Sure, I will update the status few days later.

belon_cfy · Mar 15, 2012

Code:

CPU:  0.4% user,  0.0% nice,  2.0% system,  0.2% interrupt, 97.5% idle
Mem: 4044K Active, 160K Inact, 7607M Wired, 19M Cache, 182M Free
Swap: 8192M Total, 42M Used, 8150M Free, 1304K Out

The server with 8GB RAM keeps swapping out continuously by today. Seems it is going down soon because of the same symptom is happening.

Moving swap vol from zvol to USB with freebsd-swap didn't solve the problem. My server still down with this swap configuration last time.

I have tried the following configuration but non of these can eliminate servers crashed.

Changing arc_max to below 1GB on 4GB servers.
Avoid swap partition on zvol
Increase kmem_size to 6GB on 4GB server.
Decrease kmem_size to 3GB on 4GB servers.

peetaur · Mar 15, 2012

Try setting vm.kmem_size_max to 1 or 2 GB below your physical memory. (or lower if you have memory hogging things in kernel space, like 10Gbps network drivers)

And please repost your zfs-status during the near-melt-down so we can compare numbers (eg. is arc usage higher than arc_max you set?)

belon_cfy · Mar 15, 2012

peetaur said:
Try setting vm.kmem_size_max to 1 or 2 GB below your physical memory. (or lower if you have memory hogging things in kernel space, like 10Gbps network drivers)

And please repost your zfs-status during the near-melt-down so we can compare numbers (eg. is arc usage higher than arc_max you set?)

Hi
The server with 2 x 1Gbps network port.

below is my last zfs-stats before it crashed.

Code:

 ------------------------------------------------------------------------
 ZFS Subsystem Report                           
 ------------------------------------------------------------------------

 System Information:

        Kernel Version:                         900044 (osreldate)
        Hardware Platform:                      amd64
        Processor Architecture:                 amd64

        ZFS Storage pool Version:               28
        ZFS Filesystem Version:                 5

 FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012 root
 11:39PM  up 3 days, 12:40, 0 users, load averages: 0.53, 0.54, 0.68

 ------------------------------------------------------------------------

 System Memory:

        0.10%   3.79    MiB Active,     0.01%   472.00  KiB Inact
        99.20%  3.81    GiB Wired,      0.15%   5.82    MiB Cache
        0.53%   20.75   MiB Free,       0.02%   808.00  KiB Gap

        Real Installed:                         4.00    GiB
        Real Available:                 99.49%  3.98    GiB
        Real Managed:                   96.49%  3.84    GiB

        Logical Total:                          4.00    GiB
        Logical Used:                   99.34%  3.97    GiB
        Logical Free:                   0.66%   27.03   MiB

 Kernel Memory:                                 396.07  MiB
        Data:                           94.99%  376.23  MiB
        Text:                           5.01%   19.84   MiB

 Kernel Memory Map:                             3.27    GiB
        Size:                           7.93%   265.52  MiB
        Free:                           92.07%  3.01    GiB

 ------------------------------------------------------------------------

 ARC Summary: (THROTTLED)
        Memory Throttle Count:                  4.79k

 ARC Misc:
        Deleted:                                34.77m
        Recycle Misses:                         5.20m
        Mutex Misses:                           241.28k
        Evict Skips:                            241.28k

 ARC Size:                              12.50%  363.44  MiB
        Target Size: (Adaptive)         12.50%  363.48  MiB
        Min Size (Hard Limit):          12.50%  363.48  MiB
        Max Size (High Water):          8:1     2.84    GiB

 ARC Size Breakdown:
        Recently Used Cache Size:       6.35%   23.08   MiB
        Frequently Used Cache Size:     93.65%  340.39  MiB

 ARC Hash Breakdown:
        Elements Max:                           332.55k
        Elements Current:               42.63%  141.77k
        Collisions:                             30.20m
        Chain Max:                              60
        Chains:                                 36.87k

 ------------------------------------------------------------------------

 ARC Efficiency:                                        357.24m
        Cache Hit Ratio:                91.29%  326.11m
        Cache Miss Ratio:               8.71%   31.13m
        Actual Hit Ratio:               91.20%  325.82m

        Data Demand Efficiency:         88.75%  10.72m
        Data Prefetch Efficiency:       0.00%   1.71m

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           54.05%  176.28m
          Most Frequently Used:         45.86%  149.54m
          Most Recently Used Ghost:     0.14%   457.14k
          Most Frequently Used Ghost:   1.73%   5.65m

        CACHE HITS BY DATA TYPE:
          Demand Data:                  2.92%   9.52m
          Prefetch Data:                0.00%   31
          Demand Metadata:              96.99%  316.30m
          Prefetch Metadata:            0.09%   289.97k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  3.88%   1.21m
          Prefetch Data:                5.51%   1.71m
          Demand Metadata:              89.59%  27.89m
          Prefetch Metadata:            1.03%   320.32k

 ------------------------------------------------------------------------

 L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        0
        Tried Lock Failures:                    358
        IO In Progress:                         0
        Low Memory Aborts:                      1.29k
        Free on Write:                          10.68k
        Writes While Full:                      136
        R/W Clashes:                            10
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           0

 L2 ARC Size: (Adaptive)                                1.17    GiB
        Header Size:                    1.50%   17.95   MiB

 L2 ARC Breakdown:                              986.25k
        Hit Ratio:                      18.91%  186.55k
        Miss Ratio:                     81.09%  799.70k
        Feeds:                                  200

 L2 ARC Buffer:
        Bytes Scanned:                          7.70    GiB
        Buffer Iterations:                      200
        List Iterations:                        8.84k
        NULL List Iterations:                   2.73k

 L2 ARC Writes:
        Writes Sent:                    100.00% 200

 ------------------------------------------------------------------------


 ------------------------------------------------------------------------

 VDEV cache is disabled

 ------------------------------------------------------------------------

 ZFS Tunables (sysctl):
        kern.maxusers                           384
        vm.kmem_size                            4122816512
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        329853485875
        vfs.zfs.l2c_only_size                   1041817088
        vfs.zfs.mfu_ghost_data_lsize            0
        vfs.zfs.mfu_ghost_metadata_lsize        44313088
        vfs.zfs.mfu_ghost_size                  44313088
        vfs.zfs.mfu_data_lsize                  17408
        vfs.zfs.mfu_metadata_lsize              56710656
        vfs.zfs.mfu_size                        204380672
        vfs.zfs.mru_ghost_data_lsize            157184
        vfs.zfs.mru_ghost_metadata_lsize        336623616
        vfs.zfs.mru_ghost_size                  336780800
        vfs.zfs.mru_data_lsize                  0
        vfs.zfs.mru_metadata_lsize              0
        vfs.zfs.mru_size                        43282432
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       1082368
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  762268672
        vfs.zfs.arc_meta_used                   379941128
        vfs.zfs.arc_min                         381134336
        vfs.zfs.arc_max                         3049074688
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.write_limit_override            0
        vfs.zfs.write_limit_inflated            12818632704
        vfs.zfs.write_limit_max                 534109696
        vfs.zfs.write_limit_min                 33554432
        vfs.zfs.write_limit_shift               3
        vfs.zfs.no_write_throttle               0
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                1
        vfs.zfs.mg_alloc_failures               8
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.txg.synctime_ms                 1000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.scrub_limit                     10
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.ramp_rate                  2
        vfs.zfs.vdev.time_shift                 6
        vfs.zfs.vdev.min_pending                4
        vfs.zfs.vdev.max_pending                10
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     28
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0

 ------------------------------------------------------------------------

belon_cfy · Mar 16, 2012

The storage with 8GB RAM finally down again after 6 days. Adding RAM just make it survives longer but still can't avoid memory exhaustion.

mamalos · Mar 16, 2012

Just out of curiosity (bacause I was reading it in another thread): you don't have any tmpfs filesystems mounted, I suppose...

belon_cfy · Mar 16, 2012

Seems storing a lot of small file such as email will cause the server consume all of the memory. The small files are rsync from my production servers.

Another box with vmware images are working fine for more than 25 days. These 2 servers specification are identical. Seems storing different kind of data will result to different stability on FreeBSD + ZFS.

peetaur · Mar 16, 2012

Code:

 Kernel Memory:                                 396.07  MiB
 ARC Size:                              12.50%  363.44  MiB
        vm.kmem_size                            4122816512

How terribly strange. Why should zfs-status show a different number.

Please show output from:

# grep -vE "^#|^$" /boot/loader.conf
# grep -vE "^#|^$" /etc/sysctl.conf

And actually, if you trust the numbers from zfs-status, I would say ZFS may be crashing in the end, due to zero memory, but doesn't actually seem to be the cause of the leak. I'm not sure what the right command to check is... but check the man pages for netstat to find out how to print memory stats, and post results here.

Probably:
# netstat -m

Also systat might have something more interesting. (netstat would only tell you if it was your network using the memory)

Also check top or use procstat to find which user space process uses the most memory. But I think we are looking for something eating kernel memory... something different. I don't know if top and procstat tell you anything about that.

I'm not a low level guy, like a FreeBSD developer, so I don't know where to point you to find the root of the problem. Can someone else provide suggestions?

belon_cfy · Mar 18, 2012

Hi Peetaur

Below is my netstat -m nearly three days uptime. Seems nothing wrong with the memory usage.

Code:

263/3667/3930 mbufs in use (current/cache/total)
260/2162/2422/25600 mbuf clusters in use (current/cache/total/max)
260/932 mbuf+clusters out of packet secondary zone in use (current/cache)
0/44/44/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
585K/5416K/6002K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

I have disabled primarycache on all zfs volumes. But it still consume almost all of my memory.

Code:

Mem: 8804K Active, 26M Inact, 3448M Wired, 544K Cache, 353M Free
Swap: 8192M Total, 1192K Used, 8191M Free

procstat -a showing the following result:

Code:

  PID  PPID  PGID   SID  TSID THR LOGIN    WCHAN     EMUL          COMM
    0     0     0     0     0 166 -        -         -             kernel
    1     0     1     1     0   1 root     wait      FreeBSD ELF64 init
    2     0     0     0     0   5 -        zvol:io   -             zfskern
    3     0     0     0     0   1 -        waiting_  -             sctp_iterator
    4     0     0     0     0   1 -        ccb_scan  -             xpt_thrd
    5     0     0     0     0   1 -        psleep    -             pagedaemon
    6     0     0     0     0   1 -        psleep    -             vmdaemon
    7     0     0     0     0   1 -        pgzero    -             pagezero
    8     0     0     0     0   1 -        psleep    -             bufdaemon
    9     0     0     0     0   1 -        syncer    -             syncer
   10     0     0     0     0   1 -        audit_wo  -             audit
   11     0     0     0     0   4 -        -         -             idle
   12     0     0     0     0  19 -        -         -             intr
   13     0     0     0     0   3 -        -         -             geom
   14     0     0     0     0   1 -        -         -             yarrow
   15     0     0     0     0   8 -        -         -             usb
   16     0     0     0     0   1 -        vlruwt    -             vnlru
   17     0     0     0     0   1 -        sdflush   -             softdepflush
  840     1   840   840     0   1 root     select    FreeBSD ELF64 devd
 1015     1  1015  1015     0   1 root     select    FreeBSD ELF64 syslogd
 1038     1  1038  1038     0   1 root     select    FreeBSD ELF64 rpcbind
 1162     1  1162  1162     0   1 root     pause     FreeBSD ELF64 nfsuserd
 1163  1162  1162  1162     0   1 root     select    FreeBSD ELF64 nfsuserd
 1164  1162  1162  1162     0   1 root     select    FreeBSD ELF64 nfsuserd
 1165  1162  1162  1162     0   1 root     select    FreeBSD ELF64 nfsuserd
 1166  1162  1162  1162     0   1 root     select    FreeBSD ELF64 nfsuserd
 1178     1  1178  1178     0   1 root     select    FreeBSD ELF64 mountd
 1180     1  1180  1180     0   1 root     select    FreeBSD ELF64 nfsd
 1181  1180  1180  1180     0 128 root     rpcsvc    FreeBSD ELF64 nfsd
 1187     1  1187  1187     0   1 root     select    FreeBSD ELF64 rpc.statd
 1193     1  1193  1193     0   1 root     rpcsvc    FreeBSD ELF64 rpc.lockd
 1320     1  1320  1320     0   1 root     select    FreeBSD ELF64 sshd
 1327     1  1327  1327     0   1 root     select    FreeBSD ELF64 sendmail
 1333     1  1333  1333     0   1 root     pause     FreeBSD ELF64 sendmail
 1339     1  1339  1339     0   1 root     nanslp    FreeBSD ELF64 cron
 1405     1  1405  1405  1405   1 root     ttyin     FreeBSD ELF64 getty
 1406     1  1406  1406  1406   1 root     ttyin     FreeBSD ELF64 getty
 1407     1  1407  1407  1407   1 root     ttyin     FreeBSD ELF64 getty
 1408     1  1408  1408  1408   1 root     ttyin     FreeBSD ELF64 getty
 1409     1  1409  1409  1409   1 root     ttyin     FreeBSD ELF64 getty
 1410     1  1410  1410  1410   1 root     ttyin     FreeBSD ELF64 getty
 1411     1  1411  1411  1411   1 root     ttyin     FreeBSD ELF64 getty
 1412     1  1412  1412  1412   1 root     ttyin     FreeBSD ELF64 getty
 1573  1320  1573  1573     0   1 root     select    FreeBSD ELF64 sshd
 1576  1573  1576  1576  1576   1 root     pause     FreeBSD ELF64 csh
 1707  1320  1707  1707     0   1 root     select    FreeBSD ELF64 sshd
 1710  1707  1710  1710  1710   1 root     pause     FreeBSD ELF64 csh
 1714  1710  1714  1710  1710   1 root     select    FreeBSD ELF64 top
 1715  1320  1715  1715     0   1 root     select    FreeBSD ELF64 sshd
 1742  1715  1742  1742  1742   1 root     pause     FreeBSD ELF64 csh
 1747  1742  1747  1742  1742   1 root     nanslp    FreeBSD ELF64 vmstat
 1748  1320  1748  1748     0   1 root     select    FreeBSD ELF64 sshd
 1751  1748  1751  1751  1751   1 root     pause     FreeBSD ELF64 csh
 1755  1751  1755  1751  1751   1 root     nanslp    FreeBSD ELF64 perl5.12.4
 1778  1320  1778  1778     0   1 root     select    FreeBSD ELF64 sshd
 1810  1778  1810  1810  1810   1 root     pause     FreeBSD ELF64 csh
 1830  1810  1830  1810  1810   1 root     nanslp    FreeBSD ELF64 zpool
11070  1339  1339  1339     0   1 root     piperd    FreeBSD ELF64 cron
11071  1339  1339  1339     0   1 root     piperd    FreeBSD ELF64 cron
11072  1339  1339  1339     0   1 root     piperd    FreeBSD ELF64 cron
11073  1339  1339  1339     0   1 root     piperd    FreeBSD ELF64 cron
11074 11071 11074 11074     0   1 root     nanslp    FreeBSD ELF64 perl5.12.4
11075 11070 11075 11075     0   1 root     nanslp    FreeBSD ELF64 perl5.12.4
11076 11072 11076 11076     0   1 root     nanslp    FreeBSD ELF64 perl5.12.4
11077 11073 11077 11077     0   1 root     nanslp    FreeBSD ELF64 perl5.12.4
11142  1576 11142  1576  1576   1 root     -         FreeBSD ELF64 procstat
32598  1320 32598 32598     0   1 root     select    FreeBSD ELF64 sshd
32613 32598 32613 32613 32613   1 root     ttyin     FreeBSD ELF64 csh

belon_cfy · Mar 18, 2012

Continue the post above
/etc/rc.conf

Code:

dumpdev="AUTO"
sshd_enable="YES"

# NFS
nfsv4_server_enable="YES"
rpcbind_enable="YES"
nfs_server_enable="YES"
mountd_flags="-r"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"
nfs_server_flags="-u -t -n 128"

/boot/loader.conf

Code:

zfs_enable="YES"
storage20# more /boot/loader.conf
if_vlan_load="YES"
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"

zfs-stats -a result as below:

Code:

------------------------------------------------------------------------
ZFS Subsystem Report                            Sun Mar 18 18:10:25 2012
------------------------------------------------------------------------

System Information:

        Kernel Version:                         900044 (osreldate)
        Hardware Platform:                      amd64
        Processor Architecture:                 amd64

        ZFS Storage pool Version:               28
        ZFS Filesystem Version:                 5

FreeBSD 9.0-RELEASE #0: Tue Jan 3 07:46:30 UTC 2012 root
 6:10PM  up 2 days,  4:59, 6 users, load averages: 0.01, 0.02, 0.00

------------------------------------------------------------------------

System Memory:

        0.25%   9.50    MiB Active,     0.67%   25.70   MiB Inact
        89.89%  3.37    GiB Wired,      0.01%   544.00  KiB Cache
        9.16%   351.48  MiB Free,       0.02%   860.00  KiB Gap

        Real Installed:                         4.00    GiB
        Real Available:                 97.36%  3.89    GiB
        Real Managed:                   96.22%  3.75    GiB

        Logical Total:                          4.00    GiB
        Logical Used:                   90.78%  3.63    GiB
        Logical Free:                   9.22%   377.71  MiB

Kernel Memory:                                  1.05    GiB
        Data:                           98.16%  1.03    GiB
        Text:                           1.84%   19.84   MiB

Kernel Memory Map:                              3.20    GiB
        Size:                           30.14%  987.71  MiB
        Free:                           69.86%  2.24    GiB

------------------------------------------------------------------------

ARC Summary: (THROTTLED)
        Memory Throttle Count:                  8

ARC Misc:
        Deleted:                                13.99m
        Recycle Misses:                         1.40k
        Mutex Misses:                           70.55k
        Evict Skips:                            70.55k

ARC Size:                               37.99%  1.04    GiB
        Target Size: (Adaptive)         45.24%  1.24    GiB
        Min Size (Hard Limit):          12.50%  351.65  MiB
        Max Size (High Water):          8:1     2.75    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       91.56%  1.14    GiB
        Frequently Used Cache Size:     8.44%   107.36  MiB

ARC Hash Breakdown:
        Elements Max:                           377.32k
        Elements Current:               40.02%  150.99k
        Collisions:                             13.48m
        Chain Max:                              62
        Chains:                                 21.92k

------------------------------------------------------------------------

ARC Efficiency:                                 99.58m
        Cache Hit Ratio:                51.01%  50.80m
        Cache Miss Ratio:               48.99%  48.79m
        Actual Hit Ratio:               50.91%  50.70m

        Data Demand Efficiency:         0.26%   6.40m
        Data Prefetch Efficiency:       0.00%   175

        CACHE HITS BY CACHE LIST:
          Most Recently Used:           1.18%   601.49k
          Most Frequently Used:         98.62%  50.10m
          Most Recently Used Ghost:     7.18%   3.65m
          Most Frequently Used Ghost:   75.07%  38.14m

        CACHE HITS BY DATA TYPE:
          Demand Data:                  0.03%   16.43k
          Prefetch Data:                0.00%   0
          Demand Metadata:              99.25%  50.42m
          Prefetch Metadata:            0.72%   365.62k

        CACHE MISSES BY DATA TYPE:
          Demand Data:                  13.08%  6.38m
          Prefetch Data:                0.00%   175
          Demand Metadata:              86.92%  42.41m
          Prefetch Metadata:            0.00%   168

------------------------------------------------------------------------

L2ARC is disabled

------------------------------------------------------------------------


------------------------------------------------------------------------

VDEV cache is disabled

------------------------------------------------------------------------

ZFS Tunables (sysctl):
        kern.maxusers                           384
        vm.kmem_size                            4023611392
        vm.kmem_size_scale                      1
        vm.kmem_size_min                        0
        vm.kmem_size_max                        329853485875
        vfs.zfs.l2c_only_size                   0
        vfs.zfs.mfu_ghost_data_lsize            4263424
        vfs.zfs.mfu_ghost_metadata_lsize        913382912
        vfs.zfs.mfu_ghost_size                  917646336
        vfs.zfs.mfu_data_lsize                  0
        vfs.zfs.mfu_metadata_lsize              275456
        vfs.zfs.mfu_size                        1801216
        vfs.zfs.mru_ghost_data_lsize            123222016
        vfs.zfs.mru_ghost_metadata_lsize        182882304
        vfs.zfs.mru_ghost_size                  306104320
        vfs.zfs.mru_data_lsize                  514520064
        vfs.zfs.mru_metadata_lsize              479475712
        vfs.zfs.mru_size                        1027247104
        vfs.zfs.anon_data_lsize                 0
        vfs.zfs.anon_metadata_lsize             0
        vfs.zfs.anon_size                       409600
        vfs.zfs.l2arc_norw                      1
        vfs.zfs.l2arc_feed_again                1
        vfs.zfs.l2arc_noprefetch                1
        vfs.zfs.l2arc_feed_min_ms               200
        vfs.zfs.l2arc_feed_secs                 1
        vfs.zfs.l2arc_headroom                  2
        vfs.zfs.l2arc_write_boost               8388608
        vfs.zfs.l2arc_write_max                 8388608
        vfs.zfs.arc_meta_limit                  737467392
        vfs.zfs.arc_meta_used                   605950848
        vfs.zfs.arc_min                         368733696
        vfs.zfs.arc_max                         2949869568
        vfs.zfs.dedup.prefetch                  1
        vfs.zfs.mdcomp_disable                  0
        vfs.zfs.write_limit_override            0
        vfs.zfs.write_limit_inflated            12544475136
        vfs.zfs.write_limit_max                 522686464
        vfs.zfs.write_limit_min                 33554432
        vfs.zfs.write_limit_shift               3
        vfs.zfs.no_write_throttle               0
        vfs.zfs.zfetch.array_rd_sz              1048576
        vfs.zfs.zfetch.block_cap                256
        vfs.zfs.zfetch.min_sec_reap             2
        vfs.zfs.zfetch.max_streams              8
        vfs.zfs.prefetch_disable                1
        vfs.zfs.mg_alloc_failures               8
        vfs.zfs.check_hostid                    1
        vfs.zfs.recover                         0
        vfs.zfs.txg.synctime_ms                 1000
        vfs.zfs.txg.timeout                     5
        vfs.zfs.scrub_limit                     10
        vfs.zfs.vdev.cache.bshift               16
        vfs.zfs.vdev.cache.size                 0
        vfs.zfs.vdev.cache.max                  16384
        vfs.zfs.vdev.write_gap_limit            4096
        vfs.zfs.vdev.read_gap_limit             32768
        vfs.zfs.vdev.aggregation_limit          131072
        vfs.zfs.vdev.ramp_rate                  2
        vfs.zfs.vdev.time_shift                 6
        vfs.zfs.vdev.min_pending                4
        vfs.zfs.vdev.max_pending                10
        vfs.zfs.vdev.bio_flush_disable          0
        vfs.zfs.cache_flush_disable             0
        vfs.zfs.zil_replay_disable              0
        vfs.zfs.zio.use_uma                     0
        vfs.zfs.version.zpl                     5
        vfs.zfs.version.spa                     28
        vfs.zfs.version.acl                     1
        vfs.zfs.debug                           0
        vfs.zfs.super_owner                     0

------------------------------------------------------------------------

My zpool list as below, already 83% of space is in used.

Code:

NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
zroot  3.62T  3.01T   628G    83%  1.00x  ONLINE  -

This is my backup server storing a lot of emails and web contents with daily rsync. What's wrong with the FreeBSD + ZFS on storing such data in this scenario? I have two servers with completely different hardware but storing similar data up to 3TB per server are facing the exactly same problem. This is the only reason stopping me to use FreeBSD + ZFS in production environment.

peetaur · Mar 18, 2012

Have you tried using 8-STABLE or 8.3-RC1?

belon_cfy · Mar 19, 2012

Hi Peetaur
There is no way for me to downgrade to FreeBSD 8.3. I don't have any free server to reinstall for migration.

Below is my latest memory usage, I still can't find any process consume all of my memory.

Code:

last pid: 24556;  load averages:  0.17,  0.10,  0.03  up 3+10:19:14    10:26:12
60 processes:  1 running, 59 sleeping

Mem: 7940K Active, 7288K Inact, 3664M Wired, 1252K Cache, 251M Free
Swap: 8192M Total, 19M Used, 8173M Free


  PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME    CPU COMMAND
66237 root      20    0 18404K   844K nanslp  1  63:36  0.39% vmstat 0.5
 1703 root      20    0 10052K   812K rpcsvc  0   6:30  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  2   6:30  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  3   6:27  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  0   6:26  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  1   6:25  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  0   6:22  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  1   6:22  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  0   6:21  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  1   6:19  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  2   6:18  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  0   6:17  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  0   6:17  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  1   6:15  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  2   6:13  0.00% nfsd: server (nfsd){nfsd: service}
 1703 root      20    0 10052K   812K rpcsvc  3   6:12  0.00% nfsd: server (nfsd){nfsd: master}
 1703 root      20    0 10052K   812K rpcsvc  1   6:11  0.00% nfsd: server (nfsd){nfsd: service}
66245 root      52    0 19964K  1360K nanslp  0   4:37  0.00% /usr/bin/perl /root/zfs_list.pl (perl5.12.4)
66307 root      20    0 35604K  1280K nanslp  1   3:45  0.00% zpool iostat -v 1
66238 root      20    0 68016K  1252K select  3   1:34  0.00% sshd: root@pts/3 (sshd)
66206 root      20    0 68016K  1244K select  0   0:44  0.00% sshd: root@pts/2 (sshd)
66255 root      20    0 68016K  1268K select  2   0:38  0.00% sshd: root@pts/4 (sshd)
 1849 root      20    0 20384K  1168K select  0   0:04  0.00% sendmail: accepting connections (sendmail)
66157 root      20    0 68016K  1164K select  0   0:04  0.00% sshd: root@pts/0 (sshd)
 1861 root      20    0 14260K   452K nanslp  0   0:04  0.00% /usr/sbin/cron -s
 1537 root      20    0 12184K   792K select  3   0:02  0.00% /usr/sbin/syslogd -s
 1560 root      20    0 14264K   760K select  0   0:01  0.00% /usr/sbin/rpcbind
 1715 root      52    0 14264K   612K rpcsvc  0   0:00  0.00% /usr/sbin/rpc.lockd
 1709 root      20    0   268M   684K select  3   0:00  0.00% /usr/sbin/rpc.statd
 1686 root      20    0 10052K   660K select  2   0:00  0.00% nfsuserd: slave (nfsuserd)
 1687 root      20    0 10052K   660K select  0   0:00  0.00% nfsuserd: slave (nfsuserd)
66160 root      20    0 14612K  1524K pause   0   0:00  0.00% -csh (csh)
 1685 root      20    0 10052K   660K select  1   0:00  0.00% nfsuserd: slave (nfsuserd)
 1688 root      20    0 10052K   660K select  1   0:00  0.00% nfsuserd: slave (nfsuserd)
 1855 smmsp     20    0 20384K   424K pause   2   0:00  0.00% sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (
 1700 root      20    0 12180K   624K select  1   0:00  0.00% /usr/sbin/mountd -r /etc/exports /etc/zfs/exports
 1702 root      20    0 10052K   580K select  0   0:00  0.00% nfsd: master (nfsd)
66241 root      22    0 14612K     0K pause   1   0:00  0.00% -csh (<csh>)
66270 root      22    0 14612K     0K pause   0   0:00  0.00% -csh (<csh>)
66233 root      22    0 14612K     0K pause   0   0:00  0.00% -csh (<csh>)
24493 root      36    0 19964K  2348K nanslp  0   0:00  0.00% /usr/bin/perl /root/perl_zfs_sysctl.pl (perl5.12.4)
24495 root      22    0 19964K  2320K nanslp  3   0:00  0.00% /usr/bin/perl /root/perl_uptime.pl (perl5.12.4)
24492 root      52    0 19964K  2356K nanslp  0   0:00  0.00% /usr/bin/perl /root/perl_zfs_stats.pl (perl5.12.4)
24494 root      27    0 19964K  2320K nanslp  3   0:00  0.00% /usr/bin/perl /root/perl_memstatus.pl (perl5.12.4)
 1842 root      20    0 46876K   880K select  3   0:00  0.00% /usr/sbin/sshd
 1342 root      20    0 10372K   340K select  0   0:00  0.00% /sbin/devd
 1923 root      52    0 12184K   580K ttyin   0   0:00  0.00% /usr/libexec/getty Pc ttyv4
24556 root      20    0 16700K  1888K CPU1    1   0:00  0.00% top -naCH 100000
 1919 root      52    0 12184K   580K ttyin   0   0:00  0.00% /usr/libexec/getty Pc ttyv0
 1920 root      52    0 12184K   580K ttyin   2   0:00  0.00% /usr/libexec/getty Pc ttyv1
 1925 root      52    0 12184K   580K ttyin   1   0:00  0.00% /usr/libexec/getty Pc ttyv6
 1924 root      52    0 12184K   580K ttyin   3   0:00  0.00% /usr/libexec/getty Pc ttyv5
 1921 root      52    0 12184K   580K ttyin   2   0:00  0.00% /usr/libexec/getty Pc ttyv2
 1922 root      52    0 12184K   580K ttyin   3   0:00  0.00% /usr/libexec/getty Pc ttyv3
 1926 root      52    0 12184K   580K ttyin   0   0:00  0.00% /usr/libexec/getty Pc ttyv7
24489 root      20    0 14260K  1076K piperd  1   0:00  0.00% cron: running job (cron)
 1684 root      34    0 10052K     0K pause   0   0:00  0.00% nfsuserd: master (<nfsuserd>)
24490 root      20    0 14260K  1076K piperd  1   0:00  0.00% cron: running job (cron)
24488 root      20    0 14260K  1076K piperd  0   0:00  0.00% cron: running job (cron)
24491 root      21    0 14260K  1076K piperd  1   0:00  0.00% cron: running job (cron)

netstat -m

Code:

2055/2520/4575 mbufs in use (current/cache/total)
2050/1712/3762/25600 mbuf clusters in use (current/cache/total/max)
2050/766 mbuf+clusters out of packet secondary zone in use (current/cache)
0/71/71/12800 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)
0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
4613K/4338K/8951K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

peetaur · Mar 19, 2012

You should be able to just install 8.3 in a new dataset, and then change the bootfs property to point to the right version. You should find a howto somewhere on how to run multiple versions at the same time. And try it in a VM first.

belon_cfy · Mar 20, 2012

Changing primarycache to none didn't solve my problem too, it crashed again last night because of memory exhaustion.

Limiting arc_max, kmem_size, kmem_size_max, swap partition type changed to freebsd-swap, adding 8GB thumbdrive for l2arc, upgrading ram to 8GB, lastly disabled primarycache. All of the methods above didn't solve my problem.

Any suggestion? Downgrading to FreeBSD 8.3?

alsuki · Mar 26, 2012

Hi, everyone.

Just now I saw this thread. I had the same problem about a year back, at that time I didn't know the cause, so I reinstall Linux. At that time I had a AMD64 athlon x2 with 8GB of memory and 5 TB of disk, and every 7 days the machine would hang during the night.

My point is downgrading will not solve the problem. But I would be very glad to see this problem solved, because I'm creating a script for a full install of FreeBSD 9.0 on my machine.

Sorry I can't be of any help, but keep up the good work.

Best regards,
Alsuki.

peetaur · Mar 26, 2012

alsuki said:
Hi, everyone.

Just now I saw this thread. I had the same problem about a year back, at that time I didn't know the cause, so I reinstall Linux. At that time I had a AMD64 athlon x2 with 8GB of memory and 5 TB of disk, and every 7 days the machine would hang during the night.

My point is downgrading will not solve the problem. But I would be very glad to see this problem solved, because I'm creating a script for a full install of FreeBSD 9.0 on my machine.

Sorry I can't be of any help, but keep up the good work.

Best regards,
Alsuki.

What version did you use? Lots of mysterious problems happen with ZFS in 8.2-RELEASE and earlier, and 8-STABLE until around October 2011. And I wouldn't try 9 in production until it is proven stable.

And some problems are confused for other problems. For example, if you have many snapshots and run a command that goes through all the .zfs/snapshot/* directories, for example: # find /tank/.zfs/snapshot -type d, it is well known that your system will fail due to running out of memory. And yet you can run a perfectly stable system if you just simply avoid doing that.

And to the OP, I forget if the above type of problem was ever mentioned, but you could try to figure out if something like that is happening. (not sure how... maybe just make a cronjob that logs "ps axl" and "ps aux" output, assuming that the problem process will run long enough to find).

Other random ideas that came to mind:

Maybe exporting and importing the pool or some other similar hack would clean up the memory.

Deleting all but a few snapshots would prevent the above type of problem (I would not be happy with this solution)

You should post a detailed question on the freebsd-current or freebsd-fs mailing lists to get attention of developers, because they would know how to track down the root cause of the problem. You could also try updating to the latest 9 code, in case that is what the freebsd-current people expect you to be running. (but expect some people to be annoyed if you update between questions, because then their diagnostic commands they want you to run won't apply to your new version anymore and give inconsistent information).

mamalos · Mar 27, 2012

belon_cfy,

Follow these instructions about debugging a FreeBSD kernel crash, and send the stack trace of gdb to the freebsd-stable mailing list. Developers are watching this list too, so it would be very helpful for them to obtain a stack trace of your crashing system. On the other hand, since it's a memory leak, I am not sure if your output would suffice (the output will show which part of the system crashed due to memory starvation -which is a probably a bug- but will not show which function caused the starvation). Anyway, it would be a good idea to talk to these guys, since the lists are much more technical than this forum.

obrandmueller · Apr 2, 2012

FreeBSD 9 amd64, ZFS, NFS

I'm suspecting a NAMEI leak in most probably NFS or a combination of NFS and ZFS.

I'm seeing constant growth in vmstat -z for NAMEI while in other systems (running ZFS, no NFS or running old NFS, no ZFS, FreeBSD 8) I see NAMEI always close to 0 in the USED row.

Could you confirm this for your systems showing the wrong behaviour?

- Oliver

VVB16 · Apr 3, 2012

Hi

I ran into similar (or same) problems with FreeBSD 8 from the beginning.
Solution that works for me:

apply patch to kernel:

Code:

=========================================================================
--- sys/kern/kern_malloc.c      2009-08-03 15:13:06.000000000 +0700
+++ sys/kern/kern_malloc.c      2010-05-19 11:07:09.717752297 +0700
@@ -611,8 +611,10 @@
         * to something sane. Be careful to not overflow the 32bit
         * ints while doing the check.
         */
+/*
        if (((vm_kmem_size / 2) / PAGE_SIZE) > cnt.v_page_count)
                vm_kmem_size = 2 * cnt.v_page_count * PAGE_SIZE;
+*/

        /*
         * Tune settings based on the kmem map's size at this time.
=========================================================================

Add to /boot/loader.conf:

Code:

vm.kmem_size="16G"
vfs.zfs.arc_min="1G"
vfs.zfs.arc_max="4G"

These values used on 8GB RAM server. Main tunable is vm.kmem_size. As far as I can remember real problem is memory fragmentation which leads to memory exhaustion. Patch needed to set such high vm.kmem_size, check it after reboot with sysctl -a vm.kmem_size.

peetaur · Apr 3, 2012

VVB16, so is the idea to just allocate it all now and not change it, thereby making vm.kmem_size_max obsolete?

VVB16 · Apr 3, 2012

peetaur,

No. Actual output from this box:

sysctl -a |grep vm.kmem_size

Code:

vm.kmem_size_scale: 1
vm.kmem_size_max: 329853485875
vm.kmem_size_min: 0
vm.kmem_size: 17179869184

Problem with vm.kmem_size was (and probably is) in that it's not allocated at all when needed. Maybe because system has free (virtual) memory, but this memory fragmented and at some time call for memory fails => kernel panic.

At the beginning I set vm.kmem_size to 24G (3*RAM size), later reduced it to 2*RAM just to see how it works. System works stable _with our workload_ with this settings, so I stop tuning it.

peetaur · Apr 4, 2012

BTW have you guys seen the "9-STABLE, ZFS, NFS, ggatec - suspected memory leak" thread in the freebsd-stable mailing list?

It sounds like a similar issue to yours. One guy says it is definitely ZFS. Other guy says it is definitely the new NFS server (default in 9.0 I think) and has no problems using the old NFS server (default in 8.2).

And the memory usage can be seen using # vmstat -z

FreeBSD 9 64bit with ZFS crashes every 3 days

Administrator