Solved Memory leak on FreeBSD 9.3 and FreeBSD 10?

belon_cfy · Sep 12, 2013

Latest update: https://forums.freebsd.org/viewtopic.php?f=48&t=41880&start=25#p249468

Hi,

I have a server running on 8 GB of ~~ram~~ RAM that serves as a NFS. However the memory is exhausted significantly faster than with the rest of the servers with the same configuration, but running on FreeBSD 9.1 and below. Currently the server has only 300 MB left but others with the same configuration have more than 1 GB left.

vmstat is showing the "32" used up to 6,750,713 within eight days, but all of my servers are below 2,000,000 with 400 days of uptime.

Anything abnormal on the server?

Server specification: Supermicro Xeon 1230v2, ~~8GB Ram~~ 8 GB RAM, 4 X 2 TB HDD with SSD as ZIL and L2ARC, FreeBSD 9.2-RC3 running ZFS.

My etc/rc.conf:

Code:

geli_devices="gpt/data_disk0 gpt/data_disk1 gpt/data_disk2 gpt/data_disk3 gpt/l2arc_disk0 gpt/slog_disk0"
geli_swap_flags="-e AES-CBC -l 256 -s 4096 -d"
geli_default_flags="-p -k /boot/keys/disks.key"
geli_autodetach="NO"

sshd_enable="YES"

# NFS
nfsv4_server_enable="YES"
rpcbind_enable="YES"
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 64"
mountd_flags="-r"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"

zfs_enable="YES"

/boot/loader.conf:

Code:

zfs_load="YES"
aesni_load="YES"
vfs.zfs.cache_flush_disable=1
vfs.root.mountfrom="zfs:zroot"

vfs.zfs.arc_max="4G"
vfs.zfs.write_limit_override="2G"

/etc/sysctl.conf:

Code:

vfs.zfs.l2arc_noprefetch=0
vfs.zfs.l2arc_feed_min_ms=1000
vfs.zfs.l2arc_norw=0
vfs.zfs.l2arc_feed_again=0

belon_cfy · Sep 12, 2013

vmstat part 1:

Code:

# vmstat -z
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     187,       0,     187,   0,   0
UMA Zones:             1408,      0,     187,       1,     187,   0,   0
UMA Slabs:              568,      0,  182851,    7262,1655359382,   0,   0
UMA RCntSlabs:          568,      0,    9932,       1,  191297,   0,   0
UMA Hash:               256,      0,      79,      11,      82,   0,   0
16 Bucket:              152,      0,      42,     158,     218,   0,   0
32 Bucket:              280,      0,     142,     166,     410,   1,   0
64 Bucket:              536,      0,     194,      58,     563,  96,   0
128 Bucket:            1048,      0,   10637,       1,  341785,271526,   0
VM OBJECT:              232,      0,    3696,    1616,198040921,   0,   0
MAP:                    232,      0,       8,      24,       8,   0,   0
KMAP ENTRY:             120, 276706,    7173,   25749,4537350748,   0,   0
MAP ENTRY:              120,      0,    1239,    1985,478584526,   0,   0
fakepg:                 120,      0,      10,    1106,   43970,   0,   0
mt_zone:               4112,      0,     329,      44,     329,   0,   0
16:                      16,      0,    6707,    7909,516746799,   0,   0
32:                      32,      0, 6750713, 4857722,842972203,   0,   0
64:                      64,      0,   19919,   17097,4319447531,   0,   0
128:                    128,      0,  382970,    5398,809071624,   0,   0
256:                    256,      0,    7029,   15321,1003894827,   0,   0
512:                    512,      0,    4163,    9795,569112985,   0,   0
1024:                  1024,      0,     135,    1621,86442108,   0,   0
2048:                  2048,      0,     232,    1852,453188505,   0,   0
4096:                  4096,      0,     933,    2987,232332977,   0,   0
Files:                   80,      0,     122,    1453,80400534,   0,   0
rl_entry:                40,      0,     558,     198,     558,   0,   0
TURNSTILE:              136,      0,    1231,      89,    1231,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1192,      0,      71,     523, 6902043,   0,   0
THREAD:                1160,      0,    1031,     199,    1835,   0,   0
SLEEPQUEUE:              80,      0,    1231,     161,    1231,   0,   0
VMSPACE:                392,      0,      44,    1066, 6902005,   0,   0
cpuset:                  72,      0,     465,     435,    1204,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,    1023,    1409,359086743,   0,   0
mbuf:                   256,      0,    1069,    7134,4857962764,   0,   0
mbuf_cluster:          2048,  25600,    2432,    1870,861391498,   0,   0
mbuf_jumbo_page:       4096,  12800,       0,    1381,274887034,   0,   0
mbuf_jumbo_9k:         9216,   6400,    1031,    5369,1383502445,94030,   0
mbuf_jumbo_16k:       16384,   3200,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,       5,    1675,2446699117,   0,   0
ttyinq:                 160,      0,     240,     384,    1110,   0,   0
ttyoutq:                256,      0,     126,     249,     582,   0,   0
ata_request:            328,      0,       0,       0,       0,   0,   0
ata_composite:          336,      0,       0,       0,       0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,    1152,  351156,   0,   0
VNODE:                  504,      0,    2790,     714,   28692,   0,   0
VNODEPOLL:              112,      0,       0,       0,       0,   0,   0
S VFS Cache:            108,      0,    2775,     855, 5606552,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,       6,     126,      65,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NAMEI:                 1024,      0,       0,    1168,209778631,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0
NCLNODE:                568,      0,       0,       0,       0,   0,   0
Mountpoints:            824,      0,       6,      14,       8,   0,   0
pipe:                   728,      0,      14,     506, 9057166,   0,   0
space_seg_cache:         64,      0,  622219,   59469,244206545,   0,   0
zio_cache:              944,      0,       7,   17093,3282984010,   0,   0
zio_link_cache:          48,      0,       3,   17277,3222755314,   0,   0
zio_buf_512:            512,      0,       0,       0,       0,   0,   0
zio_data_buf_512:       512,      0,       0,       0,       0,   0,   0
zio_buf_1024:          1024,      0,       0,       0,       0,   0,   0
zio_data_buf_1024:     1024,      0,       0,       0,       0,   0,   0
zio_buf_1536:          1536,      0,       0,       0,       0,   0,   0
zio_data_buf_1536:     1536,      0,       0,       0,       0,   0,   0
zio_buf_2048:          2048,      0,       0,       0,       0,   0,   0
zio_data_buf_2048:     2048,      0,       0,       0,       0,   0,   0
zio_buf_2560:          2560,      0,       0,       0,       0,   0,   0
zio_data_buf_2560:     2560,      0,       0,       0,       0,   0,   0
zio_buf_3072:          3072,      0,       0,       0,       0,   0,   0
zio_data_buf_3072:     3072,      0,       0,       0,       0,   0,   0
zio_buf_3584:          3584,      0,       0,       0,       0,   0,   0
zio_data_buf_3584:     3584,      0,       0,       0,       0,   0,   0
zio_buf_4096:          4096,      0,       0,       0,       0,   0,   0
zio_data_buf_4096:     4096,      0,       0,       0,       0,   0,   0
zio_buf_5120:          5120,      0,       0,       0,       0,   0,   0
zio_data_buf_5120:     5120,      0,       0,       0,       0,   0,   0
zio_buf_6144:          6144,      0,       0,       0,       0,   0,   0
zio_data_buf_6144:     6144,      0,       0,       0,       0,   0,   0
zio_buf_7168:          7168,      0,       0,       0,       0,   0,   0
zio_data_buf_7168:     7168,      0,       0,       0,       0,   0,   0
zio_buf_8192:          8192,      0,       0,       0,       0,   0,   0
zio_data_buf_8192:     8192,      0,       0,       0,       0,   0,   0
zio_buf_10240:        10240,      0,       0,       0,       0,   0,   0
zio_data_buf_10240:   10240,      0,       0,       0,       0,   0,   0
zio_buf_12288:        12288,      0,       0,       0,       0,   0,   0
zio_data_buf_12288:   12288,      0,       0,       0,       0,   0,   0
zio_buf_14336:        14336,      0,       0,       0,       0,   0,   0
zio_data_buf_14336:   14336,      0,       0,       0,       0,   0,   0
zio_buf_16384:        16384,      0,       0,       0,       0,   0,   0
zio_data_buf_16384:   16384,      0,       0,       0,       0,   0,   0
zio_buf_20480:        20480,      0,       0,       0,       0,   0,   0
zio_data_buf_20480:   20480,      0,       0,       0,       0,   0,   0
zio_buf_24576:        24576,      0,       0,       0,       0,   0,   0
zio_data_buf_24576:   24576,      0,       0,       0,       0,   0,   0

belon_cfy · Sep 12, 2013

vmstat Part 2:

Code:

zio_buf_28672:        28672,      0,       0,       0,       0,   0,   0
zio_data_buf_28672:   28672,      0,       0,       0,       0,   0,   0
zio_buf_32768:        32768,      0,       0,       0,       0,   0,   0
zio_data_buf_32768:   32768,      0,       0,       0,       0,   0,   0
zio_buf_36864:        36864,      0,       0,       0,       0,   0,   0
zio_data_buf_36864:   36864,      0,       0,       0,       0,   0,   0
zio_buf_40960:        40960,      0,       0,       0,       0,   0,   0
zio_data_buf_40960:   40960,      0,       0,       0,       0,   0,   0
zio_buf_45056:        45056,      0,       0,       0,       0,   0,   0
zio_data_buf_45056:   45056,      0,       0,       0,       0,   0,   0
zio_buf_49152:        49152,      0,       0,       0,       0,   0,   0
zio_data_buf_49152:   49152,      0,       0,       0,       0,   0,   0
zio_buf_53248:        53248,      0,       0,       0,       0,   0,   0
zio_data_buf_53248:   53248,      0,       0,       0,       0,   0,   0
zio_buf_57344:        57344,      0,       0,       0,       0,   0,   0
zio_data_buf_57344:   57344,      0,       0,       0,       0,   0,   0
zio_buf_61440:        61440,      0,       0,       0,       0,   0,   0
zio_data_buf_61440:   61440,      0,       0,       0,       0,   0,   0
zio_buf_65536:        65536,      0,       0,       0,       0,   0,   0
zio_data_buf_65536:   65536,      0,       0,       0,       0,   0,   0
zio_buf_69632:        69632,      0,       0,       0,       0,   0,   0
zio_data_buf_69632:   69632,      0,       0,       0,       0,   0,   0
zio_buf_73728:        73728,      0,       0,       0,       0,   0,   0
zio_data_buf_73728:   73728,      0,       0,       0,       0,   0,   0
zio_buf_77824:        77824,      0,       0,       0,       0,   0,   0
zio_data_buf_77824:   77824,      0,       0,       0,       0,   0,   0
zio_buf_81920:        81920,      0,       0,       0,       0,   0,   0
zio_data_buf_81920:   81920,      0,       0,       0,       0,   0,   0
zio_buf_86016:        86016,      0,       0,       0,       0,   0,   0
zio_data_buf_86016:   86016,      0,       0,       0,       0,   0,   0
zio_buf_90112:        90112,      0,       0,       0,       0,   0,   0
zio_data_buf_90112:   90112,      0,       0,       0,       0,   0,   0
zio_buf_94208:        94208,      0,       0,       0,       0,   0,   0
zio_data_buf_94208:   94208,      0,       0,       0,       0,   0,   0
zio_buf_98304:        98304,      0,       0,       0,       0,   0,   0
zio_data_buf_98304:   98304,      0,       0,       0,       0,   0,   0
zio_buf_102400:      102400,      0,       0,       0,       0,   0,   0
zio_data_buf_102400: 102400,      0,       0,       0,       0,   0,   0
zio_buf_106496:      106496,      0,       0,       0,       0,   0,   0
zio_data_buf_106496: 106496,      0,       0,       0,       0,   0,   0
zio_buf_110592:      110592,      0,       0,       0,       0,   0,   0
zio_data_buf_110592: 110592,      0,       0,       0,       0,   0,   0
zio_buf_114688:      114688,      0,       0,       0,       0,   0,   0
zio_data_buf_114688: 114688,      0,       0,       0,       0,   0,   0
zio_buf_118784:      118784,      0,       0,       0,       0,   0,   0
zio_data_buf_118784: 118784,      0,       0,       0,       0,   0,   0
zio_buf_122880:      122880,      0,       0,       0,       0,   0,   0
zio_data_buf_122880: 122880,      0,       0,       0,       0,   0,   0
zio_buf_126976:      126976,      0,       0,       0,       0,   0,   0
zio_data_buf_126976: 126976,      0,       0,       0,       0,   0,   0
zio_buf_131072:      131072,      0,       0,       0,       0,   0,   0
zio_data_buf_131072: 131072,      0,       0,       0,       0,   0,   0
sa_cache:                80,      0,    2722,     833,   28846,   0,   0
dnode_t:                856,      0,    5077,    3231,26308054,   0,   0
dmu_buf_impl_t:         224,      0,   69694,   39327,187063060,   0,   0
arc_buf_hdr_t:          216,      0, 3445591, 2140817,128615322,   0,   0
arc_buf_t:               72,      0,  114515,   43135,263110335,   0,   0
zil_lwb_cache:          192,      0,     320,    3040,60798759,   0,   0
zfs_znode_cache:        368,      0,    2722,     638,   28566,   0,   0
ksiginfo:               112,      0,     583,    1298,   30176,   0,   0
itimer:                 344,      0,       0,      33,       2,   0,   0
KNOTE:                  128,      0,       0,     812,  127524,   0,   0
socket:                 680,  25602,      56,     436,  342232,   0,   0
unpcb:                  240,  25600,      10,     294,   59560,   0,   0
ipq:                     56,    819,       0,       0,       0,   0,   0
udp_inpcb:              392,  25600,      18,     712,  244493,   0,   0
udpcb:                   16,  25704,      18,    1662,  244493,   0,   0
tcp_inpcb:              392,  25600,      28,     332,   38163,   0,   0
tcpcb:                  976,  25600,      28,     368,   38163,   0,   0
tcptw:                   72,   5150,       0,     650,    6910,   0,   0
syncache:               152,  15375,       0,     250,    8831,   0,   0
hostcache:              136,  15372,       4,     164,      17,   0,   0
tcpreass:                40,   1680,       0,    1092,  712835,   0,   0
sackhole:                32,      0,       0,    1111,  280985,   0,   0
sctp_ep:               1384,  25600,       0,       0,       0,   0,   0
sctp_asoc:             2288,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80064,       0,     288,       8,   0,   0
sctp_raddr:             704,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400008,       0,       0,       0,   0,   0
sctp_readq:             104, 400032,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400032,       0,       0,       0,   0,   0
sctp_asconf:             40, 400008,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400032,       0,       0,       0,   0,   0
ripcb:                  392,  25600,       0,      90,      11,   0,   0
rtentry:                200,      0,      29,      66,      29,   0,   0
selfd:                   56,      0,     939,    1014,152725323,   0,   0
SWAPMETA:               288, 1008163,     712,     146,    9232,   0,   0

belon_cfy · Sep 17, 2013

I can confirm that two of my FreeBSD-9.2-RC3 and RC4 having the same issue. The free memory dropped below 1G 1 GB within three days. However. with the same configuration of my new FreeBSD 9.1 server with the same uptime and heavier I/O doesn't suffer from the same behavior, until now it has 2 GB and above free memory. I think some kind of memory leak is happening on my FreeBSD 9.2 servers.

FreeBSD 9.1 - ZFS V28 with compression on
FreeBSD 9.2 - ZFS V5000 with LZ4 compression

SirDice · Sep 17, 2013

The biggest question is, what do you consider "free" memory? Keep in mind that FreeBSD will try to use everything, mainly for caching. But this shouldn't cause any problems. Not unless the server starts using swap excessively.

jem · Sep 17, 2013

"Free memory is wasted memory".

If it's not being used by programs then it might as well be used to cache disk I/O instead.

Perhaps 9.2-RC3+ is simply better at doing this.

belon_cfy · Sep 18, 2013

I'm afraid the memory leak will lead to memory exhaustion and eventually bring down the server since both of the FreeBSD 9.2 servers' memory usage behave unlike the rest of my FreeBSD 9.1.

The ARC is limited to 4 GB however the total memory usage is up to 7 GB and above for an NFS server only.

The swap space has been used, although not that aggressively.

belon_cfy · Sep 19, 2013

Another issue is happening since the free memory is dropped below 400 MB. The server keeps flushing the writeback cache to disks instead of buffering it in RAM and commit every five seconds. Will FreeBSD force to commit write cache to the disk when the memory is lower than 400 MB?

The server has SSD as ZIL.

The similar issue doesn't happen on another server with FreeBSD 9.1 and 2 GB RAM free at equivalent work load.

By the way, I have the following parameters in /boot/loader.conf on both servers.

Code:

vfs.zfs.arc_max="4G"
vfs.zfs.write_limit_override="2G"

J65nko · Sep 19, 2013

Have you tried the latest FreeBSD 9.2-RC4? Nobody will spend time on fixing RC3 if the problem has gone away in RC4

belon_cfy · Sep 19, 2013

Yes, the server has been upgraded to FreeBSD 9.2 RC4, problem remains.

belon_cfy · Sep 23, 2013

The server with FreeBSD 9.2-RC4 was down last night and the uptime is about ten days with only 200 MB RAM free. However the other server with FreeBSD 9.1 and the same uptime is still running fine with 2 GB of RAM free.

sun_ia32 · Sep 23, 2013

This really does sound like a problem. Did you try contacting the developers via the mailing list?

belon_cfy · Oct 6, 2013

Upgraded to FreeBSD 9.2-Release, problem still persist.

Think I have to switch it back to FreeBSD 9.1 and wait until the issue is resolved.

belon_cfy · Nov 10, 2013

Upgraded to FreeBSD 9.2-p1 and moved some of the virtual machine on it, seems the memory leak issue still there compare to the previous version.

I notice that the FreeBSD 9.2 memory usage of 32: is much higher than FreeBSD 9.1 , both server are hosted exactly the same content.

vmstat -z on FreeBSD 9.1

Code:

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

16:                      16,      0, 1627726,  950402,2742669502,   0,   0
32:                      32,      0, 1649754,  943724,3526384744,   0,   0
64:                      64,      0,  522316,  463284,20497736229,   0,   0
128:                    128,      0,   24113,   55956,3720279158,   0,   0
256:                    256,      0,   54111,   61149,4593010295,   0,   0
512:                    512,      0,   70580,  313762,2177203825,   0,   0
1024:                  1024,      0,     138,   28018,165498646,   0,   0
2048:                  2048,      0,    6364,    7226,1582801956,   0,   0
4096:                  4096,      0,    8789,   13396,1032322949,   0,   0
Files:                   80,      0,     114,     426,213572476,   0,   0

vmstat -z on FreeBSD 9.2

Code:

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

16:                      16,      0,   23240,    3136,436263549,   0,   0
32:                      32,      0, 7646055,    1766,677904734,   0,   0
64:                      64,      0,   26875,   17925,2959077356,   0,   0
128:                    128,      0,  345203,    1057,422898487,   0,   0
256:                    256,      0,   17419,   15176,578761754,   0,   0
512:                    512,      0,   14678,    3900,418832444,   0,   0
1024:                  1024,      0,     146,    4678,41878369,   0,   0
2048:                  2048,      0,     343,    2269,34511702,   0,   0
4096:                  4096,      0,    1360,    5934,63518714,   0,   0
Files:                   80,      0,     129,     771,117024521,   0,   0

zfs volumes

Code:

NAME        USED  AVAIL  RSIZE  COMP   RATIO  ATIME  SYNC      SNAP
vol        1.79T  1.75T    32K  lz4    1.41x  off    standard     0
vol/vol00  1.01T  1.75T    32K  lz4    1.34x  off    standard    30  
vol/vol01   800G  1.75T    32K  lz4    1.49x  off    standard    30

Any idea?

aupanner · Nov 11, 2013

Unless you're getting excessive swapping, it is unlikely that there is a memory leak. It may simply be that 9.2 is more aggressive about caching. Is your system locking up or crashing? Are applications failing? Is swap soaring? If not, focusing on the Free value isn't particularly helpful.

Here's my box. Not much free, runs great.

Code:

Mem: 349M Active, 659M Inact, 2672M Wired, 43M Cache, 417M Buf, 197M Free
ARC: 1939M Total, 111M MFU, 1491M MRU, 146K Anon, 30M Header, 306M Other
Swap: 16G Total, 61M Used, 16G Free

belon_cfy · Nov 12, 2013

Hi,

Yes, eventually it crashed and the service stopped working after a week.

aupanner · Nov 12, 2013

belon_cfy said:
Yes, eventually it crashed and the service stopped working after a week.

Ok here's what I'd try:

Stay on 9.1? You don't have to debug this problem unless 9.2 has something you need. Wait for 10 to come out and then try that. Maybe the problem will be magically fixed. Maybe you'll have budget to buy more memory
You don't say if you're using i386 or amd64. Apparently i386 has memory issues with ZFS like those you're describing.
On the 9.2-RELEASE-p1 box, if it crashes after 7 days, let it run for 5 days and then start comparing things against the 9.1 box.
- ps augxwwd: Can you see a process that is consuming all of your RAM?
- swapinfo: Is your swap being consumed? (2% doesn't count, 80% is an issue.)
- zfs-stats -A: Is ARC within bounds? (I.e.: if you set it for 2 GB, is it under 2 GB?)
Shot in the dark: are you using ZFS dedup? You might not have enough memory for that. See, for example:
http://forums.freebsd.org/showthread.php?t=41194
You have some old threads about memory starvation under 8.X, you might want to check /usr/src/sys and make sure the patches that gave you stability then are still applied to the codebase.

belon_cfy · Nov 13, 2013

Most of my servers are FreeBSD 9.1 except this for testing.

Below is the current memory usage, I just don't understand how come the 32 memory usage is much higher which is 10 times than the others on FreeBSD 9.1

Code:

16:                      16,      0,   10903,   15473,855690791,   0,   0
32:                      32,      0, 9388002, 1384456,1213007946,   0,   0
64:                      64,      0,   21092,   23708,5481012925,   0,   0
128:                    128,      0,  629890,   30556,892815193,   0,   0

The server is running FreeBSD 9.2-p1 64bit with no dedupe.

ARC is ok, only 4GB is in used which is same as what I have defined.

Code:

------------------------------------------------------------------------
ZFS Subsystem Report                            Wed Nov 13 10:34:16 2013
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                265.01m
        Recycle Misses:                         35.31m
        Mutex Misses:                           114.99k
        Evict Skips:                            2.15b

ARC Size:                               100.00% 4.00    GiB
        Target Size: (Adaptive)         100.00% 4.00    GiB
        Min Size (Hard Limit):          12.50%  512.00  MiB
        Max Size (High Water):          8:1     4.00    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       6.30%   257.99  MiB
        Frequently Used Cache Size:     93.70%  3.75    GiB

ARC Hash Breakdown:
        Elements Max:                           5.44m
        Elements Current:               87.29%  4.75m
        Collisions:                             267.10m
        Chain Max:                              77
        Chains:                                 131.07k

Now it has 762MB free after 6 days, will keep it run for 2 weeks because previously it crashed after 2 weeks of uptime.

Code:

last pid:  4578;  load averages:  0.22,  0.22,  0.25                                                                                                    up 6+15:04:59  10:35:37
46 processes:  1 running, 43 sleeping, 2 zombie
CPU:  0.2% user,  0.0% nice,  1.9% system,  0.0% interrupt, 97.9% idle
Mem: 7856K Active, 42M Inact, 7069M Wired, 762M Free
ARC: 4097M Total, 2827M MFU, 236M MRU, 21M Anon, 1125M Header, 28M Other
Swap: 8192M Total, 8192M Free

User23 · Nov 14, 2013

I have a nearly similar hardware/software setup and cant see any memory leak related problems.

FreeBSD 9.2-RELEASE #2 r255966, used as NFS Server and other stuff.

Supermicro X9SRi-3F, Xeon E5-2620, 32 GB, 2x raidz2 (5x 1 TB), 60 GB ZIL mirror, 500 GB L2ARC

Code:

load averages:  2.34,  1.53,  1.18                                up 10+13:40:17  12:45:57
3903 processes:2 running, 3901 sleeping
CPU:  1.2% user,  0.0% nice,  4.2% system,  0.3% interrupt, 94.3% idle
Mem: 4429M Active, 3196M Inact, 18G Wired, 3310M Buf, 5564M Free
ARC: 3938M Total, 1522M MFU, 253M MRU, 1488K Anon, 1154M Header, 1079M Other
Swap: 8192M Total, 8192M Free

vfs.zfs.arc_max is limited to 16GB

belon_cfy · Nov 15, 2013

I think it is abnormal to have 5564M free memory left with 10 days uptime and

Code:

vfs.zfs.arc_max=16GB

your system suppose to have more free memory with the same running period on FreeBSD 9.1.

Can you show me your vmstat -z output?

My system with FreeBSD 9.1 with 8 GB RAM and

Code:

vfs.zfs.arc_max=4GB

has 3GB++ free after 10 days. Today it started swapping out some data.

Code:

Mem: 23M Active, 5556K Inact, 7445M Wired, 6924K Cache, 402M Free
ARC: 3922M Total, 2713M MFU, 138M MRU, 29M Anon, 1170M Header, 18M Other
Swap: 8192M Total, 17M Used, 8175M Free

Is there anything occupying the memory?

Code:

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     187,       0,     187,   0,   0
UMA Zones:             1408,      0,     187,       1,     187,   0,   0
UMA Slabs:              568,      0,  181914,    4979,1804741298,   0,   0
UMA RCntSlabs:          568,      0,    5696,    1857,  267446,   0,   0
UMA Hash:               256,      0,      79,      11,      82,   0,   0
16 Bucket:              152,      0,      38,     137,     218,   0,   0
32 Bucket:              280,      0,     113,     153,     376,  20,   0
64 Bucket:              536,      0,     133,     133,     677, 114,   0
128 Bucket:            1048,      0,    1103,      10,  200125,403399,   0
VM OBJECT:              232,      0,    7984,    8096,738552266,   0,   0
MAP:                    232,      0,       8,      24,       8,   0,   0
KMAP ENTRY:             120, 276923,   10931,   22673,4901016283,   0,   0
MAP ENTRY:              120,      0,    1217,    2255,1746936120,   0,   0
fakepg:                 120,      0,      10,     641,    2230,   0,   0
mt_zone:               4112,      0,     329,      44,     329,   0,   0
16:                      16,      0,   10957,   15419,1118302053,   0,   0
32:                      32,      0, 9720077, 1413052,1641641837,   0,   0
64:                      64,      0,   24519,   20169,7567482623,   0,   0
128:                    128,      0,  685814,    1283,1265826932,   0,   0
256:                    256,      0,   13940,   15955,1578489466,   0,   0
512:                    512,      0,    8187,    6170,1112245854,   0,   0
1024:                  1024,      0,     122,    1514,149252478,   0,   0
2048:                  2048,      0,     428,    1280,363653023,   0,   0
4096:                  4096,      0,    1160,    2870,286123404,   0,   0
Files:                   80,      0,     123,    1407,283213419,   0,   0
rl_entry:                40,      0,    1116,     396,    1116,   0,   0
TURNSTILE:              136,      0,    1774,     186,    1774,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1192,      0,      72,    1077,26252004,   0,   0
THREAD:                1160,      0,    1615,     158,    3048,   0,   0
SLEEPQUEUE:              80,      0,    1774,     140,    1774,   0,   0
VMSPACE:                392,      0,      44,    1376,26251973,   0,   0
cpuset:                  72,      0,     495,     405,    1004,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,    1023,    1153,283692698,   0,   0
mbuf:                   256,      0,    1079,    4365,11191284667,   0,   0
mbuf_cluster:          2048,  25600,    2179,    1067,700510937,   0,   0
mbuf_jumbo_page:       4096,  12800,       0,     724,224911333,   0,   0
mbuf_jumbo_9k:         9216,   6400,    1028,    2321,4627626515,14450,   0
mbuf_jumbo_16k:       16384,   3200,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,     197,    1468,2643948400,   0,   0
ttyinq:                 160,      0,     240,     216,     435,   0,   0
ttyoutq:                256,      0,     126,     114,     229,   0,   0
ata_request:            328,      0,       0,       0,       0,   0,   0
ata_composite:          336,      0,       0,       0,       0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,    1224,  453062,   0,   0
VNODE:                  504,      0,    5912,    3080,   61328,   0,   0
VNODEPOLL:              112,      0,       0,       0,       0,   0,   0
S VFS Cache:            108,      0,   11731,    6914,16497555,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,     121,     119,     151,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NAMEI:                 1024,      0,       0,    1344,753866169,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0
NCLNODE:                568,      0,       0,       0,       0,   0,   0
Mountpoints:            824,      0,       6,      42,      12,   0,   0
pipe:                   728,      0,      15,    1090,32878248,   0,   0
space_seg_cache:         64,      0,  501039,  214641,247485326,   0,   0
zio_cache:              944,      0,    3546,    7902,4177627377,   0,   0
zio_link_cache:          48,      0,    3560,    8752,3846552305,   0,   0
sa_cache:                80,      0,    5843,    6622,   61509,   0,   0
dnode_t:                856,      0,    8368,    3712,75667990,   0,   0
dmu_buf_impl_t:         224,      0,   49435,   41685,368238154,   0,   0
arc_buf_hdr_t:          216,      0, 4941183,  708513,244460588,   0,   0
arc_buf_t:               72,      0,   98651,   13349,404810190,   0,   0
zil_lwb_cache:          192,      0,      73,    2067,63434345,   0,   0
zfs_znode_cache:        368,      0,    5854,    3456,   61229,   0,   0
ksiginfo:               112,      0,    1137,     909,   30860,   0,   0
itimer:                 344,      0,       0,      33,       2,   0,   0
KNOTE:                  128,      0,       0,     754,  135673,   0,   0
socket:                 680,  25602,      57,     495,  352847,   0,   0
unpcb:                  240,  25600,      10,     294,   72374,   0,   0
ipq:                     56,    819,       0,       0,       0,   0,   0
udp_inpcb:              392,  25600,      18,     632,  241152,   0,   0
udpcb:                   16,  25704,      18,    1662,  241152,   0,   0
tcp_inpcb:              392,  25600,      31,     529,   39306,   0,   0
tcpcb:                  976,  25600,      29,     331,   39306,   0,   0
tcptw:                   72,   5150,       2,     598,   10565,   0,   0
syncache:               152,  15375,       0,     250,    7366,   0,   0
hostcache:              136,  15372,       4,     192,      18,   0,   0
tcpreass:                40,   1680,       0,    1260,  217230,   0,   0
sackhole:                32,      0,       0,    1111,  228371,   0,   0
sctp_ep:               1384,  25600,       0,       0,       0,   0,   0
sctp_asoc:             2288,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80064,       0,     288,       7,   0,   0
sctp_raddr:             704,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400008,       0,       0,       0,   0,   0
sctp_readq:             104, 400032,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400032,       0,       0,       0,   0,   0
sctp_asconf:             40, 400008,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400032,       0,       0,       0,   0,   0
ripcb:                  392,  25600,       0,     100,      10,   0,   0
rtentry:                200,      0,      27,     125,      27,   0,   0
selfd:                   56,      0,    1561,    1022,2896510487,   0,   0
SWAPMETA:               288, 1009008,     587,     297,    5394,   0,   0

User23 · Nov 15, 2013

Code:

last pid: 79212;  load averages:  1.01,  1.20,  1.06                                          up 11+12:47:17  11:52:57
3721 processes:3 running, 3718 sleeping
CPU:  1.8% user,  0.0% nice,  3.8% system,  0.3% interrupt, 94.1% idle
Mem: 4324M Active, 3186M Inact, 19G Wired, 3162M Buf, 4950M Free
ARC: 4763M Total, 1943M MFU, 221M MRU, 2045K Anon, 1209M Header, 1465M Other
Swap: 8192M Total, 8192M Free

Code:

# vmstat -z output
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     197,       7,     197,   0,   0
UMA Zones:             1920,      0,     197,       1,     197,   0,   0
UMA Slabs:              568,      0,  454935,  299567,980797209,   0,   0
UMA RCntSlabs:          568,      0,   10750,       2,   10750,   0,   0
UMA Hash:               256,      0,      78,      12,      82,   0,   0
16 Bucket:              152,      0,     247,       3,     250,   0,   0
32 Bucket:              280,      0,     446,       2,     467,   0,   0
64 Bucket:              536,      0,     595,       0,     600, 116,   0
128 Bucket:            1048,      0,  167002,       2,  171956,590167,   0
VM OBJECT:              232,      0,  409880,   53080,759122563,   0,   0
MAP:                    232,      0,       8,      24,       8,   0,   0
KMAP ENTRY:             120, 1063641,   82427,   33110,2427352700,   0,   0
MAP ENTRY:              120,      0,  101284,   13974,1333783984,   0,   0
fakepg:                 120,      0,       0,      62,      22,   0,   0
mt_zone:               4112,      0,     241,      30,     241,   0,   0
16:                      16,      0,   52452,   79932,1432220872,   0,   0
32:                      32,      0,10767152,   14194,333873690,   0,   0
64:                      64,      0,  333234,  777134,6697646135,   0,   0
128:                    128,      0, 2021223,  243184,2259051728,   0,   0
256:                    256,      0,   98838,   72402,1799273484,   0,   0
512:                    512,      0, 1151700, 5034690,3715054440,   0,   0
1024:                  1024,      0,    7997,   28003,138173079,   0,   0
2048:                  2048,      0,   15458,    8132,2033938003,   0,   0
4096:                  4096,      0,   58744,  168447,304563970,   0,   0
Files:                   80,      0,   34728,    5592,723920888,   0,   0
TURNSTILE:              136,      0,    6154,     166,    6154,   0,   0
rl_entry:                40,      0,    4859,     769,    4859,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1192,      0,    3774,    1917,20261126,   0,   0
THREAD:                1160,      0,    5968,     185,    6139,   0,   0
SLEEPQUEUE:              80,      0,    6154,     284,    6154,   0,   0
VMSPACE:                392,      0,    3752,    2648,20261111,   0,   0
cpuset:                  72,      0,     302,     598,     490,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,   10421,    4043,1957160288,   0,   0
mbuf:                   256,      0,    1153,    4483,6615599997,   0,   0
mbuf_cluster:          2048,  25600,   14474,    4600,569773322,   0,   0
mbuf_jumbo_page:       4096,  12800,      28,    1185,210815427,   0,   0
mbuf_jumbo_9k:         9216,   6400,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,   3200,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,      51,   30354,3667270826,   0,   0
ttyinq:                 160,      0,     135,     801,    6060,   0,   0
ttyoutq:                256,      0,      72,     558,    3146,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0
VNODE:                  504,      0,  439049,   63607,1185833386,   0,   0
VNODEPOLL:              112,      0,    2877,    1974, 1167605,   0,   0
NAMEI:                 1024,      0,       0,     380,2427214002,   0,   0
S VFS Cache:            108,      0,  318080,  129466,137083558,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,  116834,  131638,1026140651,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NFSMOUNT:               632,      0,       0,       0,       0,   0,   0
NFSNODE:                656,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,    5918,     666,    8576,   0,   0
Mountpoints:            824,      0,       6,      50,       9,   0,   0
pipe:                   728,      0,    3798,    1642,19580435,   0,   0
ksiginfo:               112,      0,    5680,    1745,  859533,   0,   0
itimer:                 344,      0,       1,      21,       1,   0,   0
pfsrctrpl:              152,  10000,       0,       0,       0,   0,   0
pfrulepl:               936,      0,      63,     241,     434,   0,   0
pfstatepl:              288,  10010,    4069,    5941,28304722,4629,   0
pfstatekeypl:           288,      0,    4069,    6435,28304722,   0,   0
pfstateitempl:          288,      0,    4069,    6643,28304722,   0,   0
pfaltqpl:               240,      0,       0,       0,       0,   0,   0
pfpooladdrpl:            88,      0,       0,       0,       0,   0,   0
pfrktable:             1296,   1002,       0,       0,       0,   0,   0
pfrkentry:              160, 200016,       0,       0,       0,   0,   0
pfrkcounters:            64,      0,       0,       0,       0,   0,   0
pffrent:                 32,   5050,       0,       0,       0,   0,   0
pffrag:                  80,      0,       0,       0,       0,   0,   0
pffrcache:               80,  10035,       0,       0,       0,   0,   0
pffrcent:                24,  50022,       0,       0,       0,   0,   0
pfstatescrub:            40,      0,       0,       0,       0,   0,   0
pfiaddrpl:              120,      0,       0,       0,       0,   0,   0
pfospfen:               112,      0,     710,      16,    4970,   0,   0
pfosfp:                  40,      0,     420,      84,    2940,   0,   0
KNOTE:                  128,      0,   25286,    6469,1922518344,   0,   0
socket:                 680,  25602,   12580,    3722,149393700,   0,   0
unpcb:                  240,  25600,   10424,    3256,118822451,   0,   0
ipq:                     56,    819,       0,     819,20174256, 329,   0
udp_inpcb:              392,  25600,      32,     978,20839999,   0,   0
udpcb:                   16,  25704,      32,    2656,20839999,   0,   0
tcp_inpcb:              392,  25600,    2600,    2460, 9731211,   0,   0
tcpcb:                  976,  25600,    2123,    2097, 9731211,   0,   0
tcptw:                   72,   5150,     477,    2773, 7087191,   0,   0
syncache:               152,  15375,       0,    1275, 9666020,   0,   0
hostcache:              136,  15372,    1601,    1619,  124154,   0,   0
tcpreass:                40,   1680,       4,    1676, 1324816,  25,   0
sackhole:                32,      0,       0,    2020, 1497427,   0,   0
ripcb:                  392,  25600,       0,     120,      14,   0,   0
rtentry:                200,      0,      34,     251,      40,   0,   0
selfd:                   56,      0,   11326,    1967,113813395,   0,   0

User23 · Nov 15, 2013

Code:

SWAPMETA:               288, 4057911,       0,       0,       0,   0,   0
FFS inode:              168,      0,  260827,  149781, 8311805,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,  260827,  149918, 8311771,   0,   0
taskq_zone:              48,      0,       0,    2592, 1483331,   0,   0
space_seg_cache:         64,      0,  559444,   48436,86232641,   0,   0
zio_cache:              944,      0,      51,   97873,4365447262,   0,   0
zio_link_cache:          48,      0,      38,  106234,3339013853,   0,   0
zio_buf_512:            512,      0,       0,       0,       0,   0,   0
zio_data_buf_512:       512,      0,       0,       0,       0,   0,   0
zio_buf_1024:          1024,      0,       0,       0,       0,   0,   0
zio_data_buf_1024:     1024,      0,       0,       0,       0,   0,   0
zio_buf_1536:          1536,      0,       0,       0,       0,   0,   0
zio_data_buf_1536:     1536,      0,       0,       0,       0,   0,   0
zio_buf_2048:          2048,      0,       0,       0,       0,   0,   0
zio_data_buf_2048:     2048,      0,       0,       0,       0,   0,   0
zio_buf_2560:          2560,      0,       0,       0,       0,   0,   0
zio_data_buf_2560:     2560,      0,       0,       0,       0,   0,   0
zio_buf_3072:          3072,      0,       0,       0,       0,   0,   0
zio_data_buf_3072:     3072,      0,       0,       0,       0,   0,   0
zio_buf_3584:          3584,      0,       0,       0,       0,   0,   0
zio_data_buf_3584:     3584,      0,       0,       0,       0,   0,   0
zio_buf_4096:          4096,      0,       0,       0,       0,   0,   0
zio_data_buf_4096:     4096,      0,       0,       0,       0,   0,   0
zio_buf_5120:          5120,      0,       0,       0,       0,   0,   0
zio_data_buf_5120:     5120,      0,       0,       0,       0,   0,   0
zio_buf_6144:          6144,      0,       0,       0,       0,   0,   0
zio_data_buf_6144:     6144,      0,       0,       0,       0,   0,   0
zio_buf_7168:          7168,      0,       0,       0,       0,   0,   0
zio_data_buf_7168:     7168,      0,       0,       0,       0,   0,   0
zio_buf_8192:          8192,      0,       0,       0,       0,   0,   0
zio_data_buf_8192:     8192,      0,       0,       0,       0,   0,   0
zio_buf_10240:        10240,      0,       0,       0,       0,   0,   0
zio_data_buf_10240:   10240,      0,       0,       0,       0,   0,   0
zio_buf_12288:        12288,      0,       0,       0,       0,   0,   0
zio_data_buf_12288:   12288,      0,       0,       0,       0,   0,   0
zio_buf_14336:        14336,      0,       0,       0,       0,   0,   0
zio_data_buf_14336:   14336,      0,       0,       0,       0,   0,   0
zio_buf_16384:        16384,      0,       0,       0,       0,   0,   0
zio_data_buf_16384:   16384,      0,       0,       0,       0,   0,   0
zio_buf_20480:        20480,      0,       0,       0,       0,   0,   0
zio_data_buf_20480:   20480,      0,       0,       0,       0,   0,   0
zio_buf_24576:        24576,      0,       0,       0,       0,   0,   0
zio_data_buf_24576:   24576,      0,       0,       0,       0,   0,   0
zio_buf_28672:        28672,      0,       0,       0,       0,   0,   0
zio_data_buf_28672:   28672,      0,       0,       0,       0,   0,   0
zio_buf_32768:        32768,      0,       0,       0,       0,   0,   0
zio_data_buf_32768:   32768,      0,       0,       0,       0,   0,   0
zio_buf_36864:        36864,      0,       0,       0,       0,   0,   0
zio_data_buf_36864:   36864,      0,       0,       0,       0,   0,   0
zio_buf_40960:        40960,      0,       0,       0,       0,   0,   0
zio_data_buf_40960:   40960,      0,       0,       0,       0,   0,   0
zio_buf_45056:        45056,      0,       0,       0,       0,   0,   0
zio_data_buf_45056:   45056,      0,       0,       0,       0,   0,   0
zio_buf_49152:        49152,      0,       0,       0,       0,   0,   0
zio_data_buf_49152:   49152,      0,       0,       0,       0,   0,   0
zio_buf_53248:        53248,      0,       0,       0,       0,   0,   0
zio_data_buf_53248:   53248,      0,       0,       0,       0,   0,   0
zio_buf_57344:        57344,      0,       0,       0,       0,   0,   0
zio_data_buf_57344:   57344,      0,       0,       0,       0,   0,   0
zio_buf_61440:        61440,      0,       0,       0,       0,   0,   0
zio_data_buf_61440:   61440,      0,       0,       0,       0,   0,   0
zio_buf_65536:        65536,      0,       0,       0,       0,   0,   0
zio_data_buf_65536:   65536,      0,       0,       0,       0,   0,   0
zio_buf_69632:        69632,      0,       0,       0,       0,   0,   0
zio_data_buf_69632:   69632,      0,       0,       0,       0,   0,   0
zio_buf_73728:        73728,      0,       0,       0,       0,   0,   0
zio_data_buf_73728:   73728,      0,       0,       0,       0,   0,   0
zio_buf_77824:        77824,      0,       0,       0,       0,   0,   0
zio_data_buf_77824:   77824,      0,       0,       0,       0,   0,   0
zio_buf_81920:        81920,      0,       0,       0,       0,   0,   0
zio_data_buf_81920:   81920,      0,       0,       0,       0,   0,   0
zio_buf_86016:        86016,      0,       0,       0,       0,   0,   0
zio_data_buf_86016:   86016,      0,       0,       0,       0,   0,   0
zio_buf_90112:        90112,      0,       0,       0,       0,   0,   0
zio_data_buf_90112:   90112,      0,       0,       0,       0,   0,   0
zio_buf_94208:        94208,      0,       0,       0,       0,   0,   0
zio_data_buf_94208:   94208,      0,       0,       0,       0,   0,   0
zio_buf_98304:        98304,      0,       0,       0,       0,   0,   0
zio_data_buf_98304:   98304,      0,       0,       0,       0,   0,   0
zio_buf_102400:      102400,      0,       0,       0,       0,   0,   0
zio_data_buf_102400: 102400,      0,       0,       0,       0,   0,   0
zio_buf_106496:      106496,      0,       0,       0,       0,   0,   0
zio_data_buf_106496: 106496,      0,       0,       0,       0,   0,   0
zio_buf_110592:      110592,      0,       0,       0,       0,   0,   0
zio_data_buf_110592: 110592,      0,       0,       0,       0,   0,   0
zio_buf_114688:      114688,      0,       0,       0,       0,   0,   0
zio_data_buf_114688: 114688,      0,       0,       0,       0,   0,   0
zio_buf_118784:      118784,      0,       0,       0,       0,   0,   0
zio_data_buf_118784: 118784,      0,       0,       0,       0,   0,   0
zio_buf_122880:      122880,      0,       0,       0,       0,   0,   0
zio_data_buf_122880: 122880,      0,       0,       0,       0,   0,   0
zio_buf_126976:      126976,      0,       0,       0,       0,   0,   0
zio_data_buf_126976: 126976,      0,       0,       0,       0,   0,   0
zio_buf_131072:      131072,      0,       0,       0,       0,   0,   0
zio_data_buf_131072: 131072,      0,       0,       0,       0,   0,   0
sa_cache:                80,      0,  178178,   64912,1177516606,   0,   0
dnode_t:                856,      0, 1138457, 4920519,820265718,   0,   0
dmu_buf_impl_t:         224,      0, 1266215, 5339883,1254850926,   0,   0
arc_buf_hdr_t:          216,      0, 5434480,    1214,88435454,   0,   0
arc_buf_t:               72,      0,  170049,  573351,461319188,   0,   0
zil_lwb_cache:          192,      0,      11,    1709, 2696135,   0,   0
zfs_znode_cache:        368,      0,  178149,   64701,1177516577,   0,   0

User23 · Nov 15, 2013

The server is in use since the begin of week 45.

The Legend is wrong.
M= GB
k= MB
= KB

belon_cfy · Nov 15, 2013

User23 said:

Code:

last pid: 79212;  load averages:  1.01,  1.20,  1.06                                          up 11+12:47:17  11:52:57
3721 processes:3 running, 3718 sleeping
CPU:  1.8% user,  0.0% nice,  3.8% system,  0.3% interrupt, 94.1% idle
Mem: 4324M Active, 3186M Inact, 19G Wired, 3162M Buf, 4950M Free
ARC: 4763M Total, 1943M MFU, 221M MRU, 2045K Anon, 1209M Header, 1465M Other
Swap: 8192M Total, 8192M Free

Code:

# vmstat -z output
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     197,       7,     197,   0,   0
UMA Zones:             1920,      0,     197,       1,     197,   0,   0
UMA Slabs:              568,      0,  454935,  299567,980797209,   0,   0
UMA RCntSlabs:          568,      0,   10750,       2,   10750,   0,   0
UMA Hash:               256,      0,      78,      12,      82,   0,   0
16 Bucket:              152,      0,     247,       3,     250,   0,   0
32 Bucket:              280,      0,     446,       2,     467,   0,   0
64 Bucket:              536,      0,     595,       0,     600, 116,   0
128 Bucket:            1048,      0,  167002,       2,  171956,590167,   0
VM OBJECT:              232,      0,  409880,   53080,759122563,   0,   0
MAP:                    232,      0,       8,      24,       8,   0,   0
KMAP ENTRY:             120, 1063641,   82427,   33110,2427352700,   0,   0
MAP ENTRY:              120,      0,  101284,   13974,1333783984,   0,   0
fakepg:                 120,      0,       0,      62,      22,   0,   0
mt_zone:               4112,      0,     241,      30,     241,   0,   0
16:                      16,      0,   52452,   79932,1432220872,   0,   0
32:                      32,      0,10767152,   14194,333873690,   0,   0
64:                      64,      0,  333234,  777134,6697646135,   0,   0
128:                    128,      0, 2021223,  243184,2259051728,   0,   0
256:                    256,      0,   98838,   72402,1799273484,   0,   0
512:                    512,      0, 1151700, 5034690,3715054440,   0,   0
1024:                  1024,      0,    7997,   28003,138173079,   0,   0
2048:                  2048,      0,   15458,    8132,2033938003,   0,   0
4096:                  4096,      0,   58744,  168447,304563970,   0,   0
Files:                   80,      0,   34728,    5592,723920888,   0,   0
TURNSTILE:              136,      0,    6154,     166,    6154,   0,   0
rl_entry:                40,      0,    4859,     769,    4859,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1192,      0,    3774,    1917,20261126,   0,   0
THREAD:                1160,      0,    5968,     185,    6139,   0,   0
SLEEPQUEUE:              80,      0,    6154,     284,    6154,   0,   0
VMSPACE:                392,      0,    3752,    2648,20261111,   0,   0
cpuset:                  72,      0,     302,     598,     490,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,   10421,    4043,1957160288,   0,   0
mbuf:                   256,      0,    1153,    4483,6615599997,   0,   0
mbuf_cluster:          2048,  25600,   14474,    4600,569773322,   0,   0
mbuf_jumbo_page:       4096,  12800,      28,    1185,210815427,   0,   0
mbuf_jumbo_9k:         9216,   6400,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,   3200,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,      51,   30354,3667270826,   0,   0
ttyinq:                 160,      0,     135,     801,    6060,   0,   0
ttyoutq:                256,      0,      72,     558,    3146,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0
VNODE:                  504,      0,  439049,   63607,1185833386,   0,   0
VNODEPOLL:              112,      0,    2877,    1974, 1167605,   0,   0
NAMEI:                 1024,      0,       0,     380,2427214002,   0,   0
S VFS Cache:            108,      0,  318080,  129466,137083558,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,  116834,  131638,1026140651,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NFSMOUNT:               632,      0,       0,       0,       0,   0,   0
NFSNODE:                656,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,    5918,     666,    8576,   0,   0
Mountpoints:            824,      0,       6,      50,       9,   0,   0
pipe:                   728,      0,    3798,    1642,19580435,   0,   0
ksiginfo:               112,      0,    5680,    1745,  859533,   0,   0
itimer:                 344,      0,       1,      21,       1,   0,   0
pfsrctrpl:              152,  10000,       0,       0,       0,   0,   0
pfrulepl:               936,      0,      63,     241,     434,   0,   0
pfstatepl:              288,  10010,    4069,    5941,28304722,4629,   0
pfstatekeypl:           288,      0,    4069,    6435,28304722,   0,   0
pfstateitempl:          288,      0,    4069,    6643,28304722,   0,   0
pfaltqpl:               240,      0,       0,       0,       0,   0,   0
pfpooladdrpl:            88,      0,       0,       0,       0,   0,   0
pfrktable:             1296,   1002,       0,       0,       0,   0,   0
pfrkentry:              160, 200016,       0,       0,       0,   0,   0
pfrkcounters:            64,      0,       0,       0,       0,   0,   0
pffrent:                 32,   5050,       0,       0,       0,   0,   0
pffrag:                  80,      0,       0,       0,       0,   0,   0
pffrcache:               80,  10035,       0,       0,       0,   0,   0
pffrcent:                24,  50022,       0,       0,       0,   0,   0
pfstatescrub:            40,      0,       0,       0,       0,   0,   0
pfiaddrpl:              120,      0,       0,       0,       0,   0,   0
pfospfen:               112,      0,     710,      16,    4970,   0,   0
pfosfp:                  40,      0,     420,      84,    2940,   0,   0
KNOTE:                  128,      0,   25286,    6469,1922518344,   0,   0
socket:                 680,  25602,   12580,    3722,149393700,   0,   0
unpcb:                  240,  25600,   10424,    3256,118822451,   0,   0
ipq:                     56,    819,       0,     819,20174256, 329,   0
udp_inpcb:              392,  25600,      32,     978,20839999,   0,   0
udpcb:                   16,  25704,      32,    2656,20839999,   0,   0
tcp_inpcb:              392,  25600,    2600,    2460, 9731211,   0,   0
tcpcb:                  976,  25600,    2123,    2097, 9731211,   0,   0
tcptw:                   72,   5150,     477,    2773, 7087191,   0,   0
syncache:               152,  15375,       0,    1275, 9666020,   0,   0
hostcache:              136,  15372,    1601,    1619,  124154,   0,   0
tcpreass:                40,   1680,       4,    1676, 1324816,  25,   0
sackhole:                32,      0,       0,    2020, 1497427,   0,   0
ripcb:                  392,  25600,       0,     120,      14,   0,   0
rtentry:                200,      0,      34,     251,      40,   0,   0
selfd:                   56,      0,   11326,    1967,113813395,   0,   0

Seems your 32 usage is similar to my FreeBSD 9.2 server, previously it was 10 times lower on FreeBSD 9.1 with more than 40 days uptime. I don't know whether will it cause any issue on the server or not.

FreeBSD 9.1

Code:

ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     183,       4,     183,   0,   0
UMA Zones:              896,      0,     183,       1,     183,   0,   0
UMA Slabs:              568,      0,  126840,  146496,5974622916,   0,   0
UMA RCntSlabs:          568,      0,    9336,       2,    9336,   0,   0
UMA Hash:               256,      0,      81,       9,      81,   0,   0
16 Bucket:              152,      0,     151,      24,     152,   0,   0
32 Bucket:              280,      0,     173,       9,     173,   2,   0
64 Bucket:              536,      0,     204,       6,     204, 104,   0
128 Bucket:            1048,      0,   60106,       2,   80405,513331,   0
VM OBJECT:              232,      0,   53448,    1448,688477684,   0,   0
MAP:                    232,      0,       7,      25,       7,   0,   0
KMAP ENTRY:             120, 277171,   21338,   45622,16142747226,   0,   0
MAP ENTRY:              120,      0,    1030,    2132,1706681438,   0,   0
fakepg:                 120,      0,      10,     362,  306465,   0,   0
mt_zone:               4112,      0,     332,      11,     332,   0,   0
16:                      16,      0, 1830776, 1373992,2929906167,   0,   0
32:                      32,      0, 1789982, 1437473,4077616578,   0,   0
64:                      64,      0,  468725,  516875,23034375543,   0,   0
128:                    128,      0,   36209,   43860,4070754252,   0,   0
256:                    256,      0,   50342,   64918,4880964059,   0,   0
512:                    512,      0,   65997,  318345,2791782015,   0,   0

Today the FreeBSD 9.2 server NFS suddenly stop working and result to all the servers hosted on it were down (It was happened before within 7-14 days an above). I have no idea whether is it related to the memory issue? Possibly LZ4 compression? The server uptime is only 8 days.

Previously (Last week) it was able to run for more than 100 days on FreeBSD 9.0 and 9.1 before switching to FreeBSD 9.2.

User23 · Nov 15, 2013

belon_cfy said:
Possibly LZ4 compression?

Dedup or compression is not enabled on my server.

Solved Memory leak on FreeBSD 9.3 and FreeBSD 10?

Administrator