Solved Memory leak on FreeBSD 9.3 and FreeBSD 10?

Latest update: https://forums.freebsd.org/viewtopic.php?f=48&t=41880&start=25#p249468


Hi,

I have a server running on 8 GB of ram RAM that serves as a NFS. However the memory is exhausted significantly faster than with the rest of the servers with the same configuration, but running on FreeBSD 9.1 and below. Currently the server has only 300 MB left but others with the same configuration have more than 1 GB left.

vmstat is showing the "32" used up to 6,750,713 within eight days, but all of my servers are below 2,000,000 with 400 days of uptime.

Anything abnormal on the server?

Server specification: Supermicro Xeon 1230v2, 8GB Ram 8 GB RAM, 4 X 2 TB HDD with SSD as ZIL and L2ARC, FreeBSD 9.2-RC3 running ZFS.

My etc/rc.conf:
Code:
geli_devices="gpt/data_disk0 gpt/data_disk1 gpt/data_disk2 gpt/data_disk3 gpt/l2arc_disk0 gpt/slog_disk0"
geli_swap_flags="-e AES-CBC -l 256 -s 4096 -d"
geli_default_flags="-p -k /boot/keys/disks.key"
geli_autodetach="NO"

sshd_enable="YES"

# NFS
nfsv4_server_enable="YES"
rpcbind_enable="YES"
nfs_server_enable="YES"
nfs_server_flags="-u -t -n 64"
mountd_flags="-r"
rpc_lockd_enable="YES"
rpc_statd_enable="YES"

zfs_enable="YES"

/boot/loader.conf:
Code:
zfs_load="YES"
aesni_load="YES"
vfs.zfs.cache_flush_disable=1
vfs.root.mountfrom="zfs:zroot"

vfs.zfs.arc_max="4G"
vfs.zfs.write_limit_override="2G"

/etc/sysctl.conf:
Code:
vfs.zfs.l2arc_noprefetch=0
vfs.zfs.l2arc_feed_min_ms=1000
vfs.zfs.l2arc_norw=0
vfs.zfs.l2arc_feed_again=0
 
vmstat part 1:

Code:
# vmstat -z
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     187,       0,     187,   0,   0
UMA Zones:             1408,      0,     187,       1,     187,   0,   0
UMA Slabs:              568,      0,  182851,    7262,1655359382,   0,   0
UMA RCntSlabs:          568,      0,    9932,       1,  191297,   0,   0
UMA Hash:               256,      0,      79,      11,      82,   0,   0
16 Bucket:              152,      0,      42,     158,     218,   0,   0
32 Bucket:              280,      0,     142,     166,     410,   1,   0
64 Bucket:              536,      0,     194,      58,     563,  96,   0
128 Bucket:            1048,      0,   10637,       1,  341785,271526,   0
VM OBJECT:              232,      0,    3696,    1616,198040921,   0,   0
MAP:                    232,      0,       8,      24,       8,   0,   0
KMAP ENTRY:             120, 276706,    7173,   25749,4537350748,   0,   0
MAP ENTRY:              120,      0,    1239,    1985,478584526,   0,   0
fakepg:                 120,      0,      10,    1106,   43970,   0,   0
mt_zone:               4112,      0,     329,      44,     329,   0,   0
16:                      16,      0,    6707,    7909,516746799,   0,   0
32:                      32,      0, 6750713, 4857722,842972203,   0,   0
64:                      64,      0,   19919,   17097,4319447531,   0,   0
128:                    128,      0,  382970,    5398,809071624,   0,   0
256:                    256,      0,    7029,   15321,1003894827,   0,   0
512:                    512,      0,    4163,    9795,569112985,   0,   0
1024:                  1024,      0,     135,    1621,86442108,   0,   0
2048:                  2048,      0,     232,    1852,453188505,   0,   0
4096:                  4096,      0,     933,    2987,232332977,   0,   0
Files:                   80,      0,     122,    1453,80400534,   0,   0
rl_entry:                40,      0,     558,     198,     558,   0,   0
TURNSTILE:              136,      0,    1231,      89,    1231,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1192,      0,      71,     523, 6902043,   0,   0
THREAD:                1160,      0,    1031,     199,    1835,   0,   0
SLEEPQUEUE:              80,      0,    1231,     161,    1231,   0,   0
VMSPACE:                392,      0,      44,    1066, 6902005,   0,   0
cpuset:                  72,      0,     465,     435,    1204,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,    1023,    1409,359086743,   0,   0
mbuf:                   256,      0,    1069,    7134,4857962764,   0,   0
mbuf_cluster:          2048,  25600,    2432,    1870,861391498,   0,   0
mbuf_jumbo_page:       4096,  12800,       0,    1381,274887034,   0,   0
mbuf_jumbo_9k:         9216,   6400,    1031,    5369,1383502445,94030,   0
mbuf_jumbo_16k:       16384,   3200,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,       5,    1675,2446699117,   0,   0
ttyinq:                 160,      0,     240,     384,    1110,   0,   0
ttyoutq:                256,      0,     126,     249,     582,   0,   0
ata_request:            328,      0,       0,       0,       0,   0,   0
ata_composite:          336,      0,       0,       0,       0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,    1152,  351156,   0,   0
VNODE:                  504,      0,    2790,     714,   28692,   0,   0
VNODEPOLL:              112,      0,       0,       0,       0,   0,   0
S VFS Cache:            108,      0,    2775,     855, 5606552,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,       6,     126,      65,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NAMEI:                 1024,      0,       0,    1168,209778631,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0
NCLNODE:                568,      0,       0,       0,       0,   0,   0
Mountpoints:            824,      0,       6,      14,       8,   0,   0
pipe:                   728,      0,      14,     506, 9057166,   0,   0
space_seg_cache:         64,      0,  622219,   59469,244206545,   0,   0
zio_cache:              944,      0,       7,   17093,3282984010,   0,   0
zio_link_cache:          48,      0,       3,   17277,3222755314,   0,   0
zio_buf_512:            512,      0,       0,       0,       0,   0,   0
zio_data_buf_512:       512,      0,       0,       0,       0,   0,   0
zio_buf_1024:          1024,      0,       0,       0,       0,   0,   0
zio_data_buf_1024:     1024,      0,       0,       0,       0,   0,   0
zio_buf_1536:          1536,      0,       0,       0,       0,   0,   0
zio_data_buf_1536:     1536,      0,       0,       0,       0,   0,   0
zio_buf_2048:          2048,      0,       0,       0,       0,   0,   0
zio_data_buf_2048:     2048,      0,       0,       0,       0,   0,   0
zio_buf_2560:          2560,      0,       0,       0,       0,   0,   0
zio_data_buf_2560:     2560,      0,       0,       0,       0,   0,   0
zio_buf_3072:          3072,      0,       0,       0,       0,   0,   0
zio_data_buf_3072:     3072,      0,       0,       0,       0,   0,   0
zio_buf_3584:          3584,      0,       0,       0,       0,   0,   0
zio_data_buf_3584:     3584,      0,       0,       0,       0,   0,   0
zio_buf_4096:          4096,      0,       0,       0,       0,   0,   0
zio_data_buf_4096:     4096,      0,       0,       0,       0,   0,   0
zio_buf_5120:          5120,      0,       0,       0,       0,   0,   0
zio_data_buf_5120:     5120,      0,       0,       0,       0,   0,   0
zio_buf_6144:          6144,      0,       0,       0,       0,   0,   0
zio_data_buf_6144:     6144,      0,       0,       0,       0,   0,   0
zio_buf_7168:          7168,      0,       0,       0,       0,   0,   0
zio_data_buf_7168:     7168,      0,       0,       0,       0,   0,   0
zio_buf_8192:          8192,      0,       0,       0,       0,   0,   0
zio_data_buf_8192:     8192,      0,       0,       0,       0,   0,   0
zio_buf_10240:        10240,      0,       0,       0,       0,   0,   0
zio_data_buf_10240:   10240,      0,       0,       0,       0,   0,   0
zio_buf_12288:        12288,      0,       0,       0,       0,   0,   0
zio_data_buf_12288:   12288,      0,       0,       0,       0,   0,   0
zio_buf_14336:        14336,      0,       0,       0,       0,   0,   0
zio_data_buf_14336:   14336,      0,       0,       0,       0,   0,   0
zio_buf_16384:        16384,      0,       0,       0,       0,   0,   0
zio_data_buf_16384:   16384,      0,       0,       0,       0,   0,   0
zio_buf_20480:        20480,      0,       0,       0,       0,   0,   0
zio_data_buf_20480:   20480,      0,       0,       0,       0,   0,   0
zio_buf_24576:        24576,      0,       0,       0,       0,   0,   0
zio_data_buf_24576:   24576,      0,       0,       0,       0,   0,   0
 
vmstat Part 2:

Code:
zio_buf_28672:        28672,      0,       0,       0,       0,   0,   0
zio_data_buf_28672:   28672,      0,       0,       0,       0,   0,   0
zio_buf_32768:        32768,      0,       0,       0,       0,   0,   0
zio_data_buf_32768:   32768,      0,       0,       0,       0,   0,   0
zio_buf_36864:        36864,      0,       0,       0,       0,   0,   0
zio_data_buf_36864:   36864,      0,       0,       0,       0,   0,   0
zio_buf_40960:        40960,      0,       0,       0,       0,   0,   0
zio_data_buf_40960:   40960,      0,       0,       0,       0,   0,   0
zio_buf_45056:        45056,      0,       0,       0,       0,   0,   0
zio_data_buf_45056:   45056,      0,       0,       0,       0,   0,   0
zio_buf_49152:        49152,      0,       0,       0,       0,   0,   0
zio_data_buf_49152:   49152,      0,       0,       0,       0,   0,   0
zio_buf_53248:        53248,      0,       0,       0,       0,   0,   0
zio_data_buf_53248:   53248,      0,       0,       0,       0,   0,   0
zio_buf_57344:        57344,      0,       0,       0,       0,   0,   0
zio_data_buf_57344:   57344,      0,       0,       0,       0,   0,   0
zio_buf_61440:        61440,      0,       0,       0,       0,   0,   0
zio_data_buf_61440:   61440,      0,       0,       0,       0,   0,   0
zio_buf_65536:        65536,      0,       0,       0,       0,   0,   0
zio_data_buf_65536:   65536,      0,       0,       0,       0,   0,   0
zio_buf_69632:        69632,      0,       0,       0,       0,   0,   0
zio_data_buf_69632:   69632,      0,       0,       0,       0,   0,   0
zio_buf_73728:        73728,      0,       0,       0,       0,   0,   0
zio_data_buf_73728:   73728,      0,       0,       0,       0,   0,   0
zio_buf_77824:        77824,      0,       0,       0,       0,   0,   0
zio_data_buf_77824:   77824,      0,       0,       0,       0,   0,   0
zio_buf_81920:        81920,      0,       0,       0,       0,   0,   0
zio_data_buf_81920:   81920,      0,       0,       0,       0,   0,   0
zio_buf_86016:        86016,      0,       0,       0,       0,   0,   0
zio_data_buf_86016:   86016,      0,       0,       0,       0,   0,   0
zio_buf_90112:        90112,      0,       0,       0,       0,   0,   0
zio_data_buf_90112:   90112,      0,       0,       0,       0,   0,   0
zio_buf_94208:        94208,      0,       0,       0,       0,   0,   0
zio_data_buf_94208:   94208,      0,       0,       0,       0,   0,   0
zio_buf_98304:        98304,      0,       0,       0,       0,   0,   0
zio_data_buf_98304:   98304,      0,       0,       0,       0,   0,   0
zio_buf_102400:      102400,      0,       0,       0,       0,   0,   0
zio_data_buf_102400: 102400,      0,       0,       0,       0,   0,   0
zio_buf_106496:      106496,      0,       0,       0,       0,   0,   0
zio_data_buf_106496: 106496,      0,       0,       0,       0,   0,   0
zio_buf_110592:      110592,      0,       0,       0,       0,   0,   0
zio_data_buf_110592: 110592,      0,       0,       0,       0,   0,   0
zio_buf_114688:      114688,      0,       0,       0,       0,   0,   0
zio_data_buf_114688: 114688,      0,       0,       0,       0,   0,   0
zio_buf_118784:      118784,      0,       0,       0,       0,   0,   0
zio_data_buf_118784: 118784,      0,       0,       0,       0,   0,   0
zio_buf_122880:      122880,      0,       0,       0,       0,   0,   0
zio_data_buf_122880: 122880,      0,       0,       0,       0,   0,   0
zio_buf_126976:      126976,      0,       0,       0,       0,   0,   0
zio_data_buf_126976: 126976,      0,       0,       0,       0,   0,   0
zio_buf_131072:      131072,      0,       0,       0,       0,   0,   0
zio_data_buf_131072: 131072,      0,       0,       0,       0,   0,   0
sa_cache:                80,      0,    2722,     833,   28846,   0,   0
dnode_t:                856,      0,    5077,    3231,26308054,   0,   0
dmu_buf_impl_t:         224,      0,   69694,   39327,187063060,   0,   0
arc_buf_hdr_t:          216,      0, 3445591, 2140817,128615322,   0,   0
arc_buf_t:               72,      0,  114515,   43135,263110335,   0,   0
zil_lwb_cache:          192,      0,     320,    3040,60798759,   0,   0
zfs_znode_cache:        368,      0,    2722,     638,   28566,   0,   0
ksiginfo:               112,      0,     583,    1298,   30176,   0,   0
itimer:                 344,      0,       0,      33,       2,   0,   0
KNOTE:                  128,      0,       0,     812,  127524,   0,   0
socket:                 680,  25602,      56,     436,  342232,   0,   0
unpcb:                  240,  25600,      10,     294,   59560,   0,   0
ipq:                     56,    819,       0,       0,       0,   0,   0
udp_inpcb:              392,  25600,      18,     712,  244493,   0,   0
udpcb:                   16,  25704,      18,    1662,  244493,   0,   0
tcp_inpcb:              392,  25600,      28,     332,   38163,   0,   0
tcpcb:                  976,  25600,      28,     368,   38163,   0,   0
tcptw:                   72,   5150,       0,     650,    6910,   0,   0
syncache:               152,  15375,       0,     250,    8831,   0,   0
hostcache:              136,  15372,       4,     164,      17,   0,   0
tcpreass:                40,   1680,       0,    1092,  712835,   0,   0
sackhole:                32,      0,       0,    1111,  280985,   0,   0
sctp_ep:               1384,  25600,       0,       0,       0,   0,   0
sctp_asoc:             2288,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80064,       0,     288,       8,   0,   0
sctp_raddr:             704,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400008,       0,       0,       0,   0,   0
sctp_readq:             104, 400032,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400032,       0,       0,       0,   0,   0
sctp_asconf:             40, 400008,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400032,       0,       0,       0,   0,   0
ripcb:                  392,  25600,       0,      90,      11,   0,   0
rtentry:                200,      0,      29,      66,      29,   0,   0
selfd:                   56,      0,     939,    1014,152725323,   0,   0
SWAPMETA:               288, 1008163,     712,     146,    9232,   0,   0
 
I can confirm that two of my FreeBSD-9.2-RC3 and RC4 having the same issue. The free memory dropped below 1G 1 GB within three days. However. with the same configuration of my new FreeBSD 9.1 server with the same uptime and heavier I/O doesn't suffer from the same behavior, until now it has 2 GB and above free memory. I think some kind of memory leak is happening on my FreeBSD 9.2 servers.

FreeBSD 9.1 - ZFS V28 with compression on
FreeBSD 9.2 - ZFS V5000 with LZ4 compression
 
The biggest question is, what do you consider "free" memory? Keep in mind that FreeBSD will try to use everything, mainly for caching. But this shouldn't cause any problems. Not unless the server starts using swap excessively.
 
"Free memory is wasted memory".

If it's not being used by programs then it might as well be used to cache disk I/O instead.

Perhaps 9.2-RC3+ is simply better at doing this.
 
I'm afraid the memory leak will lead to memory exhaustion and eventually bring down the server since both of the FreeBSD 9.2 servers' memory usage behave unlike the rest of my FreeBSD 9.1.

The ARC is limited to 4 GB however the total memory usage is up to 7 GB and above for an NFS server only.

The swap space has been used, although not that aggressively.
 
Another issue is happening since the free memory is dropped below 400 MB. The server keeps flushing the writeback cache to disks instead of buffering it in RAM and commit every five seconds. Will FreeBSD force to commit write cache to the disk when the memory is lower than 400 MB?

The server has SSD as ZIL.

The similar issue doesn't happen on another server with FreeBSD 9.1 and 2 GB RAM free at equivalent work load.

By the way, I have the following parameters in /boot/loader.conf on both servers.
Code:
vfs.zfs.arc_max="4G"
vfs.zfs.write_limit_override="2G"
 
The server with FreeBSD 9.2-RC4 was down last night and the uptime is about ten days with only 200 MB RAM free. However the other server with FreeBSD 9.1 and the same uptime is still running fine with 2 GB of RAM free.
 
Upgraded to FreeBSD 9.2-Release, problem still persist.

Think I have to switch it back to FreeBSD 9.1 and wait until the issue is resolved.
 
Upgraded to FreeBSD 9.2-p1 and moved some of the virtual machine on it, seems the memory leak issue still there compare to the previous version.

I notice that the FreeBSD 9.2 memory usage of 32: is much higher than FreeBSD 9.1 , both server are hosted exactly the same content.

vmstat -z on FreeBSD 9.1
Code:
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

16:                      16,      0, 1627726,  950402,2742669502,   0,   0
32:                      32,      0, 1649754,  943724,3526384744,   0,   0
64:                      64,      0,  522316,  463284,20497736229,   0,   0
128:                    128,      0,   24113,   55956,3720279158,   0,   0
256:                    256,      0,   54111,   61149,4593010295,   0,   0
512:                    512,      0,   70580,  313762,2177203825,   0,   0
1024:                  1024,      0,     138,   28018,165498646,   0,   0
2048:                  2048,      0,    6364,    7226,1582801956,   0,   0
4096:                  4096,      0,    8789,   13396,1032322949,   0,   0
Files:                   80,      0,     114,     426,213572476,   0,   0


vmstat -z on FreeBSD 9.2
Code:
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

16:                      16,      0,   23240,    3136,436263549,   0,   0
32:                      32,      0, 7646055,    1766,677904734,   0,   0
64:                      64,      0,   26875,   17925,2959077356,   0,   0
128:                    128,      0,  345203,    1057,422898487,   0,   0
256:                    256,      0,   17419,   15176,578761754,   0,   0
512:                    512,      0,   14678,    3900,418832444,   0,   0
1024:                  1024,      0,     146,    4678,41878369,   0,   0
2048:                  2048,      0,     343,    2269,34511702,   0,   0
4096:                  4096,      0,    1360,    5934,63518714,   0,   0
Files:                   80,      0,     129,     771,117024521,   0,   0

zfs volumes
Code:
NAME        USED  AVAIL  RSIZE  COMP   RATIO  ATIME  SYNC      SNAP
vol        1.79T  1.75T    32K  lz4    1.41x  off    standard     0
vol/vol00  1.01T  1.75T    32K  lz4    1.34x  off    standard    30  
vol/vol01   800G  1.75T    32K  lz4    1.49x  off    standard    30

Any idea?
 
Unless you're getting excessive swapping, it is unlikely that there is a memory leak. It may simply be that 9.2 is more aggressive about caching. Is your system locking up or crashing? Are applications failing? Is swap soaring? If not, focusing on the Free value isn't particularly helpful.

Here's my box. Not much free, runs great.
Code:
Mem: 349M Active, 659M Inact, 2672M Wired, 43M Cache, 417M Buf, 197M Free
ARC: 1939M Total, 111M MFU, 1491M MRU, 146K Anon, 30M Header, 306M Other
Swap: 16G Total, 61M Used, 16G Free
 
belon_cfy said:
Yes, eventually it crashed and the service stopped working after a week.

Ok here's what I'd try:

  1. Stay on 9.1? You don't have to debug this problem unless 9.2 has something you need. Wait for 10 to come out and then try that. Maybe the problem will be magically fixed. Maybe you'll have budget to buy more memory ;)
  2. You don't say if you're using i386 or amd64. Apparently i386 has memory issues with ZFS like those you're describing.
  3. On the 9.2-RELEASE-p1 box, if it crashes after 7 days, let it run for 5 days and then start comparing things against the 9.1 box.
    • ps augxwwd: Can you see a process that is consuming all of your RAM?
    • swapinfo: Is your swap being consumed? (2% doesn't count, 80% is an issue.)
    • zfs-stats -A: Is ARC within bounds? (I.e.: if you set it for 2 GB, is it under 2 GB?)
  4. Shot in the dark: are you using ZFS dedup? You might not have enough memory for that. See, for example:
    http://forums.freebsd.org/showthread.php?t=41194
  5. You have some old threads about memory starvation under 8.X, you might want to check /usr/src/sys and make sure the patches that gave you stability then are still applied to the codebase.
 
Most of my servers are FreeBSD 9.1 except this for testing.

Below is the current memory usage, I just don't understand how come the 32 memory usage is much higher which is 10 times than the others on FreeBSD 9.1
Code:
16:                      16,      0,   10903,   15473,855690791,   0,   0
32:                      32,      0, 9388002, 1384456,1213007946,   0,   0
64:                      64,      0,   21092,   23708,5481012925,   0,   0
128:                    128,      0,  629890,   30556,892815193,   0,   0

The server is running FreeBSD 9.2-p1 64bit with no dedupe.

ARC is ok, only 4GB is in used which is same as what I have defined.
Code:
------------------------------------------------------------------------
ZFS Subsystem Report                            Wed Nov 13 10:34:16 2013
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                265.01m
        Recycle Misses:                         35.31m
        Mutex Misses:                           114.99k
        Evict Skips:                            2.15b

ARC Size:                               100.00% 4.00    GiB
        Target Size: (Adaptive)         100.00% 4.00    GiB
        Min Size (Hard Limit):          12.50%  512.00  MiB
        Max Size (High Water):          8:1     4.00    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       6.30%   257.99  MiB
        Frequently Used Cache Size:     93.70%  3.75    GiB

ARC Hash Breakdown:
        Elements Max:                           5.44m
        Elements Current:               87.29%  4.75m
        Collisions:                             267.10m
        Chain Max:                              77
        Chains:                                 131.07k

Now it has 762MB free after 6 days, will keep it run for 2 weeks because previously it crashed after 2 weeks of uptime.
Code:
last pid:  4578;  load averages:  0.22,  0.22,  0.25                                                                                                    up 6+15:04:59  10:35:37
46 processes:  1 running, 43 sleeping, 2 zombie
CPU:  0.2% user,  0.0% nice,  1.9% system,  0.0% interrupt, 97.9% idle
Mem: 7856K Active, 42M Inact, 7069M Wired, 762M Free
ARC: 4097M Total, 2827M MFU, 236M MRU, 21M Anon, 1125M Header, 28M Other
Swap: 8192M Total, 8192M Free
 
I have a nearly similar hardware/software setup and cant see any memory leak related problems.

FreeBSD 9.2-RELEASE #2 r255966, used as NFS Server and other stuff.

Supermicro X9SRi-3F, Xeon E5-2620, 32 GB, 2x raidz2 (5x 1 TB), 60 GB ZIL mirror, 500 GB L2ARC

Code:
load averages:  2.34,  1.53,  1.18                                up 10+13:40:17  12:45:57
3903 processes:2 running, 3901 sleeping
CPU:  1.2% user,  0.0% nice,  4.2% system,  0.3% interrupt, 94.3% idle
Mem: 4429M Active, 3196M Inact, 18G Wired, 3310M Buf, 5564M Free
ARC: 3938M Total, 1522M MFU, 253M MRU, 1488K Anon, 1154M Header, 1079M Other
Swap: 8192M Total, 8192M Free
vfs.zfs.arc_max is limited to 16GB
 
I think it is abnormal to have 5564M free memory left with 10 days uptime and
Code:
vfs.zfs.arc_max=16GB
your system suppose to have more free memory with the same running period on FreeBSD 9.1.

Can you show me your vmstat -z output?

My system with FreeBSD 9.1 with 8 GB RAM and
Code:
vfs.zfs.arc_max=4GB
has 3GB++ free after 10 days. Today it started swapping out some data.

Code:
Mem: 23M Active, 5556K Inact, 7445M Wired, 6924K Cache, 402M Free
ARC: 3922M Total, 2713M MFU, 138M MRU, 29M Anon, 1170M Header, 18M Other
Swap: 8192M Total, 17M Used, 8175M Free

Is there anything occupying the memory?

Code:
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     187,       0,     187,   0,   0
UMA Zones:             1408,      0,     187,       1,     187,   0,   0
UMA Slabs:              568,      0,  181914,    4979,1804741298,   0,   0
UMA RCntSlabs:          568,      0,    5696,    1857,  267446,   0,   0
UMA Hash:               256,      0,      79,      11,      82,   0,   0
16 Bucket:              152,      0,      38,     137,     218,   0,   0
32 Bucket:              280,      0,     113,     153,     376,  20,   0
64 Bucket:              536,      0,     133,     133,     677, 114,   0
128 Bucket:            1048,      0,    1103,      10,  200125,403399,   0
VM OBJECT:              232,      0,    7984,    8096,738552266,   0,   0
MAP:                    232,      0,       8,      24,       8,   0,   0
KMAP ENTRY:             120, 276923,   10931,   22673,4901016283,   0,   0
MAP ENTRY:              120,      0,    1217,    2255,1746936120,   0,   0
fakepg:                 120,      0,      10,     641,    2230,   0,   0
mt_zone:               4112,      0,     329,      44,     329,   0,   0
16:                      16,      0,   10957,   15419,1118302053,   0,   0
32:                      32,      0, 9720077, 1413052,1641641837,   0,   0
64:                      64,      0,   24519,   20169,7567482623,   0,   0
128:                    128,      0,  685814,    1283,1265826932,   0,   0
256:                    256,      0,   13940,   15955,1578489466,   0,   0
512:                    512,      0,    8187,    6170,1112245854,   0,   0
1024:                  1024,      0,     122,    1514,149252478,   0,   0
2048:                  2048,      0,     428,    1280,363653023,   0,   0
4096:                  4096,      0,    1160,    2870,286123404,   0,   0
Files:                   80,      0,     123,    1407,283213419,   0,   0
rl_entry:                40,      0,    1116,     396,    1116,   0,   0
TURNSTILE:              136,      0,    1774,     186,    1774,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1192,      0,      72,    1077,26252004,   0,   0
THREAD:                1160,      0,    1615,     158,    3048,   0,   0
SLEEPQUEUE:              80,      0,    1774,     140,    1774,   0,   0
VMSPACE:                392,      0,      44,    1376,26251973,   0,   0
cpuset:                  72,      0,     495,     405,    1004,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,    1023,    1153,283692698,   0,   0
mbuf:                   256,      0,    1079,    4365,11191284667,   0,   0
mbuf_cluster:          2048,  25600,    2179,    1067,700510937,   0,   0
mbuf_jumbo_page:       4096,  12800,       0,     724,224911333,   0,   0
mbuf_jumbo_9k:         9216,   6400,    1028,    2321,4627626515,14450,   0
mbuf_jumbo_16k:       16384,   3200,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,     197,    1468,2643948400,   0,   0
ttyinq:                 160,      0,     240,     216,     435,   0,   0
ttyoutq:                256,      0,     126,     114,     229,   0,   0
ata_request:            328,      0,       0,       0,       0,   0,   0
ata_composite:          336,      0,       0,       0,       0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,    1224,  453062,   0,   0
VNODE:                  504,      0,    5912,    3080,   61328,   0,   0
VNODEPOLL:              112,      0,       0,       0,       0,   0,   0
S VFS Cache:            108,      0,   11731,    6914,16497555,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,     121,     119,     151,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NAMEI:                 1024,      0,       0,    1344,753866169,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0
NCLNODE:                568,      0,       0,       0,       0,   0,   0
Mountpoints:            824,      0,       6,      42,      12,   0,   0
pipe:                   728,      0,      15,    1090,32878248,   0,   0
space_seg_cache:         64,      0,  501039,  214641,247485326,   0,   0
zio_cache:              944,      0,    3546,    7902,4177627377,   0,   0
zio_link_cache:          48,      0,    3560,    8752,3846552305,   0,   0
sa_cache:                80,      0,    5843,    6622,   61509,   0,   0
dnode_t:                856,      0,    8368,    3712,75667990,   0,   0
dmu_buf_impl_t:         224,      0,   49435,   41685,368238154,   0,   0
arc_buf_hdr_t:          216,      0, 4941183,  708513,244460588,   0,   0
arc_buf_t:               72,      0,   98651,   13349,404810190,   0,   0
zil_lwb_cache:          192,      0,      73,    2067,63434345,   0,   0
zfs_znode_cache:        368,      0,    5854,    3456,   61229,   0,   0
ksiginfo:               112,      0,    1137,     909,   30860,   0,   0
itimer:                 344,      0,       0,      33,       2,   0,   0
KNOTE:                  128,      0,       0,     754,  135673,   0,   0
socket:                 680,  25602,      57,     495,  352847,   0,   0
unpcb:                  240,  25600,      10,     294,   72374,   0,   0
ipq:                     56,    819,       0,       0,       0,   0,   0
udp_inpcb:              392,  25600,      18,     632,  241152,   0,   0
udpcb:                   16,  25704,      18,    1662,  241152,   0,   0
tcp_inpcb:              392,  25600,      31,     529,   39306,   0,   0
tcpcb:                  976,  25600,      29,     331,   39306,   0,   0
tcptw:                   72,   5150,       2,     598,   10565,   0,   0
syncache:               152,  15375,       0,     250,    7366,   0,   0
hostcache:              136,  15372,       4,     192,      18,   0,   0
tcpreass:                40,   1680,       0,    1260,  217230,   0,   0
sackhole:                32,      0,       0,    1111,  228371,   0,   0
sctp_ep:               1384,  25600,       0,       0,       0,   0,   0
sctp_asoc:             2288,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80064,       0,     288,       7,   0,   0
sctp_raddr:             704,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400008,       0,       0,       0,   0,   0
sctp_readq:             104, 400032,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400032,       0,       0,       0,   0,   0
sctp_asconf:             40, 400008,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400032,       0,       0,       0,   0,   0
ripcb:                  392,  25600,       0,     100,      10,   0,   0
rtentry:                200,      0,      27,     125,      27,   0,   0
selfd:                   56,      0,    1561,    1022,2896510487,   0,   0
SWAPMETA:               288, 1009008,     587,     297,    5394,   0,   0
 
Code:
last pid: 79212;  load averages:  1.01,  1.20,  1.06                                          up 11+12:47:17  11:52:57
3721 processes:3 running, 3718 sleeping
CPU:  1.8% user,  0.0% nice,  3.8% system,  0.3% interrupt, 94.1% idle
Mem: 4324M Active, 3186M Inact, 19G Wired, 3162M Buf, 4950M Free
ARC: 4763M Total, 1943M MFU, 221M MRU, 2045K Anon, 1209M Header, 1465M Other
Swap: 8192M Total, 8192M Free


Code:
# vmstat -z output
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     197,       7,     197,   0,   0
UMA Zones:             1920,      0,     197,       1,     197,   0,   0
UMA Slabs:              568,      0,  454935,  299567,980797209,   0,   0
UMA RCntSlabs:          568,      0,   10750,       2,   10750,   0,   0
UMA Hash:               256,      0,      78,      12,      82,   0,   0
16 Bucket:              152,      0,     247,       3,     250,   0,   0
32 Bucket:              280,      0,     446,       2,     467,   0,   0
64 Bucket:              536,      0,     595,       0,     600, 116,   0
128 Bucket:            1048,      0,  167002,       2,  171956,590167,   0
VM OBJECT:              232,      0,  409880,   53080,759122563,   0,   0
MAP:                    232,      0,       8,      24,       8,   0,   0
KMAP ENTRY:             120, 1063641,   82427,   33110,2427352700,   0,   0
MAP ENTRY:              120,      0,  101284,   13974,1333783984,   0,   0
fakepg:                 120,      0,       0,      62,      22,   0,   0
mt_zone:               4112,      0,     241,      30,     241,   0,   0
16:                      16,      0,   52452,   79932,1432220872,   0,   0
32:                      32,      0,10767152,   14194,333873690,   0,   0
64:                      64,      0,  333234,  777134,6697646135,   0,   0
128:                    128,      0, 2021223,  243184,2259051728,   0,   0
256:                    256,      0,   98838,   72402,1799273484,   0,   0
512:                    512,      0, 1151700, 5034690,3715054440,   0,   0
1024:                  1024,      0,    7997,   28003,138173079,   0,   0
2048:                  2048,      0,   15458,    8132,2033938003,   0,   0
4096:                  4096,      0,   58744,  168447,304563970,   0,   0
Files:                   80,      0,   34728,    5592,723920888,   0,   0
TURNSTILE:              136,      0,    6154,     166,    6154,   0,   0
rl_entry:                40,      0,    4859,     769,    4859,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1192,      0,    3774,    1917,20261126,   0,   0
THREAD:                1160,      0,    5968,     185,    6139,   0,   0
SLEEPQUEUE:              80,      0,    6154,     284,    6154,   0,   0
VMSPACE:                392,      0,    3752,    2648,20261111,   0,   0
cpuset:                  72,      0,     302,     598,     490,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,   10421,    4043,1957160288,   0,   0
mbuf:                   256,      0,    1153,    4483,6615599997,   0,   0
mbuf_cluster:          2048,  25600,   14474,    4600,569773322,   0,   0
mbuf_jumbo_page:       4096,  12800,      28,    1185,210815427,   0,   0
mbuf_jumbo_9k:         9216,   6400,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,   3200,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,      51,   30354,3667270826,   0,   0
ttyinq:                 160,      0,     135,     801,    6060,   0,   0
ttyoutq:                256,      0,      72,     558,    3146,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0
VNODE:                  504,      0,  439049,   63607,1185833386,   0,   0
VNODEPOLL:              112,      0,    2877,    1974, 1167605,   0,   0
NAMEI:                 1024,      0,       0,     380,2427214002,   0,   0
S VFS Cache:            108,      0,  318080,  129466,137083558,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,  116834,  131638,1026140651,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NFSMOUNT:               632,      0,       0,       0,       0,   0,   0
NFSNODE:                656,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,    5918,     666,    8576,   0,   0
Mountpoints:            824,      0,       6,      50,       9,   0,   0
pipe:                   728,      0,    3798,    1642,19580435,   0,   0
ksiginfo:               112,      0,    5680,    1745,  859533,   0,   0
itimer:                 344,      0,       1,      21,       1,   0,   0
pfsrctrpl:              152,  10000,       0,       0,       0,   0,   0
pfrulepl:               936,      0,      63,     241,     434,   0,   0
pfstatepl:              288,  10010,    4069,    5941,28304722,4629,   0
pfstatekeypl:           288,      0,    4069,    6435,28304722,   0,   0
pfstateitempl:          288,      0,    4069,    6643,28304722,   0,   0
pfaltqpl:               240,      0,       0,       0,       0,   0,   0
pfpooladdrpl:            88,      0,       0,       0,       0,   0,   0
pfrktable:             1296,   1002,       0,       0,       0,   0,   0
pfrkentry:              160, 200016,       0,       0,       0,   0,   0
pfrkcounters:            64,      0,       0,       0,       0,   0,   0
pffrent:                 32,   5050,       0,       0,       0,   0,   0
pffrag:                  80,      0,       0,       0,       0,   0,   0
pffrcache:               80,  10035,       0,       0,       0,   0,   0
pffrcent:                24,  50022,       0,       0,       0,   0,   0
pfstatescrub:            40,      0,       0,       0,       0,   0,   0
pfiaddrpl:              120,      0,       0,       0,       0,   0,   0
pfospfen:               112,      0,     710,      16,    4970,   0,   0
pfosfp:                  40,      0,     420,      84,    2940,   0,   0
KNOTE:                  128,      0,   25286,    6469,1922518344,   0,   0
socket:                 680,  25602,   12580,    3722,149393700,   0,   0
unpcb:                  240,  25600,   10424,    3256,118822451,   0,   0
ipq:                     56,    819,       0,     819,20174256, 329,   0
udp_inpcb:              392,  25600,      32,     978,20839999,   0,   0
udpcb:                   16,  25704,      32,    2656,20839999,   0,   0
tcp_inpcb:              392,  25600,    2600,    2460, 9731211,   0,   0
tcpcb:                  976,  25600,    2123,    2097, 9731211,   0,   0
tcptw:                   72,   5150,     477,    2773, 7087191,   0,   0
syncache:               152,  15375,       0,    1275, 9666020,   0,   0
hostcache:              136,  15372,    1601,    1619,  124154,   0,   0
tcpreass:                40,   1680,       4,    1676, 1324816,  25,   0
sackhole:                32,      0,       0,    2020, 1497427,   0,   0
ripcb:                  392,  25600,       0,     120,      14,   0,   0
rtentry:                200,      0,      34,     251,      40,   0,   0
selfd:                   56,      0,   11326,    1967,113813395,   0,   0
 
Code:
SWAPMETA:               288, 4057911,       0,       0,       0,   0,   0
FFS inode:              168,      0,  260827,  149781, 8311805,   0,   0
FFS1 dinode:            128,      0,       0,       0,       0,   0,   0
FFS2 dinode:            256,      0,  260827,  149918, 8311771,   0,   0
taskq_zone:              48,      0,       0,    2592, 1483331,   0,   0
space_seg_cache:         64,      0,  559444,   48436,86232641,   0,   0
zio_cache:              944,      0,      51,   97873,4365447262,   0,   0
zio_link_cache:          48,      0,      38,  106234,3339013853,   0,   0
zio_buf_512:            512,      0,       0,       0,       0,   0,   0
zio_data_buf_512:       512,      0,       0,       0,       0,   0,   0
zio_buf_1024:          1024,      0,       0,       0,       0,   0,   0
zio_data_buf_1024:     1024,      0,       0,       0,       0,   0,   0
zio_buf_1536:          1536,      0,       0,       0,       0,   0,   0
zio_data_buf_1536:     1536,      0,       0,       0,       0,   0,   0
zio_buf_2048:          2048,      0,       0,       0,       0,   0,   0
zio_data_buf_2048:     2048,      0,       0,       0,       0,   0,   0
zio_buf_2560:          2560,      0,       0,       0,       0,   0,   0
zio_data_buf_2560:     2560,      0,       0,       0,       0,   0,   0
zio_buf_3072:          3072,      0,       0,       0,       0,   0,   0
zio_data_buf_3072:     3072,      0,       0,       0,       0,   0,   0
zio_buf_3584:          3584,      0,       0,       0,       0,   0,   0
zio_data_buf_3584:     3584,      0,       0,       0,       0,   0,   0
zio_buf_4096:          4096,      0,       0,       0,       0,   0,   0
zio_data_buf_4096:     4096,      0,       0,       0,       0,   0,   0
zio_buf_5120:          5120,      0,       0,       0,       0,   0,   0
zio_data_buf_5120:     5120,      0,       0,       0,       0,   0,   0
zio_buf_6144:          6144,      0,       0,       0,       0,   0,   0
zio_data_buf_6144:     6144,      0,       0,       0,       0,   0,   0
zio_buf_7168:          7168,      0,       0,       0,       0,   0,   0
zio_data_buf_7168:     7168,      0,       0,       0,       0,   0,   0
zio_buf_8192:          8192,      0,       0,       0,       0,   0,   0
zio_data_buf_8192:     8192,      0,       0,       0,       0,   0,   0
zio_buf_10240:        10240,      0,       0,       0,       0,   0,   0
zio_data_buf_10240:   10240,      0,       0,       0,       0,   0,   0
zio_buf_12288:        12288,      0,       0,       0,       0,   0,   0
zio_data_buf_12288:   12288,      0,       0,       0,       0,   0,   0
zio_buf_14336:        14336,      0,       0,       0,       0,   0,   0
zio_data_buf_14336:   14336,      0,       0,       0,       0,   0,   0
zio_buf_16384:        16384,      0,       0,       0,       0,   0,   0
zio_data_buf_16384:   16384,      0,       0,       0,       0,   0,   0
zio_buf_20480:        20480,      0,       0,       0,       0,   0,   0
zio_data_buf_20480:   20480,      0,       0,       0,       0,   0,   0
zio_buf_24576:        24576,      0,       0,       0,       0,   0,   0
zio_data_buf_24576:   24576,      0,       0,       0,       0,   0,   0
zio_buf_28672:        28672,      0,       0,       0,       0,   0,   0
zio_data_buf_28672:   28672,      0,       0,       0,       0,   0,   0
zio_buf_32768:        32768,      0,       0,       0,       0,   0,   0
zio_data_buf_32768:   32768,      0,       0,       0,       0,   0,   0
zio_buf_36864:        36864,      0,       0,       0,       0,   0,   0
zio_data_buf_36864:   36864,      0,       0,       0,       0,   0,   0
zio_buf_40960:        40960,      0,       0,       0,       0,   0,   0
zio_data_buf_40960:   40960,      0,       0,       0,       0,   0,   0
zio_buf_45056:        45056,      0,       0,       0,       0,   0,   0
zio_data_buf_45056:   45056,      0,       0,       0,       0,   0,   0
zio_buf_49152:        49152,      0,       0,       0,       0,   0,   0
zio_data_buf_49152:   49152,      0,       0,       0,       0,   0,   0
zio_buf_53248:        53248,      0,       0,       0,       0,   0,   0
zio_data_buf_53248:   53248,      0,       0,       0,       0,   0,   0
zio_buf_57344:        57344,      0,       0,       0,       0,   0,   0
zio_data_buf_57344:   57344,      0,       0,       0,       0,   0,   0
zio_buf_61440:        61440,      0,       0,       0,       0,   0,   0
zio_data_buf_61440:   61440,      0,       0,       0,       0,   0,   0
zio_buf_65536:        65536,      0,       0,       0,       0,   0,   0
zio_data_buf_65536:   65536,      0,       0,       0,       0,   0,   0
zio_buf_69632:        69632,      0,       0,       0,       0,   0,   0
zio_data_buf_69632:   69632,      0,       0,       0,       0,   0,   0
zio_buf_73728:        73728,      0,       0,       0,       0,   0,   0
zio_data_buf_73728:   73728,      0,       0,       0,       0,   0,   0
zio_buf_77824:        77824,      0,       0,       0,       0,   0,   0
zio_data_buf_77824:   77824,      0,       0,       0,       0,   0,   0
zio_buf_81920:        81920,      0,       0,       0,       0,   0,   0
zio_data_buf_81920:   81920,      0,       0,       0,       0,   0,   0
zio_buf_86016:        86016,      0,       0,       0,       0,   0,   0
zio_data_buf_86016:   86016,      0,       0,       0,       0,   0,   0
zio_buf_90112:        90112,      0,       0,       0,       0,   0,   0
zio_data_buf_90112:   90112,      0,       0,       0,       0,   0,   0
zio_buf_94208:        94208,      0,       0,       0,       0,   0,   0
zio_data_buf_94208:   94208,      0,       0,       0,       0,   0,   0
zio_buf_98304:        98304,      0,       0,       0,       0,   0,   0
zio_data_buf_98304:   98304,      0,       0,       0,       0,   0,   0
zio_buf_102400:      102400,      0,       0,       0,       0,   0,   0
zio_data_buf_102400: 102400,      0,       0,       0,       0,   0,   0
zio_buf_106496:      106496,      0,       0,       0,       0,   0,   0
zio_data_buf_106496: 106496,      0,       0,       0,       0,   0,   0
zio_buf_110592:      110592,      0,       0,       0,       0,   0,   0
zio_data_buf_110592: 110592,      0,       0,       0,       0,   0,   0
zio_buf_114688:      114688,      0,       0,       0,       0,   0,   0
zio_data_buf_114688: 114688,      0,       0,       0,       0,   0,   0
zio_buf_118784:      118784,      0,       0,       0,       0,   0,   0
zio_data_buf_118784: 118784,      0,       0,       0,       0,   0,   0
zio_buf_122880:      122880,      0,       0,       0,       0,   0,   0
zio_data_buf_122880: 122880,      0,       0,       0,       0,   0,   0
zio_buf_126976:      126976,      0,       0,       0,       0,   0,   0
zio_data_buf_126976: 126976,      0,       0,       0,       0,   0,   0
zio_buf_131072:      131072,      0,       0,       0,       0,   0,   0
zio_data_buf_131072: 131072,      0,       0,       0,       0,   0,   0
sa_cache:                80,      0,  178178,   64912,1177516606,   0,   0
dnode_t:                856,      0, 1138457, 4920519,820265718,   0,   0
dmu_buf_impl_t:         224,      0, 1266215, 5339883,1254850926,   0,   0
arc_buf_hdr_t:          216,      0, 5434480,    1214,88435454,   0,   0
arc_buf_t:               72,      0,  170049,  573351,461319188,   0,   0
zil_lwb_cache:          192,      0,      11,    1709, 2696135,   0,   0
zfs_znode_cache:        368,      0,  178149,   64701,1177516577,   0,   0
 
The server is in use since the begin of week 45.

993pfb.png


The Legend is wrong.
M= GB
k= MB
= KB
 
User23 said:
Code:
last pid: 79212;  load averages:  1.01,  1.20,  1.06                                          up 11+12:47:17  11:52:57
3721 processes:3 running, 3718 sleeping
CPU:  1.8% user,  0.0% nice,  3.8% system,  0.3% interrupt, 94.1% idle
Mem: 4324M Active, 3186M Inact, 19G Wired, 3162M Buf, 4950M Free
ARC: 4763M Total, 1943M MFU, 221M MRU, 2045K Anon, 1209M Header, 1465M Other
Swap: 8192M Total, 8192M Free


Code:
# vmstat -z output
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     197,       7,     197,   0,   0
UMA Zones:             1920,      0,     197,       1,     197,   0,   0
UMA Slabs:              568,      0,  454935,  299567,980797209,   0,   0
UMA RCntSlabs:          568,      0,   10750,       2,   10750,   0,   0
UMA Hash:               256,      0,      78,      12,      82,   0,   0
16 Bucket:              152,      0,     247,       3,     250,   0,   0
32 Bucket:              280,      0,     446,       2,     467,   0,   0
64 Bucket:              536,      0,     595,       0,     600, 116,   0
128 Bucket:            1048,      0,  167002,       2,  171956,590167,   0
VM OBJECT:              232,      0,  409880,   53080,759122563,   0,   0
MAP:                    232,      0,       8,      24,       8,   0,   0
KMAP ENTRY:             120, 1063641,   82427,   33110,2427352700,   0,   0
MAP ENTRY:              120,      0,  101284,   13974,1333783984,   0,   0
fakepg:                 120,      0,       0,      62,      22,   0,   0
mt_zone:               4112,      0,     241,      30,     241,   0,   0
16:                      16,      0,   52452,   79932,1432220872,   0,   0
32:                      32,      0,10767152,   14194,333873690,   0,   0
64:                      64,      0,  333234,  777134,6697646135,   0,   0
128:                    128,      0, 2021223,  243184,2259051728,   0,   0
256:                    256,      0,   98838,   72402,1799273484,   0,   0
512:                    512,      0, 1151700, 5034690,3715054440,   0,   0
1024:                  1024,      0,    7997,   28003,138173079,   0,   0
2048:                  2048,      0,   15458,    8132,2033938003,   0,   0
4096:                  4096,      0,   58744,  168447,304563970,   0,   0
Files:                   80,      0,   34728,    5592,723920888,   0,   0
TURNSTILE:              136,      0,    6154,     166,    6154,   0,   0
rl_entry:                40,      0,    4859,     769,    4859,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1192,      0,    3774,    1917,20261126,   0,   0
THREAD:                1160,      0,    5968,     185,    6139,   0,   0
SLEEPQUEUE:              80,      0,    6154,     284,    6154,   0,   0
VMSPACE:                392,      0,    3752,    2648,20261111,   0,   0
cpuset:                  72,      0,     302,     598,     490,   0,   0
audit_record:           960,      0,       0,       0,       0,   0,   0
mbuf_packet:            256,      0,   10421,    4043,1957160288,   0,   0
mbuf:                   256,      0,    1153,    4483,6615599997,   0,   0
mbuf_cluster:          2048,  25600,   14474,    4600,569773322,   0,   0
mbuf_jumbo_page:       4096,  12800,      28,    1185,210815427,   0,   0
mbuf_jumbo_9k:         9216,   6400,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,   3200,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,      51,   30354,3667270826,   0,   0
ttyinq:                 160,      0,     135,     801,    6060,   0,   0
ttyoutq:                256,      0,      72,     558,    3146,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0
VNODE:                  504,      0,  439049,   63607,1185833386,   0,   0
VNODEPOLL:              112,      0,    2877,    1974, 1167605,   0,   0
NAMEI:                 1024,      0,       0,     380,2427214002,   0,   0
S VFS Cache:            108,      0,  318080,  129466,137083558,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,  116834,  131638,1026140651,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NFSMOUNT:               632,      0,       0,       0,       0,   0,   0
NFSNODE:                656,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,    5918,     666,    8576,   0,   0
Mountpoints:            824,      0,       6,      50,       9,   0,   0
pipe:                   728,      0,    3798,    1642,19580435,   0,   0
ksiginfo:               112,      0,    5680,    1745,  859533,   0,   0
itimer:                 344,      0,       1,      21,       1,   0,   0
pfsrctrpl:              152,  10000,       0,       0,       0,   0,   0
pfrulepl:               936,      0,      63,     241,     434,   0,   0
pfstatepl:              288,  10010,    4069,    5941,28304722,4629,   0
pfstatekeypl:           288,      0,    4069,    6435,28304722,   0,   0
pfstateitempl:          288,      0,    4069,    6643,28304722,   0,   0
pfaltqpl:               240,      0,       0,       0,       0,   0,   0
pfpooladdrpl:            88,      0,       0,       0,       0,   0,   0
pfrktable:             1296,   1002,       0,       0,       0,   0,   0
pfrkentry:              160, 200016,       0,       0,       0,   0,   0
pfrkcounters:            64,      0,       0,       0,       0,   0,   0
pffrent:                 32,   5050,       0,       0,       0,   0,   0
pffrag:                  80,      0,       0,       0,       0,   0,   0
pffrcache:               80,  10035,       0,       0,       0,   0,   0
pffrcent:                24,  50022,       0,       0,       0,   0,   0
pfstatescrub:            40,      0,       0,       0,       0,   0,   0
pfiaddrpl:              120,      0,       0,       0,       0,   0,   0
pfospfen:               112,      0,     710,      16,    4970,   0,   0
pfosfp:                  40,      0,     420,      84,    2940,   0,   0
KNOTE:                  128,      0,   25286,    6469,1922518344,   0,   0
socket:                 680,  25602,   12580,    3722,149393700,   0,   0
unpcb:                  240,  25600,   10424,    3256,118822451,   0,   0
ipq:                     56,    819,       0,     819,20174256, 329,   0
udp_inpcb:              392,  25600,      32,     978,20839999,   0,   0
udpcb:                   16,  25704,      32,    2656,20839999,   0,   0
tcp_inpcb:              392,  25600,    2600,    2460, 9731211,   0,   0
tcpcb:                  976,  25600,    2123,    2097, 9731211,   0,   0
tcptw:                   72,   5150,     477,    2773, 7087191,   0,   0
syncache:               152,  15375,       0,    1275, 9666020,   0,   0
hostcache:              136,  15372,    1601,    1619,  124154,   0,   0
tcpreass:                40,   1680,       4,    1676, 1324816,  25,   0
sackhole:                32,      0,       0,    2020, 1497427,   0,   0
ripcb:                  392,  25600,       0,     120,      14,   0,   0
rtentry:                200,      0,      34,     251,      40,   0,   0
selfd:                   56,      0,   11326,    1967,113813395,   0,   0

Seems your 32 usage is similar to my FreeBSD 9.2 server, previously it was 10 times lower on FreeBSD 9.1 with more than 40 days uptime. I don't know whether will it cause any issue on the server or not.

FreeBSD 9.1
Code:
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               208,      0,     183,       4,     183,   0,   0
UMA Zones:              896,      0,     183,       1,     183,   0,   0
UMA Slabs:              568,      0,  126840,  146496,5974622916,   0,   0
UMA RCntSlabs:          568,      0,    9336,       2,    9336,   0,   0
UMA Hash:               256,      0,      81,       9,      81,   0,   0
16 Bucket:              152,      0,     151,      24,     152,   0,   0
32 Bucket:              280,      0,     173,       9,     173,   2,   0
64 Bucket:              536,      0,     204,       6,     204, 104,   0
128 Bucket:            1048,      0,   60106,       2,   80405,513331,   0
VM OBJECT:              232,      0,   53448,    1448,688477684,   0,   0
MAP:                    232,      0,       7,      25,       7,   0,   0
KMAP ENTRY:             120, 277171,   21338,   45622,16142747226,   0,   0
MAP ENTRY:              120,      0,    1030,    2132,1706681438,   0,   0
fakepg:                 120,      0,      10,     362,  306465,   0,   0
mt_zone:               4112,      0,     332,      11,     332,   0,   0
16:                      16,      0, 1830776, 1373992,2929906167,   0,   0
32:                      32,      0, 1789982, 1437473,4077616578,   0,   0
64:                      64,      0,  468725,  516875,23034375543,   0,   0
128:                    128,      0,   36209,   43860,4070754252,   0,   0
256:                    256,      0,   50342,   64918,4880964059,   0,   0
512:                    512,      0,   65997,  318345,2791782015,   0,   0

Today the FreeBSD 9.2 server NFS suddenly stop working and result to all the servers hosted on it were down (It was happened before within 7-14 days an above). I have no idea whether is it related to the memory issue? Possibly LZ4 compression? The server uptime is only 8 days.

Previously (Last week) it was able to run for more than 100 days on FreeBSD 9.0 and 9.1 before switching to FreeBSD 9.2.
 
Back
Top