Solved Memory leak on FreeBSD 9.3 and FreeBSD 10?

Re: Memory leak on FreeBSD 9.2 and FreeBSD 10?

Matty said:
Did you try to set the vfs.zfs.zio.use_uma to 0?

Hi, this was interesting. I have two amd64 10-STABLE machines (which are experiencing this issue, one upgraded from 9, one installed) which have vfs.zfs.zio.use_uma=1.
Going to try this asap, L2ARC removal did not help.

/HenkeZan
 
Re: Memory leak on FreeBSD 9.2 and FreeBSD 10?

HenkeZan said:
Matty said:
Did you try to set the vfs.zfs.zio.use_uma to 0?

Hi, this was interesting. I have two amd64 10-STABLE machines (which are experiencing this issue, one upgraded from 9, one installed) which have vfs.zfs.zio.use_uma=1.
Going to try this asap, L2ARC removal did not help.

/HenkeZan
Any improvements?
 
Re: Memory leak on FreeBSD 9.2 and FreeBSD 10?

Matty said:
HenkeZan said:
Matty said:
Did you try to set the vfs.zfs.zio.use_uma to 0?

Hi, this was interesting. I have two amd64 10-STABLE machines (which are experiencing this issue, one upgraded from 9, one installed) which have vfs.zfs.zio.use_uma=1.
Going to try this asap, L2ARC removal did not help.

/HenkeZan
Any improvements?

Not really. Sorry for not replying in time.
However, this does look like the correct fix: PR 187594

Recompiling source with the patch right now. Let's see how it goes.

/Henrik
 
Re: Memory leak on FreeBSD 9.2 and FreeBSD 10?

I'm on the same boat now, my FreeBSD 10 server start pressuring out the ARC. After 20 days the ARC utilization drops to 86% instead of 100%.

Code:
------------------------------------------------------------------------
ZFS Subsystem Report                            Fri Mar 21 14:23:46 2014
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                459.07m
        Recycle Misses:                         40.47m
        Mutex Misses:                           6.02m
        Evict Skips:                            976.96m

ARC Size:                               86.80%  10.42   GiB
        Target Size: (Adaptive)         86.81%  10.42   GiB
        Min Size (Hard Limit):          12.50%  1.50    GiB
        Max Size (High Water):          8:1     12.00   GiB

ARC Size Breakdown:
        Recently Used Cache Size:       81.10%  8.45    GiB
        Frequently Used Cache Size:     18.90%  1.97    GiB

ARC Hash Breakdown:
        Elements Max:                           6.98m
        Elements Current:               84.47%  5.90m
        Collisions:                             1.06b
        Chain Max:                              73
        Chains:                                 262.14k

------------------------------------------------------------------------

Code:
last pid: 98494;  load averages:  0.70,  0.83,  0.63                                                                      up 20+15:01:31  14:26:23
36 processes:  1 running, 35 sleeping
CPU:     % user,     % nice,     % system,     % interrupt,     % idle
Mem: 4896K Active, 62M Inact, 14G Wired, 60K Cache, 1282M Free
ARC: 10G Total, 2993M MFU, 6412M MRU, 15M Anon, 1389M Header, 45M Other
Swap: 8192M Total, 8192M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1125 root         64  20    0 12272K  2700K rpcsvc  1  17.1H   0.00% nfsd
 1212 root          1  20    0 23980K  5360K select  0   0:14   0.00% sendmail
 1219 root          1  52    0 16520K  2108K nanslp  1   0:12   0.00% cron
34922 root          1  20    0 86084K  7132K select  2   0:09   0.00% sshd
 1030 root          1  20    0 14424K  1972K select  2   0:08   0.00% syslogd
 1034 root          1  20    0 16512K  2092K select  6   0:02   0.00% rpcbind
 1131 root          1  52    0 20636K  3824K rpcsvc  3   0:01   0.00% rpc.lockd
 1128 root          1  20    0   274M  3756K select  3   0:01   0.00% rpc.statd
 1104 root          1  20    0 12656K  1804K select  4   0:01   0.00% nfsuserd
 1107 root          1  20    0 12656K  1804K select  2   0:01   0.00% nfsuserd
 1105 root          1  20    0 12656K  1804K select  2   0:01   0.00% nfsuserd
 1106 root          1  20    0 12656K  1804K select  3   0:01   0.00% nfsuserd
 1193 root          1  20    0 60816K  6060K select  0   0:01   0.00% sshd
  901 root          1  20    0 13584K  4484K select  0   0:00   0.00% devd
 1215 smmsp         1  20    0 23980K  4940K pause   7   0:00   0.00% sendmail
35044 root          1  20    0 23492K  4012K pause   4   0:00   0.00% csh
 1124 root          1  20    0 16368K  3760K select  3   0:00   0.00% nfsd
98458 root          1  21    0 24580K  3952K piperd  3   0:00   0.00% perl5.16.3
98460 root          1  20    0 24580K  3928K nanslp  2   0:00   0.00% perl5.16.3
98461 root          1  20    0 24580K  3932K nanslp  6   0:00   0.00% perl5.16.3
 1122 root          1  20    0 18528K  3876K select  0   0:00   0.00% mountd
98464 root          1  20    0 27712K  2740K nanslp  2   0:00   0.00% gstat
 1264 root          1  20    0 14420K  1884K ttyin   6   0:00   0.00% getty
 1271 root          1  52    0 14420K  1884K ttyin   2   0:00   0.00% getty
 
Re: Memory leak on FreeBSD 9.2 and FreeBSD 10?

My ARC utilization is getting less and less. After 26 days it drops to 81% .

Seems the issue was started since ZFS V5000, all of my FreeBSD 9.2 and 10 are suffering on the same issue. But non of my running FreeBSD 9.1 with ZFS V28 encountered.

Code:
------------------------------------------------------------------------
ZFS Subsystem Report                            Thu Mar 27 20:21:27 2014
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                604.66m
        Recycle Misses:                         50.37m
        Mutex Misses:                           8.19m
        Evict Skips:                            1.29b

ARC Size:                               81.43%  9.77    GiB
        Target Size: (Adaptive)         81.43%  9.77    GiB
        Min Size (Hard Limit):          12.50%  1.50    GiB
        Max Size (High Water):          8:1     12.00   GiB

ARC Size Breakdown:
        Recently Used Cache Size:       93.75%  9.16    GiB
        Frequently Used Cache Size:     6.25%   625.68  MiB

ARC Hash Breakdown:
        Elements Max:                           6.98m
        Elements Current:               87.30%  6.10m
        Collisions:                             1.38b
        Chain Max:                              73
        Chains:                                 262.14k

------------------------------------------------------------------------

Code:
last pid:  3633;  load averages:  0.64,  0.63,  0.49                                                                                  up 26+20:57:17  20:22:25
36 processes:  1 running, 35 sleeping
CPU:  0.0% user,  0.0% nice,  3.3% system,  0.2% interrupt, 96.5% idle
Mem: 5064K Active, 67M Inact, 14G Wired, 1292M Free
ARC: 10G Total, 517M MFU, 8198M MRU, 40M Anon, 1437M Header, 43M Other
Swap: 8192M Total, 8192M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
 1125 root         64  20    0 12272K  2700K rpcsvc  2  22.0H   0.39% nfsd
 1212 root          1  20    0 23980K  5360K select  6   0:18   0.00% sendmail
 1219 root          1  52    0 16520K  2108K nanslp  2   0:16   0.00% cron
35863 root          1  20    0 86084K  7132K select  5   0:14   0.00% sshd
 1030 root          1  20    0 14424K  1972K select  7   0:10   0.00% syslogd
 1034 root          1  20    0 16512K  2092K select  2   0:09   0.00% rpcbind
 1131 root          1  52    0 20636K  3824K rpcsvc  1   0:01   0.00% rpc.lockd
 1128 root          1  20    0   274M  3756K select  0   0:01   0.00% rpc.statd
 1105 root          1  20    0 12656K  1804K select  5   0:01   0.00% nfsuserd
 1104 root          1  20    0 12656K  1804K select  0   0:01   0.00% nfsuserd
 1106 root          1  20    0 12656K  1804K select  5   0:01   0.00% nfsuserd
 1107 root          1  20    0 12656K  1804K select  2   0:01   0.00% nfsuserd
 1193 root          1  20    0 60816K  6060K select  7   0:01   0.00% sshd
  901 root          1  20    0 13584K  4484K select  3   0:00   0.00% devd
 1215 smmsp         1  20    0 23980K  4940K pause   5   0:00   0.00% sendmail
35878 root          1  20    0 23492K  3728K pause   3   0:00   0.00% csh
 
Re: Memory leak on FreeBSD 9.2 and FreeBSD 10?

I'm having the same issue as well - I went from 8GB to 32GB the same time I upgraded to 10.0 RELEASE and I have had 2-3 memory exhaustion situations with the same workload as before.

I'm definitely curious if the patch listed above helps - I will have to wait for a release, but at least I would know it's coming!
 
Re: Memory leak on FreeBSD 9.2 and FreeBSD 10?

Sylgeist said:
I'm having the same issue as well - I went from 8GB to 32GB the same time I upgraded to 10.0 RELEASE and I have had 2-3 memory exhaustion situations with the same workload as before.

I'm definitely curious if the patch listed above helps - I will have to wait for a release, but at least I would know it's coming!

Just wondering whether the patch will work or not because initially my server has 900MB-1.3GB free memory when ARC has been 100% populated , however after 26 days the ARC dropped below 80% but the free memory still remains the same which is 900MB-1.3GB. Assuming the situation keeps happening for the next half year, my ARC utilization will eventually drop to zero however the free memory still same as before, where is the rest of memory gone, or can we call it memory leakage?

I'm running FreeBSD 10 as a NFS server only , no additional service is running. Same situation on FreeBSD 9.2 as well, but worst.
 
Re: Memory leak on FreeBSD 9.2 and FreeBSD 10?

I have no idea any processes are eating my memory?
Code:
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               384,      0,     105,       5,     105,   0,   0
UMA Zones:             1664,      0,     105,       1,     105,   0,   0
UMA Slabs:               80,      0,  454251,  283749,9119050697,   0,   0
UMA RCntSlabs:           88,      0,   19372,      23,   19372,   0,   0
UMA Hash:               256,      0,       3,      12,       8,   0,   0
4 Bucket:                32,      0,   10875,    1250,5872830311,   0,   0
8 Bucket:                64,      0,    3689,    2821,369554009,   0,   0
16 Bucket:              128,      0,    2521,    2005,265019165,  37,   0
32 Bucket:              256,      0,    3110,    1750,10806849, 455,   0
64 Bucket:              512,      0,     576,     568,83052636, 107,   0
128 Bucket:            1024,      0,   16455,    1093,209563217, 650,   0
vmem btag:               56,      0,  488188,  291321,399107366,10979,   0
VM OBJECT:              256,      0,    7091,    4684,650793961,   0,   0
RADIX NODE:             144,      0,  237977,   21331,2665329795,  51,   0
MAP:                    240,      0,       3,      61,       3,   0,   0
KMAP ENTRY:             128,      0,       7,     396,       7,   0,   0
MAP ENTRY:              128,      0,     885,    7888,1709546842,   0,   0
VMSPACE:                448,      0,      38,     457,24569247,   0,   0
fakepg:                 104,      0,      13,    1241,   79105,  13,   0
mt_zone:               4112,      0,     349,       0,     349,   0,   0
16:                      16,      0,   15626,    8470,4075793897,   0,   0
32:                      32,      0,10404915, 2237210,4477096771,   0,   0
64:                      64,      0,   28229,   16473,22008141905,   0,   0
128:                    128,      0,  638822,  182895,5768366598,   0,   0
256:                    256,      0,   12079,   13901,7533391376,   0,   0
512:                    512,      0,    7926,   88234,1061320769,   0,   0
1024:                  1024,      0,     245,   37387,1058707599,   0,   0
2048:                  2048,      0,     335,   35411,3353981203,   0,   0
4096:                  4096,      0,    5873,   27775,664344856,   0,   0
SLEEPQUEUE:              80,      0,     811,     460,     811,   0,   0
uint64 pcpu:              8,      0,    1440,     224,    1440,   0,   0
Files:                   80,      0,     103,    1647,267649757,   0,   0
TURNSTILE:              136,      0,     811,     169,     811,   0,   0
rl_entry:                40,      0,     223,     777,     223,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1208,      0,     115,     206,24569345,   0,   0
THREAD:                1168,      0,     765,      45,    3865,   0,   0
cpuset:                  72,      0,     524,     851,    1681,   0,   0
audit_record:          1248,      0,       0,       0,       0,   0,   0
mbuf_packet:            256, 6508590,    4092,    1658,83974010,   0,   0
mbuf:                   256, 6508590,    4154,   12851,26009049059,   4,   0
mbuf_cluster:          2048, 1016968,    5750,    1964,3826753481, 302,   0
mbuf_jumbo_page:       4096, 508483,       0,     553,2143031245, 186,   0
mbuf_jumbo_9k:         9216, 150661,    4092,   10870,6449454054,5770,   0
mbuf_jumbo_16k:       16384,  84747,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,       0,    1040,14144255057,   0,   0
ttyinq:                 160,      0,     180,     695,     975,   0,   0
ttyoutq:                256,      0,      95,     520,     512,   0,   0
ata_request:            336,      0,       0,       0,       0,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
FPU_save_area:          832,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,    1328, 2053508,   0,   0
VNODE:                  472,      0,    6115,     325,  127212,   0,   0
VNODEPOLL:              112,      0,       0,       0,       0,   0,   0
BUF TRIE:               144,      0,       0,  105138,       0,   0,   0
NAMEI:                 1024,      0,       0,     156,784207591,   0,   0
S VFS Cache:            108,      0,   12622,    1273,  360825,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,     180,     204,     289,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NCLNODE:                528,      0,       0,       0,       0,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0
procdesc:               128,      0,       0,       0,       0,   0,   0
space_seg_cache:         64,      0,  355558,  641092,1790431422,1790431422,   0
zio_cache:              944,      0,       2,   20114,23931220637,23931220637,   0
zio_link_cache:          48,      0,       0,   20418,21461491080,21461491080,   0
sa_cache:                80,      0,    6047,      53,  127793,127793,   0
dnode_t:                864,      0,    9300,    7612,89213526,89213526,   0
dmu_buf_impl_t:         224,      0,  201601,  145165,482427971,482427971,   0
arc_buf_hdr_t:          216,      0, 5765856, 1145496,353793509,353793509,   0
arc_buf_t:               72,      0,  288301,   26574,918382265,918382265,   0
zil_lwb_cache:          192,      0,     610,    6730,652707907,652707907,   0
zfs_znode_cache:        368,      0,    6047,      13,  127121,127121,   0
Mountpoints:            816,      0,       7,      48,       7,   0,   0
pipe:                   744,      0,       9,     281,30778251,   0,   0
ksiginfo:               112,      0,     308,    1722,  114658,   0,   0
itimer:                 352,      0,       0,      66,       5,   0,   0
KNOTE:                  128,      0,       0,    1395,  579250,   0,   0
socket:                 696, 522255,      57,     168, 1810427,   0,   0
unpcb:                  240, 522256,      10,     662,  248955,   0,   0
ipq:                     56,  31808,       0,       0,       0,   0,   0
udp_inpcb:              392, 522260,      18,     302,  981700,   0,   0
udpcb:                   16, 522331,      18,    1488,  981700,   0,   0
tcp_inpcb:              392, 522260,      31,     259,  579732,   0,   0
tcpcb:                 1024, 522256,      29,     143,  579732,   0,   0
tcptw:                   88,  27810,       2,    1258,   23735,   0,   0
syncache:               160,  15375,       0,     500,  444097,   0,   0
hostcache:              136,  15370,       3,     490,      21,   0,   0
tcpreass:                40,  63600,       0,    1000, 2274197,   0,   0
sackhole:                32,      0,       0,    1500, 1895233,   0,   0
sctp_ep:               1408, 522256,       0,       0,       0,   0,   0
sctp_asoc:             2352,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80012,       0,     664,       8,   0,   0
sctp_raddr:             728,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400026,       0,       0,       0,   0,   0
sctp_readq:             104, 400026,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400026,       0,       0,       0,   0,   0
sctp_asconf:             40, 400000,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   0
ripcb:                  392, 522260,       0,     210,      35,   0,   0
rtentry:                200,      0,      29,     471,      29,   0,   0
selfd:                   56,      0,     356,    1277,936015855,   0,   0
SWAPMETA:               288, 2033941,       0,       0,       0,   0,   0
 
Re: Memory leak on FreeBSD 9.2 and FreeBSD 10?

User23 said:
I have tried to apply the patch but still can't find the parameter vfs.zfs.arc_freepage_percent_target, what I did wrongly?

cd /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs
wget [url=http://www.denninger.net/FreeBSD-Patches/arc-patch]http://www.denninger.net/FreeBSD-Patches/arc-patch[/url]
patch < arc-patch
cd /usr/src
make buildworld
make installworld
 
Re: Memory leak on FreeBSD 9.2 and FreeBSD 10?

Hi @belon_cfy!

We have had this exact issue on three of our machines, upgraded from 9.1 to 9.2, and I don´t know whether it´s the one or the other, or in combination, but take these away:

/boot/loader.conf
Code:
#vfs.zfs.cache_flush_disable=1

/etc/sysctl.conf
Code:
#vfs.zfs.l2arc_noprefetch=0

Making these changes have made them stable for a month now, previously going unavailable every 14 days or so, depending on load.

/Sebulon
 
Last edited by a moderator:
Re: Memory leak on FreeBSD 9.2 and FreeBSD 10?

Sebulon said:
Hi @belon_cfy!

We have had this exact issue on three of our machines, upgraded from 9.1 to 9.2, and I don´t know whether it´s the one or the other, or in combination, but take these away:

/boot/loader.conf
Code:
#vfs.zfs.cache_flush_disable=1

/etc/sysctl.conf
Code:
#vfs.zfs.l2arc_noprefetch=0

Making these changes have made them stable for a month now, previously going unavailable every 14 days or so, depending on load.

/Sebulon
Hi Sebulon,
Thanks for sharing your experience, disabling vfs.zfs.cache_flush_disable=1 and vfs.zfs.l2arc_noprefetch=0 might be the workaround solution to mitigate this issue. Will try out.
 
Last edited by a moderator:
Re: Memory leak on FreeBSD 9.2 and FreeBSD 10?

I have been tested the FreeBSD 9.3 beta release recently and it's start pressuring out my ARC, the total size of ARC has been dropped to 96% max , remaining memory retains at 200-300MB and i saw some data been swapped out. The same behavior never happened in FreeBSD 9.1 with the exact same configuration and hardware specification which is 8GB RAM, 4X 2TB HDD RAID10 and quadcore CPU.

Will keep it running and see how long it can sustain with such workload.

Below is my loader.conf.
Code:
zfs_load="YES"
aesni_load="YES"
vfs.zfs.cache_flush_disable=1
vfs.root.mountfrom="zfs:zroot"

vfs.zfs.arc_max="4G"
 
Re: Memory leak on FreeBSD 9.2 and FreeBSD 10?

My bad, it back to normal again in the morning

Code:
------------------------------------------------------------------------
ZFS Subsystem Report                            Thu Jun 26 10:49:01 2014
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                647.39m
        Recycle Misses:                         76.12m
        Mutex Misses:                           534.81k
        Evict Skips:                            5.06b

ARC Size:                               100.02% 4.00    GiB
        Target Size: (Adaptive)         100.00% 4.00    GiB
        Min Size (Hard Limit):          12.50%  512.00  MiB
        Max Size (High Water):          8:1     4.00    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       17.10%  700.69  MiB
        Frequently Used Cache Size:     82.90%  3.32    GiB

ARC Hash Breakdown:
        Elements Max:                           5.70m
        Elements Current:               79.87%  4.55m
        Collisions:                             826.70m
        Chain Max:                              135
        Chains:                                 131.07k

------------------------------------------------------------------------

Code:
------------------------------------------------------------------------
ZFS Subsystem Report                            Thu Jun 26 10:51:06 2014
------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        37.30m
        Tried Lock Failures:                    88.24m
        IO In Progress:                         577.92k
        Low Memory Aborts:                      23
        Free on Write:                          1.19m
        Writes While Full:                      781.79k
        R/W Clashes:                            373.76k
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           289.27m

L2 ARC Size: (Adaptive)                         142.91  GiB
        Header Size:                    0.70%   1.01    GiB

L2 ARC Evicts:
        Lock Retries:                           21.99k
        Upon Reading:                           0

L2 ARC Breakdown:                               589.14m
        Hit Ratio:                      49.95%  294.29m
        Miss Ratio:                     50.05%  294.85m
        Feeds:                                  2.41m

L2 ARC Buffer:
        Bytes Scanned:                          1.98    PiB
        Buffer Iterations:                      2.41m
        List Iterations:                        148.49m
        NULL List Iterations:                   18.81m

L2 ARC Writes:
        Writes Sent:                    100.00% 1.91m

------------------------------------------------------------------------

Code:
last pid: 73876;  load averages:  0.17,  0.31,  0.35                                                                              up 27+21:24:33  10:51:57
42 processes:  1 running, 40 sleeping, 1 zombie
CPU:  0.0% user,  0.0% nice,  2.5% system,  0.3% interrupt, 97.2% idle
Mem: 7564K Active, 7136K Inact, 7492M Wired, 12M Cache, 382M Free
ARC: 4096M Total, 2535M MFU, 555M MRU, 39M Anon, 1079M Header, 21M Other
Swap: 8192M Total, 22M Used, 8169M Free
 
Re: Memory leak on FreeBSD 9.2 and FreeBSD 10?

Again, the ARC utilization never reach 100%, and can't even reach 90% for more than 10 days already, looks like the server slowly loosing the ARC. Where is the rest of memory gone? Can it be considered as memory leak?

Non of my FreeBSD 9.1 server has such issue, but all of my FreeBSD 9.2 ,9.3 and 10 servers are having this issue.

FreeBSD 9.3
Code:
------------------------------------------------------------------------
ZFS Subsystem Report                            Sat Jul 26 10:34:58 2014
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                1.24b
        Recycle Misses:                         235.33m
        Mutex Misses:                           1.79m
        Evict Skips:                            12.90b

ARC Size:                               83.65%  3.35    GiB
        Target Size: (Adaptive)         83.99%  3.36    GiB
        Min Size (Hard Limit):          12.50%  512.00  MiB
        Max Size (High Water):          8:1     4.00    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       36.59%  1.23    GiB
        Frequently Used Cache Size:     63.41%  2.13    GiB

ARC Hash Breakdown:
        Elements Max:                           5.70m
        Elements Current:               75.25%  4.29m
        Collisions:                             1.71b
        Chain Max:                              135
        Chains:                                 131.07k

------------------------------------------------------------------------

Code:
------------------------------------------------------------------------
ZFS Subsystem Report                            Sat Jul 26 10:40:50 2014
------------------------------------------------------------------------

L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        96.17m
        Tried Lock Failures:                    138.35m
        IO In Progress:                         1.04m
        Low Memory Aborts:                      2.05k
        Free on Write:                          2.23m
        Writes While Full:                      1.78m
        R/W Clashes:                            614.09k
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           648.51m

L2 ARC Size: (Adaptive)                         139.00  GiB
        Header Size:                    0.69%   977.13  MiB

L2 ARC Evicts:
        Lock Retries:                           42.33k
        Upon Reading:                           12

L2 ARC Breakdown:                               1.27b
        Hit Ratio:                      50.47%  639.68m
        Miss Ratio:                     49.53%  627.86m
        Feeds:                                  4.99m

L2 ARC Buffer:
        Bytes Scanned:                          3.89    PiB
        Buffer Iterations:                      4.99m
        List Iterations:                        304.05m
        NULL List Iterations:                   33.51m

L2 ARC Writes:
        Writes Sent:                    100.00% 4.26m

------------------------------------------------------------------------

Code:
last pid: 86828;  load averages:  0.29,  0.37,  0.42                                                                          up 57+21:07:03  10:36:19
48 processes:  1 running, 45 sleeping, 2 zombie
CPU:  0.5% user,  0.0% nice,  5.4% system,  0.2% interrupt, 94.0% idle
Mem: 12M Active, 1408K Inact, 7549M Wired, 5808K Cache, 321M Free
ARC: 3478M Total, 2155M MFU, 406M MRU, 9965K Anon, 1014M Header, 20M Other
Swap: 8192M Total, 26M Used, 8165M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
 
The thing got worse today. The ARC drops to 75%. Has the memory leak issue been fixed in FreeBSD 9.3?

Code:
------------------------------------------------------------------------
ZFS Subsystem Report                            Sun Jul 27 11:58:40 2014
------------------------------------------------------------------------

ARC Summary: (HEALTHY)
        Memory Throttle Count:                  0

ARC Misc:
        Deleted:                                1.26b
        Recycle Misses:                         242.85m
        Mutex Misses:                           1.84m
        Evict Skips:                            13.19b

ARC Size:                               75.09%  3.00    GiB
        Target Size: (Adaptive)         75.10%  3.00    GiB
        Min Size (Hard Limit):          12.50%  512.00  MiB
        Max Size (High Water):          8:1     4.00    GiB

ARC Size Breakdown:
        Recently Used Cache Size:       93.75%  2.82    GiB
        Frequently Used Cache Size:     6.25%   192.24  MiB

ARC Hash Breakdown:
        Elements Max:                           5.70m
        Elements Current:               49.71%  2.83m
        Collisions:                             1.74b
        Chain Max:                              135
        Chains:                                 131.07k

------------------------------------------------------------------------
 
I have another server without L2ARC does not suffer on such issue, It is running FreeBSD 10 for more than 40 days with the same workload as others.

Guess the problem still L2ARC related.
 
Oh no! My ARC has been pushing out for no reason and only left 57%, definitely something wrong with the memory management or memory leak on FreeBSD 9.3 as well.

Code:
ARC Size:                               57.74%  2.31    GiB
        Target Size: (Adaptive)         57.74%  2.31    GiB
        Min Size (Hard Limit):          12.50%  512.00  MiB
        Max Size (High Water):          8:1     4.00    GiB
 
You should probably take this to the appropriate mailing list(s). Sounds like a developer needs to look at this.
 
Recently just restarted the server and start over again. Let's see will it happens again after 60 days. Although the machine was not dead but the ARC kept pushing out for no reason until it dropped below 50% (which is only 2GB been used up of 4GB)

It seems the memory leak issue was L2ARC related as well, regardless on FreeBSD 9.3 or 10. I have another machine with full SSD and without L2arc does not suffer on such issue. It has been running for more than 60 days with 100% of ARC utilization and 1.5GB free memory (total is 8GB with 4GB allocated to ARC).
 
Bad thing happens again, one of my FreeBSD 9.3 server down just like the others, no network and IO, nfs not working.

Considering switching back to FreeBSD 9.1 again because of this issue, the hardware was working fine previously with more than 300 days uptime with the exactly same workload.
 
Back
Top