Is L2ARC consume part of the ARC allocated memory?

B

belon_cfy

Let's say I have a system with 8GB of ram and 3GB is allocated to ARC with the parameter vfs.zfs.arc_max="3G", may I know the header size of L2ARC will use part of the memory from ARC or the remaining 5GB free memory from OS?
 
  • Thanks
Reactions: Oko
Good question. No idea, but looks like for me, I´ve capped my ARC at 22GiB:

Code: 
ARC Size:                       86.85%	19.11	GiB
	Target Size: (Adaptive)		86.85%	19.11	GiB
	Min Size (Hard Limit):       12.50%	2.75	GiB
	Max Size (High Water):       8:1	22.00	GiB

L2 ARC Size: (Adaptive)         3.88	TiB
	Header Size:                 0.27%	10.70	GiB

And my 'avail memory' is at 31GiB, so it could actually be that the header is counted "outside" of ARC (19.11+10.70=29.81), but I don´t know. Going by the notion that it is counted outside, it would be good to have a sysctl to limit the percentage of RAM that can be used for the L2_ARC header to limit it´s size, or just a hard byte limit. I mean, I have two 256GB SSD's as cache drives, and so I would have thought that the cache would only ever grow up to 512GB but lo and behold, here we are at 3.88TiB?!

/Sebulon
 
Hi Sebulon,
Is it possible your 10GB L2ARC header are part of your 19GB of ARC? Based on my calculation the consequence is same as yours.

Hopefully someone can confirm that the L2ARC header is not occupied part of ARC.
 
belon_cfy said:
Is it possible your 10GB L2ARC header are part of your 19GB of ARC? Based on my calculation the consequence is same as yours.
Click to expand...

Yes, of course it´s possible, 10GB also fits into the 19GB of ARC. But I mean, since I´ve capped my ARC at 22 and it´s a standard file server, what else (but ZFS) could be eating up the remaining 9GB of it´s available RAM? So to me, it´s more probable that the L2_ARC header is a separate count.

/Sebulon
 
Memory leak ? I have a 8GB Ram FreeBSD 10 NFS server with ARC cap at 4GB , no L2ARC , but currently the free ram is only 700MB available. Wondering where is the rest of 3.3GB ram goes?
 
What does top shows? I do not have an L2ARC and it shows

Code: 
ARC: 8192M Total, 3468M MFU, 3956M MRU, 1632K Anon, 67M Header, 699M Other

If l2arc header size is included in ARC, then it should be displayed somewhere in the above stats.
 
Ah, didn´t think about that. Mine looks like:
Code: 
Mem: 225M Active, 124M Inact, 28G Wired, 337M Cache, 1250M Free
ARC: 17G Total, 180M MFU, 4462M MRU, 185M Anon, 13G Header, 486M Other

In man for top, it says:
...
MRU: number of ARC bytes holding most recently used data
...
Header: number of ARC bytes holding headers
Click to expand...

# zfs-stats -L
Code: 
...
L2 ARC Size: (Adaptive)      4.70        TiB
	Header Size:              0.27%       13.02	GiB
...

So this means that my ARC and L2_ARC headers size should be suppressed to 22GB in total, that should leave about 8GB unused, which means there is something else wiring memory that is not ARC/L2_ARC, or it´s leaking. Is there a way to see what´s in there?

/Sebulon
 
By the way, why not just shift the header size to L2ARC SSD as well as keeping it in the memory since it might be grow up to > 10GB and risk the system in memory exhaustion situation.

I understand shifting the header size to L2ARC will slow down the performance on searching/locating the cache data, but it is still much faster than searching from spindle disk, some more it will be easier for persistent l2arc implementation and no recovery delay for restoring header data from SSD during reboot.
 
I don't use L2ARC, but here's how to see where your memory is being used:

First, get an overview from top:
Code: 
last pid:  4973;  load averages:  0.00,  0.00,  0.00   up 20+20:10:36  12:07:45
106 processes: 1 running, 102 sleeping, 3 stopped
CPU:  0.0% user,  0.0% nice,  1.0% system,  0.0% interrupt, 99.0% idle
Mem: 357M Active, 263M Inact, 3101M Wired, 60M Cache, 417M Buf, 137M Free
ARC: 2458M Total, 1309M MFU, 964M MRU, 1810K Anon, 26M Header, 157M Other
Swap: 16G Total, 125M Used, 16G Free

Then, get the (wired) kernel malloc usage from vmstat -m:
Code: 
% vmstat -m | perl -ne 'if (/(\d+)K/) { $sum += $1; } END { print int(0.5+$sum/1024),"M\n"; }'
2432M

Then, get the (wired) kernel zone memory usage from vmstat -z:
Code: 
vmstat -z | perl -ne 'if (/(\d+),\D+(\d+),\D+(\d+),\D+(\d+),/) { $sum += ($1*($3+$4)); } END {print int(0.5+($sum/(1024*1024))),"M\n";}'
512 M

Then, get the total system memory (minus bios space, etc), and available user memory from sysctl:
Code: 
% sysctl hw.physmem | awk '{print $2 / (1024*1024)"M"}'
4070.74M
% sysctl hw.usermem | awk '{print $2 / (1024*1024)"M"}'
969.766M

Finally you're in a position to do some calculations:
Code: 
4071M Total - SUM(Mem: 357M Active, 263M Inact, 3101M Wired, 60M Cache, 137M Free) => ~153M unaccounted

3101M Wired - 2432M Malloc - 512M UMA => ~157M unaccounted

970M User - SUM(Mem: 357M Active, 263M Inact, 60M Cache, 137M Free) => ~153M unaccounted

2458M ARC - SUM(1309M MFU, 964M MRU, 1810K Anon, 26M Header, 157M Other) => ~0M unaccounted

If you have a nice spreadsheet with these values for each day over a week or two you should be able to see if anything is changing in your system. @belon_cfy, I know you've been posting about memory usage for a long while but I've never seen you post enough systematic data for people to help.
 
Last edited by a moderator:
Just installed a new server with FreeBSD 10.2 , and very obvious the L2arc header will use part of the allocated ARC.

The system has 16GB ram but 11GB allocated for ARC.

Code: 
Mem: 18M Active, 35M Inact, 15G Wired, 2712K Cache, 242M Free
ARC: 11G Total, 4696M MFU, 536M MRU, 80M Anon, 6749M Header, 63M Other
Code: 
L2 ARC Summary: (HEALTHY)
        Passed Headroom:                        8.02m
        Tried Lock Failures:                    337.93k
        IO In Progress:                         83.31k
        Low Memory Aborts:                      37
        Free on Write:                          1.72m
        Writes While Full:                      190.52k
        R/W Clashes:                            7.76k
        Bad Checksums:                          0
        IO Errors:                              0
        SPA Mismatch:                           49.17m

L2 ARC Size: (Adaptive)                         889.28  GiB
        Header Size:                    0.73%   6.51    GiB

L2 ARC Evicts:
        Lock Retries:                           5.52k
        Upon Reading:                           0

L2 ARC Breakdown:                               385.54m
        Hit Ratio:                      35.94%  138.56m
        Miss Ratio:                     64.06%  246.98m
        Feeds:                                  370.29k

L2 ARC Buffer:
        Bytes Scanned:                          24.61   TiB
        Buffer Iterations:                      370.29k
        List Iterations:                        22.40m
        NULL List Iterations:                   3.61m

L2 ARC Writes:
        Writes Sent:                    100.00% 353.11k
 
You must log in or register to reply here.
Back
Top