Possible kernel memory leak FreeBSD 9.0 release

Strange output from mem.pl (well known perl script)
Code: 
SYSTEM MEMORY INFORMATION:
mem_wire:        1234513920 (   1177MB) [ 14%] Wired: disabled for paging out
mem_active:  +   3365068800 (   3209MB) [ 40%] Active: recently referenced
mem_inactive:+   3291779072 (   3139MB) [ 39%] Inactive: recently not referenced
mem_cache:   +    198041600 (    188MB) [  2%] Cached: almost avail. for allocation
mem_free:    +    211402752 (    201MB) [  2%] Free: fully available for allocation
mem_gap_vm:  +       -32768 (      0MB) [  0%] Memory gap: UNKNOWN
-------------- ------------ ----------- ------
mem_all:     =   8300773376 (   7916MB) [100%] Total real memory managed
mem_gap_sys: +    268271616 (    255MB)        Memory gap: Kernel?!
-------------- ------------ -----------
mem_phys:    =   8569044992 (   8172MB)        Total real memory available
mem_gap_hw:  +     20889600 (     19MB)        Memory gap: Segment Mappings?!
-------------- ------------ -----------
mem_hw:      =   8589934592 (   8192MB)        Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used:        4888711168 (   4662MB) [ 56%] Logically used memory
mem_avail:   +   3701223424 (   3529MB) [ 43%] Logically available memory
-------------- ------------ ----------- ------
mem_total:   =   8589934592 (   8192MB) [100%] Logically total memory

Note mem_gap_vm becomes negative :)
It's counted as
Code: 
my $mem_all       = $sysctl->{"vm.stats.vm.v_page_count"}      * $sysctl->{"hw.pagesize"};
my $mem_wire      = $sysctl->{"vm.stats.vm.v_wire_count"}      * $sysctl->{"hw.pagesize"};
my $mem_active    = $sysctl->{"vm.stats.vm.v_active_count"}    * $sysctl->{"hw.pagesize"};
my $mem_inactive  = $sysctl->{"vm.stats.vm.v_inactive_count"}  * $sysctl->{"hw.pagesize"};
my $mem_cache     = $sysctl->{"vm.stats.vm.v_cache_count"}     * $sysctl->{"hw.pagesize"};
my $mem_free      = $sysctl->{"vm.stats.vm.v_free_count"}      * $sysctl->{"hw.pagesize"};
my $mem_gap_vm    = $mem_all - ($mem_wire + $mem_active + $mem_inactive + $mem_cache + $mem_free);

Pretty strange.
 
Regarding previous mem stats.
This is probably due non atomic sysctl operation.
Something was added or freed during sysctl -a.
 
In your case you can check vmstat -m and see what exacly is growing.
But my case is a little bit different.
I see constantly and slowly increasing active memory usage (vm.stats.vm.v_active_count)
But total memory usage by userland procs is stable.
( ps aux | awk '{memory +=$6}; END {print memory } )
3316888 K

Also vmstat -m total memory usage
(vmstat -m | sed 's/K//' | awk '{a+=$3}; END {print a,"K"}')
is also stable about 28329 K.

So I don't even know where my memory gone.
I will try to downgrade a sphinxsearh due a searchd caused a kernel panic twice.
 
Interestingly - after having a good look at vmstat -m, I believe I've found my leaker - fadvise. Will gather some more data.
 
atrull said:
Interestingly - after having a good look at vmstat -m, I believe I've found my leaker - fadvise. Will gather some more data.
Click to expand...

vmstat -m output:
fadvise 30773122 961661K - 30773122 32

for real.
 
So at least you have some some evidence of leaks.
My vmstat -m totals is stable at about 28498 K
I am downgraded a sphinxsearch (searchd) to 0.9.9-release (r2117) from 2.0.5 (latest).
But our active memory usage is still growing slowly.
I am waiting for now.
 
The problem is finally solved after upgrade to 9.1-RC2.
Active memory now is stable about 2.3 Gb and not growing anymore.
So looks like some kind of bug in the kernel VM management part.
Just curious: is any body know what is changed in VM stack since 9.0 RELEASE ?

P.S. It's a last time I am installing an latest release on the production server.
The right decision was to stay at the 8.3. I really thought the this kind of bugs can't appear in such mature part of the FreeBSD kernel.

P.P.S. Old FreeBSD user and a little bit developer since 2.1.1
 
You must log in or register to reply here.
Back
Top