Possible kernel memory leak FreeBSD 9.0 release - Page 2

Remme · #26 August 25th, 2012, 07:48

Strange output from mem.pl (well known perl script)

Code:

SYSTEM MEMORY INFORMATION:
mem_wire:        1234513920 (   1177MB) [ 14%] Wired: disabled for paging out
mem_active:  +   3365068800 (   3209MB) [ 40%] Active: recently referenced
mem_inactive:+   3291779072 (   3139MB) [ 39%] Inactive: recently not referenced
mem_cache:   +    198041600 (    188MB) [  2%] Cached: almost avail. for allocation
mem_free:    +    211402752 (    201MB) [  2%] Free: fully available for allocation
mem_gap_vm:  +       -32768 (      0MB) [  0%] Memory gap: UNKNOWN
-------------- ------------ ----------- ------
mem_all:     =   8300773376 (   7916MB) [100%] Total real memory managed
mem_gap_sys: +    268271616 (    255MB)        Memory gap: Kernel?!
-------------- ------------ -----------
mem_phys:    =   8569044992 (   8172MB)        Total real memory available
mem_gap_hw:  +     20889600 (     19MB)        Memory gap: Segment Mappings?!
-------------- ------------ -----------
mem_hw:      =   8589934592 (   8192MB)        Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used:        4888711168 (   4662MB) [ 56%] Logically used memory
mem_avail:   +   3701223424 (   3529MB) [ 43%] Logically available memory
-------------- ------------ ----------- ------
mem_total:   =   8589934592 (   8192MB) [100%] Logically total memory

Note mem_gap_vm becomes negative

It's counted as

Code:

my $mem_all       = $sysctl->{"vm.stats.vm.v_page_count"}      * $sysctl->{"hw.pagesize"};
my $mem_wire      = $sysctl->{"vm.stats.vm.v_wire_count"}      * $sysctl->{"hw.pagesize"};
my $mem_active    = $sysctl->{"vm.stats.vm.v_active_count"}    * $sysctl->{"hw.pagesize"};
my $mem_inactive  = $sysctl->{"vm.stats.vm.v_inactive_count"}  * $sysctl->{"hw.pagesize"};
my $mem_cache     = $sysctl->{"vm.stats.vm.v_cache_count"}     * $sysctl->{"hw.pagesize"};
my $mem_free      = $sysctl->{"vm.stats.vm.v_free_count"}      * $sysctl->{"hw.pagesize"};
my $mem_gap_vm    = $mem_all - ($mem_wire + $mem_active + $mem_inactive + $mem_cache + $mem_free);

Pretty strange.

Remme · #27 August 25th, 2012, 08:05

Regarding previous mem stats.
This is probably due non atomic sysctl operation.
Something was added or freed during sysctl -a.

atrull · #28 August 25th, 2012, 11:29

All my plugins are here:

http://trull.org/~alex/src/FreeBSD/muninplugins/

I'll get around to uploading them sometime - had a few issues with the munin official plugin sites in the past.

Remme · #29 August 25th, 2012, 18:00

In your case you can check vmstat -m and see what exacly is growing.
But my case is a little bit different.
I see constantly and slowly increasing active memory usage (vm.stats.vm.v_active_count)
But total memory usage by userland procs is stable.
( ps aux | awk '{memory +=$6}; END {print memory } )
3316888 K

Also vmstat -m total memory usage
(vmstat -m | sed 's/K//' | awk '{a+=$3}; END {print a,"K"}')
is also stable about 28329 K.

So I don't even know where my memory gone.
I will try to downgrade a sphinxsearh due a searchd caused a kernel panic twice.

atrull · #30 August 25th, 2012, 22:01

Interestingly - after having a good look at vmstat -m, I believe I've found my leaker - fadvise. Will gather some more data.

atrull · #31 August 25th, 2012, 22:17

Quote:

Originally Posted by atrull

Interestingly - after having a good look at vmstat -m, I believe I've found my leaker - fadvise. Will gather some more data.

vmstat -m output:
fadvise 30773122 961661K - 30773122 32

for real.

Remme · #32 August 26th, 2012, 08:58

So at least you have some some evidence of leaks.
My vmstat -m totals is stable at about 28498 K
I am downgraded a sphinxsearch (searchd) to 0.9.9-release (r2117) from 2.0.5 (latest).
But our active memory usage is still growing slowly.
I am waiting for now.

mjguzik · #33 August 26th, 2012, 22:50

Quote:

Originally Posted by atrull

vmstat -m output:
fadvise 30773122 961661K - 30773122 32

for real.

It was fixed some time ago:
http://svnweb.freebsd.org/base?view=...evision=234661

In other words update to stable/8 or just copy this patch (seems to be trivial enough).

atrull · #34 August 28th, 2012, 22:00

Quote:

Originally Posted by mjguzik

It was fixed some time ago:
http://svnweb.freebsd.org/base?view=...evision=234661

In other words update to stable/8 or just copy this patch (seems to be trivial enough).

Thanks. I tried stable for a day but it proved not to be, so I'm on the patch now. Looking good.

Remme · #35 October 6th, 2012, 13:39

The problem is finally solved after upgrade to 9.1-RC2.
Active memory now is stable about 2.3 Gb and not growing anymore.
So looks like some kind of bug in the kernel VM management part.
Just curious: is any body know what is changed in VM stack since 9.0 RELEASE ?

P.S. It's a last time I am installing an latest release on the production server.
The right decision was to stay at the 8.3. I really thought the this kind of bugs can't appear in such mature part of the FreeBSD kernel.

P.P.S. Old FreeBSD user and a little bit developer since 2.1.1

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
NAMEI memory leak occur again after update.	belon_cfy	Storage	5	August 2nd, 2012 12:15
Help with net-snmp memory leak	roscoe	Web & Network Services	5	December 30th, 2010 02:29
amd64-current. some little memory leak in kernel	samspeed	FreeBSD Development	3	April 12th, 2010 23:54
Memory leak	Keshkins	General	9	April 1st, 2009 23:46
FreeBSD 7.0-RELEASE-p4 && xorg-7.3_2 huge memory leak	TheFeaR	X.Org	17	November 27th, 2008 10:54