Solved kldunload failed to release memory

I observed kldunload cannot release memory properly. I'm using amd64 FreeBSD 13.0.

Below is my sample code and make file, can refer to Skeleton Layout of a kernel module for more information.

skeleton.c

Code:
/*
 * KLD Skeleton
 * Inspired by Andrew Reiter's Daemonnews article
 */

#include <sys/types.h>
#include <sys/module.h>
#include <sys/systm.h>  /* uprintf */
#include <sys/errno.h>
#include <sys/param.h>  /* defines used in kernel.h */
#include <sys/kernel.h> /* types used in module initialization */

#define BUFFER_SIZE 10*1000*1024 // 10 MB
static char gBuffer[BUFFER_SIZE];

static int demo_init (void)
{
  for (int i=0; i<BUFFER_SIZE; i++)
  {
    gBuffer[i] = 'A';
  }
  return 0;
}

static void demo_exit (void)
{
  for (int i=0; i<BUFFER_SIZE; i++)
  {
    gBuffer[i] += 1;
  }
}

/*
 * Load handler that deals with the loading and unloading of a KLD.
 */

static int
skel_loader(struct module *m, int what, void *arg)
{
  int err = 0;

  switch (what) {
  case MOD_LOAD:                /* kldload */
    uprintf("Skeleton KLD loaded.\n");
    demo_init();
    break;
  case MOD_UNLOAD:
    uprintf("Skeleton KLD unloaded.\n");
    demo_exit();
    break;
  default:
    err = EOPNOTSUPP;
    break;
  }
  return(err);
}

/* Declare this module to the rest of the kernel */

static moduledata_t skel_mod = {
  "skel",
  skel_loader,
  NULL
};

DECLARE_MODULE(skeleton, skel_mod, SI_SUB_KLD, SI_ORDER_ANY);


Makefile
Code:
SRCS=skeleton.c
KMOD=skeleton

.include <bsd.kmod.mk>

The ko occupies a huge space (around 10MB) when loaded in memory due to its huge bss as shown in the following picture:
Picture1.jpg



freebsd-memory.sh is used to check the memory usage. This file can be downloaded using fetch https://raw.githubusercontent.com/ocochard/myscripts/master/FreeBSD/freebsd-memory.sh.

For every time load/unload skeleton.ko, can observe the memory consumed by kernel was increased by around 10 MB (4738MB - 4728MB = 10MB) as shown below.

Code:
root@freebsd-13-0-n01:~/test2 # kldload -v ./skeleton.ko
Skeleton KLD loaded.
Loaded ./skeleton.ko, id=5
root@freebsd-13-0-n01:~/test2 # kldunload -v ./skeleton.ko
Unloading skeleton.ko, id=5
Skeleton KLD unloaded.
root@freebsd-13-0-n01:~/test2 #

Picture2.jpg


The Wired memory is the Memory in use by the Kernel.

1639126983279.png


A sh file is used to repeatedly load/unload the ko for 10 times, can observe the memory consumed by kernel was increased by around 100MB (5160MB - 5062MB = 98MB) as shown below.

skeleton.sh
Code:
#!/bin/sh

max=10
for i in `seq 1 $max`
do
    echo "-------------- $i -------------- "
    kldload -v ./skeleton.ko
    sleep 3
    kldunload -v ./skeleton.ko
    sleep 3
done

Picture3.jpg


The memory cannot be released even after days.

Is there a workaround or config to solve this problem? Thanks in advance.
 
Last edited:
if i run
while true;do kldload ./skeleton.ko;sleep 1;kldunload skeleton;done
the wired stays constant so the block may not be moved imediatly put the kernel does not run out of memory
 
  • Like
Reactions: mer
Is there a real scenario where a kernel module needs to be loaded and downloaded cyclically?
I mean, it's like reinstalling the OS over and over is an end in itself. 🤷‍♂️
SWAP memory management and desktop freezing was one of the reasons I looked at FreeBSD.
 
if i run
while true;do kldload ./skeleton.ko;sleep 1;kldunload skeleton;done
the wired stays constant so the block may not be moved imediatly put the kernel does not run out of memory
Thanks Covacat for the reply, but I tried using skeleton.sh to repeatedly load/unload for 10 times, and observed the wired memory was increasing. I have updated my post with skeleton.sh and test results accordingly.
 
Not much of a programmer, certainly not a kernel developer but shouldn't that gBuffer be allocated with malloc(9) when the module is loaded and free(9) when unloaded? At least that's what the example in 9.3 does to initialize a buffer.
 
Not much of a programmer, certainly not a kernel developer but shouldn't that gBuffer be allocated with malloc(9) when the module is loaded and free(9) when unloaded? At least that's what the example in 9.3 does to initialize a buffer.
Thanks SirDice for the reply.
I intentionally use huge bss to exaggerate the issue.
Do you know whether or not there is a config/workaround to help kldunload to release memory properly? Thanks in advance.
 
Is there a real scenario where a kernel module needs to be loaded and downloaded cyclically?
I mean, it's like reinstalling the OS over and over is an end in itself. 🤷‍♂️
SWAP memory management and desktop freezing was one of the reasons I looked at FreeBSD.
Thanks CuatroTorres for the reply.
I agree with you that loading/unloading of ko so many times should not be a real use case.
But it's possible we load/unload ko for several times, so more or less this issue consumes some memory.
Considering we have FreeBSD servers running for a long time (can be several months) without reboot, it can be a potential issue to slow down the server.
 
You need to check if the memory gets unwired when memory becomes tight. Ad mentioned above by shkhln memory pressure is what frees inactive memory..
 
The covacat 's test makes sense, it's what I'd expect. Once module is unloaded all static memory is freed along with it.

How does FreeBSD behave under high memory pressure in regards to wired memory? To my understanding this memory is untouchable, you can't swap it out. So even during heavy memory demand these pages are untouched.
 
Swap?

I was kinda hoping I wouldn't have to explain it. Memory is being tracked in two places: the kernel keeps account of allocated pages (that's what utilities like top and sysctl vm stats show), while memory allocators maintain their own data structures for more granular tracking. Those allocators typically do not return pages immediately to the kernel after all allocated byte ranges within a particular page are freed. Why is that so? There is usually a good chance those pages can be reused on the next malloc call, so this is a common optimization. The term "memory pressure" refers to heuristics controlling that behavior.
 
I tested this on my amd64 FreeBSD 13.0 VM and I was able to kill the VM with the test. Test (load/unload) was done over SSH which got killed once system ran out of memory. System is pretty much dead, I can't do anything from the console either. I waited for system to sort itself out. System became responsive again. Wired memory stayed wired (almost all of the available memory) with few MBs free/inactive (lost of processes were killed), free memory is coming from them. It behaves as if that memory is leaked.

I don't know about OP but my confusion (and curiosity) stems from the fact that I'd expect the memory would not be wired once the module is unloaded. The static buffer was not allocated by malloc() so I'm assuming it's not subject of more granular tracking. As demo_init() wrote fully to those pages they are all used (so not only lazy allocated).

Under memory pressure I'd expect system to either laundry what's possible, swap out what can be and even try to free pages by killing userspace processes (OOM killer) to get some memory. But wired memory is untouchable memory. I'm not saying this as fact but as what I'd expect or thought it does.
 
In my logic above I missed the important fact while buffer is statically allocated memory had to be allocated for it by kernel. Kinda obvious now and I feel stupid. But live and learn.

Memory is allocated when object is loaded, freed when unloaded. But still, why would kernel track it as wired memory and not laundry one? I understand that memory is subject to more granular tracking and optimization but I don't understand why it's still wired.
Also my test above makes me believe that memory got lost.
 
This seems to be a FreeBSD 13 behavior (possibly bug as I don't see any feature out of this). I can't replicate this in 12.2. I can do while true ; do kldload ./test.ko && kldunload ./test.ko ; done all day long and the wired memory is oscillating in the boundary of the +10M.
 
Is there a real scenario where a kernel module needs to be loaded and downloaded cyclically?
Here in loop it's just PoC (proof of concept) that something is wrong.

Practical example of unloading and loading back modules is VirtualBox update. After upgrade often you need to unload and load the VirtualBox modules back. While you may not "feel" loosing few MBs of memory it's the idea of a leak that is a problem in general (it's a bug). And each upgrade you'd loose more and more memory.
 
interesting that on 32bit seems not to be affected. or at least the bug can't be trigger by loading / unloading the module
can anyone test on i386 ?
 
I can't replicate it on i386 either, it does behave as expected.
edit: but I was curious to see why that is ; my speculation was not worth pasting here. I did ask on PR, Mark replied with a nice explanation why that is.
 
I know it's an old thread but may be relevant; while compiling (poudriere in VM) gcc11.3.0_1 I am going through a similar scenario where the compilation stops with error 'cannot release memory...' and all the processes are kept killed.

FreeBSD RELEASE 13.1-p2
 
Back
Top