nfsd memory leak

[litrbolshik]

Hi!

I have to setup a simple NFS storage on FreeBSD 14.2. And I have memory leak when I read data from FreeBDS NFS share to my Linux host. I use mount -t nfs on Linux and rsync -avP. Please see the attachment how memory usage are growing. And yes, when memory usage come to the size of physical memory FreeBDS system hangs, it does not even answer on keyboard attached.
The setup is simple:
1. One disk with USF rootfs
2. Two disks with UFS, merged by geom to RAID-1 storage, mounted as /data on FreeBSD host.
3. One SSD disk for swap. Yes, full disk for swap.

root@fatboy:~ # gmirror status
          Name    Status  Components
mirror/mirror0  COMPLETE  ada0 (ACTIVE)
                          ada1 (ACTIVE)

I have to update system to the latest state by freebsd-update.
Initially I install the system without cache. 4GB RAM for file sharing host looks enought. After first facing the memory leak I have add the full disk for swap, enable it and put it to fstab. In my point of view cache must help, it reduce the speed of the system and gives the unlimited number of memory. But system hangs when come to the physical memory limit without even using the cache.

 

[litrbolshik]

And one more question how to debug this memory leak - I dont see growing RES for nfsd.

The htop is the only tool indicating correct memory usage. Top and freecolor say everythig is fine, free memory 80%. But when htop is above 3GB, system hangs.

 

[cracauer@]

Initially I install the system without cache.

What exactly did you do?

 

[litrbolshik]

Sorry for misspelling, without the swap.

 

[cracauer@]

Can you share a regular top(1) from when memory runs low?

 

[covacat]

if it is a kernel leak swap wont help you

 

[litrbolshik]

> if it is a kernel leak swap wont help you

I had to try, because I cant believe that release kernel may have an a leak...

 

[litrbolshik]

root@fatboy:~ # uname -a
FreeBSD fatboy 14.2-RELEASE-p1 FreeBSD 14.2-RELEASE-p1 GENERIC amd64

 

[litrbolshik]

root@fatboy:~ # sysctl hw.model
hw.model: Intel(R) Core(TM)2 Duo CPU     E4500  @ 2.20GHz

 

[litrbolshik]

Now I am trying to reproduce the hang situation and it looks like BSD caches every file it reads from disk. No reads, no leaks...

 

[litrbolshik]

After an hour of execution memory consumption stabilized at 2.73GB. Will continue watching. In previous cases hang takes about 20minutes, but the Linux file receiver was the notebook, faster than my PC.

 

[litrbolshik]

I have try to unmount /data (geom managed RAID-1) and mount again. Memory usage drops down from 2.7GB to 110MB. So the source of big memory consumption is geom? Why I don't see it in htop/top/etc? And how to fix so much memory usage by geom?

 

[SirDice]

So the source of big memory consumption is geom?

No, it's the filesystem cache. You unmounted the filesystem, which flushes the cache associated with that filesystem.

 

[litrbolshik]

Yes, I agree with you! Is it normal for UFS to have a cache 3GB size?

 

[pprocacci]

Your use of the word `normal' is subjective.

If you have the memory, the OS will eat the memory for its use -- This is normal.
What isn't `normal' is the system hanging.

You said you couldn't interact with the system via an attached keyboard.
Can you however interact with it in a different fashion? SSH perhaps?