how to find out memory leak on 14.1-RELEASE-p4

[RPUgCZUqR4qZ]

We have few servers with 14.1-RELEASE-p4, all of them have significant memory leak
Please, help us find out what exactly leaking and fix it.
Intersting observation is 14.1-RELEASE-p3 don't have the issue, with same set of services/processes/aplications.

here is top header

Mem: 135G Active, 31G Inact, 60G Laundry, 32G Wired, 56K Buf, 6099M Free
ARC: 18G Total, 5280M MFU, 10G MRU, 11M Anon, 274M Header, 2380M Other
     14G Compressed, 26G Uncompressed, 1.90:1 Ratio
Swap: 66G Total, 20G Used, 46G Free, 29% Inuse, 104K In

mem line is 135+31+60+32+6 = 264 not 272Gb total installed RAM

sysctl vfs.zfs.arc_min; sysctl vfs.zfs.arc_max
vfs.zfs.arc_min: 17179869184
vfs.zfs.arc_max: 34359738368

sysctl

hw.physmem: 291599888384
hw.realmem: 294191628288

example memory gap, it can be up to 50%

SYSTEM MEMORY INFORMATION:
mem_wire:       28933750784 (  27593MB) [ 10%] Wired: disabled for paging out
mem_active:  + 125202481152 ( 119402MB) [ 44%] Active: recently referenced
mem_inactive:+  12180725760 (  11616MB) [  4%] Inactive: recently not referenced
mem_cache:   +            0 (      0MB) [  0%] Cached: almost avail. for allocation
mem_free:    +  10700419072 (  10204MB) [  3%] Free: fully available for allocation
mem_gap_vm:  + 107080982528 ( 102120MB) [ 37%] Memory gap: UNKNOWN                 <<<<<<<<<<<<<<<<<<<=====================
-------------- ------------ ----------- ------
mem_all:     = 284098359296 ( 270937MB) [100%] Total real memory managed
mem_gap_sys: +   7501529088 (   7154MB)        Memory gap: Kernel?!
-------------- ------------ -----------
mem_phys:    = 291599888384 ( 278091MB)        Total real memory available
mem_gap_hw:  +  51997495296 (  49588MB)        Memory gap: Segment Mappings?!
-------------- ------------ -----------
mem_hw:      = 343597383680 ( 327680MB)        Total real memory installed

SYSTEM MEMORY SUMMARY:
mem_used:      320716238848 ( 305858MB) [ 93%] Logically used memory
mem_avail:   +  22881144832 (  21821MB) [  6%] Logically available memory
-------------- ------------ ----------- ------
mem_total:   = 343597383680 ( 327680MB) [100%] Logically total memory

vmstat -z and vmstat -m attached

 

[cracauer@]

20 gigabytes swap used is not pretty.

Do you still have a comparable p3 machine running?