FreeBSD 11-RELEASE-p5 - wired memory steadily growing

Possibly a nothing, but given that the system has "hung" a few times in the last couple of months, apparently due to full RAM and SWAP...

I've got a system running FreeBSD 11-RELEASE-p5 (64 bit), 16 GB of RAM and ZFS. It runs a handful of services, all of which I used to run on an Ubuntu system with 4 GB of RAM without issue.

The wired memory used on the system keeps growing steadily. I can actually watch it tick upwards. Shortly after I rebooted the system it was showing 1.2 GB wired, Now it's up to 6 GB (6 hours after booting). ZFS's ARC cache is currently at 1 GB (4 GB max) and throttled according to zfs-stats.

The ARC was consuming 4 GB up until roughly 4.5 hours after booting, at which point it started getting reduced until it hit 1 GB some 50 minutes later. The timing of the ARC reducing was closely matched by a slow increase in swap usage (from 0 then to about 4 GB now).

Now, I appreciate this may be nothing at all. It's been "some years" since I last used FreeBSD (probably at least 8) and this kind of behaviour may not be abnormal, but it doesn't feel right.

How can I diagnose this? Where can I go looking to see what's causing the steady growth in Wired memory? What other diagnostic details can I provide, or am I just worrying about nothing?
ARC generally only gets reduced if there's something else that requires the memory. To me it sounds like you have some application running that has a memory leak and it's slowly eating away at the available memory. My server at home is running 11-STABLE (recently updated), it's been up for about 5 days now, memory usage looks to be fairly consistent. What else is running on the machine?

last pid: 60441;  load averages:  0.34,  0.41,  0.41                                                        up 5+11:36:21  14:13:34
57 processes:  1 running, 56 sleeping
CPU:  9.2% user,  0.0% nice,  1.3% system,  0.1% interrupt, 89.4% idle
Mem: 627M Active, 1778M Inact, 5354M Wired, 69M Free
ARC: 3899M Total, 1586M MFU, 1639M MRU, 333K Anon, 79M Header, 596M Other
Swap: 16G Total, 1010M Used, 15G Free, 6% Inuse
The Active isn't high, which is what makes me wonder...

So, I've had/got:
  • bind9, isc DHCP server, NTPd, powerd, ssh, Samba, syslog-ng
  • Plex Media Server
  • plexpy
  • Squeezebox server
  • Nagios
  • Apache 2.4
  • snmpd
  • nullmailer
Most of those are shut down now though - only things left are those on the first line and I'm still seeing Wired creep up. Ten minutes ago:
Mem: 2387M Active, 7180M Inact, 6155M Wired, 111M Free
ARC: 1024M Total, 652M MFU, 70M MRU, 2692K Anon, 27M Header, 272M Other
Swap: 68G Total, 1806M Used, 66G Free, 2% Inuse
Mem: 2167M Active, 7330M Inact, 6222M Wired, 114M Free
ARC: 1024M Total, 639M MFU, 84M MRU, 2169K Anon, 27M Header, 273M Other
Swap: 68G Total, 1961M Used, 66G Free, 2% Inuse
Free memory is wasted memory. You should get worried only if you system really runs out of memory, starts swapping excessively and/or starts killing running processes as the result.
Oh yeah, I'm happy with used memory. I'm concerned however when Wired and swap keep growing without any apparent reason - particularly that last night, just a few hours after a reboot to bring it up to p5, it died due to 100% utilisation of RAM and swap (which I only caught thanks to Nagios).
I've canned most of the services and rebooted, let's see if that fixes things
My first suspect would be snmpd, it had memory issues in the past, on multiple occasions. But yes, disable as much as possible, see how it runs, enable one service, see how that works, then another etc.
Thanks SirDice - memory usage is now stable. Once I've got time I'll progress with basic troubleshooting - turning them on again one by one and waiting at least 24 hours for things to settle, or not. When I eventually find the offending service (or services) I'll raise a PR against it.

(Alternatively I may spin up a VM and test them there first)
Yeah, so I left Plex, Logitech Media Server, Samba, syslog-ng, bind and DHCP running and it's looking pretty stable. Wired memory is creeping up very very slightly, looks to be about 7 MB a day, but that's not the kind of growth that gives me any concern - even if that's something there with a memory leak, the system will be rebooted for an update long before that'd become an issue.