mbuf_cluster exhaustion

[einsibjani]

We have dozens of FreeBSD hosts running, most in VM's on Proxmox, all using virtio NIC's.
All are deployed with ansible, so there should be no pet configs lying around.

*One* of the hosts has, for the last 3 or 4 months started to exhaust available mbuf_clusters and becomes unresponsive.
We set `kern.ipc.nmbclusters` to 3000000, which only stalls the problem, used mbuf clusters keeps rising until it hits the
limit.

We've narrowed it down to apache24 (and php-fpm). We don't see the problem if only apache is started without php-fpm,
but then there is very little that apache serves besides it's status page.

We have plenty of other hosts running apache and php-fpm which don't display this behaviour.

What is the best way for me to debug the issue and try to pinpoint the cause?

 

[einsibjani]

This is a graph showing mbuf_cluster for the last hour.
There are multiple network services running on the host, but if I stop php-fpm, the mbuf_cluster use stops growing.

 

[JordanG]

If you want other people to help you, you should at the very least specify which version of FreeBSD is involved.

I would send an email to the freebsd-net mailing list with more detailed information about the hardware and software configuration.

If all your FreeBSD machines have the same software and only one of them has problems, I would suspect a hardware issue.