VM crashes - out of swap space

[junialter]

Hi,
I'm migrating most of my Linux VMs to FreeBSD all hosted on a proxmox system:
* Ryzen 9 5900x
* 128 GB DDR4 RAM
* ZFS Filesystem for guests on NVME Mirror

One of the VM guests is serving as a public NTP server (pool.ntp.org) and about ever 12 hours it crashes. On the console it says:
pid 2340 (find), jid 0, uid, was killed: out of swap space
sometimes it kills the getty process instead. The System is FreeBSD 13.

What's weird is that the system is far away from not having enough RAM. After a reboot it uses about 100MB out of 2000 MB RAM (screenshot). The swap isn't being used at all.
I have no idea how to approach this. In messages there is nothing being logged at all.
I'm using qemu-guest-agent and RAM balooning, dunno if it might have something todo with it.

Thank you for any hints.

 

[cmoerz]

I've encountered similar situations when using bhyve in conjunction with ZFS; in my case, ZFS's arc was gobbling up more memory than expected. Then again, you're mentioning that this happens about every 12 hours - are there any cronjobs or anything else running in the background that looks suspect?

How much swap space have you set up? Are you maybe running without any swap at all? Sorry, just realized, you're running with 2GB. Have you tried putting more swap in? Does that affect the situation?

 

[Alain De Vos]

Create a tmpfs and write a large file into it as test of the swap.

 

[junialter]

I have a cron job that runs every night for borg backup. When I run it manually nothing weird happens. The job takes only a couple of seconds.
I created a tmpfs and wrote 2 GB into it. I see the swap space being heavily used but the system keeps runninig smoothly...

 

[junialter]

I was so desperate that I reinstalled the whole system with FreeBSD 12.2 and 4 GB of swap instead of 2. When it turns out to be stable I will try to upgrade to 13.0 once again, of course after having made a ZFS snapshot.

 

[bakul]

Note that when the system is out of swap space, a *random* process will be killed. Looking at your htop display, it seems the memory hog is icinga2 what that might be.

I understand that if you are under time pressure you want the problem to just go away, but in general I would suggest *not* making such drastic changes as changing the OS release. You may end up changing multiple things and the problem might magically disappear -- until the next time. Instead make a guess what might be the problem, check things to validate your guess and then only change one thing at a time. In your case I would guess the icinga2 is the culprit so you can just capture its memory use every second (or every 10th of a second or whatever) and see if it keeps increasing.

As this is running under a VM, you can adjust the amount of memory you give it to make it die *sooner*. In general I would not give it swap space -- it will just take longer to die if there is a genuine memory leak.

The idea is that once you have a problem, you want to *narrow* down the conditions under which it occurs until there is no doubt what went wrong. You don't want it randomly disappear because the next time it happens you will go through the amount of hair pulling!

One more thing. If your host is running zfs, there is no point in giving zfs to the VM -- it will just use more memory and slow things down.

 

[usdmatt]

For something with a fairly low amount of memory like that I would limit ZFS ARC and see if the problem goes away.

/boot/loader.conf

vfs.zfs.arc_max="1G"

It's always been a problem and still doesn't seem to be much better in my experience, but ZFS will happily starve the system of RAM if you don't force it not to. Especially likely at 3am if you don't disable the various cron scripts that scan the entire filesystem such as the security checks or locate database update.

I still limit ARC on most of my servers to a value that leaves a reasonable amount spare for the other services I run on the system.

 

[PMc]

I have seen a similar thing recently, trying to compile graphics/oidn. That piece starts as many compiler processes as there are CPUs (in my case: 4), and each of them immediately occupies ca 1 GB of RAM. Running that in a bhyve with 4 GB RAM (and opportunity to swap) does not come to swapping, but kills the process. I did a workaround by significantly enlarging the oom sysctl's and limiting the ARC size (not sure which of them fixed it, but it is very likely ARC related).
I haven't seen such ARC problems on host systems; this must be related to bhyve (or due to running ZFS on both host and guest, as I need snapshoting in the guest). I think this is some VMM race condition and shouldn't happen - but here my Release is 12.2, and I thought it might be improved in 13.

 

[bakul]

this must be related to bhyve (or due to running ZFS on both host and guest, as I need snapshoting in the guest)

May be bhyve should provide snapshotting?

 

[cmoerz]

I second usdmatt's recommendation; limiting ARC will probably help you. Also, you may want to check top instead of htop. Sure, htop looks way nicer in color and has some helpful features. Unfortunately, it doesn't show you ARC or how much memory is wired. Both allocations will limit how much memory is freely available for your applications.

Does it help, if you turn off RAM balooning? Usually, that may lead to swapping issues, I believe. Since you're considering setting up 12.2 instead - I'd recommend trying a plain vanilla installation first, without any VM guest software. This way, you can clear that part of the equation of being the possible culprit.