bhyve memory requirements?

[`Orum]

At work I've been virtualizing our Windows servers to run under bhyve, and I've run into some memory-related issues. Namely, servers are crashing due to memory exhaustion, with the message: "pid XXXXX (bhyve), uid 0, was killed: out of swap space" which I assume means the host ran out of memory. For the time being, I've limited things to just running a single Windows server to aid in troubleshooting. While I will be adding more memory to the server, the current situation leaves me with a few questions; but first, let me provide some details.

Host: 4GB of memory, FreeBSD 11.1 (amd64), ZFS (prefetch disabled), no swap
Guest: Currently configured for 1GB of memory, Windows Server 2008 R2 SP1, utilizing on average ~1/3 of system memory within the guest
Memory graphs when guest crashes (via LibreNMS): Host - Guest

I realize there's not a lot a lot of breathing room with ZFS and even a single, 1 GB guest, but I would still think it should be doable. So what's going on? I'm speculating, but as the large cache from zfs doesn't leave much free memory, could it be that bhyve requests a sudden, large allocation, and before the cache memory can be freed up the allocation fails?

If so, the ideal solution would be to set FreeBSD to utilize memory for the ARC/ZIL caches until it would reduce free memory beyond a certain point; i.e. set a "reserve" amount for free memory. However, I'm not aware of any setting like this, so if there isn't, is the next best thing to set vfs.zfs.arc_max?

 

[sko]

Host: [...] no swap

You *have* to specify some swap space. Even with available RAM the system uses swap for some operations and fails if none is available.
On desktop systems firefox (amongst other software) is notoriously crashing with the weirdest crashdumps/tracebacks if there is no swap available - even on systems with 32GB RAM and more.

Also: 4GB is not enough if you want to virtualize a memory hogging OS like Windows - usually Windows refuses to install on systems with less than 4GB of RAM, let alone run anywhere near "usable" (at least as usable as this toy OS can be...) with <4GB RAM.

Install at least 8GB of RAM for virtualization testing with this single VM and at least double that for production use and _never_ over-provision memory on production systems to keep away gremlins.

 

[abishai]

I'm speculating, but as the large cache from zfs doesn't leave much free memory, could it be that bhyve requests a sudden, large allocation, and before the cache memory can be freed up the allocation fails?

Yes. Also, you can get a panic here.

 

[`Orum]

You *have* to specify some swap space. Even with available RAM the system uses swap for some operations and fails if none is available.
On desktop systems firefox (amongst other software) is notoriously crashing with the weirdest crashdumps/tracebacks if there is no swap available - even on systems with 32GB RAM and more.

I've never needed swap on servers (assuming they had enough RAM), and although I only have two FreeBSD machines running X, I've never had swap on them either. Running Firefox on them as well has been fine without swap.

Also: 4GB is not enough if you want to virtualize a memory hogging OS like Windows - usually Windows refuses to install on systems with less than 4GB of RAM, let alone run anywhere near "usable" (at least as usable as this toy OS can be...) with <4GB RAM.

Yes, 16GB more RAM is already on the way and slated to arrive tomorrow. I suspect the sudden allocation is caused by TrustedInstaller, when Windows checks for updates. But, it's not the guest that's crashing from being out of memory here--it's being terminated by the host.

Install at least 8GB of RAM for virtualization testing with this single VM and at least double that for production use and _never_ over-provision memory on production systems to keep away gremlins.

My concern is that just installing more RAM in the host won't fully resolve the issue. If the problem is that ZFS is eating up all the available memory on the host, that won't change just by adding more memory to the host.

 

[nihr43]

I've run computers for many years without swap, and big servers with only 1GB. No application should be trying to access swap directly, if thats even possible.

OP: your answer.. yes the vfs.zfs.arc_max tunable works well. I use it on bhyve servers.

 

[`Orum]

I've limited arc_max on the host to 256M, started another 1GB Windows VM (leaving the existing one running), and have the host building 11.2, and it's still floating a lot more free RAM than before (about ~512M compared to ~100M without tuning). So while it's not an ideal solution, it seems to at least hold ZFS at bay and keep enough memory free for bhyve.

Of course, I suspect the guests may still crash without some more physical memory allocated to them, but that's not FreeBSD's fault. And, that should be solved once I get some more memory in the server.