Bhyve VMs being killed

[Slable]

Hi everyone,

I am running into an issue with some of my Bhyve VMs that's quite puzzling. Currently I am using a setup where I have a VM running on the host that mounts an NFS share that resides on the host system so it can access the ZFS backed storage. The idea of this is that we can easily add disks to increase our available storage. The idea is that the VM itself will be small but can access a large pool of storage.

What I have noticed multiple times now is that whenever I try to sync a large file to the VM, the VM in question dies along with the other virtual machines I have running on my host. I'll have a large file try to sync to the VM, leave, come back the next day and then notice that ALL of my Bhyve virtual machines have died, not just the one I was syncing the file to. It's almost like Bhyve doesn't know what to do with the large file so it just kills all the virtual machines. I looked for a Bhyve log file to see if it would have any helpful information but there doesn't seem to be one. Very odd behavior and I was wondering if any of you guys had some ideas as to what might be causing this. Thanks. You guys are such a wonderful community!

 

[grehan@]

What might be happening is that the ZFS ARC cache is using a large amount of RAM, which results in bhyve VMs starting to swap, and eventually being killed if swap space is running out.

You should be able to see this by monitoring swap usage ( swapctl -shl, or the 'Swap:' row in top) and also ZFS ARC usage.

There are two ways to fix this: limit the size of the ZFS ARC cache, and increase the amount of swap space available so that all VM memory can be accounted in it.

Adding memory is another, but perhaps not necessary.

 

[Slable]

Ah, okay, thank you grehan@! The issue though is that I disabled swap on my VMs because I noticed that many of them were running very slowly when I had it enabled. They were going to swap way more often than they should be for what they're doing. It seems like I may need to choose between speed and stability unless the ZFS ARC cache is the issue which I will look into Thank you Grehan!