Other Persistent storage for bhyve guests

A

arapaima

Hi! This is an open discussion and I'm mostly curious about your general ideas and thoughts about the subject and a few struggles I'm having on my way. It's going to be a bit wall-of-texty. Sorry about that, I'll try to make it as easily digested as possible :)
It's about using small static VM images and provide persistent storage by other means.

Current bhyve guest storage
I currently use zvols for bhyve guests and extend storage by setting the zfs property volsize in the host and use growfs(8) in the guest. This has worked flawlessly so far, but growfs(8) spits out some warnings. Should I be concerned about the reliability of this procedure? Are there any UFS tunable I should be aware of that could support in this?

Build bare-minimal images instead of using official VM images
For the bhyve guests I use the official VM-IMAGES. This gives me a fullblown FreeBSD system with everything from finger(1), kerberos(8) and specialized drivers for bluetooth, atm, wireless etc. etc. Since my guests usually serves one very specific function, such as "share this filesystem", "run this webapp", "provide this network service" etc. This makes me tempted to use something like NanoBSD to build small images. With the existent helper functions and possibly with some small modifications I can build my own customized images. Either by providing static ones with prebuilt packages or do customizations when deploying new guests. NanoBSD is built on the idea of a read-only filesystem, where persistent storage for /etc is stored in a special /cfg filesystem. Any modifications made to /etc during runtime should be synchronized with /cfg to provide persistence between boots. /usr/local/etc is linked to /etc/local.

Struggle of customizing NanoBSD images
One thing that just have created problems for my self has been customizing the NanoBSD build. It's very easy breaking your build by setting various src.conf(5) variables. I wouldn't recommend anyone disabling stuff if you don't know what you are doing. I'm also a bit uncertain to which variables to use for installworld, buildworld and for both of them. As demonstrated in the example in the NanoBSD howto you have CONF_BUILD, CONF_INSTALL and CONF_WORLD. The first is exclusively used for the buildworld stage, the second for installworld and the third for both of them.
Keeping things simple would be continuing to use the full FreeBSD release image. It isn't big problem, but I feel just because I have a big house doesn't mean I want to keep furniture around that I never use.

Persistent storage for bhyve guests
The workflow of using static images to deploy various functionality in virtual or various containerized environments isn't something new. I've seen a few where iSCSI provides raw block storage, or NFS for a network mountable system etc. There are also things like GlusterFS and Ceph. But I feel a bit lost in all of this and I'm not sure about to what extent FreeBSD support is provided.
Even though I like keeping things simple, I want something flexible. This is still a single node environment so I'm not sure about the enteprise scale storage systems is the way to go.

Using one guest that provides storage for other guests
The most simple idea I can come up with is to use a large zvol attached to a file server guest that provides NFS for other guests. Something like /exports/host/local could be used to mount /usr/local for example. Maybe something like vale(4) would be interesting to look into to provide fast communication between guests and the NFS file server?
But how would a mounted NFS share perform for things like databases? I'm also a bit concerned about setting up a single point of failure where I'm exposing an NFS service to both DMZ guests and guests on a vlan used for internal services.

EOT
I would be happy hear your thoughts and experiences on storage for bhyve and also possible ways of finding a workflow with stateless virtual machines (even though this part might be a bit off-topic for the storage sub forum).
 
I will only cover what I have experimented with.
My method for creating VM's is to truncate an image and then use the FreeBSD memstick installer to populate it.
That allows me enough customization.
Another method that I have messed with is using Poudriere Image.

Poudriere image [BSD Router Project]

bsdrp.net bsdrp.net
This allows for many ways of creating a bootable image including a NanoBSD like firmware.
There are many other methods that would be good for bhyve including rawdisk and zrawdisk among others.

The only storage options I have messed with were passing a raw NVMe through in various forms..
forums.freebsd.org

NVMe passthru on Bhyve findings

I am currently using a SuperMicro X10DRL board for my Bhyve Virtualization machine. OS boots from a 64GB DOM. I use two Samsung 512GB PM953 drives in a M.2 form factor mounted on a SuperMicro AOC-SLG3-2M2. The slot where the card is installed is set to x4x4 in the BIOS for bitfurication. I am...
forums.freebsd.org forums.freebsd.org

There is a post with custom make.conf settings. My post there has the bare minimum and works on FreeBSD 12.1.
forums.freebsd.org

Share your make.conf and src.conf

Hello :) It would be nice that experienced users shared their make.conf and src.conf to help new users like me :) I know that there are man pages and I read both of them, but real user's configurations are IMHO also helpful!
forums.freebsd.org forums.freebsd.org
 
Thank you very much for taking your time to answer! I haven't made any progress with this project. Mostly because I've been busy with other stuff.
I did play around a bit with building custom images. It works pretty well. It goes something like this:

Code: 
cd /usr/src
make buildworld
make buildkernel
cd release
make vm-release
make vm-install

(if anyone else tries to follow this, make sure to read release(7) and possibly build(7). There's some variables need to be set depending on your preferences.)

This build could be further customized (src.conf(5)) and by using the NanoBSD scripts it wouldn't take too much work to go the route of static "stateless" images.

However, the whole process is very time consuming and rebuilding everything to upgrade to a new patch level isn't optimal. I've been sneaking on distcc as a way of speeding things up. Maybe I could trigger a build with webhooks or something in a similar fashion when an update is available. The goal of it all is to make it as unattended as possible. It would be real neat if FreeBSD commits were signed, and that the build server would verify those signatures upon building a new image. But that's a bit off topic for this thread :)

I'm not aware of zrawdisks. Are these zvols? I think the best way for my storage needs would be something like VirtFS. It seems to be a funded project and there have been some activity at Github. Is this planned to be included in 13-RELEASE? Has anyone tried it? Is it stable?
 
You must log in or register to reply here.
Back
Top