Default front-end for bhyve?

[Beastie7]

Any consensus for a default manager for bhyve(8)? It looks like iohyve has a small following and is currently in development from the sysutils/iocage community. Will bhyve have it's own built-in front-end for management like, for instance, pkg(8)? I'm looking to bhyve but I don't know where to start unfortunately. I watched Allan Jude's tutorial; it's scary.. and not easy. lol

 

[usdmatt]

I don't know what the end goal is for the bhyve command itself. I always wanted to be able to just run bhyve start|stop|etc vmname, and just have a config file for each vm containing everything needed to run the machine just like most other hypervisors. Having worked on my own frontend for a while now, I'm starting to wonder if it makes sense for bhyve to stay more or less as it is (as the 'backend' that most users don't touch), and for users to choose a separate frontend - There's a lot more to running machines than just starting and stopping them (networking, configuring console access, etc). I do expect bhyve to get support for reading all the guest device settings from a config file at some point though, rather than passing them all on the command line every time. I believe Allan Jude is involved with that so it might not be too dissimilar from his bhyve-ucl work (If ucl makes it into base).

There are various frontends available at the moment - iohyve, vmrc, bhyve-ucl, vm-bhyve. My favourite out of those used to be iohyve, until I wrote my own vm-bhyve. I really liked the way iohyve worked, which encouraged me to write my own, but didn't like it being tied to ZFS (I might want to use NFS or UFS-on-iSCSI-on-ZFS at some point - or hell even run bhyve on a system with just plain UFS). I also found the way it handled maintaining and restarting the guests to be a bit pathological. (That's improved recently). iohyve was developed by one guy as far as I can tell, who is now working on a branch of ioCage to build it in so ioCage can do jails or bhyve.

With mine I wanted it to just have a simple config file for each VM (I think config files make more sense for a generic frontend than ZFS properties), and handle at much as possible automatically (tap & console devices are all created dynamically as needed). That's great for 99% of cases, but not as much if you actually want control over how those devices are configured.

I have no idea whether any of these will make it into base as a standard frontend, or whether the FreeBSD devs will provide one. It may just be that users will choose their own frontend if they don't want to talk directly to bhyve, and download them from ports/pkg. This is similar to jails really - It's not that straight forward to actually make full use of jails by hand, with most users choosing to download iocage/ezjail/etc.

 

[ondra_knezour]

Yesterday I came across this on the FreeBSD wiki

1. Xinuos has done lots of work on running different OSes on bhyve. Patches are in progress and will arrive in FreeBSD when complete and tested.

2. In order to make FreeBSD more usable as a host we need more management tools. Xinuos is working on tools for managing bhyve guests, which will arrive in FreeBSD when complete and tested.

 

[usdmatt]

A lot of bhyve users can't wait for that UEFI/OS support work to appear in CURRENT. Several FreeBSD users on twitter have been talking about having it running it for months.

Wasn't aware of the second point. I'll be intrigued if they do actually pull a bhyve management tool into base. In a way it seems to make more sense to keep it simple like jails, and let people choose a management tool from ports that suits their needs if they want. I expect (hope) at some point we may even get a 'FreeNAS for virtualisation', with the same polish as FreeNAS, but for running bhyve guests and/or jails.

 

[Beastie7]

I don't know what the end goal is for the bhyve command itself. I always wanted to be able to just run bhyve start|stop|etc vmname, and just have a config file for each vm containing everything needed to run the machine just like most other hypervisors. Having worked on my own frontend for a while now, I'm starting to wonder if it makes sense for bhyve to stay more or less as it is (as the 'backend' that most users don't touch), and for users to choose a separate frontend - There's a lot more to running machines than just starting and stopping them (networking, configuring console access, etc). I do expect bhyve to get support for reading all the guest device settings from a config file at some point though, rather than passing them all on the command line every time. I believe Allan Jude is involved with that so it might not be too dissimilar from his bhyve-ucl work (If ucl makes it into base).

There are various frontends available at the moment - iohyve, vmrc, bhyve-ucl, vm-bhyve. My favourite out of those used to be iohyve, until I wrote my own vm-bhyve. I really liked the way iohyve worked, which encouraged me to write my own, but didn't like it being tied to ZFS (I might want to use NFS or UFS-on-iSCSI-on-ZFS at some point - or hell even run bhyve on a system with just plain UFS). I also found the way it handled maintaining and restarting the guests to be a bit pathological. (That's improved recently). iohyve was developed by one guy as far as I can tell, who is now working on a branch of ioCage to build it in so ioCage can do jails or bhyve.

With mine I wanted it to just have a simple config file for each VM (I think config files make more sense for a generic frontend than ZFS properties), and handle at much as possible automatically (tap & console devices are all created dynamically as needed). That's great for 99% of cases, but not as much if you actually want control over how those devices are configured.

I have no idea whether any of these will make it into base as a standard frontend, or whether the FreeBSD devs will provide one. It may just be that users will choose their own frontend if they don't want to talk directly to bhyve, and download them from ports/pkg. This is similar to jails really - It's not that straight forward to actually make full use of jails by hand, with most users choosing to download iocage/ezjail/etc.

Agreed, I'm in favor of a more filesystem agnostic approach to Jail/Bhyve management. I have a lot of idea for jails on embedded devices, for example, that could necessitate UFS. However, I don't really understand the use of multiple front-ends to what will likely produce the same end result. Like pkg(8) and zfs(8) I'd like for some consensus on a standard, yet modular way of administering Jails, and bhyve. I hate communal fragmentation of tools on standard technologies. To me its' a stupid Linuxism, and a waste of little man power we have. Also, why sh instead of C? Just curious.

I wish the Foundation or some developer with authority could sit down and implement this concept.

 

[usdmatt]

It appears FreeBSD devs are working on a bhyve frontend which will be the 'official' one according to the wiki, so we'll have to see what they come up with. Maybe it'll be perfect and no one will ever use anything else.

You're always going to have different solutions to the same problem that have different ideas though. Even with ZFS which you mention, you can run ZFSGuru or FreeNAS or probably other abstractions if you don't actually want to (or know how to) go on a console and run zfs commands directly. We have different operating systems, database engines, web servers, all with different ideas on what they are trying to achieve. Users just pick the one that meets their needs best.

With my front end, I wanted everything to be as easy as possible, with no knowledge of nmdm/tap/bridge/etc required, and without having to know all the loader commands for each OS; which is no good if you actually want control over all that. iohyve is basically an iocage clone for bhyve, tied to ZFS, but will likely eventually allow you to easily clone VMs instantly, move between hosts using send/recv and do other cool ZFS-specific stuff.. vmrc is a fairly light wrapper where the guest config files contain pretty much everything you would usually have to type in by hand, making the config files much more complex, but allowing you control over loader arguments, networking/bridge setup etc. Each one has different merits depending on what you're after.

I'm no C programmer which is partly why I used sh. Also, I took inspiration from the code behind iohyve which is sh. It's actually my first 'proper' shell script, but it's fairly easy to pick up. All of these utilities seem to be sh though, as do all the jail ones, and even base FreeBSD stuff like bsdinstall. The common theme between all these things is that really they're just wrappers around a bunch of built in utilities, and sh is actually pretty good for that, as it's literally just a a way of running existing commands with a few control statements thrown in. The iocage dev was interviewed on BSDNow recently, and said how surprised he was what he could do with shell, after rewriting his code from perl. With C you really want to be using some sort of API to access the features you are using; A C program that's constantly executing external utilities sounds messy to me. I don't really think using C would of actually made for much of a better end product. I could of made something a lot easier and neater using perl or another higher level scripting language, but for the exact same reason iocage moved from perl to sh, I wanted something without dependencies.

 

[Beastie7]

It appears FreeBSD devs are working on a bhyve frontend which will be the 'official' one according to the wiki, so we'll have to see what they come up with. Maybe it'll be perfect and no one will ever use anything else.

You're always going to have different solutions to the same problem that have different ideas though. Even with ZFS which you mention, you can run ZFSGuru or FreeNAS or probably other abstractions if you don't actually want to (or know how to) go on a console and run zfs commands directly. We have different operating systems, database engines, web servers, all with different ideas on what they are trying to achieve. Users just pick the one that meets their needs best.

With my front end, I wanted everything to be as easy as possible, with no knowledge of nmdm/tap/bridge/etc required, and without having to know all the loader commands for each OS; which is no good if you actually want control over all that. iohyve is basically an iocage clone for bhyve, tied to ZFS, but will likely eventually allow you to easily clone VMs instantly, move between hosts using send/recv and do other cool ZFS-specific stuff.. vmrc is a fairly light wrapper where the guest config files contain pretty much everything you would usually have to type in by hand, making the config files much more complex, but allowing you control over loader arguments, networking/bridge setup etc. Each one has different merits depending on what you're after.

Oh I was referring to standard FreeBSD technologies and CLI front-ends. For third party stuff though, yeah definitely.

I'm no C programmer which is partly why I used sh. Also, I took inspiration from the code behind iohyve which is sh. It's actually my first 'proper' shell script, but it's fairly easy to pick up. All of these utilities seem to be sh though, as do all the jail ones, and even base FreeBSD stuff like bsdinstall. The common theme between all these things is that really they're just wrappers around a bunch of built in utilities, and sh is actually pretty good for that, as it's literally just a a way of running existing commands with a few control statements thrown in. The iocage dev was interviewed on BSDNow recently, and said how surprised he was what he could do with shell, after rewriting his code from perl. With C you really want to be using some sort of API to access the features you are using; A C program that's constantly executing external utilities sounds messy to me. I don't really think using C would of actually made for much of a better end product. I could of made something a lot easier and neater using perl or another higher level scripting language, but for the exact same reason iocage moved from perl to sh, I wanted something without dependencies.

Hmmm, so for an API (if I'm understanding this correctly), would something like libpkg need to be developed? According to its' GitHub page it's some sort of library interface. I'm not sure how the internals of Jails works or if it has it's own API like implementation, but your point about an API really clears things up though. I supposed wrapping sh around C based utilities is easier and less bloating. Damn, I need to study this stuff. lol