I upvoted your original post as it is refreshing to see an opinion from a veteran open source guy like you. Unfortunately, this forum is full of noise. As somebody who is familiar with BSD ecosystem as long as you if not longer I feel that I need to challenge some of your claims. I make no claim that they are free of OpenBSD bias.
I remember when I first started to use (now non existing)
bsdforums.org, I also made a lot of noise
BSD operating systems are at this point so far apart with such different group chemistries that even something as simple as cross OS bug fixing is a challenge. IIlja van Sprundel gave a wonderful presentation on the recent DEF CON which speaks volumes
https://media.defcon.org/DEF CON 25/DEF CON 25 presentations/DEFCON-25-Ilja-van-Sprundel-BSD-Kern-Vulns.pdf
Expecting that FreeBSD and OpenBSD guys work on a common thing is a bit naive. Frankly, I feel that both VMM and bhyve are ill conceived projects.
At least its not yet illegal to dream.
As an avid OpenBSD user I felt that VMM which is the youngest of the two is making my beloved OS unnecessary complicated and cumbersome with minimal benefit. As I user I always felt that I would benefit far more from having sysjails than from full blown OS level virtualization. Kristaps Johnson taught us that Jails are not safe (later was discovered that sysjails were neither so they got killed)
http://www.nycbsdcon.org/2006/speakers.html#Johnson
although convenient as I can attest as a consumer of FreeBSD jails. They suffer the same network problems like bhyve in more realistic deployment scenarios
https://savagedlight.me/2014/03/07/freebsd-jail-host-with-multiple-local-networks/
I still feel that there is some hope for OpenBSD jail like system as we can read BIND Broker by tedu
https://www.tedunangst.com/flak/post/bind-broker
VMM are reality whether I like it or not. I tried them and they feel very much Xen Dom0 like. For me that is a good thing. Xen Dom0 (Alpine Linux) is my favourite hypervisor. I think that one of developers motivation was that Qemu even without kernel acceleration is moving into Linux only direction.
Yes, Jails are also great, the only thing I miss in them is 'live migration' to other FreeBSD hosts. This is where Solaris Zones shine, also SmartOS (Illumos distribution) has nice (free) Solaris Zones implementation with CPU Overbursting and other features described here in real world usage:
http://containersummit.io/events/sf-2015/videos/wolf-of-what-containers-on-wall-street
I am very familiar with VirtualBox and KVM. VirtualBox is desktop virtualization. KVM is more classical level 2 type Hypervisor. I would not run a server in the VirtualBox but I concur that it is very useful for a web developer who must test his product on multiple OSs and browsers. VirtualBox and Xen are as far apart as it gets so VMM are not really useful for somebody who needs VirtualBox. FreeBSD is not officially supported host for VirtualBox and my personal experience confirms that. I would not run VirtualBox on FreeBSD.
I would use VirtualBox on server only if it would be rock stable, unfortunately on FreeBSD it is not. KVM is not bad, especially on OVIRT solution (open source project for upstream RHV - Red Hat Virtualization).
There is experimental Xen dom0 on FreeBSD, but its VERY experimental. If You want to run Windows or Linux or any other OS on your desktop without rebooting, then VirtualBox on FreeBSD seems to be least PITA solution.
There is also other product based on Xen dom0 - Oracle VM for x86, and its FREE (which is strange in Oracle world). Offers separate Oracle VM Manager and features like Live Migration, clonning, snapshots, using SAN network etc. Kinda open source VMware ESXi in many aspects. It also has great 'feature' for Oracle Databases, it is treated by Oracle as HARD PARTITIONING solution which helps You to cut costs on Oracle Databases licensing. On VMware You have to license all hosts under all vCenter managers ...
KVM is ok for server deployment but lacks hot migration comparing to Xen and even more think like block device provisioning where you can directly pass not just HDD but also other things like GPU computing cards directly to Xen host. I think that Red Hat requires now subscription for KVM Windows hosts (please see 7.4 below the release announcement) which means that I Xen will soon be my only option for Windows server as a virtual host.
KVM does support live migration, for RHV there is open source project called OVIRT and its totally free, its another 'open source VMware ESXi' product. You can also use KVM in OpenStack solution, but that also takes ages to jump in to (as a big project).
Unlike OpenBSD I feel that FreeBSD project has bet its entire future on the a super cumbersome, patent incumbent file system ZFS which required reimplementation of the large part of Solaris kernel. FreeBSD is massively larger than Open with many unfinished things (see my rant in the thread what I would like to see done differently on FreeBSD). Things actually worked for FreeBSD and I must admit that I am heavy ZFS users and most large data people I know (I know quite a few) swear by Free (both BSD and NAS).
Yes, FreeBSD project imports all possible 'useful' software that is available on mostly compatible license, also DTrace same from Solaris, but its not as heavy used as ZFS for sure.
FreeBSD decision is further vindicated by the recent Red Hat admission that BTRFS is a vaporware
https://access.redhat.com/documenta...4_Release_Notes-Deprecated_Functionality.html
That officially confirms what many of us knew for a quite some time know that Linux has no modern file system (although early 90s SGI creation XFS with both hardware and software RAID is supper stable). That leaves Solaris, FreeBSD, and DragonFlyBSD as the only three legit storage OSs. Oracle can of course always pull the plug on FreeBSD and DragonFly is minuscule (so much about your cooperation between Open and Free as FreeBSD kicked one of its most charismatic developers which recently was repeated with John Marino).
Yes, from the time of Poetteringation of Red Hat ecosystem everything in Red Hat Linux seems to go the wrong way, illogical way, bad way and that decision is also in that taste (to deprecate BTRFS).
As I digged the Internet why Red Hat made such decision, its was that
'they already had their internal XFS developers while they had NONE BTRFS developers' and that
'BTRFS is changing too quickly to be usable on release once a year version'. They also mention ZFS in their docs, but the only thing keep taking them away from ZFS is its license - CDDL. I can not understand now much Linux zealots cherish that bullshit GPL licence above technology, fortunately in BSD world the license is least interesting thing overlined in two/three sentences and BSD people can focus on technology without all this licensing bullshit.
After abandoning the BTRFS Red Hat will be 'doing' project Stratis, outlined here:
https://stratis-storage.github.io/StratisSoftwareDesign.pdf
In short words, they will use device-mapper and LVM2 with XFS on top of that and write some utilities on python to create 'pools' like things over this. It feels to me so much wrong and useless, but that is how enterprise works, You need features not matter what garbage is under the hood.
I also keep fingers crossed for HAMMER2 and I regret that people in FreeBSD project could not talk enought to continue work together without additional forks like Dragonfly BSD. But thats life.
Why am I taking so much about ZFS when the topic is bhyve. Because just like with Jails, Bhyve are infinitely more useful combined with ZFS underneath even with all network limitations you pointed. Personally I have not given a Bhyve try as I am experimenting with various DomU options on Alpine Linux. As adverse as Linux is to the third party kernel modules ZFS kernel modules do exist for ZFS and Alpine Linux does support DomU installation on the top of ZFS pool. That seems to be winner for me.
I once checked Alpine Linux, and as much I like the goals and features of this project, its sill Linux, which keeps me away from it. I sometimes use ZFS even on Linux systems that officially does not want to have ANYTHING common with it (like Red Hat Linux) and it works flawlessly.
In retrospect I think that all BSDs were way to late for Virtualization party. FreeBSD was too late in part due to interesting Jail concept so much championed by Solaris zones and poorly imitated with Linux containers (docker is another laughable "brake trough" of Linux community. Maybe only NetBSD got it right by porting mature Xen technology instead of developing its own hypervisor but due to the current sorry state of the BSD (the headline for the incoming 8.0 release is support for USB 3.0) I am not sure how well maintained is Xen on NetBSD. One thing for sure I would not use NetBSD in production for anything at this time when the future of the project is so uncertain.
Bhyve and VMM could and will change that as nothing new then KVM or VirtualBox havent been invented, so its only time to catch up
Regards,
vermaden
