Thank for linking to that. I guess the only way is to install an illumos-based OS and try it out. I was interested mainly in the "feel". I remember, coming from Linux, the first time I sat at a FreeBSD shell, I said to myself, "Wow, this is so much better". But it was difficult to point what was different.
For me it was the _MUCH_ more structured and "engineered" feel of FreeBSD compared to linux. This starts with simple things like filesystem-hygiene which has become even worse on linux since systemd scatters stuff in places where they definitely _don't_ belong (e.g. executables under /lib).
Also the integration and interoperability of basic tools and technologies (ZFS) is something that just isn't there on linux and it makes working with the system so much easier and more efficient.
These points are even more true for illumos - at least that's my experience. We're using smartOS for virtualization since ~2 years and I'm running omniOS in a few VMs, mainly for testing. Lots of services I previously ran in jails are now running in zones (dhcp, DNS, radius, ldap....), mainly because migration between hosts is much easier than for jails.
Tools like vmadm/zoneadm (and its siblings) or dladm are very powerful; crossbow is just awesome and even for very sophisticated setups still quite easy to manage compared to e.g. netflow.
SMF is quite a beast, but I've started to like some aspects of it. Like other tools on illumos it has a quite steep learning curve, but after some adjustment period it really makes sense. One thing I just recently discovered and immediately fell in love with, is the ability to get the path of the logfile for a service with
svcs -L <service>
- no more searching where a service puts its logfile or how it is named (yes, looking at you BIND...), just do
less `svcs -L pkgsrc/bind`
.
Regarding Oracle Solaris: Essentially all engineering teams for Solaris' key technologies left after the Oracle invasion or at the latest when Oracle re-closed the Solaris source. All further development of the OS and technologies like ZFS then happened outside of Oracle. So for me it's not only an ethical decision not to use Oracle Solaris, but also a purely technical - especially when it comes to ZFS. You just can't use Oracle ZFS properly with any other OS - it's a one-way-ticket if you import a OpenZFS pool in Oracle.
We're backing up our VM snapshots from smartOS to a FreeBSD storage server and NAS, which _just works™_ A colleague is/was running a Linux desktop with ZFS and was also backing up to the same Servers and still uses his data pool with TrueOS. My storage server at home was migrated from an old debian linux installation to FreeBSD (and omniOS for a few weeks of testing) basically by just exporting/importing the storage pool.
Another thing I'm looking forward to is the bhyve port to smartOS/illumos. Combined with the ongoing development of vmadm for FreeBSD by ProjectFiFo it might become a real thing to seamlessly migrate VMs between FreeBSD and smartOS even using the same tools on both systems!