general/other Server advice - migration to BSD or Debian Linux?

Good morning all!
I recognise this is a BSD forum, but as you are all more knowledgeable than me on these matters, I value your opinion.
I have a server running Proxmox (Linux based, not BSD), and at the moment I am having issues which I believe are related to some flaky hardware. Since this will mean I need to make some significant effort to resolve this, I think it is a good time to decide whether to reconfigure my server and home network to something a bit simpler.
Current configuration:
  • Proxmox, running on the hardware, with the following VMs:
    • TrueNAS scale (4 disk passthrough - spinning rust)
    • Nextcloud, on a Debian VM, also with some small websites I host for myself, and for the local community band
    • A Debian install, with several small docker applications and Immich (also in Docker). I am running Asterisk on this as well to connect to HamsOverIP and HamshackHotline.
    • Jellyfin
Since TrueNAS Scale could host the three other VMs, I think the use of Proxmox is superfluous. However, I would like to do the configuration "on the metal" and avoid the use of Proxmox or TrueNAS Scale, and simply put the VMs on a host operating system - either BSD or Linux. I know much more about the latter (used for 20+ years on a daily basis), but understand the merits of BSD for server applications, even if I know much less about BSD at the moment.

I believe the following would be a good route to take:
  • FreeBSD running on the hardware, with the following VMs:
    • Debian - running the docker applications etc, as above
    • Jellyfin - also running in a Debian VM
The NAS services with the 4 hard drives would be managed by the base system. Nextcloud could be installed in the base system as well, if that's better than using a VM, and the websites could be run in jails. I could also look at not using Nextcloud - it is for file sharing and calendars, so I am not using anything like all of it's features and something simpler could suffice.

What do you think? Am I heading in the correct direction, or have I completely misunderstood what I should be doing?

Thanks for reading!
 
I do assume you will replace your flaky hardware first.

FreeBSD ships with ZFS included. So it is a first class citizen, and well documented.

FreeBSD has its own hypervisor named bhyve which can be used to runs VMs. There are several wrappers around to manage it, like CBSD or vm-bhyve. Personally I've been using vm-bhyve since ages and am quite happy with it. It also supports ZFS out of the box, which means you can make a quick snapshot of the VM file for backups. Bhyve exposes the VM via VNC to you, if you want to.

FreeBSD does not have docker, but jails. Jails are the container solutions builtin to FreeBSD. Again there are some frontends around, I've been using ezjail-admin for ages. Jails is different compared to Docker that it just gives you a container, nothing more nothing less. It is up to you to update and maintain the services in it. Jails has no equivalent to docker-compose.yml files.

Nextcloud could easily run in a jails container. For Jellyfin you will need a VM though, because it does not officially support FreeBSD. In case you need transcoding you should ensure to use the proper CPU type in the VM in case your CPU has hardware support for it.
 
Whats the logic behind using Docker or NAS in a VM? That sort of defeats the purpose.

A simple setup would be running all of your applications in a Jail(s) on top a bunch of ZFS datasets. If you required an app that only runs on Linux, then you can throw in Bhyve. With ZFS + Jails you get an HCI-like configuration.
 
Whats the logic behind using Docker or NAS in a VM? That sort of defeats the purpose.

A simple setup would be running all of your applications in a Jail(s) on top a bunch of ZFS datasets. If you required an app that only runs on Linux, then you can throw in Bhyve. With ZFS + Jails you get an HCI-like configuration.
Thanks for the response! It's a good question, and my thoughts on how to proceed are based on me not understanding the possibilities of jails and docker in FreeBSD.
  • I agree that running a NAS in a VM is not worthwhile. Just because it can be done, doesn't mean it should!
  • I understand that Docker can not run natively in FreeBSD, but needs to run in a VirtualBox environment. I'm not so fond of VirtualBox, so I was considering how to avoid using this. For the applications currently running in Docker, some of them seem to be docker only installations - such as immich. Some of the others may run natively in FreeBSD - NodeRED, grafana? I need to explore more.
I have a Dell Wyse 5750 available - I think I will first of all replace the motherboard in the main server (just been to Preston to fetch it, along with 4 x 8TB drives, which will go in once everything is confirmed working again), and get it running again in its current configuration. Meanwhile, I will look at setting up the Wyse 5750 with FreeBSD and explore ZFS, Jails and bhyve. This could give me some experience in setting everything up without taking downtime on the current setup. I have a good collection of Michael Lucas's books which I picked up as a bundle some months back - these should be good guidance, alongside the official documentation.
 
For Jellyfin you will need a VM though, because it does not officially support FreeBSD. In case you need transcoding you should ensure to use the proper CPU type in the VM in case your CPU has hardware support for it.
Which doesn't mean that Jellyfin won't run on FreeBSD out of the box, though. I'd be more concerned with the thumbnail generator middleware's (Skia) screwing up - independend of the OS it's running on - then with Jellyfin itself on FreeBSD. It's a dotnet application and we now have decent dotnet support. Properly providing multimedia content into a true VM instance, rather then a container, is going to come at a significant cost in terms of either performance or setup effort, though.

AFAIK if your CPU isn't properly supported on a FreeBSD host (Intel Core i 12th gen or newer, Xeon Scalable 4th gen or newer, AMD zen4 or newer, ...), then bhyve hosted virtual machines are not going to be able to overcome the host OS's shortcomings, either.

I understand that Docker can not run natively in FreeBSD, but needs to run in a VirtualBox environment. I'm not so fond of VirtualBox, so I was considering how to avoid using this. For the applications currently running in Docker, some of them seem to be docker only installations - such as immich. Some of the others may run natively in FreeBSD - NodeRED, grafana? I need to explore more.
I might be religious here but IMO, if a software cannot be used outside of a Docker Container, then it shouldn't be used in the first place. Docker is supposed to be a containerization solution, not a software distribution solution, and it also lacks most requirements we put into distribution systems. E.g. on Debian, when you deploy a deb package, apt/dpkg will automatically verify the package's cryptographic signature against the pgp credential associated with the repository the package was downloaded from, deb files contain changelogs, apt will tell you what it's going to do besides the deployment order you gave it prior of doing so, it will provide a migration path if your configuration files need modifications, and so on. If you're mistrustive of the stuff debian's providing you, you can check debians' gitlab server and review the CI logs to see what was done in order to build the deb file the CI published (the packages are signed and published by the CI so no debian maintainer could abuse his position to sabotage a package w/o anyone noticing), then make your own decision whether you want to install the debian provided package or build it from source (or apt source) yourself instead. What does docker have to compare? Unsinged SHA256 hashes, no changelog, no migration strategies, image creation intransparence on all major docker registries, ...

If it was just used as a containerization solution and only for that, then the content of the Dockerfile(s) could be turned into a sequence of docker run/copy/mount/... commands on a stateful container, which could then be converted into a sequence of lxc or bastille or raw jail commands if you decided to migrate to another containerization solution. Or you could wrap it all up nicely in ansible or similar orchestration frameworks, to have them either generate the command sequence appropriate for the containerization solution, or generate the Dockerfile / Bastillefile template instead and then invoke docker build / bastille create/template, which scales alot better then handwritten dockerfile stuff, especially if you have a lot of containers.

... which is also what I'd recommend: create some automation that can deploy your debian-based setup completely. Once you have it and you're sure it works, create a virtual machine with FreeBSD on the Debian system and start wrapping the Debian specific commands to enable the orchastration to use FreeBSD codepaths in the VM to do the same thing(s) it does with Debian codepaths on the host. Try getting more and more services that don't hard-depend on Docker into jails, and only when you're done figuring out what services you can't move into jails, and you're still confident you want to make the switch to FreeBSD, reinstall to FreeBSD, pull up a VM with Debian inside on the new FreeBSD installation and run your orchastration for all containers that could go into jails on the host and all those that couldn't go into jail to orchastrate Docker containers into the Debian VM instead.
 
Jellyfin works fine inside a jail.

As far as immich, it is possible to run inside a jail, but needs a lot of work to get running.

If you do decide to switch and get immich running somehow, I’d appreciate a walkthrough as I haven’t been able to get it, although others have at times.
 
New motherboard installed on the server this morning, and it seems to have resolved the flakiness issues I was experiencing - time will tell...

I have also set up and installed FreeBSD on the Dell Wyse 5750, and will now spend time with learning more about ZFS, Bhyve and so on.

xibo - I agree with your points, but I will probably aim to get stuff running first, then work on better (non-docker) installation later. Your migration path looks good - I am retiring soon from paid work, so I can put some effort into it going forward. What ever I do, I will be planning the setup more concretely this time, rather than the haphazard approach I used when first setting everything up! Also, I will document everything!
victort - I will let you know if I manage to get immich working natively, but initially I may go down the jail route.
Beastie7 - thanks for the freshports tip off!

Thanks for all your feedback!
 
Back
Top