Hi everyone,
I am planning to build a new virtualization server with bhyve since I read a lot positive articles and posts about it.
I don't have too much experience with FreeBSD yet, but am a longtime GNU/Linux user.
But I'm getting more and more fed up with the whole Linux ecosystem feeling "bloated" (which is way out of scope for this thread) and want to go with FreeBSD for my new server.
Please keep in mind this is a home-lab project and not a production system for a company.
1) I am eyeing the following hardware and picked the components with performance headroom, power efficiency and affordability in mind. I want the machine to last for several years without having to do major changes to hardware aside from a possible upgrade of the CPU or more RAM.
Are there any reasons not to run it on the following hardware or suggestions for "better" hardware w.r.t. my wishes above?
AMD Zen5 (Ryzen 9700x or maybe 9600x)
AsRock or Asus Mainboard with AMD B850 chipset
ECC UDIMM RAM, most likely 32 GB to start with
2 SSDs as boot drive for the host and guests (raid0)
2 HDDs exclusively for NAS
Intel X710 NIC (2x 10g SFP+ and SR-IOV capable NIC is required, working ASPM would be great)
The servers job will be to run
- OPNsense router / hw firewall in one VM, for which I have a dedicated Intel Core i5 machine right now.
- Debian 13 based NAS (NFS, SMB, rsync, sftp) for < 10 users and long term storage of my data. I have a dedicated (but aging) Supermicro X11 server for that right now.
- One VM for a small webserver and services facing the internet. I have a dedicated Raspberry Pi 5 for this now.
- Another Linux VM for various containers and non-critical tasks for internal services like receiving scanned pdfs, performing ocr and various "smart home" stuff. These tasks are fulfilled by my NAS and Raspberry right now.
- Headroom for guests as "playground" for new projects. One possible idea is building a PXE boot server with iVentoy, but that does not have to be running 24/7.
2) The OPNsense VM will use one of the SFP+ ports to face the internet (with a GPON SFP) and the other SFP+ port to face my local network.
The port facing my local net is planned to be virtualized with SR-IOV and shared among the bhyve host and the VMs.
Any onboard NIC ports will be used for oob/mgmt.
I don't see any lurking caveats here and if nobody has arguments to not do this, I don't have any further questions.
3) For my NAS I would like to be able to spin down the HDDs for power efficiency when I don't need it.
Monday through Friday I need my NAS 18:00 - 23:00 on a non-regular basis and Saturday/Sunday 08:00 - 23:00.
I know there are different opinions on stopping and starting spinning discs, but for power efficiency I'd like the HDDs to be powered down when I don't need it and if I power it up, I want it to power down the discs at midnight (when I won't be using it for at least a few hours).
HDDs get polled by all kind of things like smartd, systemd and who knows.
For the sake of keeping it simple I don't want to mess with configurations of various daemons and start pulling my hair why the disks are still spinning up, so I'd want to give the NAS VM direct access to the HDDs and just stop that VM when I don't need the NAS.
3.1) Who has done something similar or can recommend specific dos or don'ts?
3.2) In which way should I pass the HDDs to the VM to give it exclusive access?
3.3) Do I possibly need a dedicated PCIE HBA (in IT-mode) to achieve what I want?
3.4) Can I automate starting the NAS-VM when a client wants to access it or can a VM be powered up with WoL?
4) Since I haven't too much experience with FreeBSD and none to speak of with bhyve yet, what articles/posts would you recommend me to read before I actually start setting this up?
5) The server will run in the basement without display and keyboard. I'm not sure if I need IPMI or if I should keep a Raspberry Pi for KVM around or if I will be fine with remote access to the host and VMs.
I can live with walking down a few stairs with a small display and keyboard once a year, but I definitely want to be able to manage and monitor the server remotely.
What do you think is necessary, convenient or unnecessary regarding remote management?
So any answers to my questions, sharing experiences and hints on anything to consider would be very much appreciated.
Peter
I am planning to build a new virtualization server with bhyve since I read a lot positive articles and posts about it.
I don't have too much experience with FreeBSD yet, but am a longtime GNU/Linux user.
But I'm getting more and more fed up with the whole Linux ecosystem feeling "bloated" (which is way out of scope for this thread) and want to go with FreeBSD for my new server.
Please keep in mind this is a home-lab project and not a production system for a company.
1) I am eyeing the following hardware and picked the components with performance headroom, power efficiency and affordability in mind. I want the machine to last for several years without having to do major changes to hardware aside from a possible upgrade of the CPU or more RAM.
Are there any reasons not to run it on the following hardware or suggestions for "better" hardware w.r.t. my wishes above?
AMD Zen5 (Ryzen 9700x or maybe 9600x)
AsRock or Asus Mainboard with AMD B850 chipset
ECC UDIMM RAM, most likely 32 GB to start with
2 SSDs as boot drive for the host and guests (raid0)
2 HDDs exclusively for NAS
Intel X710 NIC (2x 10g SFP+ and SR-IOV capable NIC is required, working ASPM would be great)
The servers job will be to run
- OPNsense router / hw firewall in one VM, for which I have a dedicated Intel Core i5 machine right now.
- Debian 13 based NAS (NFS, SMB, rsync, sftp) for < 10 users and long term storage of my data. I have a dedicated (but aging) Supermicro X11 server for that right now.
- One VM for a small webserver and services facing the internet. I have a dedicated Raspberry Pi 5 for this now.
- Another Linux VM for various containers and non-critical tasks for internal services like receiving scanned pdfs, performing ocr and various "smart home" stuff. These tasks are fulfilled by my NAS and Raspberry right now.
- Headroom for guests as "playground" for new projects. One possible idea is building a PXE boot server with iVentoy, but that does not have to be running 24/7.
2) The OPNsense VM will use one of the SFP+ ports to face the internet (with a GPON SFP) and the other SFP+ port to face my local network.
The port facing my local net is planned to be virtualized with SR-IOV and shared among the bhyve host and the VMs.
Any onboard NIC ports will be used for oob/mgmt.
I don't see any lurking caveats here and if nobody has arguments to not do this, I don't have any further questions.
3) For my NAS I would like to be able to spin down the HDDs for power efficiency when I don't need it.
Monday through Friday I need my NAS 18:00 - 23:00 on a non-regular basis and Saturday/Sunday 08:00 - 23:00.
I know there are different opinions on stopping and starting spinning discs, but for power efficiency I'd like the HDDs to be powered down when I don't need it and if I power it up, I want it to power down the discs at midnight (when I won't be using it for at least a few hours).
HDDs get polled by all kind of things like smartd, systemd and who knows.
For the sake of keeping it simple I don't want to mess with configurations of various daemons and start pulling my hair why the disks are still spinning up, so I'd want to give the NAS VM direct access to the HDDs and just stop that VM when I don't need the NAS.
3.1) Who has done something similar or can recommend specific dos or don'ts?
3.2) In which way should I pass the HDDs to the VM to give it exclusive access?
3.3) Do I possibly need a dedicated PCIE HBA (in IT-mode) to achieve what I want?
3.4) Can I automate starting the NAS-VM when a client wants to access it or can a VM be powered up with WoL?
4) Since I haven't too much experience with FreeBSD and none to speak of with bhyve yet, what articles/posts would you recommend me to read before I actually start setting this up?
5) The server will run in the basement without display and keyboard. I'm not sure if I need IPMI or if I should keep a Raspberry Pi for KVM around or if I will be fine with remote access to the host and VMs.
I can live with walking down a few stairs with a small display and keyboard once a year, but I definitely want to be able to manage and monitor the server remotely.
What do you think is necessary, convenient or unnecessary regarding remote management?
So any answers to my questions, sharing experiences and hints on anything to consider would be very much appreciated.
Peter
