Planning a new server with virtual machines

zirias@

zirias@

Developer
It's time to think about a new server I will use in my private network, and I'd like to base it on FreeBSD this time. As my experience with FreeBSD so far is only a VM on my Linux Desktop for experimenting and the new installation on my notebook that's still work in progress, both running 11.0-CURRENT, I'm looking for some advice here.

First a quick description of my current server:
  • Runs XEN 3.2 with a minimal Linux (Debian) dom0 and custom patched kernel using some old SuSE modifications for managing the hardware and providing the volumes on encrypted storage
  • Hosts three Debian paravirtualized VMs, one as a router/firewall, one providing all internal services and one for services that are accessible from the outside (kind of "poor-mans DMZ")
  • The router/firewall VM has exclusive control over the single physical NIC
  • Additionaly hosts a Windows Server 2008R2 hardware-virtualized VM
My requirements (or should I say wishlist) for a new server would be:
  • stable and reliable (so, probably not running -CURRENT)
  • provide the separation of internal services, external services and firewall in a way comparable to what I have now (especially in terms of security)
  • I also want that virtual Windows server again, it's nice for testing stuff, e.g. some builds on win32, or for hosting some windows applications without the need for a local VM.
  • Full disk encryption (but this doesn't seem an issue, geli happily encrypts my whole ada0 on my notebook now)
  • Ideally run a PBX using some FXS/FXO card to connect POTS lines to a SIP provider.
Some thoughts and doubts I had so far:
  • For the separation of firewall, internal and external services, maybe jails could do? Would there be a way to disconnect the host system from the physical network while allowing one jail access?
  • Am I correct to assume both xen dom0 support and bhyve support for windows guests are only available in -CURRENT so far? Is there a recommended alternative?
Thanks for any advice!
 
I haven't tried it, but I believe the code to support Windows was merged for 10.3. Bear in mind that at the moment there's no VGA console in bhyve, so it's not possible to just 'look at the screen' to see what it's doing. The only access to Windows is via the emergency serial console or RDP. VGA is supposed to arrive very soon, although I don't know if that will require CURRENT.
 
Almost all new development happens in -CURRENT first. Once the issues have been ironed out features are MFC'ed to a -STABLE branch.

That said, bhyve should work in 10.3 but it is still considered an experimental feature. It should run fine but maybe not on your hardware. You'll have to try to find out.
 
usdmatt said:
I haven't tried it, but I believe the code to support Windows was merged for 10.3. Bear in mind that at the moment there's no VGA console in bhyve, so it's not possible to just 'look at the screen' to see what it's doing. The only access to Windows is via the emergency serial console or RDP.
Click to expand...
Would be exactly what I'm looking for! My current server doesn't even have any graphics card -- serial console for emergency. I use the Windows server exclusively via RDP :)

SirDice said:
Almost all new development happens in -CURRENT first. Once the issues have been ironed out features are MFC'ed to a -STABLE branch.

That said, bhyve should work in 10.3 but it is still considered an experimental feature. It should run fine but maybe not on your hardware. You'll have to try to find out.
Click to expand...
That's good news for me, as I would only need it for this Windows Server, and this is the least important part of the planned setup -- as long as the rest is working stable, that would be fine.

Any thoughts on the other aspects, anyone? Thanks!
 
As far as I know XEN only runs on -CURRENT. And even there it's still quite unstable. There's definitely a lot of work happening but, unless you're a hardcore developer, I wouldn't try it just yet.

I recently got a nice, shiny, SuperMicro machine with dual XEON E5620 (4 cores per package) and 48 GB ECC RAM. It's a bit loud for my living room so I'll need to find a different solution for it but I've been testing it with sysutils/vm-bhyve and it works really well. It's extremely fast (compared to any of my other machines) and it runs bhyve really well. At the moment it's running a recent -CURRENT but I'm going to try 10.3-RELEASE to see how well it works.
 
I have another question on this. After reading a lot of documentation, my plan now would be:
  • Use bhyve for the router/firewall machine and pass it the PCI device of my real NIC
  • Use jails for my internal and external/DMZ servers
  • wire up between those jails and the firewall machine using virtual NICs (that bhyve should expose as virtio NICs)
What I'm missing here: How would I create a standalone virtual NIC with a MAC, but no real hardware backing it? I do a similar thing on my current XEN/Linux based server using these XEN virtual networks, but I'm pretty sure there must be a native FreeBSD function I just happen to keep overlooking in the docs?
 
Zirias said:
What I'm missing here: How would I create a standalone virtual NIC with a MAC, but no real hardware backing it? I do a similar thing on my current XEN/Linux based server using these XEN virtual networks, but I'm pretty sure there must be a native FreeBSD function I just happen to keep overlooking in the docs?
Click to expand...
I haven't tried it myself but you could simply create a tap(4) interface and not bind it to the bridge. sysutils/vm-bhyve allows you to easily create networks that aren't attached to a physical network. It simply creates tap interfaces and bridge(4) them together. As long as you don't bind the bridge(4) to a physical adapter it should, more or less, be a "host-only" network.
 
That's what I'm looking for, so I hope it works :) Thanks so far, as soon as I really start building this server (the first hardware parts arrived), I'll give feedback!
 
With quite some delay (also caused by delays in the construction of my new house), I'm about to start the installation -- the shiny new machine is sitting on the floor next to me :)

I have now two detail questions:

  1. I want one virtual machine (either jail or bhyve) dedicated to poudriere as a build server. As I also plan to use this for testing my own ports, it should build packages for -CURRENT as well. In poudriere logs, I saw a warning before that the used jail is newer than the host OS. What's the impact of this? Would it be safe to build -CURRENT packages in a jail on my main version 11 machine or should I better go for bhyve and have my buildserver run -CURRENT?
  2. Is there any advise on how to decide how many virtual CPUs the bhyve guests should get? E.g. does it make sense to give all VMs as many virtual CPUs as physical cores are available?
TIA, Felix
 
1) You can't build packages for a newer system than the one poudriere is running on. This has to do with the way jails work. You can only run 'lower' version jails. So running a 10.3-RELEASE jail on 11.0 works but not the other way around. This limitation also applies to poudriere. If you're going to create a specific VM for poudriere consider creating a VM per version. Build -CURRENT packages on a -CURRENT VM, 10.3 packages on a 10.3 VM, etc. If you really want to get creative you can set up a VM with devel/jenkins and use that to dynamically spawn VMs and kick off poudriere builds. Have a look here if you're interested in that: https://wiki.freebsd.org/Jenkins

2) No, that's going to cause performance problems. In general I tend not to use more than 4 cores (assuming the host has a lot more) for a VM. The hypervisor needs to wait for the cores to be available before it can hand over control to the VM. It's however perfectly fine to create 4 VMs, each with 4 cores, on a host with 8 cores for example.
 
You must log in or register to reply here.
Back
Top