Server configuration

[zader]

Just wanted get some build advice / best practice for a hybrid server.

My goal is to run a mix of vm’s and jails on the same box. The box its self is very beefy ie 2cpu 36 core 128g ram..

The only real options i have come up with is ..

#1 - pass 1 nic to the host and set up the basic jails i need like vsftp and a famp stack .. then use the second nic as a tap with jails..

Or

#2 - use all vms and create my jails on vms..?

Guessing theres other ways to .. but what would be a place to start?

Thanks

 

[usdmatt]

If you are planning to use jails then I don't think it makes sense to add overhead by running them inside a VM.

I don't use jails much but I don't see any problem with creating a bridge on the host and connecting both jails and virtual machines to it. There's various ways you can configure the networking though; Not sure which is the most efficient and it's somewhat down to preference.

 

[SirDice]

I'd use VMs for everything, much more versatile, you're not limited to the same OS (or version) for example. And you can treat a VM much more like a 'standalone' machine. That makes it easier to update/upgrade each one individually.

And I agree with usdmatt using a jail on a VM is overkill, besides adding a lot of complexity doesn't add any real security benefits. Adding complexity only makes it more prone to failures which would actually be detrimental to your security, we're all human and mistakes are easily made.

I'm personally a big fan of sysutils/vm-bhyve to handle management of the VMs.

root@hosaka:~ # vm list
NAME            DATASTORE  LOADER     CPU  MEMORY  VNC           AUTOSTART  STATE
case            default    bhyveload  2    4096M   -             No         Stopped
freebsd11-img   default    uefi       1    512M    -             No         Stopped
jenkins         default    bhyveload  4    6144M   -             Yes [5]    Running (4802)
kdc             default    uefi       2    2048M   0.0.0.0:5901  Yes [1]    Running (52777)
lady3jane       default    uefi       2    4096M   -             No         Stopped
tessierashpool  default    bhyveload  4    12288M  -             Yes [4]    Running (12209)
build11         stor10k    bhyveload  4    8192M   -             No         Stopped
funky           stor10k    bhyveload  4    8192M   -             Yes [7]    Running (3420)
gitlab          stor10k    bhyveload  4    8192M   -             Yes [8]    Running (69978)
gitlab-runner   stor10k    bhyveload  4    8192M   -             Yes [9]    Running (38242)
kibana          stor10k    bhyveload  4    6144M   -             Yes [2]    Running (50054)
phabricator     stor10k    bhyveload  4    12288M  -             No         Stopped
plex            stor10k    bhyveload  4    6144M   -             Yes [6]    Running (70523)
sdgame01        stor10k    grub       2    4096M   -             No         Stopped
wintermute      stor10k    bhyveload  4    8192M   -             Yes [3]    Running (61536)

 

[zader]

Hi guys,

Thanks for the input. sofar my challenge is the networking.. would the single bridge be the best approach? or is there perhaps a"how to" i missed that shows the best way to set it up? Is it as simple as making the bridge and assigning a cidar ip range to it? then use those ips for the vms?

cheers

 

[SirDice]

Thanks for the input. sofar my challenge is the networking.. would the single bridge be the best approach? or is there perhaps a"how to" i missed that shows the best way to set it up? Is it as simple as making the bridge and assigning a cidar ip range to it? then use those ips for the vms?

What's the best way is going to depend on your own situation. You're not limited to a single bridge, you can have as many as you like. You can have multiple bridges each on their own VLAN, or add NAT to a (virtual) switch. You can create quite complex setups.

https://github.com/churchers/vm-bhyve/wiki/Virtual-Switches

At the moment I'm using a single bridge, but I'm going to put servers into their own VLAN and split some things up. My home network is currently just a single flat /24 address space.

root@hosaka:~ # vm switch list
NAME    TYPE    IFACE     ADDRESS  PRIVATE  MTU  VLAN  PORTS
public  manual  bridge10  n/a      no       n/a  n/a   n/a

 

[`Orum]

And I agree with usdmatt using a jail on a VM is overkill, besides adding a lot of complexity doesn't add any real security benefits. Adding complexity only makes it more prone to failures which would actually be detrimental to your security, we're all human and mistakes are easily made.

I agree that jails should almost always be directly on the host instead of run inside a VM, but there are a few legitimate circumstances where you might want to do it. Namely, if you want to run a different FreeBSD kernel (either version or drivers/options) inside the VM, or if you want live-migration which (AFAIK) only has support in bleeding-edge versions of FreeBSD/bhyve.