Anyone try VIMAGE with jails?

stratacast1

stratacast1

I'm trying to switch from Ubuntu Server using LXD (shiny, polished Linux Containers) to FreeBSD with Jails for my server, and I'm trying my darnedest to replicate what I like about LXD. So far, I see there's more work in setting up jails and a deeper understanding of BSD that I haven't acquired, but diving in blindly is sometimes what I do to learn. So far, it seems jails have checked all the boxes for me, except there isn't a lot of info on jails getting their own networking stack, whereas LXD does offer that out of the box. I've read some places saying using VIMAGE is still very experimental and I want to know if anyone has done that and what their experience is? And if they did do this, did they do if_bridge or netgraph and which one they find better?


PS- this is a home server that I run Nextcloud, Plex, SAMBA and a separate container for my web development dabbling. I'm experimenting with all of this in VirtualBox right now since my old desktop gets WAY too hot when compiling code
 
Hey SirDice, thank you for responding. I already came across that, and am using it. I was more wondering what people thought of its stability and if they prefer the if_bridge or netgraph way better
 
I personally haven't used VIMAGE jails, I don't have a need for them. If I need that kind of control I'm more inclined to use a bhyve VM. But I do believe FreeNAS uses VIMAGE for their jails by default, so it should be fairly stable.
 
Sweet thank you, I wanted to use FreeNAS but I really want to know how FreeBSD works in general with no life preservers, plus FreeNAS had too many errors when I would do something like set nameservers on the host or make a zpool on a flash drive and vanilla FreeBSD hasn't done that to me yet.

As for VIMAGE jails, I want to be certain I'm setting things up as secure as possible and I think separate networking stacks does that for me and the rest can already be handled out of the box with jails and ZFS. I decided to go with if_bridge because it's more of a familiar tool to me, so now I just have to learn jail management with the jail.conf method and without ezjail and I should be steps closer to switching my home server to FreeBSD!
 
I'm running several VIMAGE jails in production since 10.3 without any major issues. That said: most of my jails are not using VIMAGE, because ~90% of all use cases really don't require a dedicated network stack.
I've also tried netgraph, which is great for building complex, virtualized networks, but for my use cases simple bridges are sufficient, faster to set up and far less error-prone especially with automated deployments (ansible).

Overall, jails without VIMAGE are easier to manage, especially network-wise. VIMAGE + bridges + VLANs has some oddities and can be rather frustrating - it's far easier to just attach the jail to the hosts VLAN interface as this is guaranteed to 'just work™'
 
That's good to know, sko. I'm fine with accepting some complexity if it means having some more security. But I should ask someone who knows more than I do: can the host listen to container traffic without using VIMAGE + bridge for separate networking stacks? I find it interesting that all you're doing otherwise is just aliasing a new address to the container so to speak
 
I know there's already the possibility of the host listening on the bridge and capturing traffic, I guess I'm just wondering if there's more that the host has access to without/without separate networking stacks that makes it risky to not have them more separate
 
The host can observe anything going on in jails including network traffic, even with VIMAGE interfaces. The jails cant - they are fully isolated from one another regardless if they use VIMAGE or the "native" interface.
Some system tools are especially jail-aware, so the host can observe, control or manage its jails: e.g. pkg(8) can be used from the host system to update packages within a jail. top(1) is also jail-aware.

The reason why linux containers *need* separate network stacks is the overall bad/nonexistent segregation of "containers". There is no such concept in Linux like Jails in FreeBSD or Zones in Solaris - its just namespaces, cgroups and a lot of wallpaper plastered over all the air gaps in between. With Jails and Zones you start with a blank sheet - everything the Jail/Zone should be able to do has to be added - Linux containers are designed the other way around and for every part where better enforced segregation is needed, the whole part has to be rewritten and sometimes needs to be completely virtualized (e.g. the network stack).
Never think of Linux containers as securely contained and multi-tenancy secure "virtual hosts" - they just aren't. OTOH you can do interesting things with them if you exploit the various ways of interaction between each other and the host that containers can do.
There was a nice blogpost about exactly this distinction a while ago: https://blog.jessfraz.com/post/containers-zones-jails-vms/

On FreeBSD the dedicated network stack IMHO is intended for special cases like (ancient) services/daemons that work on such a low/obscure level, that they absolutely need a full network stack and/or virutalized device to work.
 
Thank you for the information sko! This is really going to make life easier in creating Jails in FreeBSD. Now that I'm back from a trip, I can continue trying things out. Thank you again!
 
stratacast1 said:
Thank you for the information sko! This is really going to make life easier in creating Jails in FreeBSD. Now that I'm back from a trip, I can continue trying things out. Thank you again!
Click to expand...

Check out if you have a possibility of jails to cross-talk: http://freebsd.1045724.x6.nabble.com/Firewalling-jails-and-lo0-td6120599.html
Yes, on a host you can see all of jails traffic. But we want a way to flexibly limit it, which lo0 will not let you.

I guess we should distinguish between a need for network stack in jail (not common, true), and a need to treat some of local traffic (between jails), as less trustworthy. But we set skip on lo0, don't we?
 
I wanted to share. If you put each jail on its own loX without giving it an external ip (ip from non-loopback), then inter-jail communication can be filtered while keeping skip on lo0. PF can pass packets from external ips inside, given proper rules, I assume.
No VIMAGE required! And yes, it feels like a little hack. :cool:
 
You must log in or register to reply here.
Back
Top