Help with jail bridges appreciated

~Luna

New Member

Reaction score: 2
Messages: 4

Hi all,

I'm relatively new to FreeBSD and intend to use it in production when I am comfortable enough with it. To see more about my background, please see this post. I know being new to FreeBSD isn't a valid reason to ask questions that are answered in the documentation, but I really can't figure bridges out. Also not after reading the documentation and looking at tens of posts on the internet. Also it seems a lot of posts are outdated since FreeBSD changed over time. For example it seems that /etc/jail.conf didn't exist in earlier versions. I hope this is not a case of RTFM all along ;-). Also I should mention that I am less comfortable with networking in general than for example server management/hardening and software. So part of it might be my lack of understanding.

Goal
To use FreeBSD with jails (both service jails and 'full' jails, although that's probably irrelevant). Some jails have public IPv4 and IPv6 addresses and some of them have internal RFC1918 addresses (I guess with something like NAT). So two use-cases:

1. The FreeBSD host has a public IPv4 and IPv6 address that can be used to NAT (or some other mechanism) to internal jails (with private address space) that aren't publicly available (but can access the internet themselves), unless you use some proxy in front of it (which can of course also be jailed. Jail everything!). This is more or less how Docker, LXD/LXC and others are being used in the default setup.

2. Some of the jails on the FreeBSD host are publicly available with their own IPv4 and IPv6 address. I don't mind them having an internal RFC1918 address for local communication as well, but this is no requirement as of now.

Question
How would one configure this in host and jails (FreeBSD 12)? Any config examples of these use-cases that are more elaborate than "make a bridge to ..."?

And to close off: I know of the existence of iocage, ezjail etc. But by joining the FreeBSD community (from Linux) I hope to remove abstraction layers/dependencies instead of replacing them with new ones ;-). Also I like to know the inner workings of systems I use to some extend, and using something like iocage will make me lazy/not do that.

If someone needs more info, has questions or what not, ask! And of course I appreciate any response, thanks!
 

pkc

Member

Reaction score: 2
Messages: 84

There are certainly multiple ways to accomplish these things and I am no expert.

Personally, I use ipfw with what I would call vnet jails (jails which have vnet=1 or something like that in their configuration). If you haven't seen yet, this vnet parameter gives your jail its own network stack, and is enabled by the VIMAGE kernel option, which should be in GENERIC in 12.0 (so no need to do anything to enable it on your system.

In-kernel NAT can be accomplished with ipfw and jails to provide jails with an internal network, internet access, and a secure stateful firewall. It can get a little tricky and subtle depending on what exactly your desired setup is.

As far as first steps, you will want to create a bridge interface with ifconfig bridge0 create. After that you can configure the bridge much as you would other interfaces, for example ifconfig bridge0 inet 10.0.1.0/24.

Personally, for vnet jails, I then use epair interfaces to connect the jails to the bridge. This basically simulates a layer-2 (ethernet) hub and spoke topology with your layer 2 device / switch at the center (your bridge0), which also has a layer 3 (IP) address and so is capable of both switching among your bridges, and routing out of the bridge network. Creating an epair device actually creates two devices; one attaches to the bridge and one goes in your jail. The details should be in the ifconfig manpage (search for "addm" and "vnet"). You will assign an IP address in the bridge's subnet (say 10.0.1.0/24) to the "b" end of the epair interface, the one that goes into the jail.

When creating such a setup, you will want to have tcpdump handy for the inevitable troubleshooting, so that you can track the progress of data as it moves through the network to identify where it might be getting stopped.

If you're using ipfw I recommend turning on the "firewall interface", for example ipfw0, so that you can monitor packets which have been blocked with a "log" action using tcpdump. I forget how to do this exactly since I've had it enabled for a while, but details should be available on the internet.

Setting up a stateful firewall and NAT with jails may take time. Depending on the circumstances, you may need two state labels instead of the one that is normally used with ipfw NAT gateways. I can advise you on that if/when you get to that point.

For now I would say the best first step would be to get one or more jails attached to a bridge interface and establish "IP connectivity" (be able to ping) among the jails and the bridge.
 
OP
OP
L

~Luna

New Member

Reaction score: 2
Messages: 4

Thanks for the reply pkc! I have read it and will experiment with your suggested setup for the next week or so. So if I do not respond it's not that I lost interest, I am just trying to get my head around it ;-).

You already use something similar it seems, do you mind sharing the configuration files (rc.conf, jail.conf etc.) with me? It might get me started quicker/understand it quicker. Via PM is also fine. Well for now thanks again for the thorough post!
 

getopt

Aspiring Daemon

Reaction score: 411
Messages: 609

~Luna
you come along with the term "jail bridge". Where did you pick that up or did you fabricate it yourself?

pkc refers to bridging interfaces. Also see https://en.wikipedia.org/wiki/Bridging_(networking) if that is what you mean.

Before making any more input please clarify on your use of the term "jail bridge". While the purpose of a jail is separation, bridging relates to interfaces.
 

PMc

Aspiring Daemon

Reaction score: 180
Messages: 513

Adding "me too" to the question from getopt - I also don't understand these "jail bridges".
The remainder of the description sounds very much like what I have running: multiple jails, some of them with local RFC1918 addresses, some with externally reachable address.

I am not using any of the new fancy features like VIMAGE or setfib, as the machine was built many years ago.
Instead I am doing it all with ipfw, that means the jails are set up just as usual, and ipfw does forwarding and feeds multiple NATs.
I am also using a local DNS server to keep things consistent and well-ordered on the higher levels where name resolution comes into play.

In my opinion, this all starts with a precise understanding of the intended topology, i.e. starting with the external links we have, and then planning where these should end up and how they should be processed. Therefore, mere config examples would not be much helpful, because they do not explain the topology that is behind.
 
OP
OP
L

~Luna

New Member

Reaction score: 2
Messages: 4

Thanks both for responding.
~Luna
you come along with the term "jail bridge". Where did you pick that up or did you fabricate it yourself?

pkc refers to bridging interfaces. Also see https://en.wikipedia.org/wiki/Bridging_(networking) if that is what you mean.

Before making any more input please clarify on your use of the term "jail bridge". While the purpose of a jail is separation, bridging relates to interfaces.
Hi. Yes you are completely right. I mean interface bridges meant for jails :). Does this clarify to you what I mean? I have some reading to do then. Any more tips are welcome of course.

Adding "me too" to the question from getopt - I also don't understand these "jail bridges".
The remainder of the description sounds very much like what I have running: multiple jails, some of them with local RFC1918 addresses, some with externally reachable address.

I am not using any of the new fancy features like VIMAGE or setfib, as the machine was built many years ago.
Instead I am doing it all with ipfw, that means the jails are set up just as usual, and ipfw does forwarding and feeds multiple NATs.
I am also using a local DNS server to keep things consistent and well-ordered on the higher levels where name resolution comes into play.

In my opinion, this all starts with a precise understanding of the intended topology, i.e. starting with the external links we have, and then planning where these should end up and how they should be processed. Therefore, mere config examples would not be much helpful, because they do not explain the topology that is behind.
That indeed seems like my use-case, although I use pf for firewalling.
 

rigoletto@

Daemon
Developer

Reaction score: 1,041
Messages: 2,101

TLDR;

If you want to make jails to communicate between themselves for some reason, e.g. database in one jail and clients in another, you can use loop(4).
 
Top