Solved Jails on a very restricted local network (only 1 local IP)

Alan Jiménez

Alan Jiménez

I am trying to set up a set of servers with FreeBSD using Jails, and I am struggling to find a proper solution. I am a bit of a noob, so I think the biggest part of the picture I'm missing is what the heck my options are.

I'll try to make it brief: I imagine a FreeBSD server would have a reverse proxy running on the main host, and a Jail for each Apache/FTP instance I want to create. And probably also a Jail for MySQL for a while. Perhaps also other services would be added in the future.
I'm confident I'm not the first to come up with something like this. However, I am at the moment trying this on virtual machines @ DigitalOcean. They provide one public IP and, more importantly, one single private network IP per host, so I can't just start creating Jails with each their private IPs on said private network.

So, my options as far as I can see:

1 - I have read on one or more posts here that it should be possible for me to just pass traffic to a Jail from the main host, even if that jail only has a "lo1" network interface. Is this really even possible (I have not yet succeeded in finding out how to) and if so, would this be a/the way to go?

2 - I have also read (in the ezjails page of the FreeBSD docs, I think) that a jail could have the same IP address as the host without (too many) problems. Would this be a viable alternative?

3 - My idea (and this might be a far fetched thing) was to add a tap device to the host (say, with IP 192.168.2.1 which is different from the already assigned local network's subnet), and then add a tap device for each jail and giving it an address in the same 192.168.2.0/24 subnet, thereby creating a virtual internal network.
I would then setup firewall and NAT on the main host that would grant and remove internet access to the jails as needed and, most importantly, a reverse proxy could route HTTP requests and the like into this virtual internal network (I found that for this to work I would have to play with setfib, since jails and main host will need to have different default routers, an issue I'll need to read on if I continue on this venue).
I think it would then be possible to set firewalls on all jails for added security (would this not be a good idea?), and perhaps even set up SSH tunnels for communicating with other jails to/from other servers, which will quite probably be a requirement in the future if things work out

Are any of these three options (and specially the latter) sound at all, or am I barking mad? If my idea turns out to be barking mad, what would be a better way to go?

Thanks!

/Alan
 
1) Yes, I have it set up on my VPS. Keep in mind you can only forward each port just once. If you want multiple sites, spread out on multiple jails, on the same IP address and port you will have to use a reverse proxy like net/haproxy on the host.

2) Not sure if it's possible. I certainly don't recommend it even if it is.

3) Doubtful it will work. And it looks very complicated which makes it very prone to errors.
 
I use 3) without any problems or complications. I even put security/strongswan on the host and can SSH to any jail from anywhere.
Code: 
# Virtual interfaces
cloned_interfaces="lo1"
ifconfig_lo1="inet 192.168.0.1 netmask 255.255.255.0 up"
ifconfig_lo1_alias0="inet 192.168.0.2 netmask 255.255.255.255"
ifconfig_lo1_alias1="inet 192.168.0.3 netmask 255.255.255.255"
ifconfig_lo1_alias2="inet 192.168.0.4 netmask 255.255.255.255"
ifconfig_lo1_alias3="inet 192.168.0.5 netmask 255.255.255.255"
ifconfig_lo1_alias4="inet 192.168.0.6 netmask 255.255.255.255"
ifconfig_lo1_alias5="inet 192.168.0.7 netmask 255.255.255.255"
ifconfig_lo1_alias6="inet 192.168.0.8 netmask 255.255.255.255"
ifconfig_lo1_alias7="inet 192.168.0.9 netmask 255.255.255.255"
To allow jail to access internet I use
nat on $ext from $jail:network to any -> $main
to expose jailed daemon
rdr on $ext proto tcp from any to $main port https -> $j_www port https

If you plan to secure jail-to-jail communications, be sure to set rules as no state

Actually I don't understand, why this solution is not recommended, I'm getting
1. Ability to use the same port for each jail
2. Manage jails without jexec or login to main host.
3. Ability to add new jails without touching configuration of existing ones including firewall rules.
4. Ability to secure jail-to-jail communications

But I agree, with this setup the pf.conf can be huge (but plain).
 
That configuration is more like #1. I have a similar setup. Although I use sysutils/ezjail to easily manage the jails:

Code: 
root@armitage:~ # ezjail-admin list
STA JID  IP              Hostname                       Root Directory
--- ---- --------------- ------------------------------ ------------------------
ZR  1    192.168.21.3/24 webtrees                       /jails/webtrees
ZR  2    192.168.21.10/24 mysql                          /jails/mysql
ZR  3    192.168.21.4/24 mail                              /jails/mail
ZR  4    192.168.21.20/24 dayz                           /jails/dayz
All jails are bound to lo1. I have net/haproxy running on the host which is configured to use the different jails as backends. That way I can run separate instances of Apache, each on a jail, while only having one, external, internet IP address. HAProxy switches between them based on the HTTP 1.1 Host header.
 
Hmm.. I notice that all Jails I come across (also in this post) have reasonable 192.168.x.0/24 or so IP addresses. This is what I was thinking I couldn't do, because I am using DigitalOcean, and they only allow me 1 IP address in a local network (vtnet1: inet 10.133.190.179 netmask 0xffff0000). My current jail that I am experimenting with right now only has a loopback IP address (127.0.1.1) and has no internat access, since I thought this was my only option - or, as I explained in option #2 above, give the jail the same IP address 10.133.190.179, which SirDice discouraged, and I personally also don't like the sound of either.
Have I perhaps misunderstood the way jails work? Could I simply pull an IP address out of my sleeve and stick it to the jail, perhaps 192.168.2.10, in this environment and the jail will work fine as far as networking is concerned?

I feel I'm asking some pretty basic questions about jails, but I have been through the hand book and other ressources (man pages, etc.), and I haven't found any specific info on this. Let me know if I've missed something, or if you know of any other literature I may read and learn from. By the way, I am also using Ezjails.

I sincerely appreciate your comments. Thank you kindly!

/Alan
 
Alan Jiménez said:
Could I simply pull an IP address out of my sleeve and stick it to the jail, perhaps 192.168.2.10, in this environment and the jail will work fine as far as networking is concerned?
Click to expand...
In essence, yes. You can use whatever IP addresses you like. I would suggest using a range that's not used anywhere else on the network as that might cause a bit of problem with routing. Because that network is bound to a local interface (lo1 or tap(4)) you will have to use NAT on the host, which effectively 'hides' the range from the rest of the network.

Think of the host as a firewall/router and the jails as machines on a LAN. The 'LAN' is basically the local interface (lo1 or tap(4)) and the 'WAN' connection of the firewall/router is connected to the rest of the network.
 
Sorry about my confusion, but this sounds to me a lot like my option #3?

Would I have to give the host another network address in the same subnet (192.168.2.1 would be my first choice), and tell the jails to use that as a default gateway (I'd have to use setfib for this)? And then I would be slowly creating an internal virtual network of jails in the host, using the host as NAT/Firewall and reverse proxy? The only difference I see is that I don't need to use tap devices, as I though I would have to?
 
Alan Jiménez said:
Sorry about my confusion, but this sounds to me a lot like my option #3?

Would I have to give the host another network address in the same subnet (192.168.2.1 would be my first choice), and tell the jails to use that as a default gateway (I'd have to use setfib for this)? And then I would be slowly creating an internal virtual network of jails in the host, using the host as NAT/Firewall and reverse proxy? The only difference I see is that I don't need to use tap devices, as I though I would have to?
Click to expand...

Stop thinking jails as virtual machines, they are not quite that. Jails are just sligthly glorified chroot(8) environments that do not have their own network stacks. This means all networking functionalities for jails are handles by the host including routing.
 
Alan Jiménez said:
Sorry about my confusion, but this sounds to me a lot like my option #3?
Click to expand...
Number 1 and 3 are similar but not the same. Your #3 would have a tap(1) interface per jail. So if you have 3 jails you'll get 3 tap(4) interfaces. Whereas #1 uses a single, local, interface for all jails.

Would I have to give the host another network address in the same subnet (192.168.2.1 would be my first choice), and tell the jails to use that as a default gateway (I'd have to use setfib for this)?
Click to expand...
No, this is not necessary. The host knows what network is attached to lo1 so it knows where to find it. Routing is done by the host, not the jails.
 
OK, I think I managed to get option 1 sort of running now. The strange thing about this, I mean, what I think was hard for me to get my head around has been to what extent a jail is (or is not) a virtual machine.

As kpa rightly said, I needed to stop thinking so rigidly in this direction (thanks for that). Still, I found I had to set up NAT with PF so I could get the jails online so I could install stuff.

I'll get to grips with it in time. For now, I think I have enough to get going.

Many thanks to all of you. I truly appreciate the time you've put into helping me out here!

/Alan
 
You must log in or register to reply here.
Back
Top