Jail networking

[perkypork]

I am using ezjails to create jails on a Dell R710 with 4 network interfaces. 1 nic is being used for basejail traffic and the remaining 3 nics are being used for different types of jails. All 3 nics are connected to the public internet. I am splitting the jails over 3 nics to maximise throughput.

How in this scenario am I able to use all 3 jail nics for traffic? It seems like I am going to have to configure multiple gateways. Is someone able to offer any alternative options available to me?

 

[perkypork]

Not really looking to use lagg because I loose some flexibility i.e. I cannot use different subnets on different interfaces, etc. Thanks for giving me a pointer though.

 

[perkypork]

To add further information to this post. I am using Jails for this server.

I have 4 NICs on my server. I use one nic for base system management. I would like to use of the other 3 nics with potentially a different "gateway". When I say gateway, I don't think I mean it in the technical sense, I mean that I want a jail to respond back via the interface its been given. I have done a ton of reading about this but I am stuck. I don't want to use setfib(1) unless there is no other way but I don't understand static routes properly.

Maybe if I give some examples of something I would like to achieve.

One of my base systems has an IP of 192.168.123.192. My VPN gives me an IP of 10.0.123.2. The default gateway on the base system is a public IP. I would like to be able to access the base system IP from my VPN connection. Currently I the base system does not respond to any traffic from the 10.0.123.0/24 subnet. Am I able to get this working via static routes for the base system interface?

I would like to create a jail that has an IP on the local network - 192.168.123.45 but using a different interface than the base system. The only way I can access the local network is via a VPN which hands out IPs from the 10.0.123.0/24. The default gateway on the base system is a public IP. Am I able to get this working via static routes for the base system interface?

If I am able to configure the above via static routes could someone please explain how I would do this in rc.conf?

 

[asteriskRoss]

Hi perkypork. I'm a bit confused about your network architecture and where your VPN fits in. A sketch diagram might help.

I am using ezjails to create jails on a Dell R710 with 4 network interfaces. 1 nic is being used for basejail traffic and the remaining 3 nics are being used for different types of jails. All 3 nics are connected to the public internet. I am splitting the jails over 3 nics to maximise throughput.

It sounds like the four interfaces on your server are connected to the same physical network. Is that correct?

My VPN gives me an IP of 10.0.123.2.

What is the "me" in this description? An interface on the server or a client machine? Where is the VPN endpoint in relation to the network the server is on?

Information that may, or may not be useful: Jails (including those created using sysutils/ezjail) can be configured with multiple interfaces. Have a look at the excellent ezjail section of the FreeBSD handbook (written by wblock@). So, if you wanted two jails on totally different networks to be able to communicate without routing traffic externally, you could configure a shared local interface in addition to the external interfaces.

bthomson is right to suggest lagg(4). If you want to maximise throughput, you could aggregate all your interfaces together using lacp, loadbalance or roundrobin configuration and then assign the jails addresses on the aggregated interface (which you might call lagg0). The aggregation means that if one of your jails is generating a lot of network traffic but the other two are quiet, the traffic will be split across all the interfaces. If you do as you suggest and just assign a jail an address on one interface (without aggregation), that interface could be very busy whilst your other interfaces sit doing nothing.

 

[perkypork]

Hi perkypork. I'm a bit confused about your network architecture and where your VPN fits in. A sketch diagram might help.

It sounds like the four interfaces on your server are connected to the same physical network. Is that correct?

My question has changed a little since I first posted. What I would like to do now is have the ability for each interface to be connected to a different network and have the jails be able to respond to traffic on their specific IP/network.

My OpenVPN connection is a pfsense firewall with a tunnel network of 10.0.123.0/24 which pushes a route to the 192.168.123.0/24 network. The OpenVPN connection is the only way I can access this system remotely. The private network where this server resides is 192.168.123.0/24, the base system has an IP in this range. If I leave the default gateway on the base system to be the gateway of the local network (192.168.123.0/24) then I can access the base system from the VPN. The only problem with doing this is any jails with a public IP stop responding to traffic. If I change it so that the default gateway is one of our public gateways then I am unable to access the base system on its private IP from my VPN connection. I would to know how to configure FreeBSD so traffic is responded to on all interfaces, not an OR scenario like I currently have.

I will do a drawing up when I get to the office.

 

[perkypork]

I decided that I would try this out on one of our systems instead of trying to understand everything before configuring it. I added a static route to see if it would work:

route add -net 10.0.123.0/24 192.168.123.254

It started working, YEY!

The reason I was asking about this is I have found it hard to find information about this that comes from the angle of "What I want to achieve" instead of the technical.

For someone who might be looking for more of a practical example. By adding the above static route to my system, when traffic comes from my VPN connection, the system knows where to respond to it. Previously it would receive the traffic from 10.0.123.0/24 on its local IP 192.168.123.192 and respond via the defaultrouter (a public IP), which was never going to work. Adding the static route meant that instead of replying via the defaultrouter it now responded via 192.168.123.254. 192.168.123.254 is the address of our pfsense system and that system knows how to communicate with the VPN network 10.0.123.0/24. Please correct me if I am wrong.

 

[perkypork]

I think I jumped the gun.

I added the static route above and while I was now able to access any server on the local network (192.168.123.0/24) from the VPN, none of those servers could access the internet. To solve this, I switched things around and made 192.168.123.254 the defautrouter (our local gateway) and added a static route to our WAN gateway. This seems to work for those servers with a public IP providing HTTP/HTTPS traffic but SSHing to any of these servers fails.

I am unsure how to solve this issue, any help would be appreciated!

 

[asteriskRoss]

perkypork, I see you have also started Thread jails-configuration-questions.50387 and Thread dual-gateways.49126, where you appear to be tackling the same issue. Perhaps it would be sensible to consolidate these?

 

[perkypork]

perkypork, I see you have also started Thread jails-configuration-questions.50387 and Thread dual-gateways.49126, where you appear to be tackling the same issue. Perhaps it would be sensible to consolidate these?

This is a good question asteriskRoss. One of the reasons I didn't continue asking questions in those posts is that I didn't realise they were related. I have struggled finding information on how to achieve what I want to achieve. Much of the documentation I have found discusses things in a technical way, which requires that you know how you want to achieve something - being a newbie I have no idea how to achieve something, only what I want to achieve.

This post jails-configuration-questions.50387 was about the best way of implementing jails.

This post dual-gateways.49126 was about understanding the differences between Linux and FreeBSD networking. Coming from the Linux world, I was unaware that multiple gateways are not required in most scenarios. In my Linux days, I would just set a gateway for each interface and be done with it.

The current post jail-networking.50639 was about understanding how I should configure my networking in jails. This post and the dual gateways post could be combined but then newbies who use these questions to work out what they want to achieve may be confused by the combination of migration from Linux (centos) and Jails networking.

 

[kpa]

This post dual-gateways.49126 was about understanding the differences between Linux and FreeBSD networking. Coming from the Linux world, I was unaware that multiple gateways are not required in most scenarios. In my Linux days, I would just set a gateway for each interface and be done with it.

I don't remember how Linux handles gateways but in FreeBSD there is no direct connection between interfaces and gateways in the routing table, you can't "set a gateway" for an interface because it just doesn't make sense in any way. Instead the gateways are always tied to routes in the routing table.

 

[asteriskRoss]

perkypork, a network diagram of what you want to achieve would still be helpful It doesn't need to be pretty!

 

[perkypork]

perkypork, a network diagram of what you want to achieve would still be helpful It doesn't need to be pretty!

asteriskRoss - its not pretty.

I now have everything working except the local jails. When I try to install a port, the jail doesn't know how to get on the net. This is because my defaultrouter is a public IP. I have put static routes for the VPN networks on pfsense, which means we can access the local jails and the base system from the VPN. The public jails have no issues so far.

All that is left is getting the jails with a local ip to be able to get ports.

 

[asteriskRoss]

I'm pleased you have most of what you want working. If you need any further input on the routing side it would be great if you could post the routing tables ( netstat -r) as well as the interface configuration ( ifconfig) on your server and the pfSense box so I and other forum members can make sure we understand your set up.

Regarding the jails connected to your local network, if you don't have public IPv4 addresses to spare and you're using private IPv4 addresses (you indicated they are on 192.168.123.0/24), you'll need to configure Network Address Translation (NAT) to present a routable IP address to the public Internet. You could do this on your server (in conjunction with a firewall, using your server as a router) or using the pfSense box as your router.

An alternative (not hassle-free) would be to host your own package repository on the server, which might be more efficient since otherwise you may be downloading and compiling the same ports multiple times. If this is the way you choose, I highly recommend ports-mgmt/poudriere (see also the section in the FreeBSD handbook). Once you have your repository built, you could share it among your jails by using nullfs(5) to mount the directory tree in each jail or even set up a jail with a webserver on your local network to serve the package repository over HTTP.

 

[asteriskRoss]

I should add that if you're using binary packages rather than ports and you just have a couple of packages to install in a jail, an easy option is to obtain the packages and install them from disk rather than over the network with pkg add <package_name>.tgz. This can even be done with ports-mgmt/pkg itself.

 

[perkypork]

I should add that if you're using binary packages rather than ports and you just have a couple of packages to install in a jail, an easy option is to obtain the packages and install them from disk rather than over the network with pkg add <package_name>.tgz. This can even be done with ports-mgmt/pkg itself.

Thanks asteriskRoss!

I am currently building from source. Ideally I would like to use poudriere but at the moment, I am holding that on the backburner until I get my jails stuff sorted out. I have attached my routing table and network setup.

 

[perkypork]

I have not run out of public IPv4 addresses, I use the private IPs for staging servers and the public IPs for live servers. The base system has a private IP.

I VPN in to get access to the staging servers. The staging servers get internet access for ports etc through pfsense (vpn) box.

 

[perkypork]

asteriskRoss - its not pretty.

I now have everything working except the local jails. When I try to install a port, the jail doesn't know how to get on the net. This is because my defaultrouter is a public IP. I have put static routes for the VPN networks on pfsense, which means we can access the local jails and the base system from the VPN. The public jails have no issues so far.

All that is left is getting the jails with a local ip to be able to get ports.

My only (but big) issue with the above configuration is if I need to restart networking for a new IP to be read from rc.conf. I must run the command service netif restart & service routing restart or I am unable to access the base system from the VPN. Ideally I would have my default router to be the private IP of the pfsense (VPN) box and have a static route for the internet. This would allow me not to worry about being locked out because of a typo.

 

[asteriskRoss]

Suggestion: Configure two different routing tables with setfib(1)
I know from your other posts you had been trying to avoid using multiple routing tables, but in this case it sounds like it might be a good option. setfib(1) allows you to have multiple routing tables and you can specify which one you want to use for each jail (see also the man page for setfib(2) to configure options in your kernel or /boot/loader.conf).

Your production jails and staging systems (in which I'm including your host system, which you said also has an IP on the 192.168.123.0/24 network) have different routing requirements. The default route for your production systems needs to be via your public gateway, whereas the default route for your staging systems needs to be via the pfSense box, which can then handle NAT routing to the Internet.

The ezjail configuration file has a configuration option for specifying which routing table to use (jail_${jail_name}_fib if I remember correctly), which is equivalent to setting the exec.fib parameter in jail.conf(5) for a non-ezjail jail.

Option involving no static routes
With the default gateway of your staging systems being the pfSense box, I don't believe you would need static routes on either your staging systems or the pfSense box. The pfSense box has addresses on the 10.0.123.0/24 network (for your VPN) and also the 192.168.123.0/24 network (for your staging systems), so the pfSense's routing tables will be automatically populated to route traffic between these two networks. Anything else would be routed out via NAT to the public Internet. I'll caveat this by saying that I don't use pfSense so I'm not sure what special configuration (if any) it requires.

With the default gateway of your production jails being your public gateway, these can respond appropriately to incoming requests without issue via this default route. For remote access from your VPN, you could connect via ssh(1) to your host system (on the 192.168.123.0/24 network) then attach to your jails with ezjail-admin console <jailname> or with jexec(8). This wouldn't require any static routes.

Option to allow direct SSH connections to production jails from the VPN
If you wanted to connect with ssh(1) directly to your production jails via the VPN, a reasonable option would be to assign them an address on an interface connected to the 192.168.123.0/24 network as well as their public IP. In my opinion it would be better to run your SSH servers listening only on the private addresses so they can be accessed via the VPN but not from the public Internet. You would then need to configure a static route so that the production jails route traffic for the 10.0.123.0/24 network (for VPN connections) via the pfSense box.

Does that make sense? I hope I've understood your network and requirements correctly

Edit: Various typo corrections

 

[perkypork]

asteriskRoss - I cannot thank you enough for helping me out with this! The best I can do is say "in the short time I have been using FreeBSD I am a huge convert, the OS is the best server OS I have used. What makes FreeBSD even better is the great community it has. People like yourself and all the other people on the forums who have helped me very patiently, is something very special for an open source project"

I will implement multiple routing tables as it seems like this is my only option. Thanks again!

 

[junovitch@]

This should get you started. I skimmed the thread and haven't read all the details but at least wanted to throw this out there to get you rolling.
echo 'net.fibs=4' >> /boot/loader.conf

Set a default route on each FIB.
/etc/rc.conf

static_routes="fib1default fib2default fib3default"
route_fib1default="default 1.1.1.1 -fib 1"
route_fib2default="default 2.2.2.2 -fib 2"
route_fib3default="default 3.3.3.3 -fib 3"

Now just set your jails to use the FIB you want them to be on. This is the exec.fib parameter in jail.conf. Just set it to the FIB number you want and all traffic in that jail will use the default gateway you assigned on that FIB.

 

[perkypork]

junovitch - Thanks! I have had a crack at implementing 2 fibs.

I tried your initial instructions and found a similar issue to below, so changed it to the below to see if the issue fixed itself. I think I like the below better because it gives a little more control but more importantly (as a newbie) I need to know what is going on instead of things happening automatically so I learn stuff.

echo 'net.fibs=2' >> /boot/loader.conf
echo 'net.add_addr_allfibs=0' >> /boot/loader.conf

I then setup my static routes (see screenshot).

Almost everything seems to be working at this point, except for the firewall. I use IPFW and the base system is now unable to ssh or ping to any jail with a public IP on the same box (permission denied). Before implementing fibs I had no issues. I have not changed the firewall rules at all. My firewall rules are in attached screenshot. I am not sure what I need to change to get it working properly.

 

[perkypork]

I get the feeling my ssh (IPFW) issue has something to do with the fact that the base system can now see two routing tables and my IPFW config only allows communication from the base system from its local IP. I want each system to only communicate on its allocated IP and the base system only has a private IP.

 

[junovitch@]

When you are using echo 'net.add_addr_allfibs=0' >> /boot/loader.conf things get a big odd. Since the local address don't get added that means the replies to the local host are taking a round trip to the default router which likely is configured to not send the packet back out the same interface it was received on. If you use that tunable, try something along the lines of this to give a path back for the reply traffic that won't require the round trip to the gateway.

route_fib1network="-net 192.168.102.0/24 -interface em0 -fib 1"

Try to use tcpdump(1) at on your gateway and on lo0 to gain an insight on traffic leaving and staying local on the box. Tweak your routes until you get the behavior you are looking for.

 

[perkypork]

I have done the below, which you can see in the screenshots (above) of my configuration.

When you are using echo 'net.add_addr_allfibs=0' >> /boot/loader.conf things get a big odd. Since the local address don't get added that means the replies to the local host are taking a round trip to the default router which likely is configured to not send the packet back out the same interface it was received on. If you use that tunable, try something along the lines of this to give a path back for the reply traffic that won't require the round trip to the gateway.

route_fib1network="-net 192.168.102.0/24 -interface em0 -fib 1"

 

[junovitch@]

Ok, sorry about that. I've looked at the pictures now. It looks like you've got the routes for both FIBs to see each other. Check with something like tcpdump -i lo0 -n port 22 to see if you see the traffic floating around on the loopback and particularly if you see anything like R flags for RST's being returned.