Jail networking

I ran tcpdump -i lo0 -n port 22 and got no traffic. It seems the traffic is being blocked by my firewall as I am getting a permission denied response. I only get this if I am trying to ssh into jails from the base system. I can happily SSH to any IP (public or private) not on the base system.

To explain this a little further. If you look at my routing tables, fib 0 contains routing information for the public ip range on the box. It seems that when I try to ssh from the base (which uses a private IP and has firewall rules only allowing traffic in/out for this private ip) its trying to use bce1 which its not able to do.
 
While I wait for some generous person to help me out, I went and reconfigured my system following junovitch initial instructions (again) for getting setfib(1) up and running. I still have the same issue. I am unable to ssh from the base system to a jail (running on the base) with a public ip. Sadly after Googling/reading for a few hours I cannot seem to find a way to fix it.
 
After a number of hours trying to work out how to fix my issue, it seems I have fixed it but I have no idea how. Very annoying, especially since I wanted to document what I did for others.

I have to build another one of these boxes, I will do my steps all over again and document exactly what I did.
 
perkypork, which address(es) did you configure your SSH servers to listen on (in /etc/ssh/sshd_config)? Did you assign your production webservers an address on interface for the private network as well as on the interface for the public network? Posting the relevant bits from the ezjail configuration file for one of the jails could be useful.
 
At a minimum you have a snapshot of some of your previous configurations from the post. Have you checked for differences against that? The FIB approach does take some time experimenting to get the right behavior and tcpdump(1) is going to be you friend. Do you see SSH sessions using the loopback now or do you see any of them using your external interfaces?
 
perkypork, which address(es) did you configure your SSH servers to listen on (in /etc/ssh/sshd_config)? Did you assign your production webservers an address on interface for the private network as well as on the interface for the public network? Posting the relevant bits from the ezjail configuration file for one of the jails could be useful.

I am running 2 ssh servers, one on base and another with a public IP. Both are configured to only listen on the IP I have allocated them.

I only allocated either a local or public IP (never both) to any one jail.
 
At a minimum you have a snapshot of some of your previous configurations from the post. Have you checked for differences against that? The FIB approach does take some time experimenting to get the right behavior and tcpdump(1) is going to be you friend. Do you see SSH sessions using the loopback now or do you see any of them using your external interfaces?

That is a good point. I have also documented what I have done and I guess I will find where I went wrong (or right) by going through that again.

I noticed that with the FIB setup, if I do not have all networking and firewalls for a jail running on Base before I start the jail, it can take some time for the jail to have network access (approx 5-10 minutes). Its sort of like things are running slowly, though my box has 92GB RAM and 12 cores. This issue is not a show stopper, just something to remember.

I have yet to master TCPdump, though I am working on understanding it.

I see no ssh sessions on loopback, not a single thing shows while running tcpdump -i lo0 -n port 22

Everything seems to be working great at the moment. My only remaining issue is running two Postgres 9.4.1 jails. I get the "semctl failed: Invalid argument" error message. I will start another post about this as its off topic.
 
As a further follow up, I can say with as much certainty that is possible for running my Jail setup for a few days, that it works, flawlessly.

I am currently running 12 jails, all running different software configurations and different firewall requirements. All of these jails are used to provide a single software platform i.e. they provide different functions for a single software platform). These jails used to be all virtual machines and administering them was extreme painful and time consuming. It used to take approx 1 day to bring up 2 servers, it now takes approx 1 day to bring up 12 servers (jails). In the future, with a little scripting, it should take minutes to bring up these 12 servers (jails).
 
Back
Top