Jail to jail communication failing (lo1 to wan interface)

P

perkypork

I was hoping someone could help me out to solve this issue as its been driving me a little crazy for a while.

I have two jails on a host. One jail (mysql) has a public ip (different to the host) on the public interface (em0) and the other jail (http) has an IP on the lo1 interface (say 10.0.0.1). The jail on the lo1 interface is able to access the outside world using pf nat.

When they try to talk with each other, the communication fails because pf blocks the traffic. I have no idea why the communication isn't working because based on my rules it should. All I can see from pf logs is the mysql jail communicating with the http jail via the lo1 interface. For some reason I never see the initial communication from the http jail to the mysql jail, my hunch is the initial communication is being natted but I am not seeing it in the logs.

I do not want to use "set skip on lo1" to solve this issue as it would break a number of other things. Does anyone have any ideas on how I might be able to solve this one?

EDIT: punctuation and grammar
 
perkypork said:
When they try to talk with each other, the communication fails because pf blocks the traffic. I have no idea why the communication isn't working because based on my rules it should.
Click to expand...
I will probably help immensely if you showed us your rules.
 
SirDice said:
I will probably help immensely if you showed us your rules.
Click to expand...

I would do this but they are ridiculously complex rules in many anchors and rdr rules, they would probably confuse more than help. It doesn't seem to matter what rules I have, the traffic doesn't flow the way I envisage it should or maybe I am misunderstanding how the traffic should flow. If the traffic flowed the way I expected it to, it would work with the rules I currently have. All my rules are structured like each jail has access to only one interface with a single IP. Traffic between jails on the same interface work fine but traffic between jails on different interfaces do not work properly.

What I would love to understand is how I should expect traffic to flow in this situation so that I can write the proper pf rules.

This is all that I am seeing in the logs. It seems the http jail is able to establish a connection with the mysql jail but the mysql jail is responding to the jail http on the lo1 interface.
block out on lo1: AAA.BBB.CCC.DDD.53306 > 10.0.158.115.49288
On further investigation the http jail was establishing its communication via lo0 but the mysql jail is responding via the lo1 interface.
 
I have done some further testing today, which produced some results I am unsure of.

If I set "set skip on lo0" in pf.conf, the issue above occurs.

If I set "pass log on lo0 from any to any" and comment out "set skip on lo0", the issue I describe above is fixed BUT the host is unable to communicate with other jails with a public IP.
 
In the hopes that someone might be able to help me.

I think my issue is not in regards to pf but in regards to routing. If I use vimage this issue is non existent but because of the bugs in vimage I haven't been that keen to use it. I was wondering if there was another way to implement vimage style routing (using epair or netgraph) without having to use vimage, a poor mans vimage so to say?
 
You must log in or register to reply here.
Back
Top