PF Connect to jail's webserver from same jail

pentago

pentago

I have a host with 5 jails for couple of apps and a database server.

When from one webserver jail I try to curl URL of app in another or same jail i get:
Code: 
curl: (7) Failed to connect to app1.com port 80: Connection refused

By watching at these PF rules, could someone tell me why are my connections from jail to jail over web getting refused?
I have configured PF with these rules:

Code: 
# Macros
nic = "re0"
ip1 = "xxx.xxx.xxx.xxx"
ip2 = "xxx.xxx.xxx.xxx"
ip3 = "xxx.xxx.xxx.xxx"
ip4 = "xxx.xxx.xxx.xxx"
db = "192.168.1.2"
app1 = "192.168.1.3"
app2 = "192.168.1.4"
app3 = "192.168.1.5"
app4 = "192.168.1.6"
app5 = "192.168.1.7"
icmp_types = "{echoreq,unreach}"

# Options
set skip on lo
set block-policy drop
set loginterface $nic

# Normalization
scrub in no-df random-id

# NAT
nat pass on $nic from $db -> $ip1

nat pass on $nic from $app1 -> $ip2
rdr on $nic inet proto tcp to $ip2 port {25,80,443,30000} -> $app1

nat pass on $nic from $app2 -> $ip2
rdr on $nic inet proto tcp to $ip2 port {10000} -> $app2

nat pass on $nic from $app3 -> $ip3
rdr on $nic inet proto tcp to $ip3 port {25,80,443,50000} -> $app3

nat pass on $nic from $app4 -> $ip4
rdr on $nic inet proto tcp to $ip4 port {25,80,443,60000} -> $app4

nat pass on $nic from $app5 -> $ip4
rdr on $nic inet proto tcp to $ip4 port {80,443,20000} -> $app5


# Filtering
block in log
antispoof quick for $nic inet
pass in proto tcp to port {25,80,443,10000,20000,30000,40000,50000,60000}
pass inet proto icmp all icmp-type $icmp_types
pass out
 
I cannot comment on your firewall rules (I'm not an expert), but I saw here a problem that seem similar to yours.
Indeed
Code: 
curl: (7) Failed to connect to app1.com port 80: Connection refused
IMHO seem more a server related problem than a firewall one.
To me, when PF is blocking a connection, curl complains
Code: 
curl: (7) Couldn't connect to server
 
How are your jails configured? I mean to what interface are they bound?
 
SirDice said:
How are your jails configured? I mean to what interface are they bound?
Click to expand...
All jails are bound to lo1 interface and I'm using PF to NAT their traffic out so they could access web and be accessed through set of public IP's i have.

Any suggestions?
Thanks a ton!
 
Are you trying to connect to the external IP address or the internal IP address? You should use the internal address. Connecting to the external address from an internal host will cause issues because of the way PF handles packets.
 
SirDice said:
Are you trying to connect to the external IP address or the internal IP address? You should use the internal address. Connecting to the external address from an internal host will cause issues because of the way PF handles packets.
Click to expand...

Yes I'm trying to access app through external IP, in fact not really me but developers who work on applications I host in jails.
So what exactly is the problem with this? I mean, why wouldn't I be able to connect to app through its domain name instead of local IP?
 
Because the packet never actually leaves the external interface. Even if it did the rdr will only work on packets coming in from the 'outside', so it's never sent back to the internal IP address.
 
Google for "nat reflection" for one technique to overcome the limitation. PfSense for example implements it as an optional feature. The other common way to overcome the problem is "split horizon" DNS where the internal clients are served a different set of IP address for the same FQDNs.
 
SirDice said:
Because the packet never actually leaves the external interface. Even if it did the rdr will only work on packets coming in from the 'outside', so it's never sent back to the internal IP address.
Click to expand...

Hmm, NAT reflection in this case gives me pretty much OpenBSD PF related stuff and syntax..
Also, I absolutely need to have access to all my jails from at least one jail.
What would I have to do in order to make this happen?

I wonder why this would be the issue since it's pretty much straightforward accessing any URL/website from jail, why would it be problem to access domain which is hosted on same server?
 
pentago said:
I wonder why this would be the issue since it's pretty much straightforward accessing any URL/website from jail, why would it be problem to access domain which is hosted on same server?
Click to expand...
From pf.conf(5):
Code: 
     [b]Translation rules apply only to packets that pass through the specified
     interface[/b], and if no interface is specified, translation is applied to
     packets on all interfaces.  For instance, redirecting port 80 on an
     external interface to an internal web server will only work for connec-
     tions originating from the outside.  [b]Connections to the address of the
     external interface from local hosts will not be redirected, since such
     packets do not actually pass through the external interface.[/b]  Redirec-
     tions cannot reflect packets back through the interface they arrive on,
     they can only be redirected to hosts connected to different interfaces or
     to the firewall itself.
 
SirDice said:
From pf.conf(5):
Code: 
     [b]Translation rules apply only to packets that pass through the specified
     interface[/b], and if no interface is specified, translation is applied to
     packets on all interfaces.  For instance, redirecting port 80 on an
     external interface to an internal web server will only work for connec-
     tions originating from the outside.  [b]Connections to the address of the
     external interface from local hosts will not be redirected, since such
     packets do not actually pass through the external interface.[/b]  Redirec-
     tions cannot reflect packets back through the interface they arrive on,
     they can only be redirected to hosts connected to different interfaces or
     to the firewall itself.
Click to expand...

Hmm didn't know that.
OK then, do you have any suggestion on what exactly I'd need to do in order to triage this problem?
Is this solved on pf.conf or networking configuration level?

Don't have idea on where to start solving this.

Thanks a ton for answers so far!
 
The simplest solution is to use a "split-horizon" DNS as kpa suggested. Internal clients will then resolve hostnames to internal addresses and external clients resolve the hostnames to external addresses.
 
You must log in or register to reply here.
Back
Top