Solved Cannot access other jail via www

fred974

fred974

Hi Guys,

I am running a few webjails using sysutils/iocage and one of the webjail is used as a cdn where we host all our librairies used by our other jails like font awsome and other js scripts etc..

The problem that we have is that we are unable to access any of the files.
Code: 
fetch https://cdn.mydomain.co.uk/wp/themes/flatsome.zip
fetch: https://cdn.mydomain.co.uk/wp/themes/flatsome.zip: Connection refused
Code: 
wp plugin install https://cdn.mydomain.co.uk/wp/plugins/gplkit.zip --activate
Downloading install package from https://cdn.mydomain.co.uk/wp/plugins/gplkit.zip...
Warning: Download failed. "cURL error 7: Failed to connect to cdn.mydomain.co.uk port 443: Connection refused"
Finally, when we add this line to our css file, the font isn't loading
Code: 
@import url(https://cdn.mydomain.co.uk.co.uk/font-awesome/4.6.3/css/font-awesome.min.css)
;

The wordpress cron sare also not working..
Code: 
wp cron test
Error: WP-Cron spawn failed with error: cURL error 7: Failed to connect to myotherdomain.com port 80: Connection refused
I have read that having allow.raw_sockets=1 enabled could fix this problem..but I don't know for sure..
How do I check the curent status of raw_sockets?

I also need to mention that I have 1 public IP so I use a reverse proxy to send the correct domain to the right jail

Thank you in advance
 
Connection refused usually means there's nothing listening on the receiving end. So it looks like the web server on cdn.mydomain.co.uk isn't running.
 
Hi SirDice
The web server is definetly running at all side of the story..
The FreeBSD production server that host all the jail are in a UK datacentre.. I can run all the command above sucessfully from our server in the USA.
So the problem is defenetly internal to the FreeBSD host/jail.
I'll copy my /etc/pf.conf here as it could be firewall issue but I asked an experience user to check it already and was told firewall is good
Code: 
ExtIf   = "bce0"
JailIf  = "lo1"

www_ext    =   "91.203.xxx.xxx"
www_int    =   "10.8.20.xxx"
ftp_ext    =   "91.203.xxx.xxx"
ftp_int    =   "10.8.20.12"
Sophimail  = "91.203.xxx.xxx"
webservices = "{80, 443, 8080, 8443}"
mailservices = "{25, 993, 995, 465, 143, 587}"
netbios_tcp = "{135, 139, 445}"
netbios_udp = "{135, 139, 445}"
IPv4_icmp_types="echoreq"

table <TRUSTED> persist file "/etc/pf-files/trusted.pftable"
table <BLOCKEDZONE> persist file "/etc/pf-files/blockedzones.pftable"
table <BLOCKTEMP> counters file "/etc/pf-files/fail2ban.pftable"
table <BLOCKNETS> {224.0.0.22, 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, \
                    169.254.0.0/16, 192.0.2.0/24, \
                    192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24, \
                    169.254.0.0/16, 0.0.0.0/8, 240.0.0.0/4, 255.255.255.255/32}



### all incoming traffic on external interface is normalized and fragmented
### packets are reassembled.
scrub in on $ExtIf all fragment reassemble

nat on $ExtIf from $JailIf:network  to any -> ($ExtIf)

# Hiawatha
# --- redirect http traffic to the internal web proxy server ---
rdr on $ExtIf inet proto tcp from  !($ExtIf) to $www_ext port http  -> $www_int port  http
rdr on $ExtIf inet proto tcp from  !($ExtIf) to $www_ext port https -> $www_int port  https
rdr on $ExtIf inet proto tcp from  !($ExtIf) to $www_ext port 8443  -> $www_int port  8443
rdr on $ExtIf inet proto tcp from  !($ExtIf) to $www_ext port 8080  -> $www_int port  8080

# Pure-ftpd
# --- redirect ftp traffic to the internal ftp server ---
rdr on $ExtIf inet proto tcp from  !($ExtIf) to $ftp_ext port 21 -> $ftp_int port 21
rdr on $ExtIf inet proto tcp from  !($ExtIf) to $ftp_ext port 49000:51000 -> $ftp_int port 49000:51000

# Anchors
rdr-anchor "openvpn"


### set a default deny everything policy.
block log all

### exercise antispoofing on the external interface, but add the local
### loopback interface as an exception, to prevent services utilizing the
### local loop from being blocked accidentally.
set skip on lo

antispoof for $ExtIf inet

### get rid quick of Internet noise like microsoft netbios service.
### This accounts to 80% of dropped traffic. We don't need to log this also
block in quick on $ExtIf proto tcp from any to any port $netbios_tcp
block in quick on $ExtIf proto udp from any to any port $netbios_udp

anchor "openvpn"

### Quick blocks
block drop in log quick on $ExtIf from <BLOCKTEMP> to any
block drop in log quick on $ExtIf proto tcp from <BLOCKEDZONE> to any
block drop in log quick on $ExtIf proto tcp from <BLOCKNETS> to any

### $ExtIf inbound

# --- pass icmp echo
pass in log on $ExtIf inet proto icmp all icmp-type $IPv4_icmp_types

# --- pass incoming http/https traffic --
pass in log on $ExtIf inet proto tcp  from !($ExtIf)    to $www_int port $webservices

# --- pass incoming ssh traffic --
pass in quick log on $ExtIf proto tcp from <TRUSTED>  to $ExtIf port 22

# --- pass incoming ftp traffic ---
pass in log on $ExtIf inet proto tcp  from !($ExtIf) to $ftp_int port 21
pass in log on $ExtIf inet proto tcp  from !($ExtIf) to $ftp_int port 49000:51000

# --- pass incoming mail traffic ---
pass in log on $ExtIf inet proto tcp from any to $Sophimail port $mailservices
pass in log on $ExtIf inet proto tcp from any to $Sophimail port $webservices

### $ExtIf outbound
pass out log on $ExtIf inet proto { tcp, udp, icmp } from any to any modulate state
 
If your firewall was blocking the traffic you tend to get a "connection timed out" instead of a "connection refused". A "connection refused" means you're receiving a RST in response to a SYN packet. This typically happens when you try to connect to a closed port (i.e. there's nothing listening).
 
SirDice I understand what you are saying and yes I would agree if I couldn't fetch the data from another location.. but I can
Problem only occur on the 1 machine..
The jails are cloned interface so could it be networking problem?
Code: 
 cloned_interfaces="${cloned_interfaces} lo1"                          # allows loopback isolation in the jail
  ipv4_addrs_lo1="10.8.20.10-49/29"
Do you think allow.raw_sockets has any part to play in my problem?
Could you please sugest and command I could run to test if port is open/closed or any other usefull command I caould run

Thank you
 
fred974 said:
Do you think allow.raw_sockets has any part to play in my problem?
Click to expand...
No, that's not relevant. These are regular TCP connections and don't require raw_sockets.

Could you please sugest and command I could run to test if port is open/closed or any other usefull command I caould run
Click to expand...
The quintessential tool to debug network issues is tcpdump(1). Run it on the external interface and make sure you can actually see the connection come in. Then run it on the 'internal' interface to see if the firewall (specifically the redirection) is working properly.
 
SirDice
Public ip 91.203.xxx.xxx
Reverse proxy IP: 10.8.20.14
cdn.mydomain.co.uk IP: 10.8.20.16
seconddomain.com IP:10.8.20.17

As all www request goes trough the reverse proxy and I will run fetch https://cdn.mydomain.co.uk/wp/themes/flatsome.zip from seconddomain.com could you please help me refine my command as I have got too much traffic running tcpdump -ni bce0 port 443
I tried tcpdump -ni bce0 dst 10.8.20.16 and got nothing back
 
Something like this will probably work, tcpdump -ni bce0 host 10.8.20.16 and port 443 (you don't want to filter on a destination because you also want to see the return traffic when source and destination will be reversed)
 
Hi SirDice
You must be tired of me but I'm really stuck...

I have been running fetch https://cdn.mydomain.co.uk/wp/themes/flatsome.zip from various jail and monitoring if any request arrived to the proxy
tcpdump -ni lo0 host 10.8.20.14 and port 443
Code: 
22:46:32.752788 IP 10.8.20.17.443 > 10.8.20.14.16047: Flags [.], seq 112261:128593, ack 1397, win 1275, options [nop,nop,TS val 925937582 ecr 741016460], length 16332
22:46:32.752795 IP 10.8.20.17.443 > 10.8.20.14.16047: Flags [P.], seq 128593:140341, ack 1397, win 1275, options [nop,nop,TS val 925937582 ecr 741016460], length 11748
The same command on the external eth is not returning anything..
tcpdump -ni bce0 host 10.8.20.14 and port 443

I then run the same command fetch https://cdn.mydomain.co.uk/wp/themes/flatsome.zip from my server in the USA and monitored my public IP
tcpdump -ni bce0 host 91.203.xxx.xxx and port 443
Code: 
22:56:20.566362 IP 208.95.3.xxx.56094 > 91.203.72.237.443: Flags [.], ack 1, win 1026, options [nop,nop,TS val 568619938 ecr 2209686453], length 0
22:56:20.590359 IP 208.95.3.xxx.56094 > 91.203.72.237.443: Flags [P.], seq 1:518, ack 1, win 1026, options [nop,nop,TS val 568619962 ecr 2209686453], length 517

Not sure where to go from here..
I can see that outside traffic reach cdn server but internal request aren't..

Could it be that my nat is wrong?
sudo pfctl -s nat
Code: 
nat on bce0 inet from 10.8.20.10 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.11 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.13 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.14 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.15 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.16 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.17 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.18 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.19 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.20 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.21 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.22 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.23 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.24 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.25 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.26 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.27 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.28 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.29 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.30 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.31 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.32 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.33 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.34 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.35 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.36 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.37 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.38 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.39 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.40 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.41 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.42 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.43 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.44 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.45 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.46 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.47 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.48 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.49 to any -> (bce0) round-robin
nat on bce0 inet from 10.8.20.12 to any -> (bce0) round-robin
rdr on bce0 inet proto tcp from ! (bce0) to 91.203.xxx.xxx port = http -> 10.8.20.14 port 80
rdr on bce0 inet proto tcp from ! (bce0) to 91.203.xxx.xxx port = https -> 10.8.20.14 port 443
rdr on bce0 inet proto tcp from ! (bce0) to 91.203.xxx.xxx port = 8443 -> 10.8.20.14 port 8443
rdr on bce0 inet proto tcp from ! (bce0) to 91.203.xxx.xxx port = 8080 -> 10.8.20.14 port 8080
rdr-anchor "openvpn" all

Could you please assist me a bit further?

Thank you
 
I spent weeks trying to solve this problem and trying so manay advance debuging were the probem was infact very small.
I added the cdn hostname in the webjails /etc/hosts file for which I wanted to use the cdn for and problem is now solved :)
Code: 
10.8.20.16 mydomain.co.uk   www.cdn.mydomain.co.uk   cdn.mydomain.co.uk
I hope this post will help another newby out there...
 
You must log in or register to reply here.
Back
Top