Solved firewall rules for connecting reverse proxy to nginx jails

H

Haxo

Even though I have been using freebsd on my desktop and laptop for many years, I have no experience in server configuration.
After much research, trial and error, I was finally able to put together a webtoaster.

The services run in jails and each one with a private IP, in this case a jail that contains net/haproxy is configured in http mode to receive internet traffic on a single public IP and send it to the correct server according to its domain name. the domains are self contained within their own jail running each jail an www/nginx server. one domain per jail, one nginx instance within each jail. the traffic is filtered with PF on the host that contains all the jails.

Running haproxy without domain / jails backend, it works fine (I know it works because it waits for the 30s timeout to say: 504, the server does not answer the request). on the other hand nginx also works well and answers web requests, only if I disable haproxy). the point is that I don't understand how to configure PF to connect haproxy to the different jails according to the domain name of the web requests that arrive at the host.

My firewall is configured very basic, below I send the code. It seems to me that this set of PF rules works fine but something is missing to connect the haproxy traffic to the jails with nginx and my limited knowledge does not allow me to solve this, I appreciate the attention to this post, I would like to receive comments:

This is host's rc.conf
Code: 
$ sudo cat /etc/rc.conf
clear_tmp_enable="YES"
syslogd_flags="-ss"
sendmail_enable="NONE"
hostname="webtoaster"
ifconfig_vtnet0="DHCP"
enable_gateway="YES"
sshd_enable="YES"
ntpdate_enable="YES"
ntpd_enable="YES"
powerd_enable="YES"
powerd_flags="-a hiadaptive"
zfs_enable="YES"
pf_enable="YES"
pflog_enable="YES"
bastille_enable="YES"
cloned_interfaces="lo1"
ifconfig_lo1_name="bastille0"
ipv4_addrs_lo1="192.168.0.0-255/24"

This is my PF ruleset:
Code: 
$ sudo cat /etc/pf.conf
#
#PF FreeBSD firewall
#
vtnet0 = "vtnet0"
scrub in all fragment reassemble max-mss 1440
icmp_types = "{ echoreq unreach }"
table <jails> persist
nat on $vtnet0 from <jails> to any -> ($vtnet0)
## static rdr example
rdr pass inet proto tcp from any to any port {80, 443} -> 192.168.0.254
table <bruteforce> persist
table <webcrawlers> persist
table <rfc6890> { 0.0.0.0/8 10.0.0.0/8 100.64.0.0/10 127.0.0.0/8 169.254.0.0/16          \
                  172.16.0.0/12 192.0.0.0/24 192.0.0.0/29 192.0.2.0/24 192.88.99.0/24    \
                  192.168.0.0/16 198.18.0.0/15 198.51.100.0/24 203.0.113.0/24            \
                  240.0.0.0/4 255.255.255.255/32 }
# ---- Unwanted ports and IPs
junk_ports="{ 135,137,138,139,445,68,67,3222 }"
junk_ip="224.0.0.0/4"
# /----
set skip on lo0
#scrub in all fragment reassemble max-mss 1440
antispoof quick for $vtnet0
block in quick on $vtnet0 from <rfc6890>
block return out quick on egress to <rfc6890>
block all
# ---- Block junk logs
block quick proto { tcp, udp } from any to $junk_ip
block quick proto { tcp, udp } from any to any port $junk_ports
# /----
# ---- block SMTP out
block quick proto tcp from $vtnet0 to any port 25
# /----
pass in on $vtnet0 proto tcp to port { 22 } \
    keep state (max-src-conn 15, max-src-conn-rate 3/1, \
        overload <bruteforce> flush global)
pass in on $vtnet0 proto tcp to port { 80 443 } \
    keep state (max-src-conn 45, max-src-conn-rate 9/1, \
        overload <webcrawlers> flush global)
pass out proto { tcp udp } to port { 22 53 80 123 443 }
pass inet proto icmp icmp-type $icmp_types
# ---- The end

For handling the jails I usesysutils/bastille. The goal is to be able to add new sites easily and quickly with their own isolated environment. I seek the addition of PF rules to be simple and minimalist, but that it complies with an easy, readable and secure configuration.

Questions:
Does the ruleset look reasonable?
Does it meet the goals efficiently?
Does it protect my server in the right way?
Any tweaks advised?
and finally, how could connect the (frontend) HAProxy jail with (backend) nginx Jails?
 
I would like to thank the people who contributed their knowledge and good will to help me resolve this issue. To quamenzullo who shared with me personally how he resolved his thread about a jails network:

"each-website-in-its-own-jail" (thread)

and rghq who shared his solution in this thread:

"multiple-jails-communication-with-each-other" (thread)

both members of the FreeBSD community.

I should also mention the help of Neo (member of another unix / unix like community), who shared his knowledge and personal configuration of one of his networks.

The contributions of these three guys spurred my search, got me out of intellectual stagnation and I was able to solve this thread (with their help).

It was necessary to add a few more lines to /etc/pf.conf on host.

Code: 
...
lo1="bastille0"
proxy="192.168.0.254"
webservers="192.168.0.0/24"
...
pass in quick on $lo1 proto tcp from { $proxy $webservers } to { $proxy $webservers } port 8080 keep state
pass out quick on $lo1 proto tcp from { $proxy $webservers } to { $proxy $webservers } port 8080 keep state
...

I configure net/haproxy jail as proxy layer 7 to use a single public IP and direct traffic to domains/jails that listen for http requests:

Code: 
proxy: ~ #cat /usr/local/etc/haproxy.conf
# Extra firewall / nat rules are required.
# HAProxy configuration file ".
#

global
        maxconn 4096
        user haproxy
        group haproxy
        daemon

defaults
        http mode
        timeout client 5s
        timeout server 5s
        timeout connect 5s

frontend Web_proxy
        bind 192.168.0.254:80
        acl CYBERIABRAINSTORM_url hdr_dom (host) -i cyberiabrainstorm.com
        acl CYBERIACLOUD_url hdr_dom (host) -i cyberiacloud.com

        use_backend CYBERIABRAINSTORM if CYBERIABRAINSTORM_url
        use_backend CYBERIACLOUD if CYBERIACLOUD_url

CYBERIABRAINSTORM backend
        http mode
        server web-1 192.168.0.1:8080

CYBERIACLOUD backend
        http mode
        server web-2 192.168.0.2:8080

On the side of the jails that contains the www/nginx web server, the key fragment of the configuration is the following:

Code: 
cyberiabrainstorm: ~ #cat /usr/local/etc/nginx/nginx.conf
...
  server {
        listen *: 8080;
        server_name cyberiabrainstorm.com;

        location / {
            root / var / www;
            index index.html index.htm;
        }
...

This solves the problem that I did not know how to make it work for a set of jails for web serving.

When I started this post, I was not asking for a cut and paste solution and fire up my server with a ready made fast and free answer, I am interested in learning, I was just waiting for clues or suggestions to study and find the most viable solution for my case.
 
You must log in or register to reply here.
Back
Top