Jail's nat redirect does not work

[gerger]

Hi all,

I created a jail using Bastille. I would like to redirect traffic to the jail with haproxy.
Somehow, the redirect does not work, I am not able to reach my python server externally.

Here is the list of jails given by Bastille:

# bastille list
 JID             IP Address      Hostname                      Path
 vanilla         10.17.89.1      vanilla                       /usr/local/bastille/jails/vanilla/root

My haproxy.conf:

defaults
  mode http
  timeout client 10s
  timeout connect 5s
  timeout server 10s
  timeout http-request 10s

frontend something.xyz
  bind :80
  acl host_vanilla hdr(host) -i vanilla.something.xyz

  use_backend vanilla if host_vanilla

backend vanilla
  server node1 10.17.89.1:8000

My pf.conf

ext_if="em0"

set block-policy return
scrub in on $ext_if all fragment reassemble
set skip on lo

table <jails> persist
nat on $ext_if from <jails> to any -> ($ext_if:0)
rdr-anchor "rdr/*"

block in all
pass out quick keep state
pass in inet proto tcp from any to any port ssh flags S/SA keep state

I've also created an rdr for the jail, which shows in ifconfig's out:

em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=481209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
    ether ca:05:6a:1c:96:1b
    inet .172.212.92 netmask 0xffffff00 broadcast 79.172.212.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bastille0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 fe80::1%bastille0 prefixlen 64 scopeid 0x3
    inet 10.17.89.1 netmask 0xffffffff
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

I started a python server inside the jail, which I can reach whith curl

curl 10.17.89.1:8000

I've configured pf based on Bastille's documentation, added the rdr, configured haproxy...
What did I miss?

Thanks.

 

[SirDice]

Run haproxy on the host. Or add a redirect to port 80 to the jail.

 

[gerger]

Haproxy runs on the host, not inside the jail.
Sorry for the misunderstanding.

 

[SirDice]

Maybe actually allow incoming port 80 in PF? You have a block in all that blocks everything.

pass in proto tcp from any to ($ext_if) port 80

 

[gerger]

Ok, I totally missed that, thanks.
However, I'm still not able to reach the server inside the jail. Now I get a 503 Service Unavailable error.
It got me thinking -- can that be because the rdr rules are failing? I mean, shouldn't I be able to reach the jail's server as
localhost:8000 (instead of <jail ip>:8000) when I have a redirect for that port?

 

[SirDice]

Now I get a 503 Service Unavailable error.

That means it's offline. Did you add a check to haproxy to see if the backend is up or not? If all backends are down haproxy will return a 503.

It got me thinking -- can that be because the rdr rules are failing? I mean, shouldn't I be able to reach the jail's server as
localhost:8000 (instead of <jail ip>:8000) when I have a redirect for that port?

Remove all those redirects. You don't need them. The proxy (running on the host) will create a new connection to the jail. There's nothing to redirect here. The host can connect to the jail's IP directly.

Also note that you must use a HTTP/1.1 connection with Host: header using vanilla.something.xyz. Or else your backend will not be selected. You probably also want to define a default_backend in haproxy.conf. If an incoming request doesn't match your acl host_vanilla hdr(host) -i vanilla.something.xyz haproxy won't know what to do with it.

 

[cmoerz]

I'm wondering - if you're using pf for redirecting traffic - would that require IP forwarding? Did you enable that already?
If you're using haproxy, you probably won't need that, I assume?

 

[SirDice]

You probably want to add this to the defaults section of haproxy.conf:

        stats enable
        stats uri /haproxy?stats
        stats realm Statistics
        stats auth haproxy_admin:supersecretpassword

Then point your browser to http://<your external IP>/haproxy?stats

 

[gerger]

That means it's offline. Did you add a check to haproxy to see if the backend is up or not? If all backends are down haproxy will return a 503.


Remove all those redirects. You don't need them. The proxy (running on the host) will create a new connection to the jail. There's nothing to redirect here. The host can connect to the jail's IP directly.

Also note that you must use a HTTP/1.1 connection with Host: header using vanilla.something.xyz. Or else your backend will not be selected. You probably also want to define a default_backend in haproxy.conf. If an incoming request doesn't match your acl host_vanilla hdr(host) -i vanilla.something.xyz haproxy won't know what to do with it.

I opened the port in pf to see, if I can reach the jail's server and it works, so the server is online and I guess that rules out my idea that the rdr is not working.

 

[SirDice]

I opened the port in pf to see, if I can reach the jail's server and it works

What did you open? There's nothing to open except the incoming connection to port 80. There are no rules needed for the host to connect to the jail, you have a set skip on lo, so this traffic is already allowed.

I guess that rules out my idea that the rdr is not working.

What redirection? You don't need one.

 

[gerger]

What did you open? There's nothing to open except the incoming connection to port 80. There are no rules needed for the host to connect to the jail, you have a set skip on lo, so this traffic is already allowed.

I added this line to pf.conf

pass in inet proto tcp from any to any port 8000 flags S/SA keep state

And now I can reach the jail's server through <external-ip>:8000

What redirection? You don't need one.

I meant the rdr I added with Bastille that redirects port 8000 to the jail's port 8000 (according to the documentation).
Do I not need this?

Edit: that rule should look like

rdr pass on em0 inet proto tcp from any to any port = 8000 -> 10.17.89.1 port 8000

 

[SirDice]

pass in inet proto tcp from any to any port 8000 flags S/SA keep state

Don't need it. It's already allowed due to set skip on lo.

I meant the rdr I added with Bastille that redirects port 8000 to the jail's port 8000 (according to the documentation).
Do I not need this?

No, you don't need this. It's haproxy that runs on the host that will make the connection. There is no redirection required for this. Haproxy is a reverse proxy. It will make a new connection.

 

[gerger]

It seems I had a problem with haproxy. The service status was "running", but the /haproxy?stats was not reachable so I started looking for logs, which aren't any by default... I configured logging for it in conf and syslogd and somehow it now works...

SirDice thanks for the help, it cleared quite a few things.

 

[SirDice]

so I started looking for logs, which aren't any by default...

I have this:

global
        daemon
        log /dev/log local2

Then a syslog rule:

local2.*                                        /var/log/haproxy.log

Or use

  log             192.168.1.230 local0

To log to a remote syslog host on 192.168.1.230.

The logs are quite cryptic though. To have a real-time view of things the status page is much easier.

For the status you could also do something like this:

frontend stats
        bind 127.0.0.1:8404
        stats enable
        stats uri /stats

That creates a separate frontend you can run on a different port and/or IP, specifically for the status page. That might be easier to access.