I run multiple jails on a physical FreeBSD 13.1 server. For simplicity this is what it looks like:
lb01 runs nginx that I've been using as a TCP/UDP proxy as a frontend to services running in other jails that I'd like to be available from outside of my network.
app01 runs such a service on port 33333/tcp.
Currently, in non-transparent proxy mode, the traffic flow looks as follows:
This works without issues, however I'd like to enable nginx transparent mode so that service on app01:33333/tcp is aware of the real client IP.
This changes (or rather *should change*) the traffic flow as follows:
Jail app01 uses lb01 as its default gateway, so in the latter case the replies from app01 to the spoofed client IP address (5.5.5.5), arrive at lb01 and then without any pf rules, obviously get sent on to 5.5.5.5.
Because app01 is not to the original request from the client sent from src_port1 but to the proxied request with a src_port2, the client IP drops the packet.
That obviously doesn't work as intended, I'm not expecting magic here.
What I've been failing to do is to use PF to redirect the incoming response from app01 on epair1b to the nginx running locally, instead of being normally routed through the ext_if.
The original instructions published by nginx for Linux are rather simple but I can't work out an equivalent on FreeBSD:
I created a separate routing table used in one of the attempts described later:
The rules below (tested separately, of course) stop the response from 10.0.0.20:<src_port2> from trying to leave epair2b but still don't cause the response to be intercepted by nginx, which complains about the lack of response from app01:
I've tested various other rules, to no avail but I'd like to omit them here as this post is already quite long.
I've been reading man pages of pf, pf.conf and docs section on firewall and advanced networking. So far that hasn't helped me find what I'm missing.
Could someone please point me in the right direction?
Thank you.
Code:
Physical machine:
em0: 10.0.0.10/24
Jail lb01:
epair1b: 10.0.0.20/24
epair2b: 1.2.3.4/24 (public)
Jail app01:
epair3b: 10.0.0.30/24lb01 runs nginx that I've been using as a TCP/UDP proxy as a frontend to services running in other jails that I'd like to be available from outside of my network.
app01 runs such a service on port 33333/tcp.
Currently, in non-transparent proxy mode, the traffic flow looks as follows:
Code:
5.5.5.5:<src_port1> -> 1.2.3.4:33333 (request from external client to nginx proxy)
10.0.0.20:<src_port2> -> 10.0.0.30:33333 (request from nginx on lb01 to app01)
10.0.0.20:<src_port2> <- 10.0.0.30:33333 (response from app01 to nginx on lb01)
5.5.5.5:<src_port1> <- 1.2.3.4:33333 (respose from nginx proxy on lb01 to external client)This changes (or rather *should change*) the traffic flow as follows:
Code:
5.5.5.5:<src_port1> -> 1.2.3.4:33333 (request from external client to nginx proxy)
5.5.5.5:<src_port2> -> 10.0.0.30:33333 (request from nginx on lb01 to app01, with a spoofed client IP)
5.5.5.5:<src_port2> <- 10.0.0.30:33333 (response from app01 to the client IP)
5.5.5.5:<src_port1> <- 1.2.3.4:33333 (respose from nginx proxy on lb01 to external client)Jail app01 uses lb01 as its default gateway, so in the latter case the replies from app01 to the spoofed client IP address (5.5.5.5), arrive at lb01 and then without any pf rules, obviously get sent on to 5.5.5.5.
Because app01 is not to the original request from the client sent from src_port1 but to the proxied request with a src_port2, the client IP drops the packet.
That obviously doesn't work as intended, I'm not expecting magic here.
What I've been failing to do is to use PF to redirect the incoming response from app01 on epair1b to the nginx running locally, instead of being normally routed through the ext_if.
The original instructions published by nginx for Linux are rather simple but I can't work out an equivalent on FreeBSD:
Code:
# ip rule add fwmark 1 lookup 100
# ip route add local 0.0.0.0/0 dev lo table 100
# iptables -t mangle -A PREROUTING -p tcp -s 172.16.0.0/28 --sport 80 -j MARK --set-xmark 0x1/0xffffffffI created a separate routing table used in one of the attempts described later:
Code:
# setfib 1 netstat -rnW
Routing tables (fib: 1)
Internet:
Destination Gateway Flags Nhop# Mtu Netif Expire
default link#1 US 1 16384 lo0
127.0.0.1 link#1 UHS 2 16384 lo0The rules below (tested separately, of course) stop the response from 10.0.0.20:<src_port2> from trying to leave epair2b but still don't cause the response to be intercepted by nginx, which complains about the lack of response from app01:
Code:
int_if="epair1b"
ext_if="epair2b"
pass in quick on $int_if proto tcp from $int_if:network port 33333 to any flags any rtable 1
rdr pass on $int_if inet proto tcp from $int_if:network port 33333 to any -> 127.0.0.1
pass in quick on $int_if route-to (lo0 127.0.0.1) proto tcp from $int_if:network port 33333 to anyI've tested various other rules, to no avail but I'd like to omit them here as this post is already quite long.
I've been reading man pages of pf, pf.conf and docs section on firewall and advanced networking. So far that hasn't helped me find what I'm missing.
Could someone please point me in the right direction?
Thank you.