PF Port forwarding: PostgreSQL behind gateway not accessible

jbo@

Developer
Hi,

Simple setup:
Code:
Internet ------ Host [A] ------ Host [B]
Host A:
  • Is a simple firewall/gateway/router/DNS server
  • Runs PF & HAproxy
  • Has the private network IP address 192.168.7.1
  • Connects to the ISP's gateway on the other end
Host B

My goal is to be able to access the PostgreSQL database(s) from the public internet.
I setup PostgreSQL successfully in a way that I can access it from any other machine on the same private network successfully.
The next logical step was to modify my host A's pf configuration and add some port forwarding to it:
Code:
rdr pass on $if_wan0 proto tcp from any to any port 5432 -> 192.168.7.239
At this point I expected to be able to just login into the database server from anywhere I'd like too. But oh boy was I mistaken.
An entire day later I am still unable to figure out what's wrong here.

Symptoms:
  • Host A is a gateway that is already running in that very network almost two years. It also runs HAproxy and manages several web servers behind it (in the same network as my new PostgreSQL instance is located. Everybody is able to call up those websites --> The overall networking & routing appears to be working.
  • Running tcpdump() on host A I can see the incoming packages on the external interface on port 5432. However, I can't see any packages with port 5432 on the interface connected to the internal network - This makes me suspect that it's a PF configuration issue.

Here's the full /etc/pf.conf of host A:
Code:
# Define interfaces
if_lan0="igb0"   # Management
if_lan1="igb1"   # DNS access
if_lan2="igb2"   # Client gateway 1
if_wan0="igb3"   # Swisscom modem
if_pfsync="igb4" # PFsync
if_loc0="lo0"    # Loopback

# Define networks
serversnet = $if_lan2:network

# Define ports
allowed_ports_in_tcp  = "{ ssh, http, https }"
allowed_ports_out_tcp = "{ ssh, http, https }"
allowed_ports_udp     = "{ domain, ntp }"
allowed_icmp_types    = "{ echoreq, unreach }"

# Options
set block-policy drop

# Scrub
scrub in all

# Ignore loopback interface
set skip on $if_loc0

# NAT
nat on $if_wan0 inet from $serversnet to any -> ($if_wan0) static-port

# Redirects
rdr pass on $if_wan0 proto tcp from any to any port 5432 -> 192.168.7.239 port 5432
rdr pass on $if_wan0 proto tcp from any to any port 10051 -> 192.168.7.14 port 10051
rdr pass on $if_wan0 proto tcp from any to any port 2200  -> 192.168.7.235 port 22
rdr pass on $if_wan0 proto tcp from any to any port 2201 -> 192.168.8.16 port 22

# Deal with bruteforcers
table <bruteforce> persist
block quick from <bruteforce>

# Antispoof everything!
antispoof for {$if_lan0, $if_lan1, $if_lan2, $if_wan0}

# Let the whitelisting begin...
block all

# Whitelisting...
pass quick on $if_pfsync proto pfsync keep state (no-sync)
pass from {self, $serversnet} to any keep state
pass quick proto { tcp, udp } to port $allowed_ports_udp
pass inet proto icmp icmp-type $allowed_icmp_types

pass in on {$if_lan1 $if_wan0} proto tcp from any to any port {http, https, 8006, 54899, 54900} keep state

pass in quick on $if_lan2 proto tcp from $serversnet to $if_wan0:network port $allowed_ports_in_tcp keep state
pass proto tcp from $serversnet to port $allowed_ports_out_tcp keep state
pass in quick on {$if_lan0, $if_wan0} proto tcp from any to any port 22 flags S/SA keep state (max-src-conn 10, max-src-conn-rate 50/3600, overload <bruteforce> flush global)

pass out on $if_lan2 proto tcp from any to $serversnet port 22 keep state

On host B, I ensure that PostgreSQL is listening on the correct interface (postgresql.conf):
Code:
listen_addresses = 'localhost, 192.168.7.239'

And the corresponding pg_hba.conf allows remote connections using password authentication:
Code:
# TYPE  DATABASE        USER            ADDRESS                 METHOD

# "local" is for Unix domain socket connections only
local   all             all                                     trust
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
# IPv6 local connections:
host    all             all             ::1/128                 trust
# Allow replication connections from localhost, by a user with the
# replication privilege.
local   replication     all                                     trust
host    replication     all             127.0.0.1/32            trust
host    replication     all             ::1/128                 trust

host    all             all             0.0.0.0/0               md5

So, given the symptoms i think it's pretty safe to assume that it's not a PostgreSQL configuration issue but instead something related to the routing/filtering/forwarding on host A.

I'd be thankful for any kind of input that might help tracking down & solving this problem.

At this point I'd also like to thank DanDare from freenode for already having taken the time looking into this.
 
Right. There's no rule allowing the traffic out on $if_lan2. The traffic is allowed in via rdr pass but because you have a block all rule it's not allowed to go out $if_lan2.

I would recommend not using rdr pass and use separate rdr and pass rules to avoid ambiguity.

As a side-note, restrict access to port 10051, which I assume is Zabbix.
 
Interesting - that makes sense.

I adjusted the last line in my /etc/pf.conf to allow traffic to leave $if_lan2 on/with port 5432:
Code:
pass out on $if_lan2 proto tcp from any to $serversnet port {22, 5432} keep state

However, I'm still out of luck. I also still don't see any traffic on port 5432 coming out of igb3 / $if_lan2.
 
Did you remove the pass from the rdr pass lines? Add additional rules to allow the traffic to enter, something like this:
Code:
pass in on $if_wan0 proto tcp from any to 192.168.7.239 port 5432

Keep in mind that NAT happens first so the destination address needs to be the internal address.

The biggest problem with rdr pass is that the traffic is passed (on that interface) and all other rules are then ignored. Fine for really simple, basic rules but not for more complex situations.
 
Thank you for the note regarding rdr pass - I think I have to dive a bit deeper into the corresponding documentation to understand this better.

What I did:
  • Remove the pass from the rdr lines
  • Explicitly allow outgoing traffic on $if_lan2 for port 5432 to $serversnet
  • Explicitly allow incoming trafic on $if_wan0 for port 5432
This makes my /etc/pf.conf look like this:
Code:
# Define interfaces
if_lan0="igb0"   # Management
if_lan1="igb1"   # DNS access
if_lan2="igb2"   # Client gateway 1
if_wan0="igb3"   # Swisscom modem
if_pfsync="igb4" # PFsync
if_loc0="lo0"    # Loopback

# Define networks
serversnet = $if_lan2:network

# Define ports
allowed_ports_in_tcp  = "{ ssh, http, https }"
allowed_ports_out_tcp = "{ ssh, http, https }"
allowed_ports_udp     = "{ domain, ntp }"
allowed_icmp_types    = "{ echoreq, unreach }"

# Options
set block-policy drop

# Scrub
scrub in all

# Ignore loopback interface
set skip on $if_loc0

# NAT
nat on $if_wan0 inet from $serversnet to any -> ($if_wan0) static-port

# Redirects
rdr on $if_wan0 proto tcp from any to any port 5432 -> 192.168.7.239 port 5432
rdr pass on $if_wan0 proto tcp from any to any port 10051 -> 192.168.7.14 port 10051
rdr pass on $if_wan0 proto tcp from any to any port 2200  -> 192.168.7.235 port 22
rdr pass on $if_wan0 proto tcp from any to any port 2201 -> 192.168.8.16 port 22

# Deal with bruteforcers
table <bruteforce> persist
block quick from <bruteforce>

# Antispoof everything!
antispoof for {$if_lan0, $if_lan1, $if_lan2, $if_wan0}

# Let the whitelisting begin...
block all

# Whitelisting...
pass quick on $if_pfsync proto pfsync keep state (no-sync)
pass from {self, $serversnet} to any keep state
pass quick proto { tcp, udp } to port $allowed_ports_udp
pass inet proto icmp icmp-type $allowed_icmp_types

pass in on {$if_lan1 $if_wan0} proto tcp from any to any port {http, https, 8006, 54899, 54900} keep state

pass in quick on $if_lan2 proto tcp from $serversnet to $if_wan0:network port $allowed_ports_in_tcp keep state
pass proto tcp from $serversnet to port $allowed_ports_out_tcp keep state
pass in quick on {$if_lan0, $if_wan0} proto tcp from any to any port 22 flags S/SA keep state (max-src-conn 10, max-src-conn-rate 50/3600, overload <bruteforce> flush global)

pass in on $if_wan0 proto tcp from any to any port 5432
pass out on $if_lan2 proto tcp from any to $serversnet port {22, 5432} keep state

I left the pass in the other (existing rdr entries for now.
Unfortunately, still no luck. Nothing changed from the behavior point of view.

What am I misunderstanding?
 
Just to make sure, you do have gateway_enable="YES" in rc.conf? I assume it's there as it's working for your other things.

Double check with tcpdump(1) and verify there's actually something coming in on $if_wan0. Run a similar check on your $if_lan2. And make sure you are connecting from outside your network or else you will run into another problem (hairpinning).
 
Just to make sure, you do have gateway_enable="YES" in rc.conf?
Yep, that's there (and works as you assumed).

Double check with tcpdump(1) and verify there's actually something coming in on $if_wan0.
I already did that - I just did it again to confirm that the result is still as expected.
When trying to connect to the database from an external network, I can see the packages coming in (the client sends the same connection request four times before it decides to give up):
Code:
root@silver1:~ # tcpdump -i igb3 'port 5432'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb3, link-type EN10MB (Ethernet), capture size 262144 bytes
16:11:29.656671 IP adsl-xxx-xxx.dsl.init7.net.56619 > 172.31.255.6.postgresql: Flags [S], seq 2371430245, win 64240, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0
16:11:30.657508 IP adsl-xxx-xxx.dsl.init7.net.56619 > 172.31.255.6.postgresql: Flags [S], seq 2371430245, win 64240, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0
16:11:32.661228 IP adsl-xxx-xxx.dsl.init7.net.56619 > 172.31.255.6.postgresql: Flags [S], seq 2371430245, win 64240, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0
16:11:36.665544 IP adsl-xxx-xxx.dsl.init7.net.56619 > 172.31.255.6.postgresql: Flags [S], seq 2371430245, win 64240, options [mss 1400,nop,wscale 8,nop,nop,sackOK], length 0
^C
4 packets captured
1334 packets received by filter
0 packets dropped by kernel

At the same time, I don't see anything on the interface of the servers network (igb2 -> $if_lan2):
Code:
root@silver1:~ # tcpdump -i igb2 'port 5432'
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb2, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
372 packets received by filter
0 packets dropped by kernel

Just to mention the obvious: I do always properly reload the pf rule set. Sometimes I even restart pf completely just to be sure that it's not a caching issue or similar side effect caused by alive connections or anything like that.

In case of this helps: Here's my ifconfig of host A:
Code:
ix0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether ac:1f:6b:6e:1f:6a
        hwaddr ac:1f:6b:6e:1f:6a
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: no carrier
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether ac:1f:6b:6e:1f:6b
        hwaddr ac:1f:6b:6e:1f:6b
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: no carrier
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether ac:1f:6b:6e:1a:7c
        hwaddr ac:1f:6b:6e:1a:7c
        inet 192.168.8.12 netmask 0xffffff00 broadcast 192.168.8.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether ac:1f:6b:6e:1a:7d
        hwaddr ac:1f:6b:6e:1a:7d
        inet 192.168.1.5 netmask 0xffffff00 broadcast 192.168.1.255 vhid 1
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        carp: MASTER vhid 1 advbase 1 advskew 100
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether ac:1f:6b:6e:1a:7e
        hwaddr ac:1f:6b:6e:1a:7e
        inet 192.168.7.1 netmask 0xffffff00 broadcast 192.168.7.255 vhid 2
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        carp: MASTER vhid 2 advbase 1 advskew 100
igb3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether ac:1f:6b:6e:1a:7f
        hwaddr ac:1f:6b:6e:1a:7f
        inet 172.31.255.6 netmask 0xfffffffc broadcast 172.31.255.7 vhid 3
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        carp: MASTER vhid 3 advbase 1 advskew 100
igb4: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether ac:1f:6b:6e:1a:80
        hwaddr ac:1f:6b:6e:1a:80
        inet 192.168.255.1 netmask 0xffffff00 broadcast 192.168.255.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether ac:1f:6b:6e:1a:81
        hwaddr ac:1f:6b:6e:1a:81
        inet 192.168.7.12 netmask 0xffffff00 broadcast 192.168.7.255
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
pfsync0: flags=41<UP,RUNNING> metric 0 mtu 1500
        groups: pfsync
        pfsync: syncdev: igb4 syncpeer: 224.0.0.240 maxupd: 128 defer: off
pflog0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 33160
        groups: pflog
 
Oh, check and make sure your IP hasn't accidentally ended up in the bruteforce table.
 
I've done that - not black-listed.
Also, I am able to access websites hosted by webservers behind the same firewall in the same server network without any problems.

I've done several other things such as ensuring that there's no other service on host A which listens to port 5432 using netstat -4 -l. Furthermore, I've re-double-checked my HAproxy config (also running on host A) to ensure that nothing fishy is going on there.

Any wild guesses?
 
Looking at my configuration, I would guess "block all" is too low in the file. The only reason your other rdrs work is that they match passes underneath "block all". Try moving the single rdr for postgres below it and see if it works
 
As per my understanding it's not possible to move a rdr rule further down as then PF will complain about not sticking to the order.
 
Well, I thought I was looking close enough, but I guess not. Sorry

I gave up trying to figure out where FreeBSD's pf spec is located (all links seem to point to newer, non-compatible OpenBSD implementations) and just ran OpenBSD instead since I wanted to use PF.
 
You also have two interfaces on the same subnet (that you are trying to rdr into): igb2 and igb5; but only have a pass out rule for igb2. Everything is blocked on igb5. Just because pf will block all traffic on igb5 doesn't mean the system won't attempt to send traffic out it.

Code:
igb2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        inet 192.168.7.1 netmask 0xffffff00 broadcast 192.168.7.255 vhid 2

igb5: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        inet 192.168.7.12 netmask 0xffffff00 broadcast 192.168.7.255

Perhaps also check route show 192.168.7.239

What is igb5 for? Did you really mean to have two interfaces on the same subnet? I'm not sure what's going to happen here, but the interplay of pf, rdr, and multiple interfaces (and the destination (host b) subnet/router settings) muddies the water.

This could just be a red herring, but if you don't need igb5 right now, turn it off while figuring out your pf rules, and then carefully add it back in when you are set.
 
As per my understanding it's not possible to move a rdr rule further down as then PF will complain about not sticking to the order.

That's correct. The rdr is applied, and then the filtering rules are applied to the rewritten packet. Using rdr pass can help simplify rules, but it then makes it impossible futher filter (for example, your blacklist would not be applied) the redirected packets.
 
What is igb5 for? Did you really mean to have two interfaces on the same subnet? I'm not sure what's going to happen here, but the interplay of pf, rdr, and multiple interfaces (and the destination (host b) subnet/router settings) muddies the water.
This could just be a red herring, but if you don't need igb5 right now, turn it off while figuring out your pf rules, and then carefully add it back in when you are set.
Sir, Thank you very much! This was indeed the issue. If I take igb5 down everything is working.

The reason for having igb5 as a second interface on the same subnet: I run two of these firewall machines (within the scope of this thread referred to as 'Host A') in failover configuration using carp(). These machines are named silver1 and silver2. igb3 is used to connect to the upstream ISP gateway and igb2 is used to connect to the private network. CARP is used on those two interfaces to ensure that either of the machines can keep the network alive. So far so good.
In the private network (192.168.7.0/24) there is also a net-mgmt/zabbix34-server running to monitor all hosts (also some of the outside world). The zabbix server is listening on 192.168.7.14 and I want to monitor both silver1 and silver2 at the same time. Therefore, I brought up igb5 on both machines, assigning silver1's igb5 to 192.168.7.12 and silver2's igb5 to 192.168.7.18. This way I am able to monitor both of them constantly. This would not be the case if I simply monitor on 192.168.7.1 as that would mean that only one of the two hosts (the current CARP master) can be monitored.

I can only assume there is a much better solution to this?

Thanks for your help - It's very appreciated!
 
I’ve never used carp, but this looks like a nice how-to, especially for live pf fail-over.

I think I would likely add another subnet for monitoring; each network card can have more than one IP.
 
Back
Top