OpenVPN and pf - the route/NAT issue

[fbsd_fuse]

Hi,

I'm having a problem with OpenVPN and pf on a FreeBSD 10.3 box that I'd really appreciate some help with. From searching around it seems this is a classic issue with OpenVPN and pf, but none of the fixes I've seen using anchor sections or reply-to rule mixes with "no nat" lines have helped.

A picture tells a thousand words... here's a diagram of my setup in question:

In words I've got a remote pfsense box connected over an OpenVPN tunnel over the Internet to a FreeBSD 10.3 server. There's hosts on a network behind the FreeBSD server and hosts behind the pfsense box are trying to connect to them through the VPN tunnel.

Here's the pf.conf I'm using..

### Interfaces ###
ExtIf ="em0"
BackIf ="em1"
TunIf ="tun0"

### Services ###
my_ssh_services = "{ 22, 53, 2222, 2525 }"
my_web_services = "{ 80, 443 }"

### Hosts / Networks / Groups ###
outside_ip ="111.22.33.44"
outside_ip6 ="111:222:333:44::5"
back_ip ="192.168.19.1"
remote_net = "192.168.1.0/24"
back_net = "192.168.19.0/24"
vpn_net = "192.168.20.0/24"

### Options ###
set block-policy drop
set fingerprints "/etc/pf.os"
set ruleset-optimization none

### Timeouts ###
set optimization normal
set timeout { tcp.closing 60, tcp.established 7200 }

### Queues, States and Types ###
TcpState ="flags S/SA modulate state"
PlainState ="flags S/SA keep state"
UdpState ="keep state"

### Stateful Tracking Options (STO) ###
OpenSTO = "(max 90000, source-track rule, max-src-conn 1000, max-src-nodes 256)"
SmtpSTO = "(max   200, source-track rule, max-src-conn   10, max-src-nodes 256, max-src-conn-rate 200/30)"
SshSTO  = "(max   100, source-track rule, max-src-conn   64, max-src-nodes 100, max-src-conn-rate 100/30,  overload <childrens> flush global)"
WebSTO  = "(max  4096, source-track rule, max-src-conn 256, max-src-nodes 512, max-src-conn-rate 500/100, overload <childrens> flush global)"

scrub log on $ExtIf all reassemble tcp fragment reassemble
scrub out on $ExtIf no-df random-id

set skip on lo0

nat on $ExtIf from $vpn_net to any -> $outside_ip static-port
nat on $ExtIf from $back_net to any -> $outside_ip static-port

no rdr

# Final rule - goes first.
block in log all

# Inbound
# SSH
pass in on $ExtIf proto tcp from any to { $outside_ip, $outside_ip6 } \
port $my_ssh_services $TcpState $SshSTO

# Web
pass in on $ExtIf proto tcp from any to { $outside_ip, $outside_ip6 } \
port $my_web_services $TcpState $WebSTO

# Allow anything from remote network to backend network
pass in quick on $TunIf reply-to $TunIf from $remote_net to $back_net $UdpState
pass in quick on $TunIf reply-to $TunIf proto tcp from $remote_net to $back_net $TcpState
pass in quick on $TunIf reply-to $TunIf inet proto icmp from $remote_net to $back_net

# This server outbound
pass out on $ExtIf from $outside_ip to any $UdpState
pass out on $ExtIf proto tcp from $outside_ip to any $TcpState
pass out on $ExtIf inet proto icmp from $outside_ip to any
pass out on $ExtIf inet6 proto ipv6-icmp from $outside_ip6 to any

pass out on $BackIf from $back_ip to any $UdpState
pass out on $BackIf proto tcp from $back_ip to any $TcpState
pass out on $BackIf inet proto icmp from $back_ip to any

# Allow outbound from VPN clients
pass in on $TunIf from $vpn_net to any $UdpState
pass in on $TunIf proto tcp from $vpn_net to any $TcpState
pass in on $TunIf inet proto icmp from $vpn_net to any

# All outbound from backend network
pass in on $BackIf from $back_net to !$remote_net $UdpState
pass in on $BackIf proto tcp from $back_net to !$remote_net $TcpState
pass in on $BackIf inet proto icmp from $back_net to !$remote_net

# End of config

Here's what happens when the remote network host tries to connect to the webserver on the backend network:

00:00:00.000000 rule 20..16777216/0(match): block in on em1: 192.168.19.45.443 > 192.168.1.102.64765: Flags [S.], seq 3220851104, ack 1789169774, win 65535, options [mss 1100,nop,wscale 9,sackOK,TS val 98185860 ecr 903902510], length 0

pf on the FreeBSD server is blocking the return SYN/ACK at the em1 interface.

I don't get the logic of how this happens - pf should be building state from the inbound connection it first sees on tun0 and that state should allow the return traffic on em1. I have seen explanations on other threads/forums that relates this to the way OpenVPN injects the traffic, routes or such.

As I say I have tried some different rule ordering and rule structure. I can say the OpenVPN setup is ok - I can ping across it from the remote network to FreeBSD's em1 interface and see OpenVPN on the FreeBSD server building routes, learning host IPs and such.

Can someone point out the pf.conf changes I need to make?

Here's what I'm trying to allow through the server:

• remote_net clients to access backend_net servers (no NAT required)
  Currently Broken
• backend_net servers to access the Internet (NAT required)
  Currently Working
• other OpenVPN clients to route to the Internet (NAT required)
  Currently Working

Thanks in advance.

 

[SirDice]

Have you tried reloading PF when the tunnel comes up/down? The "problem" is that the interface gets created when OpenVPN starts and this happens after the firewall rules have been loaded.

From the top of my head, OpenVPN has an option in its config to run a script when the tunnel comes up or goes down. Start a script that simply reloads the firewall rules. That usually does the trick.

 

[fbsd_fuse]

Have you tried reloading PF when the tunnel comes up/down?

Do you mean reloading the rules/configuration? If so, yes. I have done that both manually and via client connect/disconnect scripts in OpenVPN.

Out of interest here's what the log shows when I enable logging on the VPN tunnel rules:

[NOPARSE]00:00:00.000000 rule 46..16777216/0(match): pass in on tun0: 192.168.1.4.27664 > 192.168.19.45.80: Flags [S], seq 812878581, win 65535, options [mss 1100,nop,wscale 6,sackOK,TS val 544658507 ecr 0], length 0
00:00:00.000074 rule 20..16777216/0(match): block in on em1: 192.168.19.45.80 > 192.168.1.4.27664: Flags [S.], seq 540192702, ack 812878582, win 65535, options [mss 1100,nop,wscale 9,sackOK,TS val 130500640 ecr 544658507], length 0
[/NOPARSE]

pf is recognising tun0 and accepting the inbound initiation.

I haven't tried reloading the pf module itself... but that would be a poor result if that was required.