IPFW Stateful firewall with OpenVPN and in-kernel NAT

T

toastingwitharowaway

Having a bit of a time getting stateful firewall with OpenVPN and in-kernel NAT to work, which is a few lines of iptables rules on Linux:

Code: 
*nat
:pREROUTING ACCEPT [0:0]
:pOSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE
-A POSTROUTING -s 10.8.0.0/24 -j SNAT --to-source $WAN_IP
COMMIT

Idea is to allow clients to connect to a server on its public IP running VPN on port 1194. OpenVPN routes its traffic on 10.8.0.0/24 through tun0. Then, that traffic goes out the same interface (vmx0) of the initial incoming connection.

/etc/sysctl.conf additions are:

Code: 
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1

net.inet.ip.portrange.hifirst=25000
net.inet.ip.portrange.hilast=49151

net.inet.ip.redirect=0
net.inet.icmp.drop_redirect=1

net.inet.tcp.sendspace: 65536
net.inet.tcp.recvspace: 65536
net.local.stream.sendspace=65536
net.local.stream.recvspace=65536

net.inet.icmp.maskrepl=0
net.inet.tcp.path_mtu_discovery=1
net.inet.tcp.sack.enable=1
net.inet.icmp.icmplim=1000
net.inet.tcp.syncookies=1
net.inet.ip.fw.dyn_buckets=65536
net.inet.ip.fw.dyn_max=65536
kern.ipc.soacceptqueue=20480
net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.finwait2_timeout=10000

kern.ipc.shmmax=68719476736
kern.ipc.shmall=16777216
kern.ipc.shm_use_phys=1
kern.threads.max_threads_per_proc=16384

net.inet.ip.fastforwarding=1
net.inet.ip.forwarding=1
net.inet.ip.fw.one_pass=0
net.inet.ip.fw.verbose=1
net.inet.ip.sourceroute=1

Relevant additions in /etc/rc.conf are:

Code: 
firewall_nat_enable="YES"
firewall_nat_interface="vmx0"
firewall_nat_flags="unreg_only reset"

# ipfw nat show config
Code: 
ipfw nat 1 config if vmx0 log unreg_only reset

# ipfw -a list
Code: 
00100  0  0 allow ip from any to any via lo0
00200  0  0 allow log ip from any to any via tun0
00400  7  420 nat 1 log ip from any to any dst-port 1194 in via vmx0
00500  0  0 check-state
00600  7  420 skipto 65000 tcp from any to any dst-port 1194 in via vmx0 setup keep-state
00700  0  0 skipto 65000 udp from any to any dst-port 1194 in via vmx0 keep-state
00800  0  0 deny ip from 0.0.0.0/8 to any via vmx0
01000  0  0 deny ip from 127.0.0.0/8 to any via vmx0
01100  0  0 deny ip from 169.254.0.0/16 to any via vmx0
01200  0  0 deny ip from 172.16.0.0/12 to any via vmx0
01300  0  0 deny ip from 192.0.2.0/24 to any via vmx0
01400  0  0 deny ip from 192.168.0.0/16 to any via vmx0
01500  0  0 deny ip from 204.152.64.0/23 to any via vmx0
01600  0  0 deny ip from 224.0.0.0/3 to any via vmx0
01700 25 1498 allow ip from me to any dst-port 20,21,22,25,53,80,123,443,465,587,1194,2401,3128,8443,9418 out via vmx0 keep-state
01800  0  0 skipto 65000 tcp from any to any out via vmx0 setup keep-state
01900  0  0 skipto 65000 udp from any to any out via vmx0 keep-state
02000  0  0 allow tcp from any 20-21 to me dst-port 25000-49151 in via vmx0 setup keep-state
02100 54 6224 allow tcp from any to me dst-port 22 in via vmx0 setup keep-state
02200  0  0 allow log tcp from any to me dst-port 1194 in via vmx0 setup keep-state
02250  0  0 allow log udp from any to me dst-port 1194 in via vmx0 keep-state
02300 67 3723 allow tcp from any to me dst-port 3128 in via vmx0 setup keep-state
02400  0  0 allow tcp from $NAGIOS_SERVER to me dst-port 5666 in via vmx0 keep-state
02500  0  0 allow icmp from $NAGIOS_SERVER to me in via vmx0 keep-state
65000  0  0 nat 1 ip from any to any out via vmx0
65100  0  0 deny log ip from me to any out via vmx1
65200  0  0 deny log ip from any to me in via vmx1
65300  0  0 deny log ip from me to any out via vmx2
65400  0  0 deny log ip from any to me in via vmx2
65500 12  712 deny log ip from any to any
65535  0  0 deny ip from any to any

I can see packets getting NAT'd, then denied, in /var/log/security:

Code: 
Apr 22 22:39:06 $HOST kernel: ipfw: 400 Nat TCP $MY_IP:56640 $SERVER_IP:1194 in via vmx0
Apr 22 22:39:06 $HOST kernel: ipfw: 65500 Deny TCP $MY_IP:56640 $SERVER_IP:1194 in via vmx0

Had another version of the firewall setup without the check-state and skipto rules, which allowed clients to get an IP from the OpenVPN server. However, those clients couldn't have traffic routed through it nor ping any of the IPs on the 10.8.0.0/24 network. With this setup, connections to the server on port 1194 time out.

All the other services on this machine (SSH, squid, NRPE) work fine. It must be something small and stupid that's missing and can't see what it is. This is FreeBSD 10.2-RELEASE-p9/amd64. If any further information is required, just ask.
 
Last edited by a moderator:
I setup a NAT'ing stateful firewall utilizing ipfw(8) and in-kernel NAT for a very similar purpose. Although, instead of OpenVPN, I use a L2TP/IPsec-VPN solution.

Your firewall rule #400 looks strange to me, why do you limit incoming internet packets to port 1194? My respective rule is simply:
Code: 
nat 1 ip4 from any to any in recv $WAN_INTERFACE
I cannot tell anything about OpenVPN, however, in order that VPN clients that are connected to my L2TP/IPsec server can see each other and other clients on the local network, as well as, can connect to the internet via NAT, I need to enable proxy arp in the L2TP setup. Does OpenVPN provide proxy arp?
 
OpenVPN itself does not provide proxy arp and proxy arp is almost never used with it because it is assumed that the "tunnel" network uses a subnet that does not overlap with any of the subnets used on either side of the tunnel.
 
Apologies for the late response.
obsigna said:
I setup a NAT'ing stateful firewall utilizing ipfw(8) and in-kernel NAT for a very similar purpose. Although, instead of OpenVPN, I use a L2TP/IPsec-VPN solution.

Your firewall rule #400 looks strange to me, why do you limit incoming internet packets to port 1194? My respective rule is simply:
Code: 
nat 1 ip4 from any to any in recv $WAN_INTERFACE
[...]
Click to expand...
Thanks for your suggestion. Your guide was what I used as a template for this configuration. I changed the rule to:
Code: 
00401 nat 1 log ip4 from any to any in recv vmx0
removing the previous rule 400. The behaviour remains unchanged though.

ctaranotte said:
Openvpn nats the traffic through tun0 (in and out).

Please post the output of #netstat -r
Click to expand...
There are a lot of IPs bound to the WAN interface (vmx0), so this is an abridged version:
netstat -r4 | head -12
Code: 
Routing tables

Internet:
Destination  Gateway  Flags  Netif Expire
default  $WAN_GATEWAY  UGS  vmx0
10.8.0.0  10.8.0.2  UGS  tun0
10.8.0.1  link#5  UHS  lo0
10.8.0.2  link#5  UH  tun0
$LAN_NETWORK  link#2  U  vmx1
$LAN_IP  link#2  UHS  lo0
$WAN_NETWORK/26  link#1  U  vmx0
$WAN_IP  link#1  UHS  lo0
Your responses are appreciated.
 
You might notneed NAT for incoming connections on OpenVPN port 1194, use something like this
Code: 
allow ip from any to me 1194 in recv $EXTIF
allow ip from me to any out xmit $EXTIF
 
Thanks for your recommendation. I had that originally (in a non-NAT config) but it wasn't working. I can connect and get an IP from the OpenVPN server, but packets won't get routed outside the machine. That second line affects the outbound traffic from the squid service on that machine too.
 
After adding the keep-state keyword to those lines, connections to port 1194 can happen again and connections exit the squid service. However, traffic doesn't exit via OpenVPN still.
 
I got a similar setup working with one box acting as a gateway for two others. Thus, I copied and modified that firewall config for this task and it's here:

Code: 
#!/bin/sh

cmd="ipfw -q add"
natcmd="ipfw -q nat"
wan="vmx0"
lan="vmx1"
other="vmx2"
vpn="tun0"
wanip="[redacted]"
wannet="$wanip/26{[redacted]}"
vpnnet="10.8.0.0/24"
vpnhosts="10.8.0.1,10.8.0.2,10.8.0.3,10.8.0.4,10.8.0.5,10.8.0.6,10.8.0.7,10.8.0.8,10.8.0.9,10.8.0.10"
ks="keep-state"
sks="setup $ks"

pm="[redacted]"

# squid proxy
squidproxy="[redacted]"

ipfw -q -f flush

# allow loopback to flow freely
$cmd 10 allow all from any to any via lo0
$cmd 11 allow all from any to any via tun0

# block all traffic on the LAN interface
$cmd 12 deny all from any to any via $lan

# Configure outgoing NAT
$natcmd 200 config log same_ports ip $wanip

# Configure incoming 1:1 NAT
$natcmd 10 config log same_ports ip $wanip redirect_addr $vpnhosts $wanip

# Allow incoming connections
$cmd 15 nat 10 log ip from not $pm to $wanip 53,80,443 in recv $wan
$cmd 17 allow tcp from any to $wanip 22 in recv $wan $sks
$cmd 20 allow log tcp from any to $wanip 1194 in recv $wan $sks
$cmd 23 allow tcp from any to $wannet 3128 in recv $wan $sks
$cmd 25 allow log ip from any to $vpnnet 53,80,443 out xmit $vpn

$cmd 30 allow log ip from $vpnnet 53,80,443 to any in recv $vpn established
$cmd 35 nat 10 log ip from $vpnnet 53,80,443 to any out xmit $wan established

# Allow outgoing connections
$cmd 40 allow log ip from $vpnnet to any in recv $vpn
$cmd 45 nat 200 log ip from $vpnnet to any out xmit $wan

$cmd 46 nat 200 log ip from any to $wanip in recv $wan
$cmd 47 allow log ip from any to $vpnnet out xmit $vpn
$cmd 49 check-state log

## LAN rules

## inbound

# globo4
$cmd 50 allow tcp from [redacted]/32 to me 5666 in via $wan $ks
$cmd 52 allow icmp from [redacted]/32 to me in via $wan $ks

## outbound

### WAN rules

## deny

# IPv4 bogons
$cmd 90 deny all from 0.0.0.0/8 to any via $wan
#$cmd 92 deny all from 10.0.0.0/8 to any via $wan
#$cmd 94 deny all from 127.0.0.0/8 to any via $wan
$cmd 96 deny all from 169.254.0.0/16 to any via $wan
$cmd 98 deny all from 172.16.0.0/12 to any via $wan
$cmd 100 deny all from 192.0.2.0/24 to any via $wan
#$cmd 102 deny all from 192.168.0.0/16 to any via $wan
$cmd 104 deny all from 204.152.64.0/23 to any via $wan
$cmd 106 deny all from 224.0.0.0/3 to any via $wan

### allow

## inbound

# FTP client

$cmd 120 allow tcp from any 20-21 to $wanip 25000-49151 in via $wan $sks

# SSH
$cmd 130 allow tcp from any to $wanip 22 in via $wan $sks

# OpenVPN
$cmd 135 allow log tcp from any to $wanip 1194 in via $wan $sks

# HTTPS
$cmd 140 allow tcp from $pm to $wannet 3128 in via $wan $sks

## outbound

# FTP, SSH (CVS), SMTP, DNS (lookups), HTTP, NTP, HTTPS (downloads), submission, CVS
$cmd 150 allow all from me to any 20,21,22,25,53,80,123,443,587,2401 out via $wan $ks

# deny everything else (default block)
$cmd 190 deny ip from any to any via $other
$cmd 200 deny log ip from any to any

Everything is working apart from the traffic from the outside world coming back in the WAN interface and hitting the NAT (VPN clients). That means:

  • All the services on the WAN interface (SSH, OpenVPN, squid, NRPE) work
  • Clients can connect to the squid service on the WAN interface and the traffic transits it fine
  • For the OpenVPN clients, they can connect to the server, get an address, and I can ping them from the server
  • The traffic sent from the OpenVPN-connected clients I can see in the security log is making it through to the WAN interface:
Code: 
Jul 12 23:25:57 [server] kernel: ipfw: 20 Accept TCP [wanip]:1194 [my_wanip]:41409 out via vmx0
Jul 12 23:25:57 [server] kernel: ipfw: 20 Accept TCP [my_wanip]:41409 [wanip]:1194 in via vmx0
Jul 12 23:25:57 [server] kernel: ipfw: 45 Nat UDP 10.8.0.6:58328 8.8.4.4:53 out via vmx0
Jul 12 23:25:57 [server] kernel: ipfw: 49 UNKNOWN UDP [wanip]:58328 8.8.4.4:53 out via vmx0
Jul 12 23:25:57 [server] kernel: ipfw: 20 Accept TCP [wanip]:1194 [my_wanip]:41409 out via vmx0

That's after the connection has been established to the OpenVPN service and viewing http://FreeBSD.org/ in the web browser on the client. It's trying to hit DNS and that packet is making it to the WAN interface. I can see from tcpdump on the WAN interface that the DNS server is responding and sending that packet to the WAN interface of the server.

For packet statistics on the tunnel interface, we can see while the input packets increase, the output packets remain unchanged:

netstat -I tun0
Code: 
Name  Mtu Network  Address  Ipkts Ierrs Idrop  Opkts Oerrs  Coll
tun0  1500 <Link#5>  9807  0  0  182  0  0
tun0  - fe80::250:56f fe80::250:56ff:fe  0  -  -  2  -  -
tun0  - 10.8.0.1/32  10.8.0.1  0  -  -  0  -  -

(That 182 number doesn't increase as one would think it would.) It's like there's a loop and I'm not sure where it is. Can anyone help with this? I feel that a solution is close. Also, much of the ruleset is crappy, I know.
 
You must log in or register to reply here.
Back
Top