OpenVPN IPFW NAT FreeBSD

jasonhirsh

Active Member

Reaction score: 1
Messages: 101

I am back with old question new server. I realize IPFW is kind of old and semi useless but so am I.

I am trying, once again, to get the ability to use OpenVPN to ssh to the server and also redirect the desktop internet through the OpenVPN to the internet.

Results of ifconfig:

Code:
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
	ether 00:13:8f:e5:e4:15
	inet 209.160.65.133 netmask 0xfffff800 broadcast 209.160.71.255
	inet 209.160.68.112 netmask 0xffffffff broadcast 209.160.68.112
	media: Ethernet autoselect (10baseT/UTP <full-duplex>)
	status: active
rl0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 00:18:e7:08:27:dd
	media: Ethernet autoselect
	status: no carrier
ipfw0: flags=8801<UP,SIMPLEX,MULTICAST> metric 0 mtu 65536
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 
	inet6 ::1 prefixlen 128 
	inet 127.0.0.1 netmask 0xff000000 
	nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	inet 10.8.0.1 --> 10.8.0.2 netmask 0xffffffff 
	Opened by PID 4345
tap0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=80000<LINKSTATE>
	ether 00:bd:ae:f1:11:00
My server.conf for OpenVPN is:

Code:
user root
port 1194
proto udp
dev tun
push "redirect-gateway def1 bypass-dhcp"
ca /usr/local/etc/openvpn/keys/ca.crt
cert /usr/local/etc/openvpn/keys/server.crt
key /usr/local/etc/openvpn/keys/server.key
dh /usr/local/etc/openvpn/keys/dh1024.pem
server  10.8.0.0 255.255.255.0
route 10.8.0.2 255.255.255.0
ifconfig-pool-persist ipp.txt
keepalive 10 120
tls-auth /usr/local/etc/openvpn/keys/ta.key 0 # This file is secret
cipher BF-CBC        # Blowfish (default)
comp-lzo
user nobody
group nobody
persist-key
persist-tun
status /var/log/openvpn.log
verb 9
mute 10
My rc.conf is:

Code:
 defaultrouter="209.160.64.1"
  hostname="tuna.theoceanwindow-bv.com"
  ifconfig_re0="inet 209.160.65.133  netmask 0xfffff800"
  ifconfig_re0_alias="inet 209.160.68.112 netmask 0xffffffff"
  linux_enable="YES"
  firewall_enable="YES"
  firewall_script="/etc/ipfw.rules"
  firewall_logging="YES"
  sshd_enable="YES"
  webmin_enable="YES"
  mysql_enable="YES"
  apache22_enable="YES"
  named_enable="YES"
  gateway_enable="YES"
  openvpn_if="tap"
  openvpn_enable="YES"
  openvpn_configfile="/usr/local/etc/openvpn/server.conf"
  clamsmtpd_enable="YES"
  clamav_clamd_enable="YES"
  clamav_freshclam_enable="YES"
  clamav_milter_enable="YES"
  dovecot_enable="YES"
  ntpd_enable="YES"
  inetd_enable="YES"
  amavisd_enable="YES"
  natd_enable="YES"
  natd_interface="re0"
  winbindd_enable="YES"
  #postgrey_enable="YES"
  #postgrey_pidfile="/var/run/postgrey.pid"
  #postgrey_flags="--pidfile=${postgrey_pidfile} --inet=127.0.0.1:6000 -d
  samba_enable="YES"
  proftpd_enable="YES"
  #ftpd_enable="YES"
  squid_enable="YES"
  sshd_enable="YES"
I am trying to add IPFW rules by hand based on an earlier suggestion in this forum from @varda but when I try either
ipfw nat 1 config if re0
or
ipfw nat 1 config ip 209.160.68.112
or
ipfw nat 1 config ip 209.160.65.133
I get
Code:
ipfw: setsockopt(IP_FW_NAT_CFG): Invalid argument
I have Googled and can't find much which isn't in Russian. Any idea how I messed up this time?
 
Last edited by a moderator:
A

Anonymous

Guest


jasonhirsh said:
Code:
ipfw: setsockopt(IP_FW_NAT_CFG): Invalid argument
I have googled and can't find much which isn't in russian
Any idea how i messed up THIS time??
Call #kldload ipfw_nat.ko before using any in-kernel NAT rules.
 
OP
OP
J

jasonhirsh

Active Member

Reaction score: 1
Messages: 101

Thanks! 50% there, error code now gone I am able to manually input the following with no errors:
Code:
ipfw nat 1 config if re0
ipfw nat 1 ip from 10.8.0.0/8 to any out via re0E
ipfw nat 1 ip from any to any in via re0
No errors but still no tunnel to the internet. Any help for the rest of the trip?

Code:
nat 1 ip from any to any in via re0
 
A

Anonymous

Guest


jasonhirsh said:
...any help for the rest of the trip??
Perhaps not, since I have no experience with OpenVPN. I installed an L2TP/IPsec dial-in VPN server using a combination of security/ipsec-tools+net/mpd5.

Initially, I had also the problem, that client VPN connections were restricted to endpoints at the server only, namely those connections were neither passed into the internet, nor could access anything else on the internal network. I needed to enable proxy-arp in the mpd5 configuration, and then it worked.

Perhaps OpenVPN offers a similar configuration option. See also: http://en.wikipedia.org/wiki/Proxy_ARP. Here comes what the mpd manual tells about this setting:

Code:
proxy-arp
   When this option is enabled, if after link negotiation the peer’s IP address
   is determined to lie on a local subnet, then mpd will arrange for the local
   machine to install a proxy ARP entry for the remote machine’s IP address.
   For example, suppose the local machine lies on a LAN with address
   192.168.1.10, and another machine will be connecting via mpd and using
   the LAN address 192.168.1.20. Then these commands would set up
   proxy ARP for the remote machine:
          set iface enable proxy-arp
          set ipcp ranges 192.168.1.10/32 192.168.1.20/32
   The default is disable.
If OpenVPN does not offer the Proxy ARP technique, then you need to fall back to routing table customizations, unfortunately, I cannot be of any help with that.
 
OP
OP
J

jasonhirsh

Active Member

Reaction score: 1
Messages: 101

Well that might be a solution if all else fails. OpenVPN has been a several year battle. My Android POS doesn't seem to like L2TP/IPse.
 

throAU

Aspiring Daemon

Reaction score: 147
Messages: 910

Another alternative to proxy-arp is to give your VPN connections their own subnet. E.g., if your LAN is 10.1.1.0/24, then give your VPNs 10.1.2.0/24, and ensure your gateway/router (if it is not your FreeBSD machine) has either a static route for that network pointing at your FreeBSD machine, learns it via a routing protocol or whatever.

You'd also need to ensure your FreeBSD machine is routing packets, by enabling the relevant configuration option (which currently escapes me).

I "just don't like" proxy-arp myself :). It also enables me to do specific firewall rules for remote VPN machines if desired.
 
OP
OP
J

jasonhirsh

Active Member

Reaction score: 1
Messages: 101

I am still playing with the IPFW. I know it's old and klunky but I have at least an idiot's understanding. My Freebsd FreeBSD box is a leased server that I am trying to use as a gateway for the internet. The OpenVPN configuration builds a network 10.8.0.0 with gateway at 10.8.0.5. Not quite sure how I route from the OpenVPN subnnet to the NAT.
 
OP
OP
J

jasonhirsh

Active Member

Reaction score: 1
Messages: 101

Ok, maybe it's time for back to basics: let me throw a couple of questions out
  1. In most of the discussions I see a reference to an external and internal IP when doing the setup. The server I am working on is essentially a standalone machine with no intranet. I would assume that prior to the OpenVPN or NAT it had for all intents and purposes no internal IP?
  2. If 1 is true then when I configure OpenVPN and it gives the server an IP, that subnet also becomes the server's intranet and since the server is also the gateway, its IP is also the gateway IP?
  3. Looking at the netstat -nr

    Code:
    Internet:
    Destination        Gateway            Flags    Refs      Use  Netif Expire
    default            209.160.64.1       UGS         8    64994    re0
    10.8.0.0/24        10.8.0.2           UGS         0        0   tun0
    10.8.0.1           link#5             UHS         0        0    lo0
    10.8.0.2           link#5             UH          0        0   tun0
    127.0.0.1          link#4             UH          0     2413    lo0
    209.160.64.0/21    link#1             U           0        0    re0
    209.160.65.133     link#1             UHS         0      200    lo0
    209.160.68.112     link#1             UHS         0        0    lo0 =>
    209.160.68.112/32  link#1             U           0        0    re0
    
    Internet6:
    Destination                       Gateway                       Flags      Netif Expire
    ::1                               ::1                           UH          lo0
    fe80::%lo0/64                     link#4                        U           lo0
    fe80::1%lo0                       link#4                        UHS         lo0
    ff01:4::/32                       fe80::1%lo0                   U           lo0
    ff02::%lo0/32                     fe80::1%lo0                   U           lo0
I don't really see a route for my VPN subnet 10.8.0.0/24 to get in or out. At first I thought that I needed to add a route. Reading the manual for NAT I seem to now come to the conclusion that the natd is properly configured and should establish the route. Is that correct?
 

kpa

Beastie's Twin

Reaction score: 1,796
Messages: 6,307

The routing table does have routes for 10.8.0.0/24, the subnet is directly connected and those entries are enough. Remember that the routing table is always from the point of view of a host sending traffic to other hosts, your VPN server knows how to reach hosts in the 10.8.0.0/24 subnet but it can not have any say in how other hosts connected to that subnet connect to it and what is in their routing table. What you need to tell to the clients connecting to your OpenVPN servers is that they should redirect all their traffic to the VPN tunnel and use the tunnel ip address 10.8.0.1 (may be the address 10.8.0.2 as well) as their default gateway. This is done with "push options" in the configuration like this:

Code:
push "redirect-gateway def1"
That should all that is needed if you have a working NAT for the 10.8.0.0/24 subnet.
 
OP
OP
J

jasonhirsh

Active Member

Reaction score: 1
Messages: 101

Thank you for your response. I realize my thread wanders. The issue is not getting the traffic to the OpenVPN server but getting it from there to the internet and back again. When I fire up the client all the internet goes away. I thought it might be DNS so I also added the push-dns.

I have been led to believe that with a single NIC a NAT is necessary which I am attempting with IPFW. I appear to have the NAT up, not sure if OpenVPN is talking to it or if NAT is talking to anyone. I am now exploring IPFW NAT divert as a solution.

As you can see from my server.conf, I am pushing the redirect-gatway.
Code:
local x.x.x.x
user root
port 1194
proto udp
dev tun
push "redirect-gateway def1"
push "dhcp-option DNS 8.8.8.8"
ca /usr/local/etc/openvpn/keys/ca.crt
cert /usr/local/etc/openvpn/keys/server.crt
key /usr/local/etc/openvpn/keys/server.key
dh /usr/local/etc/openvpn/keys/dh1024.pem    
server  10.8.0.0 255.255.255.0               
route 10.8.0.2 255.255.255.0                 
ifconfig-pool-persist ipp.txt                
keepalive 10 120                             
tls-auth /usr/local/etc/openvpn/keys/ta.key 0 # This file is secret
cipher BF-CBC        # Blowfish (default)    
comp-lzo                                     
#ping 10                                     
#max-clients 10                              
user nobody                                  
group nobody                                                          
persist-key
persist-tun
status /var/log/openvpn.log
verb 9
My netstat -nr with open client attached is as follows:

Code:
Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            209.160.64.1       UGS        19   414969    re0
10.8.0.0/24        10.8.0.2           UGS         0        0   tun0
10.8.0.1           link#5             UHS         0        0    lo0
10.8.0.2           link#5             UH          0        0   tun0
127.0.0.1          link#4             UH          0    48273    lo0
209.160.64.0/21    link#1             U           0        0    re0
209.160.65.133     link#1             UHS         0     1965    lo0
209.160.68.112     link#1             UHS         0        0    lo0 =>
209.160.68.112/32  link#1             U           0        0    re0
Now what is really interesting is when I open the VPN client I lose all internet connectivity but my SSH continues.

Again thanks for the input.
 

kpa

Beastie's Twin

Reaction score: 1,796
Messages: 6,307

Is the client using the pull option? Post the routing table from the client, the routing table of the server will not change when when the client connects.
 
OP
OP
J

jasonhirsh

Active Member

Reaction score: 1
Messages: 101

I added pull to the client with no success. The client.conf is
Code:
client
dev tun
pull
proto udp
remote x.x.x.x 1194
resolv-retry infinite
nobind
user nobody
group nobody
persist-key
persist-tun
ca ./keys/tuna/ca.crt
cert ./keys/tuna/megacore.crt
key ./keys/tuna/megacore.key
ns-cert-type server
tls-auth ./keys/tuna/ta.key 1
cipher BF-CBC
comp-lzo
verb 3
mute 5
netstat -nr from client with OpenVPN active:

Code:
Internet:
Destination        Gateway            Flags        Refs      Use   Netif Expire
0/1                10.8.0.5           UGSc            2        0    tun0
default            10.0.1.1           UGSc           17       19     en0
default            10.0.1.1           UGScI           1        0     en2
10.0.1/24          link#4             UCS             9        0     en0
10.0.1/24          link#6             UCSI            4        0     en2
10.0.1.1           1c:7e:e5:35:5b:7e  UHLWI           1        0     en0   1200
10.0.1.1           1c:7e:e5:35:5b:7e  UHLWI           2      147     en2   1182
10.0.1.50          0:14:51:72:bb:fc   UHLWI           0        0     en2   1184
10.0.1.100         127.0.0.1          UHS             1    28843     lo0
10.0.1.108         e0:b9:ba:7:79:85   UHLWI           0       11     en0    820
10.0.1.109         127.0.0.1          UHS             1    49980     lo0
10.0.1.110         90:84:d:f4:e3:5e   UHLWI           0       11     en0   1041
10.0.1.150         d4:9a:20:fd:8f:c0  UHLWI           3     3557     en0   1184
10.0.1.150         d4:9a:20:fd:8f:c0  UHLWI           0     5859     en2   1152
10.0.1.255         ff:ff:ff:ff:ff:ff  UHLWbI          0       11     en0
10.0.1.255         link#6             UHLWbI          1       74     en2
10.8.0.1/32        10.8.0.5           UGSc            0        0    tun0
10.8.0.5           10.8.0.6           UH              7        0    tun0
10.37.129/24       link#9             UC              4        0   vnic1
10.37.129.2        0:1c:42:0:0:9      UHLWI           0        1     lo0
10.37.129.255      ff:ff:ff:ff:ff:ff  UHLWbI          2        5   vnic1
10.211.55/24       link#8             UC              4        0   vnic0
10.211.55.2        0:1c:42:0:0:8      UHLWI           0        1     lo0
10.211.55.255      ff:ff:ff:ff:ff:ff  UHLWbI          0        5   vnic0
127                127.0.0.1          UCS             0        0     lo0
127.0.0.1          127.0.0.1          UH             12  1343320     lo0
128.0/1            10.8.0.5           UGSc            2        0    tun0
169.254            link#4             UCS             0        0     en0
209.160.65.133/32  10.0.1.1           UGSc            3        0     en0
The server address in OpenVPN is 10.8.0.1 of interest. I can not contact the client from the server after I SSH into the server.

hmmm
 
A

Anonymous

Guest


jasonhirsh said:
...I have been led top believe that with a single nic that a NAT is necessary which I am attempting with IPFW...
I am leaving save ground now, so please bear with me if the following suggestion turns out not to be viable. Anyway, here it comes. According to the output of ifconfig in your initial post, you have an unused/unconnected NIC sitting in the machine.

Code:
rl0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=8<VLAN_MTU>
	ether 00:18:e7:08:27:dd
	media: Ethernet autoselect
	status: no carrier
Using ifconfig, you can perhaps make this into a recognized NIC, even if it does not connect to anywhere.

#ifconfig rl0 inet 10.8.0.1 netmask 255.255.255.0

It may happen that the system refuses this NIC because its status will continue to be "no carrier". In this case, perhaps others know how to force it going active (without plugging something in). If this would work out, then you could proceed with a standard 2-NIC-NAT setup.
 
OP
OP
J

jasonhirsh

Active Member

Reaction score: 1
Messages: 101

I have already run up a nice bill having my host correct my server, I do appreciate the suggestion.

I tried ifconfig rl0 inet 10.8.0.1 netmask 255.255.255.0 and got
Code:
ifconfig: ioctl (SIOCAIFADDR): File exists
 
A

Anonymous

Guest


Another thing to try:

# sysctl net.inet.ip.fw.one_pass=0
# sysctl net.inet.tcp.tso=0
 
OP
OP
J

jasonhirsh

Active Member

Reaction score: 1
Messages: 101

Sort of a good news bad news. The server didn't hate it, but it didn;t fix it either.
 

kpa

Beastie's Twin

Reaction score: 1,796
Messages: 6,307

Can you summarise what works and what does not. Can you for example ssh from the client to the server using the 10.8.0.1 address? Can you query the vpn server from the client with dig(1) like this:
dig @10.8.0.1 http://www.google.com

@@rolfheinrich, with OpenVPN there's no need to have the address on any physical adapter, the address bound to the TUN/TAP adapter is enough even if the server has only one physical NIC in use.
 
Last edited by a moderator:
OP
OP
J

jasonhirsh

Active Member

Reaction score: 1
Messages: 101

When VPN is up I can:

  1. ssh to the server 10.8.0.1 address
  2. My ssh session to the server which was stable before remains
  3. I can sort of pull up my servers web page when I use IP/index.html but js doesn't work
  4. The first time I tried to ssh from the server back through the VPN I got a message about securty credentials; I accepted and it died
I can not
  1. ssh to the server using the host name
  2. dig @10.8.0.1 http://www.google.com does nothing
  3. I can not bring up any other web sites (so I don't think there is a DNS problem)
  4. My mail clients no longer work
Not sure what else to test. Now what interests/confuses me is this part of the netstat -nr:

Code:
10.8.0.1/32        10.8.0.5           UGSc            1        0    tun0
10.8.0.5           10.8.0.6           UH             16        4    tun0
Isn't this saying that 10.8.0.1 is connected to 10.8.0.5 and that it in turn is connected to 10.8.0.6?

10.8.0.6 is the client

the log of the vpn handshake showed the following

Code:
2013-05-08 12:43:27 ROUTE default_gateway=10.0.1.1
2013-05-08 12:43:27 TUN/TAP device /dev/tun0 opened
2013-05-08 12:43:27 MANAGEMENT: >STATE:1368031407,ASSIGN_IP,,10.8.0.6,
2013-05-08 12:43:27 /sbin/ifconfig tun0 delete
2013-05-08 12:43:27 NOTE: Tried to delete pre-existing tun/tap instance -- No Problem if failure
2013-05-08 12:43:27 /sbin/ifconfig tun0 10.8.0.6 10.8.0.5 mtu 1500 netmask 255.255.255.255 up
2013-05-08 12:43:27 /sbin/route add -net 209.160.65.133 10.0.1.1 255.255.255.255
                                        route: writing to routing socket: File exists
                                        add net 209.160.65.133: gateway 10.0.1.1: File exists
2013-05-08 12:43:27 /sbin/route add -net 0.0.0.0 10.8.0.5 128.0.0.0
                                        add net 0.0.0.0: gateway 10.8.0.5
2013-05-08 12:43:27 /sbin/route add -net 128.0.0.0 10.8.0.5 128.0.0.0
                                        add net 128.0.0.0: gateway 10.8.0.5
2013-05-08 12:43:27 MANAGEMENT: >STATE:1368031407,ADD_ROUTES,,,
2013-05-08 12:43:27 /sbin/route add -net 10.8.0.1 10.8.0.5 255.255.255.255
                                        add net 10.8.0.1: gateway 10.8.0.5
2013-05-08 12:43:27 GID set to nobody
2013-05-08 12:43:27 UID set to nobody
2013-05-08 12:43:27 Initialization Sequence Completed
do i have a phantom gateway in 10.8.0.5?
 

kpa

Beastie's Twin

Reaction score: 1,796
Messages: 6,307

The routing table entries are normal for a tun(4) device set up for OpenVPN, they may look confusing but it's the nature of the tun(4) device that makes them appear like that. If you were using a tap(4) device instead you would see much more familiar route set up with only one gateway address.

I only just noticed that you're pushing the google DNS 8.8.8.8 to the client so ignore what I said about DNS.

This line looks very odd to me, what kind network set up you have on the client?

Code:
2013-05-08 12:43:27 /sbin/route add -net 209.160.65.133 10.0.1.1 255.255.255.255
                                        route: writing to routing socket: File exists
                                        add net 209.160.65.133: gateway 10.0.1.1: File exists
This is where the client makes sure that the route to the local default gateway is preserved regardless of the options pushed by the OpenVPN server but here it's assigning a gateway 10.0.1.1 for address 209.160.65.133/32 which looks pretty bizarre to me.
 
OP
OP
J

jasonhirsh

Active Member

Reaction score: 1
Messages: 101

That is the IP of the server I am connecting the VPN to (ie. remote x.x.x.x).
 
OP
OP
J

jasonhirsh

Active Member

Reaction score: 1
Messages: 101

OK, correction to what works and doesn't work through VPN. I can access the web pages hosted on the server by hostname not just IP (does have some issues with js; go figure) and the mail server located on that sever and all its accounts work. No off-server accounts. It keeps seeming like I get there and die.

This is true with IPFW/NAT running and not running.
 

kpa

Beastie's Twin

Reaction score: 1,796
Messages: 6,307

I guess I remembered wrong how redirect-gateway def1 works, the route is correct in that case. No idea why it says "File exists" though.

I have to say all this would be so much easier to debug if you were using pf(4) for NAT. It would be basically a rule this simple:

Code:
nat on $ext_if inet from 10.8.0.0/24 to any -> $ext_if
 
OP
OP
J

jasonhirsh

Active Member

Reaction score: 1
Messages: 101

I have been sticking with IPFW because
  • I have a semi-working version (though with my current issues it may be less working then I thought it was)
  • I have a working knowledge of it
  • the syntax in PF has me kind of intimidated
I would assume the rule you stated would allow any traffic on the interface assigned to the variable from that subnet to go and where and return to that interface?
 

kpa

Beastie's Twin

Reaction score: 1,796
Messages: 6,307

That's just the NAT part, you would have to have rules for access as well. Most simple case would be just

Code:
pass all
 
OP
OP
J

jasonhirsh

Active Member

Reaction score: 1
Messages: 101

So I did translate that right.

I have seen a preference on here for PF over IPFW. I had IPFW compiled in my kernel. From what I gather in the documentation I do not need to go to that trouble for PF, although I will lose some advanced features. It seems like I just have to change rc.conf. I am getting no where with OpenVPN/NAT/IPFW so let me step back and try to learn the rudiments of PF and go with a OpenVPN/NAT/IP approach.
 
Top