Solved Routing issue with IPSEC (Windows works, Linux doesn't)

Hello, I've had this problem since at least 9.0 but it angered me enough now so I come here asking for help.

I have an IPSec network net to net.
other side 10.11.12.0/24

FreeBSD/strongswan 192.168.1.1/24
on 192.168.1.1 there is one windows computer and one Linux and some androids.

If I just fire up the tunnel and don't add any routes on the FreeBSD machine, both Windows and Linux/Android can ping 10.11.12.0/24. But the FreeBSD machine itself can't. So, I add a route like this route add -net 10.11.12.0/24 192.168.1.1.
Fine, now FreeBSD can ping 10.11.12.0/24 just fine. BUT Linux/Android now can't anymore, Windows still works fine though.

If I do a traceroute on Linux to 10.11.12.13 it shows the first gateway being 192.168.1.106 (which is its own IP). If I tcpdump(8) on FreeBSD I see a lot of those which looks fishy...
Code:
15:58:16.553497 ARP, Request who-has 10.11.12.13 tell 192.168.1.106, length 46
15:58:17.556957 ARP, Request who-has 10.11.12.13 tell 192.168.1.106, length 46
15:58:18.560170 ARP, Request who-has 10.11.12.13 tell 192.168.1.106, length 46
15:58:19.563379 ARP, Request who-has 10.11.12.13 tell 192.168.1.106, length 46
Strongswan shouldn't be blamed here, I've used Racoon for at least 2 years with this problem.

Best regards
Martin
 
192.168.1.1 and 10.11.12.13. you don't mean external right? I don't see that matters :)

Scenario 1:

Code:
root@fjuttsi:~ # netstat -r
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            ua-85-230-56-1.cus UGS         re0
85.230.56.0/22     link#2             U           re0
c-d53be655.015-26- link#2             UHS         lo0
localhost          link#3             UH          lo0
192.168.1.0        link#1             U           em0
192.168.1.1        link#1             UHS         lo0

All computers behind the FreeBSD router can ping 10.11.12.13, but not FreeBSD itself.

Scenario 2:

Code:
Internet:
Destination        Gateway            Flags      Netif Expire
default            ua-85-230-56-1.cus UGS         re0
10.11.12.0         192.168.1.1        UGS         em0
85.230.56.0/22     link#2             U           re0
c-d53be655.015-26- link#2             UHS         lo0
localhost          link#3             UH          lo0
192.168.1.0        link#1             U           em0
192.168.1.1        link#1             UHS         lo0

FreeBSD can now ping, and Windows behind it, but not Linux/Android.
 
Last edited by a moderator:
In scenario 1 does it work if you specify the internal IP of the FreeBSD host as source address (e.g. ping -S 192.168.1.1 10.11.12.13, or -S 192.168.1.106 if that is also assigned to it)?

If so, you either have to tell FreeBSD to use this specific IP within 192.168.1.0/24 to send packets to 10.11.12.0/24 (otherwise the default route will cause it to send packets with it's external IP, which won't match the IPsec policy that's only between the two private subnets), or you configure a second IPsec tunnel to the remote host between the external IP and the 10.11.12.0/24 subnet.

As far as I know forcing a specific source address for a route is not that easy on FreeBSD (on Linux, on the other hand, every route can have a source address assigned). strongSwan, for instance, uses a TUN device if virtual IPs are assigned by the server to force a specific source address. Something like that might work here too (i.e. set up a TUN device with an IP from 192.168.1.0/24 and install a route to 10.11.12.0/24 via that interface).

The problem with the second option is, of course, that the return path could be trickier. If the remote VPN server is not the default gateway on 10.11.12.0/24 you'll have to take some additional measures so hosts in the remote subnet send response packets addressed to the FreeBSD host's external IP to the VPN gateway instead of their default gateway.
 
I have (almost) the exact problem as Marantz.

I have a FreeBSD 10.1 box doing different stuff like nat, wireless ap, samba and also, relevant for this issue, strongswan 5.3 for a IPSEC site-to-site tunnel.

My FreeBSD box has an internal ip 192.168.254.1 and the net behind is 192.168.254.0/24. On my network I have one windows-box and one linux-box. Both these boxes on my LAN have the FreeBSD-server (192.168.254.1) as a default gateway.

On the other end of the IPSEC-tunnel there is a 192.168.250.0/24 net, and a host with ip 192.168.250.1, which I try to ping.

Everything looks good in ipsec statusall and I can see that the tunnel is established between 192.168.254.0/24 -- 192.168.250.0/24.

From my windows-box everything works perfect and I can ping 192.168.250.1 without problems. The problem is that the FreeBSD-server and the linux-box can't ping 192.168.250.1. The FreeBSD server can ping through the IPSEC-tunnel to 192.168.250.1 if I use ping -S 192.168.254.1 192.168.250.1.

So on the FreeBSD-server there is a problem with routing. When strongswan starts it inserts a route to 192.168.250.0/24 (like expected) but with my external ip as gateway.

Code:
netstat -r
Routing tables

Internet:
Destination        Gateway            Flags      Netif Expire
default            external-gw UGS      vlan10
192.168.254.0         link#7             U       bridge0
nuc                link#7             UHS         lo0
192.168.250.0         external-gw US       vlan10
external-net       link#5             U        vlan10
external-ip link#5             UHS         lo0
localhost          link#4             UH          lo0

Just for clarity I also post my /usr/local/etc/ipsec.conf

Code:
cat /usr/local/etc/ipsec.conf
conn tunnel
        keyexchange=ikev1
        esp=sha256-aes256-modp2048
        ike=sha256-aes256-modp2048
        leftid=yyy.se
        fragmentation=yes
        dpdaction=restart
        closeaction=restart
        leftsendcert=always
        leftsubnet=192.168.254.0/24
        leftcert=yyy.crt
        right=xxx.xxx.xxx.xxx
        rightid=xxx.se
        rightcert=xxx.crt
        rightsubnet=192.168.250.0/24
        auto=start

Obviusly the route inserted by strongswan doesn't work, by removing strongswan's routing entry and inserting my own route ( route add -net 192.168.250.0/24 192.168.254.1) everything works as expected. With a correct routing entry in the routing table the FreeBSD-server now can reach 192.168.250.1.

With FreeBSD set up properly, the windows-box is still able to reach 192.168.250.1.

On the FreeBSD-server I can see the decrypted FreeBSD-server and windows-box traffic destined for 192.168.250.1 flowing over enc0 in tcpdump and the cryptated traffic flowing over the external interface, and everything seems to be working like a charm.

The only problem is that the Linux box still can't ping 192.168.250.1, and it doesn't seem to matter whether I have a routing entry for the 192.168.250.0/24 net or not on the FreeBSD-server (it seems to be working for Marantz as long as he doesn't have a route in the routing table on his FreeBSD-box).

No matter how I try can't get linux-box traffic destined for 192.168.250.1 to use the ipsec-tunnel.

As with Marantz I see strange arp traffic on the FreeBSD-server from the Linux-box (with ip 192.168.254.2) when trying to ping 192.168.250.1. It seems like the Linux-box thinks that 192.168.250.1 is on the same LAN or something like that and tries to find an arp-entry for it.

Code:
tcpdump -i internalinterface
08:58:13.186012 ARP, Request who-has 192.168.250.1 tell 192.168.254.2, length 46
08:58:14.188799 ARP, Request who-has 192.168.250.1 tell 192.168.254.2, length 46

What to make of this?

I got so desperate that I tried setting up strongswan and the IPSEC-tunnel on a linux gateway instead (with the exact same strongswan configuration). With strongswan on a Linux gateway, the linux-box on the network (192.168.254.2) actually can ping the remote net and 192.168.250.1.

Am I doomed to Ubuntu or does someone have any ideas on how to solve this? :)
 
Ok, now I removed the 192.168.250.0/24 routing entry on the FreeBSD-server and rebooted the linux-box on the lan. After the reboot of the linux-box, traffic from the linux-box destined to 192.168.254.1 actually gets routed by the FreeBSD-server through the IPSEC-tunnel.

As soon as I add the route to the 192.168.250.0/24-net on the FreeBSD-server again, traffic from the linux-box is not routed through the IPSEC-tunnel, and the linux-box starts sending these strange arp requests.

My problem is therefor exactly the same as Marantz describes.
 
I can't get it to work with tun interfaces either. Here is a tcpdump(1), notice that one ping gets through and works. This tcpdump is on LAN interface on FreeBSD (192.168.1.1) when pinging from Linux (192.168.1.106).

Code:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on em0, link-type EN10MB (Ethernet), capture size 65535 bytes
12:31:13.001737 IP 192.168.1.6.netbios-ns > 10.10.10.10.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; UNICAST
12:31:13.244105 IP 192.168.5.1.52480 > 255.255.255.255.7417: UDP, length 173
12:31:14.530665 IP 192.168.1.6.netbios-ns > 192.168.1.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
12:31:14.881600 IP6 fe80::e10c:e865:3807:825a.54498 > ff02::c.1900: UDP, length 146
12:31:15.295004 IP 192.168.1.6.netbios-ns > 192.168.1.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
12:31:16.059340 IP 192.168.1.6.netbios-ns > 192.168.1.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
12:31:16.106316 ARP, Request who-has 192.168.1.12 tell 192.168.1.6, length 46
12:31:16.304210 IP 192.168.5.1.52480 > 255.255.255.255.7417: UDP, length 173
12:31:16.823680 ARP, Request who-has 192.168.1.12 tell 192.168.1.6, length 46
12:31:17.837763 ARP, Request who-has 192.168.1.12 tell 192.168.1.6, length 46
12:31:17.885734 IP6 fe80::e10c:e865:3807:825a.54498 > ff02::c.1900: UDP, length 146
12:31:18.749895 IP 192.168.1.106.mdns > 224.0.0.251.mdns: 0*- [0q] 1/0/0 (Cache flush) TXT "deviceid=00:01:2E:27:05:93" "model=Xbmc,1" "srcvers=101.28" "features=0x20F7" (130)
12:31:19.364190 IP 192.168.5.1.52480 > 255.255.255.255.7417: UDP, length 173
12:31:19.874539 IP 192.168.1.106.mdns > 224.0.0.251.mdns: 0*- [0q] 1/0/0 (Cache flush) TXT "deviceid=00:01:2E:27:05:93" "model=Xbmc,1" "srcvers=101.28" "features=0x20F7" (130)
12:31:20.163884 IP 192.168.1.106 > 10.11.12.13: ICMP echo request, id 49929, seq 0, length 64
12:31:20.163930 IP 192.168.1.1 > 192.168.1.106: ICMP redirect 10.11.12.13 to host 10.11.12.13, length 36
12:31:20.170225 IP 10.11.12.13 > 192.168.1.106: ICMP echo reply, id 49929, seq 0, length 64
12:31:21.163971 ARP, Request who-has 10.11.12.13 tell 192.168.1.106, length 46
12:31:21.894829 IP6 fe80::e10c:e865:3807:825a.54498 > ff02::c.1900: UDP, length 146
12:31:22.001022 IP 192.168.1.106.mdns > 224.0.0.251.mdns: 0*- [0q] 1/0/0 (Cache flush) TXT "deviceid=00:01:2E:27:05:93" "model=Xbmc,1" "srcvers=101.28" "features=0x20F7" (130)
12:31:22.164808 ARP, Request who-has 10.11.12.13 tell 192.168.1.106, length 46
12:31:22.424296 IP 192.168.5.1.52480 > 255.255.255.255.7417: UDP, length 173
12:31:23.168145 ARP, Request who-has 10.11.12.13 tell 192.168.1.106, length 46

This is tcpdump when Windows (192.168.1.5) is pinging.

Code:
   192.168.1.5 > 10.11.12.13: ICMP echo request, id 1, seq 8, length 40
12:43:43.893545 IP (tos 0x0, ttl 64, id 30991, offset 0, flags [none], proto ICMP (1), length 56)
    192.168.1.1 > 192.168.1.5: ICMP redirect 10.11.12.13 to host 10.11.12.13, length 36
        IP (tos 0x0, ttl 128, id 23762, offset 0, flags [none], proto ICMP (1), length 60)
    192.168.1.5 > 10.11.12.13: ICMP echo request, id 1, seq 8, length 40
12:43:43.899803 IP (tos 0x0, ttl 254, id 46529, offset 0, flags [none], proto ICMP (1), length 60)
    10.11.12.13 > 192.168.1.5: ICMP echo reply, id 1, seq 8, length 40
12:43:44.355386 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.100 tell 192.168.1.5, length 46
12:43:44.355396 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.1.12 tell 192.168.1.5, length 46
12:43:44.895465 IP (tos 0x0, ttl 128, id 23899, offset 0, flags [none], proto ICMP (1), length 60)
    192.168.1.5 > 10.11.12.13: ICMP echo request, id 1, seq 9, length 40
12:43:44.895507 IP (tos 0x0, ttl 64, id 31003, offset 0, flags [none], proto ICMP (1), length 56)
    192.168.1.1 > 192.168.1.5: ICMP redirect 10.11.12.13 to host 10.11.12.13, length 36
        IP (tos 0x0, ttl 128, id 23899, offset 0, flags [none], proto ICMP (1), length 60)
    192.168.1.5 > 10.11.12.13: ICMP echo request, id 1, seq 9, length 40
12:43:44.901837 IP (tos 0x0, ttl 254, id 61590, offset 0, flags [none], proto ICMP (1), length 60)
    10.11.12.13 > 192.168.1.5: ICMP echo reply, id 1, seq 9, length 40
12:43:45.243902 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.13 tell 192.168.1.106, length 46
12:43:45.896552 IP (tos 0x0, ttl 128, id 24013, offset 0, flags [none], proto ICMP (1), length 60)
    192.168.1.5 > 10.11.12.13: ICMP echo request, id 1, seq 10, length 40
12:43:45.896596 IP (tos 0x0, ttl 64, id 31011, offset 0, flags [none], proto ICMP (1), length 56)
    192.168.1.1 > 192.168.1.5: ICMP redirect 10.11.12.13 to host 10.11.12.13, length 36
        IP (tos 0x0, ttl 128, id 24013, offset 0, flags [none], proto ICMP (1), length 60)
    192.168.1.5 > 10.11.12.13: ICMP echo request, id 1, seq 10, length 40
12:43:45.902980 IP (tos 0x0, ttl 254, id 16329, offset 0, flags [none], proto ICMP (1), length 60)
    10.11.12.13 > 192.168.1.5: ICMP echo reply, id 1, seq 10, length 40
12:43:46.244114 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.11.12.13 tell 192.168.1.106, length 46
12:43:46.898137 IP (tos 0x0, ttl 128, id 24145, offset 0, flags [none], proto ICMP (1), length 60)
    192.168.1.5 > 10.11.12.13: ICMP echo request, id 1, seq 11, length 40
12:43:46.898178 IP (tos 0x0, ttl 64, id 31022, offset 0, flags [none], proto ICMP (1), length 56)
    192.168.1.1 > 192.168.1.5: ICMP redirect 10.11.12.13 to host 10.11.12.13, length 36
        IP (tos 0x0, ttl 128, id 24145, offset 0, flags [none], proto ICMP (1), length 60)
    192.168.1.5 > 10.11.12.13: ICMP echo request, id 1, seq 11, length 40
12:43:46.904437 IP (tos 0x0, ttl 254, id 1947, offset 0, flags [none], proto ICMP (1), length 60)
    10.11.12.13 > 192.168.1.5: ICMP echo reply, id 1, seq 11, length 40
^C
 
This is solved by setting
Code:
sysctl net.inet.ip.redirect=0
Thanks to Andrey V. Elsukov who figured that out on freebsd-net mailing list.

His explanation,

I think it is because FreeBSD sends ICMP redirects when it finds that
route to the given destination goes back to the receiving interface.
 
Having come across this problem myself, but using the newer .conf configurations (https://wiki.strongswan.org/projects/strongswan/wiki/strongswanconf), I thought I'd post my solution for the routing from router host. The route_via_internal by default creates the routes to the LAN interface, which solves the same problem solved above by performing "route add -net 192.168.250.0/24 192.168.254.1". It doesn't create routes to the internal ip address, but it does assign the route to the internal interface, which is sufficient to get it picked up by the ipsec rerouting.

JSON:
     plugins {
         kernel-pfkey {
            route_via_internal = yes
         }
     }


Code:
$netstat -r
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            y.y.y.y            UGS        igb0
10.1.1.0/24        y.y.y.y            US         igb1 <----- due to route_via_internal
10.4.4.0/24        link#2             U          igb1
10.4.4.1           link#2             UHS         lo0

Here, 10.4.4.0 is the internal network, and 10.1.1.0 is a vpn endpoint. Interface igb0 is external and igb1 is internal, but normally the route to 10.1.1.0 above would point to igb0 (and pings from the router running strongswan would fail). This solves that entirely.
 
Back
Top