Solved Strongswan routing issue

korund

korund

Good day, gentlemen.

I have recently faced a problem when a device connected to a FreeBSD server via StrongSwan can't route outside its subnet. Let me explain in details:

FreeBSD 10.0-RELEASE, kernel compiled with (full config: http://pastebin.com/5PQFXqhx):
Code: 
options IPSEC
options IPSEC_NAT_T
device crypto
device cryptodev

Forwarding turned on in sysctl.conf:
Code: 
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1

We have two interfaces. The first one is 192.168.1.100, the router forwards ports 500 and 4500 from the Internet to this IP (so we can connect from anywhere to the "sweet home" network). The second one is 10.0.0.1 which is used for Samba and other services.

/etc/rc.conf:
Code: 
#--- Network
hostname="bsdserver"
ifconfig_igb0="inet 192.168.1.100 netmask 255.255.255.0"
ifconfig_igb1="inet 10.0.0.1 netmask 255.255.255.0"
defaultrouter="192.168.1.1"
gateway_enable="YES"
ipv6_gateway_enable="YES"

2014_05_01_185452.png


ipsec.conf:
Code: 
config setup
    strictcrlpolicy = no
ca bsdserver
    cacert = /usr/local/etc/ipsec.d/cacerts/bsdserver_CA.der
conn android
    auto = add
    dpdaction = clear
    keyexchange = ikev2
    esp=aes-aes256-sha-modp1024,aes256-sha512-modp4096
    ike=aes-aes256-sha-modp1024,aes256-sha512-modp4096
    mobike = yes
    left = 192.168.1.100
    leftsubnet = 0.0.0.0/0,::0/0
    leftauth = pubkey
    leftid = "111.111.111.111" # real "white" ip here
    leftcert = /usr/local/etc/ipsec.d/certs/111.111.111.111_cert.der
    leftfirewall = yes
    leftsourceip = %config4,%config6
    right = %any
    rightsourceip = 10.0.0.3/32
    rightauth = pubkey
    rightid = "C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com"
    rightcert = /usr/local/etc/ipsec.d/certs/android_cert.der

This is the output from ipsec start --nofork:
Code: 
Starting strongSwan 5.1.1 IPsec [starter]...
no netkey IPsec stack detected
no KLIPS IPsec stack detected
no known IPsec stack detected, ignoring!
00[DMN] Starting IKE charon daemon (strongSwan 5.1.1, FreeBSD 10.0-RELEASE, amd64)
00[KNL] unable to set UDP_ENCAP: Invalid argument
00[NET] enabling UDP decapsulation for IPv6 on port 4500 failed
00[CFG] loading ca certificates from '/usr/local/etc/ipsec.d/cacerts'
00[CFG]   loaded ca certificate "C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com" from '/usr/local/etc/ipsec.d/cacerts/UNIT_CA.der'
00[CFG] loading aa certificates from '/usr/local/etc/ipsec.d/aacerts'
00[CFG] loading ocsp signer certificates from '/usr/local/etc/ipsec.d/ocspcerts'
00[CFG] loading attribute certificates from '/usr/local/etc/ipsec.d/acerts'
00[CFG] loading crls from '/usr/local/etc/ipsec.d/crls'
00[CFG] loading secrets from '/usr/local/etc/ipsec.secrets'
00[CFG]   loaded RSA private key from '/usr/local/etc/ipsec.d/private/111.111.111.111_key.der'
00[LIB] loaded plugins: charon test-vectors curl aes des blowfish rc2 sha1 sha2 md4 md5 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp xcbc cmac hmac attr kernel-pfkey kernel-pfroute resolve socket-default stroke updown eap-identity eap-aka eap-aka-3gpp2 eap-md5 eap-mschapv2 eap-dynamic eap-tls eap-ttls eap-peap xauth-generic xauth-eap whitelist addrblock
00[LIB] unable to load 4 plugin features (4 due to unmet dependencies)
00[JOB] spawning 16 worker threads
charon (49667) started after 220 ms
16[CFG] received stroke: add connection 'android'
16[CFG] adding virtXXl IP address pool 10.0.0.3/32
16[CFG] 'android' has both left- and rightsourceip, but IKE can negotiate one virtXXl IP only, ignoring local virtXXl IP
16[CFG]   loaded certificate "C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com" from '/usr/local/etc/ipsec.d/certs/111.111.111.111_cert.der'
16[CFG]   loaded certificate "C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com" from '/usr/local/etc/ipsec.d/certs/android_cert.der'
16[CFG] added configuration 'android'
16[NET] received packet: from 22.222.222.22[16483] to 192.168.1.100[500] (660 bytes)
16[ENC] parsed IKE_SA_INIT request 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) ]
16[IKE] 22.222.222.22 is initiating an IKE_SA
16[IKE] IKE_SA (unnamed)[1] state change: CREATED => CONNECTING
16[IKE] local host is behind NAT, sending keep alives
16[IKE] remote host is behind NAT
16[IKE] sending cert request for "C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com"
16[ENC] generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(MULT_AUTH) ]
16[NET] sending packet: from 192.168.1.100[500] to 22.222.222.22[16483] (337 bytes)
16[NET] received packet: from 22.222.222.22[16549] to 192.168.1.100[4500] (1964 bytes)
16[ENC] parsed IKE_AUTH request 1 [ IDi CERT N(INIT_CONTACT) CERTREQ AUTH CPRQ(ADDR ADDR6 DNS DNS6) N(ESP_TFC_PAD_N) SA TSi TSr N(MOBIKE_SUP) N(NO_ADD_ADDR) N(MULT_AUTH) N(EAP_ONLY) ]
16[IKE] received cert request for "C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com"
16[IKE] received end entity cert "C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com"
16[CFG] looking for peer configs matching 192.168.1.100[%any]...22.222.222.22[C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com]
16[CFG] selected peer config 'android'
16[CFG]   using trusted ca certificate "C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com"
16[CFG] checking certificate status of "C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com"
16[CFG] certificate status is not available
16[CFG]   reached self-signed root ca with a path length of 0
16[CFG]   using trusted certificate "C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com"
16[IKE] authentication of 'C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com' with RSA signature successful
16[IKE] processing INTERNAL_IP4_ADDRESS attribute
16[IKE] processing INTERNAL_IP6_ADDRESS attribute
16[IKE] processing INTERNAL_IP4_DNS attribute
16[IKE] processing INTERNAL_IP6_DNS attribute
16[IKE] received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding
16[IKE] peer supports MOBIKE
16[IKE] authentication of '111.111.111.111' (myself) with RSA signature successful
16[IKE] IKE_SA android[1] established between 192.168.1.100[111.111.111.111]...22.222.222.22[C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com]
16[IKE] IKE_SA android[1] state change: CONNECTING => ESTABLISHED
16[IKE] scheduling reauthentication in 9802s
16[IKE] maximum IKE_SA lifetime 10342s
16[IKE] sending end entity cert "C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com"
16[IKE] peer requested virtXXl IP %any
16[CFG] assigning new lease to 'C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com'
16[IKE] assigning virtXXl IP 10.0.0.3 to peer 'C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com'
16[IKE] peer requested virtXXl IP %any6
16[IKE] no virtXXl IP found for %any6 requested by 'C=XX, ST=XX, L=XX, O=ORG, OU=UNIT, CN=111.111.111.111, E=mail@host.com'
16[IKE] CHILD_SA android{1} established with SPIs c11f2b7e_i f7ac6573_o and TS 0.0.0.0/0 ::/0 === 10.0.0.3/32
16[CHD] updown: /usr/local/libexec/ipsec/_updown: iptables: not found
16[CHD] updown: /usr/local/libexec/ipsec/_updown: iptables: not found
16[ENC] generating IKE_AUTH response 1 [ IDr CERT AUTH CPRP(ADDR) N(ESP_TFC_PAD_N) SA TSi TSr N(AUTH_LFT) N(MOBIKE_SUP) N(ADD_4_ADDR) ]
16[NET] sending packet: from 192.168.1.100[4500] to 22.222.222.22[16549] (1596 bytes)

Looking at the tcpdump output: 10.0.0.3 is an IP that Android client got assigned by StrongSwan.

DNS:
Code: 
22:21:43.971797 IP 10.0.0.3.55123 > ns1.kyivstar.net.domain: 41221+ A? www.cdn.viber.com. (35)
22:26:00.484696 IP 10.0.0.3.25462 > gi-dns1-kie3.kyivstar.net.domain: 436+ A? mtalk.google.com. (34)
22:25:59.225291 IP 10.0.0.3.53999 > ns1.kyivstar.net.domain: 56585+ A? graph.facebook.com. (36)

OK, pinging "bsdserver" works:
Code: 
22:26:06.460464 IP 22-222-222-22-gprs.kyivstar.net.16445 > bsdserver.local.sae-urn: UDP-encap: ESP(spi=0xc289c560,seq=0x96), length 132
22:26:06.460749 IP bsdserver.local.sae-urn > 46-211-183-74-gprs.kyivstar.net.16445: UDP-encap: ESP(spi=0xe77a2567,seq=0x1f), length 132

But pinging 8.8.8.8 doesn't:
Code: 
22:30:51.179070 IP 10.0.0.3 > google-public-dns-a.google.com: ICMP echo request, id 3, seq 6, length 64
22:30:52.098315 IP 22-222-222-22-gprs.kyivstar.net.16445 > bsdserver.local.sae-urn: UDP-encap: ESP(spi=0xc289c560,seq=0xce), length 132
22:30:52.098825 IP 10.0.0.3 > google-public-dns-a.google.com: ICMP echo request, id 3, seq 7, length 64
22:30:52.754209 STP 802.1d, Config, Flags [none], bridge-id 8000.b0:48:7a:c2:f2:92.8001, length 43
22:30:53.078436 IP 22-222-222-22-gprs.kyivstar.net.16445 > bsdserver.local.sae-urn: UDP-encap: ESP(spi=0xc289c560,seq=0xcf), length 132
22:30:53.079070 IP 10.0.0.3 > google-public-dns-a.google.com: ICMP echo request, id 3, seq 8, length 64
22:30:53.358693 IP 22-222-222-22-gprs.kyivstar.net.16445 > bsdserver.local.sae-urn: isakmp-nat-keep-alive
22:30:53.738572 IP 22-222-222-22-gprs.kyivstar.net.16445 > bsdserver.local.sae-urn: UDP-encap: ESP(spi=0xc289c560,seq=0xd0), length 116
22:30:53.739164 IP 10.0.0.3.25984 > gi-dns1-kie3.kyivstar.net.domain: 1638+ A? www.cdn.viber.com. (35)
22:30:54.098740 IP 22-222-222-22-gprs.kyivstar.net.16445 > bsdserver.local.sae-urn: UDP-encap: ESP(spi=0xc289c560,seq=0xd1), length 132
22:30:54.099248 IP 10.0.0.3 > google-public-dns-a.google.com: ICMP echo request, id 3, seq 9, length 64
22:30:54.754083 STP 802.1d, Config, Flags [none], bridge-id 8000.b0:48:7a:c2:f2:92.8001, length 43

I feel like problem isn't exactly IPsec related. Any bright ideas about possible solution? :beer
 
Update: this is what the route looks like on the Android side when a/the client is connected.

Screenshot_2014_05_01_22_18_44.png
 
korund said:
...
Forwarding turned on in sysctl.conf:
Code: 
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
...
Click to expand...

I cannot guarantee, whether the above is part of your issue. Anyway, since FreeBSD 9.2 that setting is no more sufficient for a 100 % trouble-free gateway operation. Remove this from /etc/sysctl.conf and instead place into /etc/rc.conf the following:
Code: 
gateway_enable="YES"
ipv6_gateway_enable="YES"
For me this resolved a routing issue, that suddenly occurred with net/mpd5 after I upgraded from FreeBSD 9.1 to 9.2.

The explanation of the problem is here: http://lists.freebsd.org/pipermail/freebsd-net/2013-October/037011.html

Since devd is more useful for a desktop machine than it is less useful for a server, I disabled it on my server by putting devd_enable="NO" into /etc/rc.conf.
 
obsigna said:
korund said:
...
Forwarding turned on in sysctl.conf:
Code: 
net.inet.ip.forwarding=1
net.inet6.ip6.forwarding=1
...
Click to expand...

I cannot guarantee, whether the above is part of your issue. Anyway, since FreeBSD 9.2 that setting is no more sufficient for a 100 % trouble-free gateway operation. Remove this from /etc/sysctl.conf and instead place into /etc/rc.conf the following:
Code: 
gateway_enable="YES"
ipv6_gateway_enable="YES"
For me this resolved a routing issue, that suddenly occurred with net/mpd5 after I upgraded from FreeBSD 9.1 to 9.2.

The explanation of the problem is here: http://lists.freebsd.org/pipermail/freebsd-net/2013-October/037011.html

Since devd is more useful for a desktop machine than it is less useful for a server, I disabled it on my server by putting devd_enable="NO" into /etc/rc.conf.
Click to expand...

Thanks for suggestion, both are removed.
 
Update:

I can ping -S 10.0.0.1 8.8.8.8 from FreeBSD side and it works. Also, from android side, ping 192.168.1.100 (which is FreeBSD ip) works as well.
But for some reason pinging anything further than server is unsuccessful .

I have tried:
Code: 
ext_if="igb0"
int_if="igb1"

nat on $ext_if from $int_if:network to any -> ($ext_if)
in /etc/pf.conf but it doesn't help :x
 
Update:

I have installed squid and can confirm that setting android client to use 10.0.0.1 as proxy server works. Without it - no news, packets failed to reach something further than FreeBSD server.
 
There is a bug in 10.0-RELEASE with how how the kernel is tagging the mbuf allocated with IPsec packets as it gets tagged to skip firewalling. Hence PF/IPFW/IPF can't NAT what it can't see. The short answer is that you need to upgrade to the latest 10.1-RC or use an older version of FreeBSD.

Long answers:
https://bugs.freebsd.org/185876 - The PR with the technical details.
https://forums.freebsd.org/viewtopic.php?f=7&t=45691 - Same issue and the troubleshooting that helped find it.
 
You must log in or register to reply here.
Back
Top