Solved IPSEC - racoon - gif - packet routing issues - transfer stall/fail

andrei693

andrei693

Hi all,

I have an IPSEC tunnel between a FreeBSD-10.0-RELEASE and a Fortinet appliance. I have some issues with data transfers through the IPSEC tunnel.

The tunnel is between hosts a.b.c.d and d.e.f.g and is connecting 10.0.0.1/24 with 192.168.1.1/27.

The issue that I'm having is that scp transfer stalls at 2112kb. On the FreeBSD box I see:
Code: 
enc0
00:00:00.000002 (authentic,confidential): SPI : IP a.b.c.d > e.f.g.h: IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [.], seq 65138:66522, ack 2264, win 92, options [nop,nop,TS val 4021781 ecr 315264465], length 1384 (ipip-proto-4)
00:00:00.000106 (authentic,confidential): SPI : IP e.f.g.h > a.b.c.d: IP 192.168.1.2.22 > 10.0.0.5.38171: Flags [.], ack 49914, win 331, options [nop,nop,TS val 315264465 ecr 4021780], length 0 (ipip-proto-4)
00:00:00.000010 (authentic,confidential): SPI : IP e.f.g.h > a.b.c.d: IP 192.168.1.2.22 > 10.0.0.5.38171: Flags [.], ack 52682, win 331, options [nop,nop,TS val 315264465 ecr 4021780], length 0 (ipip-proto-4)
00:00:00.000404 (authentic,confidential): SPI : IP e.f.g.h > a.b.c.d: IP 192.168.1.2.22 > 10.0.0.5.38171: Flags [.], ack 55450, win 331, options [nop,nop,TS val 315264465 ecr 4021780], length 0 (ipip-proto-4)
00:00:00.000009 (authentic,confidential): SPI : IP e.f.g.h > a.b.c.d: IP 192.168.1.2.22 > 10.0.0.5.38171: Flags [.], ack 58218, win 331, options [nop,nop,TS val 315264465 ecr 4021781], length 0 (ipip-proto-4)
00:00:00.001140 (authentic,confidential): SPI : IP e.f.g.h > a.b.c.d: IP 192.168.1.2.22 > 10.0.0.5.38171: Flags [.], ack 60986, win 331, options [nop,nop,TS val 315264466 ecr 4021781], length 0 (ipip-proto-4)
00:00:00.000018 (authentic,confidential): SPI : IP e.f.g.h > a.b.c.d: IP 192.168.1.2.22 > 10.0.0.5.38171: Flags [.], ack 63754, win 374, options [nop,nop,TS val 315264466 ecr 4021781], length 0 (ipip-proto-4)
00:00:00.000009 (authentic,confidential): SPI : IP e.f.g.h > a.b.c.d: IP 192.168.1.2.22 > 10.0.0.5.38171: Flags [.], ack 66522, win 419, options [nop,nop,TS val 315264466 ecr 4021781], length 0 (ipip-proto-4)
00:00:16.348676 (authentic,confidential): SPI : IP 10.0.0.5.38160 > 192.168.1.2.22: Flags [.], seq 66522:67906, ack 2264, win 92, options [nop,nop,TS val 4025869 ecr 315255548], length 1384
00:00:35.668008 (authentic,confidential): SPI : IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [.], seq 66522:67906, ack 2264, win 92, options [nop,nop,TS val 4034786 ecr 315264466], length 1384

I don't understand why the next seq 66522:67906 is delayed with 16 seconds on the enc interface while the packets have arrived on vlan10 long before that. If I check the local vlan interface on the firewall where 10.0.0.5 is I can see replies coming a lot earlier than that:

Code: 
vlan10
00:00:00.000002 IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [P.], seq 77594:78978, ack 2264, win 92, options [nop,nop,TS val 4021781 ecr 315264465], length 1384
00:00:00.000785 IP 192.168.1.2.22 > 10.0.0.5.38171: Flags [.], ack 60986, win 331, options [nop,nop,TS val 315264466 ecr 4021781], length 0
00:00:00.000017 IP 192.168.1.2.22 > 10.0.0.5.38171: Flags [.], ack 63754, win 374, options [nop,nop,TS val 315264466 ecr 4021781], length 0
00:00:00.000009 IP 192.168.1.2.22 > 10.0.0.5.38171: Flags [.], ack 66522, win 419, options [nop,nop,TS val 315264466 ecr 4021781], length 0
00:00:00.000165 IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [.], seq 78978:80362, ack 2264, win 92, options [nop,nop,TS val 4021781 ecr 315264466], length 1384
00:00:00.000002 IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [.], seq 80362:81746, ack 2264, win 92, options [nop,nop,TS val 4021781 ecr 315264466], length 1384
00:00:00.000181 IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [.], seq 81746:83130, ack 2264, win 92, options [nop,nop,TS val 4021781 ecr 315264466], length 1384
00:00:00.000002 IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [.], seq 83130:84514, ack 2264, win 92, options [nop,nop,TS val 4021781 ecr 315264466], length 1384
00:00:00.000002 IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [.], seq 84514:85898, ack 2264, win 92, options [nop,nop,TS val 4021781 ecr 315264466], length 1384
00:00:00.000002 IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [.], seq 85898:87282, ack 2264, win 92, options [nop,nop,TS val 4021781 ecr 315264466], length 1384
00:00:00.201286 IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [.], seq 66522:67906, ack 2264, win 92, options [nop,nop,TS val 4021832 ecr 315264466], length 1384
00:00:00.407997 IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [.], seq 66522:67906, ack 2264, win 92, options [nop,nop,TS val 4021934 ecr 315264466], length 1384
00:00:00.815008 IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [.], seq 66522:67906, ack 2264, win 92, options [nop,nop,TS val 4022138 ecr 315264466], length 1384
00:00:01.631995 IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [.], seq 66522:67906, ack 2264, win 92, options [nop,nop,TS val 4022546 ecr 315264466], length 1384
00:00:03.264010 IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [.], seq 66522:67906, ack 2264, win 92, options [nop,nop,TS val 4023362 ecr 315264466], length 1384
00:00:06.527996 IP 10.0.0.5.38171 > 192.168.1.2.22: Flags [.], seq 66522:67906, ack 2264, win 92, options [nop,nop,TS val 4024994 ecr 315264466], length 1384
00:00:03.500009 IP 10.0.0.5.38160 > 192.168.1.2.22: Flags [.], seq 66522:67906, ack 2264, win 92, options [nop,nop,TS val 4025869 ecr 315255548], length 1384

If anyone has any idea why this happens please shed some light.
 
Last edited by a moderator:
Hi, I remember having the same problem a few years ago. I tried lowering the MTU on the FreeBSD box but that didn't help. I ended up lowering the ng_tcpmss() value and it worked, although I can't remember how I did it.
 
I don't see anything that stands out but It would help to have some ifconfig output for the interfaces in question. Additionally, do you have any firewall enabled? There was a bug in 10.0-RELEASE that impacted IPSEC packet processing regardless of firewall in use and cause some odd behavior with regards to firewalls.
 
I'm using PF as firewall. Tried with set skip on both enc0 and gif0.

Code: 
# ifconfig enc0
enc0: flags=141<UP,RUNNING,PROMISC> metric 0 mtu 1536
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
# ifconfig gif0
gif0: flags=8151<UP,POINTOPOINT,RUNNING,PROMISC,MULTICAST> metric 0 mtu 1280
  tunnel inet a.b.c.d --> e.f.g.h
  inet 10.0.0.1 --> 192.168.1.2 netmask 0xffffff00
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
# ifconfig vlan44
vlan10: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=303<RXCSUM,TXCSUM,TSO4,TSO6>
  ether 00:25:90:e1:00:ad
  inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255   
  inet6 fe80::225:90ff:fee1:ad%vlan44 prefixlen 64 scopeid 0xd
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
  vlan: 10 parent interface: igb1

I'm thinking if it's worth to try and update to latest 10-STABLE but I'm going to leave this project by the end of next week. For the time being I made another IPSEC tunnel between Fortinet devices on both sides, that one works. I'll keep trying to fix this with FreeBSD too.
 
Alright. Well if you are skipping filtering on the interface that should in theory result in the same thing as the issue. See PR 185876 for details. Basically, incoming IPSEC packets got tagged with M_SKIP_FIREWALL mbuf(9) tag which would result in the packets never getting passed to any firewall as they traversed the system. It would result in wierd symptoms like packets leaving on an interface that does NAT still using an internal IP address. Given the information you've shown so far I don't think that would be a factor. However, it would absolutely make sense to update to at least 10.1-RELEASE given that 10.0-RELEASE's EOL date is 28 February 2015. I'm not aware of any other bugs that could be a factor here but at least that would be a good clean slate to start troubleshooting the issue on.
 
Thanks junovitch. I've rebuild world and now on RELEASE-10.1-p4 I don't see the issue anymore. I will test next week without the set skip on gif0/enc0.
 
You must log in or register to reply here.
Back
Top