tun0 created by security/openfortivpn stopped passing traffic after upgrade from 13.2 to 14.0

S

szabig

Dear Community,

I have been using security/openfortivpn to connect to my company's corporate network for more than half a year now, and never had a problem with it. However, after a complete upgrade from 13.2 to 14.0, the traffic through the tun0 interface created by openfortivpn stopped passing.

So, I fired up a VM with 13.2 and another one with 14.0, and checked on those as well - the problem seems to be real, but I decided to ask here first as there is still a possibility that I am doing something wrong.

My config is fairly simple. I only set the host, the port, the username, the trusted-cert and set-dns = 0.
It seems to work as far as connecting goes - on both versions, connection is successful, routes are added, the tun0 interface comes up.

(xxx.xxx.xxx.xxx is obviously my corporate gateway IP)

13.2:
Code: 
$ ifconfig tun0
tun0: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet 10.12.1.130 --> xxx.xxx.xxx.xxx netmask 0xffffffff
        groups: tun
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 691

14.0:
Code: 
$ ifconfig tun0
tun0: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=80000<LINKSTATE>
        inet 10.12.1.130 --> xxx.xxx.xxx.xxx netmask 0xffffffff
        groups: tun
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        Opened by PID 1896

However, when I try to reach the network, like execute a simple ping, there is a big difference between the two:

13.2:
Code: 
$ sudo tcpdump -i tun0
Password:
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on tun0, link-type NULL (BSD loopback), capture size 262144 bytes
09:49:56.086938 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 26627, seq 20, length 64
09:49:56.090581 IP 10.12.1.130.44214 > 10.12.14.1.domain: 23899+ PTR? 130.1.12.10.in-addr.arpa. (42)
09:49:56.099329 IP 10.12.14.1 > 10.12.1.130: ICMP echo reply, id 26627, seq 20, length 64
09:49:56.102204 IP 10.12.14.1.domain > 10.12.1.130.44214: 23899 NXDomain* 0/1/0 (92)
09:49:56.102933 IP 10.12.1.130.41163 > 10.12.14.1.domain: 44969+ PTR? 1.14.12.10.in-addr.arpa. (41)
09:49:56.113332 IP 10.12.14.1.domain > 10.12.1.130.41163: 44969 NXDomain* 0/1/0 (91)
09:49:57.128101 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 26627, seq 21, length 64
09:49:57.140184 IP 10.12.14.1 > 10.12.1.130: ICMP echo reply, id 26627, seq 21, length 64
09:49:58.167455 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 26627, seq 22, length 64
09:49:58.177998 IP 10.12.14.1 > 10.12.1.130: ICMP echo reply, id 26627, seq 22, length 64
09:49:59.217471 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 26627, seq 23, length 64
09:49:59.229943 IP 10.12.14.1 > 10.12.1.130: ICMP echo reply, id 26627, seq 23, length 64

14.0:
Code: 
$ sudo tcpdump -i tun0
Password:
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on tun0, link-type NULL (BSD loopback), snapshot length 262144 bytes
08:44:17.336355 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 60167, seq 8, length 64
08:44:18.342343 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 60167, seq 9, length 64
08:44:19.378614 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 60167, seq 10, length 64
08:44:20.380962 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 60167, seq 11, length 64
08:44:21.388591 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 60167, seq 12, length 64
08:44:22.450852 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 60167, seq 13, length 64
08:44:23.474984 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 60167, seq 14, length 64
08:44:24.532355 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 60167, seq 15, length 64
08:44:25.547038 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 60167, seq 16, length 64
08:44:26.564679 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 60167, seq 17, length 64
08:44:27.625610 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 60167, seq 18, length 64
08:44:28.644863 IP 10.12.1.130 > 10.12.14.1: ICMP echo request, id 60167, seq 19, length 64

I even tried to turn on the debug.if_tun_debug sysctl kernel flag to see what is going on inside the tun driver while executing the ping.

13.2:
Code: 
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunwrite
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunoutput
tun0: starting
tun0: tunpoll
tun0: tunpoll q=1
tun0: read
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunwrite
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunoutput
tun0: starting
tun0: tunpoll
tun0: tunpoll q=1
tun0: read
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunwrite
tun0: tunpoll
tun0: tunpoll waiting

14.0:
Code: 
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunoutput
tun0: starting
tun0: tunpoll
tun0: tunpoll q=1
tun0: read
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunoutput
tun0: starting
tun0: tunpoll
tun0: tunpoll q=1
tun0: read
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting
tun0: tunpoll
tun0: tunpoll waiting

As can be seen, the tunwrite is not called in the latter case, so the driver perceives the packet as being sent without getting a reply. However, the scenario is the same here, the network, the gateway, the configuration - the only difference is the FreeBSD version. I also checked a Linux machine with openssl-3.0 to see if the updated openssl was the problem, but the Linux setup worked fine as well.
Finally, I checked the network interface where the tunnelled packet is supposed to go out, and in case of 13.2, I can see the packets going out (and the replies coming back) while in case of 14.0, there is nothing going in or out.
I am really out of ideas about what to check here, but it seems that there is some issue within the kernel between the tun and the network interfaces.
Could somebody, please, give me a hand?

Many thanks in advance!
 
I don't have much beyond:
Have you disabled all firewalling? pf, ipfw, anything else?

EDIT:
How about the routing table when tun0 is up on both?
I recall a bug a while ago in openvpn where a route was not getting added correctly. Vague recollections about the route not actually going through the tun0 interface, my limited notes say "...route was pointing at the wrong end of the tunnel"
 
Well, yes and no - for my original setup, PF is running (but only with NAT rules), but for the VMs, there are no firewalls.
But the same applies to both the successful (FreeBSD 13.2) and the failed (FreeBSD 14.0) scenarios, so that should not be the problem.
However, it should be straight-forward to reproduce if anyone here has access to a FortiGate router - or prove the problem is on my end by showing that openfortivpn works just as well with 14.0 as it did with 13.2.
 
szabig said:
Well, yes and no - for my original setup, PF is running (but only with NAT rules), but for the VMs, there are no firewalls.
But the same applies to both the successful (FreeBSD 13.2) and the failed (FreeBSD 14.0) scenarios, so that should not be the problem.
However, it should be straight-forward to reproduce if anyone here has access to a FortiGate router - or prove the problem is on my end by showing that openfortivpn works just as well with 14.0 as it did with 13.2.
Click to expand...
So I think we can rule out firewall, how about showing the routing table after tun0 is up on 13 and then on 14? I think the behavior you say on 14 could result from not having the correct route for tun0.
 
You are right, there is a difference between the two routing tables, but it is so small that it didn't catch my eye until I saved them to show you and running a diff on them for checking.
The route to the gateway is not set up properly:

Code: 
101c101

< xxx.xxx.xxx.xxx      link#3             UHS        tun0

---

> xxx.xxx.xxx.xxx      192.168.56.129     UGHS     vtnet0

(xxx.xxx.xxx.xxx is the corporate gateway, 192.168.56.129 is my private gateway).

The reason for this is given by the output of openfortivpn.

For 13.2, there are no problems:
Code: 
# openfortivpn
VPN account password:
INFO:   Connected to gateway.
INFO:   Authenticated.
INFO:   Remote gateway has allocated a VPN.
INFO:   Got addresses: [10.12.1.130], ns [10.12.14.1, 0.0.0.0], ns_suffix [xxx.com]
INFO:   Negotiation complete.
INFO:   Interface tun0 is UP.
INFO:   Setting new routes...
delete host xxx.xxx.xxx.xxx: gateway tun0
add host xxx.xxx.xxx.xxx: gateway 192.168.56.129
...
...

However, for 14.0, the output looks like this:
Code: 
# openfortivpn
VPN account password:
INFO:   Connected to gateway.
INFO:   Authenticated.
INFO:   Remote gateway has allocated a VPN.
INFO:   Got addresses: [10.12.1.131], ns [10.12.14.1, 0.0.0.0], ns_suffix [xxx.com]
INFO:   Negotiation complete.
INFO:   Interface tun0 is UP.
INFO:   Setting new routes...
delete host xxx.xxx.xxx.xxx: gateway tun0 fib 0: gateway uses the same route
add host xxx.xxx.xxx.xxx: gateway 192.168.56.129 fib 0: route already in table
...
...

Obviously the addition of the route fails because of the deletion failure above it. What does the error "fib 0: gateway uses the same route" mean, and why would it come up for 14.0 and not for 13.2?
Thanks much!
 
  • Like
Reactions: mer
I tried to google the error message "fib 0: gateway uses the same route" with only very few (and fairly old) results, like this one:
forums.freebsd.org

Can't delete a route in FreeBSD 9.2

Hello, I'm trying FreeBSD 9.2-RELEASE and faced with one trouble. I'm using the MPD5.7 client to connect to the internet provider. After connecting the script is run that deletes the route to the VPN server and adds the new route via the local gateway. This is script #!/bin/sh route delete...
forums.freebsd.org forums.freebsd.org
The problem described there looks pretty similar to mine but without any definite solution.
Any hint about how to handle it would be much appreciated!
 
szabig that sounds/looks similar to what I ran into. My limited notes I did route del followed by route add to fix things. Then poking around on openvpn forums/bugs it sounded like an openvpn bug. I do recall that upgrading packages at some point fixed it.
 
Well, I tried to manually play around with the routing table to fix the issue, but I ran into a sort of catch22 completely unrelated to openfortivpn...
I am probably misunderstanding the concept of the tun interface as the bellow explained behavior doesn't seem right to me - and I understand that I may be at the wrong topic/forum for this discussion (pointing me to the right one would also be appreciated).

When I create a tun interface and bring it up with a source and a destination address, a route is automatically added for each of these addresses through the tun interface. If I then try to change the route of the destination address (which I would suppose should not be through the tunnel as that address is a public address where the tunnel points to), I am unable to do so because I get the above mentioned message (as of 14.0 ... it did work in 13.2):

Code: 
$ sudo ifconfig tun0 create
$ sudo ifconfig tun0 10.1.1.1 10.2.2.2 netmask 255.255.255.255 up
$ ifconfig tun0
tun0: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=80000<LINKSTATE>
    inet 10.1.1.1 --> 10.2.2.2 netmask 0xffffffff
    groups: tun
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
$ netstat -r
...
default            192.168.0.1        UGS       wlan0
10.1.1.1           link#2             UHS         lo0
10.2.2.2           link#6             UH         tun0
localhost          link#2             UH          lo0
...
$ sudo route delete 10.2.2.2 -iface tun0
delete host 10.2.2.2: gateway tun0 fib 0: gateway uses the same route
$ sudo route add 10.2.2.2 192.168.0.1
add host 10.2.2.2: gateway 192.168.0.1 fib 0: route already in table

However, if I create the route first, and I try to set up the interface address after that, I cannot do that either because I get another error:

Code: 
$ sudo ifconfig tun0 create
$ sudo route add 10.2.2.2 192.168.0.1
add host 10.2.2.2: gateway 192.168.0.1
$ sudo ifconfig tun0 10.1.1.1 10.2.2.2 netmask 255.255.255.255 up
ifconfig: ioctl (SIOCAIFADDR): File exists
$ ifconfig tun0
tun0: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=80000<LINKSTATE>
    groups: tun
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

How is this scenario supposed to work? Is it not the intention of a tunnel to take some private address and tunnel it through a public gateway, which is accessible through an interface other than the tunnel? How is it possible to achieve that if neither of the above mentioned approaches are acceptable?

Many thanks to anyone who can take the time to help me understand this mechanism.
 
szabig said:
Well, I tried to manually play around with the routing table to fix the issue, but I ran into a sort of catch22 completely unrelated to openfortivpn...
I am probably misunderstanding the concept of the tun interface as the bellow explained behavior doesn't seem right to me - and I understand that I may be at the wrong topic/forum for this discussion (pointing me to the right one would also be appreciated).

When I create a tun interface and bring it up with a source and a destination address, a route is automatically added for each of these addresses through the tun interface. If I then try to change the route of the destination address (which I would suppose should not be through the tunnel as that address is a public address where the tunnel points to), I am unable to do so because I get the above mentioned message (as of 14.0 ... it did work in 13.2):

Code: 
$ sudo ifconfig tun0 create
$ sudo ifconfig tun0 10.1.1.1 10.2.2.2 netmask 255.255.255.255 up
$ ifconfig tun0
tun0: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=80000<LINKSTATE>
    inet 10.1.1.1 --> 10.2.2.2 netmask 0xffffffff
    groups: tun
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
$ netstat -r
...
default            192.168.0.1        UGS       wlan0
10.1.1.1           link#2             UHS         lo0
10.2.2.2           link#6             UH         tun0
localhost          link#2             UH          lo0
...
$ sudo route delete 10.2.2.2 -iface tun0
delete host 10.2.2.2: gateway tun0 fib 0: gateway uses the same route
$ sudo route add 10.2.2.2 192.168.0.1
add host 10.2.2.2: gateway 192.168.0.1 fib 0: route already in table

However, if I create the route first, and I try to set up the interface address after that, I cannot do that either because I get another error:

Code: 
$ sudo ifconfig tun0 create
$ sudo route add 10.2.2.2 192.168.0.1
add host 10.2.2.2: gateway 192.168.0.1
$ sudo ifconfig tun0 10.1.1.1 10.2.2.2 netmask 255.255.255.255 up
ifconfig: ioctl (SIOCAIFADDR): File exists
$ ifconfig tun0
tun0: flags=1008051<UP,POINTOPOINT,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 1500
    options=80000<LINKSTATE>
    groups: tun
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

How is this scenario supposed to work? Is it not the intention of a tunnel to take some private address and tunnel it through a public gateway, which is accessible through an interface other than the tunnel? How is it possible to achieve that if neither of the above mentioned approaches are acceptable?

Many thanks to anyone who can take the time to help me understand this mechanism.
Click to expand...

im sorry my english, but I recently had problems with openfortivpn, which didn't happen with openconnet.
If you can perform the test using:


cd /usr/ports/security/openconnect/ && make install clean

or

pkg install openconnect

doas openconnect --protocol=fortinet test.domain.com:port
 
csbatista said:
im sorry my english, but I recently had problems with openfortivpn, which didn't happen with openconnet.
If you can perform the test using:


cd /usr/ports/security/openconnect/ && make install clean

or

pkg install openconnect

doas openconnect --protocol=fortinet test.domain.com:port
Click to expand...

Thanks for the info!
I was unaware that openconnect supports fortinet VPN connections, but I tried it, and it works for me too.
Meanwhile, I patched openfortivpn because I realized that PINNED routes can be deleted by sending the 'route delete' message through PF_ROUTE raw sockets instead of the route command, so I managed to get that working as well.
There must be a more elegant way of doing it, but this is what I came up with:

Code: 
    struct rtmsg rtmes;
    struct sockaddr_in info;
    int s, wb;
    char *p;

    memset(&rtmes, '\0', sizeof rtmes);
    rtmes.m_rtm.rtm_version = RTM_VERSION;
    rtmes.m_rtm.rtm_type = RTM_DELETE;
    rtmes.m_rtm.rtm_addrs = RTA_DST;
    rtmes.m_rtm.rtm_index = 0;
    rtmes.m_rtm.rtm_seq = 0;
    rtmes.m_rtm.rtm_pid = getpid();
    rtmes.m_rtm.rtm_flags = RTF_UP | RTF_STATIC | RTF_PINNED | RTF_HOST;

    memset(&info, '\0', sizeof info);
    info.sin_family = AF_INET;
    if (check_ip(dst_ip) > 0) {
        info.sin_addr.s_addr = inet_addr(dst_ip);
    }
    else {
        fprintf(stderr, "Error: Invalid interface IP address!\n");
        return -1;
    }
    info.sin_len = sizeof(info);

    p = rtmes.m_space;
    p += memcpy_roundup(p, &info, info.sin_len);
    rtmes.m_rtm.rtm_msglen = p - (char *)&rtmes;

    s = socket(PF_ROUTE, SOCK_RAW, 0);
    if (s < 0) {
        fprintf(stderr, "socket(): %s\n", strerror(errno));
        return -1;
    }

    wb = write(s, &rtmes, rtmes.m_rtm.rtm_msglen);

    if (wb < 0) {

        if (rtmes.m_rtm.rtm_errno == 0)
            fprintf(stderr, "%s: Delete route failed: errno: %s\n",
                       dst_ip, strerror(errno));
        else
            fprintf(stderr, "%s: Delete route failed: %s\n",
                       dst_ip, strerror(rtmes.m_rtm.rtm_errno));

    }
    close(s);
    return 0;

I am still hoping for a discussion or any pointers to some reading about this topic as I dug a little bit into the code of the routing mechanism and the way ppp sets up the tun interface (I realized that in case of openfortivpn it is not set up by ifconfig but by ppp), and I would like to understand why tunnel destination address routes are PINNED by default.
 
szabig said:
Thanks for the info!
I was unaware that openconnect supports fortinet VPN connections, but I tried it, and it works for me too.
Meanwhile, I patched openfortivpn because I realized that PINNED routes can be deleted by sending the 'route delete' message through PF_ROUTE raw sockets instead of the route command, so I managed to get that working as well.
There must be a more elegant way of doing it, but this is what I came up with:

Code: 
    struct rtmsg rtmes;
    struct sockaddr_in info;
    int s, wb;
    char *p;

    memset(&rtmes, '\0', sizeof rtmes);
    rtmes.m_rtm.rtm_version = RTM_VERSION;
    rtmes.m_rtm.rtm_type = RTM_DELETE;
    rtmes.m_rtm.rtm_addrs = RTA_DST;
    rtmes.m_rtm.rtm_index = 0;
    rtmes.m_rtm.rtm_seq = 0;
    rtmes.m_rtm.rtm_pid = getpid();
    rtmes.m_rtm.rtm_flags = RTF_UP | RTF_STATIC | RTF_PINNED | RTF_HOST;

    memset(&info, '\0', sizeof info);
    info.sin_family = AF_INET;
    if (check_ip(dst_ip) > 0) {
        info.sin_addr.s_addr = inet_addr(dst_ip);
    }
    else {
        fprintf(stderr, "Error: Invalid interface IP address!\n");
        return -1;
    }
    info.sin_len = sizeof(info);

    p = rtmes.m_space;
    p += memcpy_roundup(p, &info, info.sin_len);
    rtmes.m_rtm.rtm_msglen = p - (char *)&rtmes;

    s = socket(PF_ROUTE, SOCK_RAW, 0);
    if (s < 0) {
        fprintf(stderr, "socket(): %s\n", strerror(errno));
        return -1;
    }

    wb = write(s, &rtmes, rtmes.m_rtm.rtm_msglen);

    if (wb < 0) {

        if (rtmes.m_rtm.rtm_errno == 0)
            fprintf(stderr, "%s: Delete route failed: errno: %s\n",
                       dst_ip, strerror(errno));
        else
            fprintf(stderr, "%s: Delete route failed: %s\n",
                       dst_ip, strerror(rtmes.m_rtm.rtm_errno));

    }
    close(s);
    return 0;

I am still hoping for a discussion or any pointers to some reading about this topic as I dug a little bit into the code of the routing mechanism and the way ppp sets up the tun interface (I realized that in case of openfortivpn it is not set up by ifconfig but by ppp), and I would like to understand why tunnel destination address routes are PINNED by default.
Click to expand...
Would you be interested in commiting this patch upstream? I'm trying to setup openfortivpn and would love to enjoy a more robust routing support
 
You must log in or register to reply here.
Back
Top