multiple fibs leaking traffic

sakoula

sakoula

Hi,
I have a server with:
- one NIC (ale0) with IP 192.168.1.102
- 2 FIBS
- running openvpn and transmission (bittorrent) on FIB 1. (VPN IP is 10.10.13.210)

Code: 
|---------------|
|               |-- ale0 (192.168.1.102)
|  FreeBSD 9.1  |
|    2 FIBS     |
|               |
|               |-- tap0 (openvpn on FIB 1 10.10.13.210)
|---------------|

I expect all the transmission traffic to go over the tap0 interface and not see any traffic on ale0 with IP 10.10.13.210 (vpn).

tcpdump proves me wrong:
Code: 
# tcpdump -nn -i ale0 host 10.10.13.210
09:36:36.892074 IP 10.10.13.210.51415 > 1.1.11.173.52449: Flags [R], seq 2085203014, win 0, length 0
09:36:37.344177 IP 10.10.13.210.51415 > 1.1.90.74.65156: Flags [R], seq 2816628297, win 0, length 0
09:36:37.438990 IP 10.10.13.210.30351 > 1.1.181.127.28575: Flags [R], seq 3842924655, win 0, length 0
09:36:37.526022 IP 10.10.13.210.51415 > 1.1.252.83.58814: Flags [R], seq 265210318, win 0, length 0
09:36:37.673292 IP 10.10.13.210.51415 > 1.1.248.8.53386: Flags [R], seq 2313023041, win 0, length 0
09:36:37.676229 IP 10.10.13.210.14310 > 1.1.109.39.12678: Flags [R], seq 3644214725, win 0, length 0
09:36:37.738994 IP 10.10.13.210.51415 > 1.1.188.160.50152: Flags [R], seq 224522817, win 0, length 0
09:36:38.286168 IP 10.10.13.210.51415 > 1.1.165.50.60590: Flags [R], seq 3222894287, win 0, length 0
09:36:38.305551 IP 10.10.13.210.51415 > 1.1.12.47.57376: Flags [R], seq 1047394120, win 0, length 0
09:36:38.385334 IP 10.10.13.210.51415 > 1.1.110.247.53719: Flags [R], seq 2571011398, win 0, length 0

These are TCP RESET packets send to peers over the ale0 but they are supposed to be sent over the tap0!!! Am I doing something wrong? Are TCP reset packets supposed to be broadcasted or something on all interfaces?

I do not see anything wrong in the syslog. Is it a bug or a feature of TCP?

More info:

# ifconfig
Code: 
ale0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=c319a<TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MCAST,WOL_MAGIC,VLAN_HWTSO,LINKSTATE>
  ether 90:e6:ba:f3:30:6f
  inet 192.168.1.102 netmask 0xffffff00 broadcast 192.168.1.255
  nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0xc
  inet 127.0.0.1 netmask 0xff000000
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=80000<LINKSTATE>
  ether 00:bd:cd:af:04:00
  inet6 fe80::2bd:cdff:feaf:400%tap0 prefixlen 64 scopeid 0xd
  inet 10.10.13.210 netmask 0xffffff00 broadcast 10.10.13.255
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  Opened by PID 1931


# setfib 0 netstat -nr
Code: 
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.1.1        UGS         0    11164   ale0
127.0.0.1          link#12            UH          0     1275    lo0
10.10.13.0/24      link#13            U           0        4   tap0
10.10.13.210       link#13            UHS         0        0    lo0
192.168.1.0/24     link#7             U           0       36   ale0
192.168.1.102      link#7             UHS         0        0    lo0

# setfib 1 netstat -nr
Code: 
# setfib 1 netstat -nr
Routing tables

Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
0.0.0.0/1          10.10.13.25      UGS         0   427115   tap0 =>
default            192.168.1.1        UGS         0      187   ale0
127.0.0.1          link#12            UH          0        7    lo0
128.0.0.0/1        10.10.13.25      UGS         0   129183   tap0
10.10.13.0/24      link#13            U           0      161   tap0
10.10.13.25/32     192.168.1.1        UGS         0   556385   ale0
192.168.1.0/24     link#7             U           0       68   ale0

The commands I use to start the daemons:
Code: 
setfib 1 /sbin/route add default 192.168.1.1
setfib 1 /usr/local/etc/rc.d/openvpn onestart
setfib 1 /usr/local/etc/rc.d/transmission onestart
 
This route is a little odd:
Code: 
0.0.0.0/1          10.10.13.25      UGS         0   427115   tap0 =>
Where does it come from? I'm guessing this should be 0.0.0.0/0 aka default.
 
SirDice said:
This route is a little odd:
Code: 
0.0.0.0/1          10.10.13.25      UGS         0   427115   tap0 =>
Where does it come from? I'm guessing this should be 0.0.0.0/0 aka default.
Click to expand...

That is OpenVPN's way of setting up default route over the VPN tunnel. It sets routes for 0.0.0.0/1 and 128.0.0.0/1 instead of just 0.0.0.0/0. I can't remember the exact reason for doing it that way though.

Edit: It's probably to be able to keep the 0.0.0.0/0 route as it is, those two routes are more specific than 0.0.0.0/0 so they override it completely.
 
Hi,

I did some more debugging:

I captured one of the problematic packets with source IP 10.10.13.210 to the peer 2.2.217.151. This packet should travel over the OpenVPN device (tap0) but instead it travels over the
ethernet device (ale0):

Code: 
type    ,No.,Time,        Source      ,Destination  ,Protocol,Length,Info
ethernet,526,87.164747000,10.10.13.210,2.2.217.151   ,TCP     ,54    ,51415 > 60163 [RST] Seq=1 Win=0 Len=0

I captured the whole conversation related to 2.2.217.151 peer:

Code: 
type    ,No.,Time,        Source      ,Destination ,Pro,Len,Info
openvpn ,521,87.024718000,2.2.217.151 ,10.10.13.210,TCP,66 ,60163 > 51415 [SYN] Seq=0 Win=8192 Len=0 MSS=1336 WS=256 SACK_PERM=1
openvpn ,522,87.024735000,10.10.13.210,2.2.217.151 ,TCP,66 ,51415 > 60163 [SYN, ACK] Seq=0 Ack=1 Win=65535 Len=0 MSS=1336 WS=64 SACK_PERM=1
openvpn ,523,87.163613000,2.2.217.151 ,10.10.13.210,TCP,60 ,60163 > 51415 [ACK] Seq=1 Ack=1 Win=66560 Len=0
openvpn ,524,87.163664000,10.10.13.210,2.2.217.151 ,TCP,54 ,51415 > 60163 [FIN, ACK] Seq=1 Ack=1 Win=66752 Len=0
openvpn ,525,87.164735000,2.2.217.151 ,10.10.13.210,TCP,171,60163 > 51415 [PSH, ACK] Seq=1 Ack=1 Win=66560 Len=117
ethernet,526,87.164747000,10.10.13.210,2.2.217.151 ,TCP,54 ,51415 > 60163 [RST] Seq=1 Win=0 Len=0
openvpn ,529,87.301777000,2.2.217.151 ,10.10.13.210,TCP,61 ,60163 > 51415 [PSH, ACK] Seq=118 Ack=2 Win=66560 Len=7
openvpn ,530,87.301788000,10.10.13.210,2.2.217.151 ,TCP,54 ,51415 > 60163 [RST] Seq=2 Win=0 Len=0
openvpn ,531,87.302095000,2.2.217.151 ,10.10.13.210,TCP,60 ,60163 > 51415 [FIN, ACK] Seq=125 Ack=2 Win=66560 Len=0
openvpn ,532,87.302106000,10.10.13.210,2.2.217.151 ,TCP,54 ,51415 > 60163 [RST] Seq=2 Win=0 Len=0

To my understanding the key events are:
  • - no:524 server (transmission) sends a FIN to the peer over the OpenVPN device (tap0). sounds good :)
  • - no:525 peer ACKs back over the OpenVPN device (tap0). sounds good :)
  • - no:526 server (transmission) sends a TCP reset to the peer over ethernet device (ale0) with Seq=1. no way this should travel over the OpenVPN device (tap0)! not happy :(
  • - no:529 peer ACKs again over the OpenVPN device (tap0). sounds good :)
  • - no:530 server (transmission) sends the right TCP reset to the peer over the OpenVPN device (tap0) with Seq=2. Now super happy :)

So for some reason the first RST leaked(?) to the other fib. Is there a reason that it may be connected to some TCP state machine feature/bug?

Of course I run the whole setup behind a "home" router with NAT.

Thanks again!!!!
 
You must log in or register to reply here.
Back
Top