Setting rules for routed packets

[legyndir]

I have two virtual machines running on VMware with bridge mode network interfaces. I want one of them (FreeBSD) to act as a gateway between the host OS and another VM, which also has bridge interfaces. Simply the configuration is as follows

Client (Windows)           Gateway (FreeBSD)          Server (Ubuntu)     
10.41.1.31   <--------------> 10.41.1.118(em2)                                    
10.41.0.187                   10.41.2.118(em1)<---------------> 10.41.2.121(eth0)
                              10.41.0.118(em0)

Mac addresses for the machines:
FreeBSD

# ifconfig
em0: ether 00:0c:29:dc:23:4e
     inet 10.41.0.118 netmask 0xffffff00 broadcast 10.41.0.255
     
em1: 00:0c:29:dc:23:58
     inet 10.41.2.118 netmask 0xffffff00 broadcast 10.41.2.255
     
em2: 00:0c:29:dc:23:62
     inet 10.41.1.118 netmask 0xffffff00 broadcast 10.41.1.255

Ubuntu

eth0: Link encap:Ethernet  HWaddr 00:0c:29:80:1e:f2 
            inet addr:10.41.2.121  Bcast:10.41.2.255  Mask:255.255.255.0

Windows: E0-2A-82-FB-0D-83

tracert in Windows confirms the gateway is working as expected:

C:\Windows\system32>tracert -d 10.41.2.121
  1    <1 ms    <1 ms    <1 ms  10.41.1.118
  2    <1 ms    <1 ms    <1 ms  10.41.2.121

I have also set the routing rules in Ubuntu for this trace, to use FreeBSD as a gateway for IP 10.41.1.31.

Now the reason is I want to analyze packets running through the gateway with a sniffer program, which takes diverted packets from interfaces. For this, I have installed the following firewall rules:

00500 40 3847 divert 44444 ip from any to any via em2 in
00510 224 42382 divert 44444 ip from any to any via em1 in
00520 33 3607 divert 44444 ip from any to any via em0 in
65535 8468 1543968 allow ip from any to any

To test if diversion works, I just ignore every single packet with payloads I receive in my sniffer program (so that TCP handshake messages can be transmitted), instead of injecting them back. But I still see original packets going through, when I run tcpdump on the destination machine (Ubuntu).

> tcpdump  -i eth0 -s0  -nn tcp port 443
16:23:31.146387 IP 10.41.1.31.21745 > 10.41.2.121.443: Flags [S], seq 1703311588, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
16:23:31.175457 IP 10.41.2.121.443 > 10.41.1.31.21745: Flags [S.], seq 2000005748, ack 1703311589, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
16:23:31.176048 IP 10.41.2.121.443 > 10.41.1.31.21745: Flags [S.], seq 2000005748, ack 1703311589, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
16:23:31.176167 IP 10.41.1.31.21745 > 10.41.2.121.443: Flags [.], ack 1, win 16425, length 0
16:23:31.176391 IP 10.41.1.31.21745 > 10.41.2.121.443: Flags [.], ack 1, win 16425, length 0
16:23:31.177656 IP 10.41.1.31.21745 > 10.41.2.121.443: Flags [P.], seq 1:195, ack 1, win 16425, length 194
16:23:31.178011 IP 10.41.2.121.443 > 10.41.1.31.21745: Flags [.], ack 195, win 123, length 0
16:23:31.180558 IP 10.41.2.121.443 > 10.41.1.31.21745: Flags [.], ack 195, win 123, length 0

And this is what I see on FreeBSD

> tcpdump -e -i em1 -s0 -nn  tcp port 443
16:23:31.089340 e0:2a:82:fb:0d:83 > 00:0c:29:80:1e:f2, ethertype IPv4 (0x0800), length 66: 10.41.1.31.21745 > 10.41.2.121.443: Flags [S], seq 1703311588, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
16:23:31.117970 00:0c:29:80:1e:f2 > 00:0c:29:dc:23:58, ethertype IPv4 (0x0800), length 66: 10.41.2.121.443 > 10.41.1.31.21745: Flags [S.], seq 2000005748, ack 1703311589, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
*16:23:31.118600 00:0c:29:dc:23:62 > e0:2a:82:fb:0d:83, ethertype IPv4 (0x0800), length 66: 10.41.2.121.443 > 10.41.1.31.21745: Flags [S.], seq 2000005748, ack 1703311589, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
16:23:31.118604 e0:2a:82:fb:0d:83 > 00:0c:29:dc:23:62, ethertype IPv4 (0x0800), length 60: 10.41.1.31.21745 > 10.41.2.121.443: Flags [.], ack 1, win 16425, length 0
*16:23:31.118767 00:0c:29:dc:23:58 > 00:0c:29:80:1e:f2, ethertype IPv4 (0x0800), length 54: 10.41.1.31.21745 > 10.41.2.121.443: Flags [.], ack 1, win 16425, length 0
16:23:31.120787 e0:2a:82:fb:0d:83 > 00:0c:29:80:1e:f2, ethertype IPv4 (0x0800), length 248: 10.41.1.31.21745 > 10.41.2.121.443: Flags [P.], ack 1, win 16425, length 194
16:23:31.120794 00:0c:29:80:1e:f2 > 00:0c:29:dc:23:58, ethertype IPv4 (0x0800), length 60: 10.41.2.121.443 > 10.41.1.31.21745: Flags [.], ack 195, win 123, length 0
*16:23:31.121566 00:0c:29:dc:23:62 > e0:2a:82:fb:0d:83, ethertype IPv4 (0x0800), length 60: 10.41.2.121.443 > 10.41.1.31.21745: Flags [.], ack 195, win 123, length 0

This packet in the middle, with 194 bytes of payload goes directly to the destination. This packet, along with the first SYN packet has the Ethernet address of the original source/destination addresses in its Ethernet header fields. And I can not see neither of them in my sniffer. The packets marked with a star(*) are created in the sniffer and injected into the network

Forwarding rules are also set with sysctl:

# sysctl -a |grep forward |grep "ip" |grep net
net.inet.ip.forwarding: 1
net.inet.ip.fastforwarding: 0
net.inet6.ip6.forwarding: 1

Thanks in advance.

 

[legyndir]

I was playing a bit more with the rules, tracing deeper in.

I enabled logging on the rules:

sysctl net.inet.ip.fw.verbose_limit=100000
# tried these rules also with keep-state, nothing changed..
ipfw -q add 00500 divert 44444 log ip from any to any via em2 in 
ipfw -q add 00510 divert 44444 log ip from any to any via em1 in

And here is the log at the start of the connection:

Oct 25 16:58:52 legyndir kernel: ipfw: 510 Divert 44444 TCP 10.41.2.121:443 10.41.1.31:15590 in via em1
Oct 25 16:58:52 legyndir kernel: ipfw: 600 Accept TCP 10.41.2.121:443 10.41.1.31:15590 in via em1
Oct 25 16:58:52 legyndir kernel: ipfw: 600 Accept TCP 10.41.2.121:443 10.41.1.31:15590 out via em2
Oct 25 16:58:52 legyndir kernel: ipfw: 500 Divert 44444 TCP 10.41.1.31:15590 10.41.2.121:443 in via em2
Oct 25 16:58:52 legyndir kernel: ipfw: 600 Accept TCP 10.41.1.31:15590 10.41.2.121:443 in via em2
Oct 25 16:58:52 legyndir kernel: ipfw: 600 Accept TCP 10.41.1.31:15590 10.41.2.121:443 out via em1
Oct 25 16:58:52 legyndir kernel: ipfw: 500 Divert 44444 TCP 10.41.1.31:15590 10.41.2.121:443 in via em2

Now there are two things I don't get here. First, how can the connection in the first line can also be matched by both rules, 510 and 600. Interface and source/destination addresses are all same, so shouldn't it just match the first rule it sees and not even try 600?

Second, as can be seen in the configuration I posted before, the first packet the sniffer gets should be incoming from em2, be forwarded out from em1, and the reply should follow the exact opposite route. Yet it seems here that we have just the opposite case here. How can this be possible when I start the connection myself, via a HTTP request which should initially trigger a TCP connection from client to server?