I may be missing something basic, but I cannot seem to make two hosts to communicate via VXLAN interfaces. Basic connectivity checks (pings) are ok between two hosts, but neither UDP, nor TCP connections work. Apologies for the long post, but I wanted to document how everything is configured and the basic tests/checks I already performed.
Basic configuration for both hosts (FreeBSD 12.1-RELEASE-p7) are:
##### physical interface #######
Checks and tests
1. Ping: works
##### ping host A from host B #####
TCPDUMP captures from host A:
Basically, everything works, nothing particularly interesting.
2. UDP: does not work
Host A: setup a listening UDP socket on VXLAN IP 192.168.20.2 port 9999:
Verify that the socket is listening:
Host B: send a UDP payload:
Packet capture from host A shows that the packet arrives, but it is not forwarded to the `nc` process:
3. TCP does not work (obviously)
Host A:
Verify that the socket is listening:
Host B:
Try to establish a TCP handshake:
TCPDUMP captures show that host B sends SYN packets and host A receives them from host B. Host A's nc process is oblivious to them.
What am I doing wrong?
Basic configuration for both hosts (FreeBSD 12.1-RELEASE-p7) are:
- on the same backbone physical subnet 192.168.1.0/24.
- on the same VXLAN id 1 and subnet 192.168.20.0/24.
- firewall is disabled `pfctl -d` on both hosts.
##### physical interface #######
Code:
# ifconfig bge0
bge0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8009b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE>
ether 00:1e:c9:35:07:a5
inet 192.168.1.117 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
##### VXLAN interface ######
# ifconfig vxlan0
vxlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1450
options=80000<LINKSTATE>
ether a6:2b:49:31:47:79
hwaddr 58:9c:fc:10:ff:d1
inet 192.168.20.2 netmask 0xffffff00 broadcast 192.168.20.255
groups: vxlan
vxlan vni 1 local 192.168.1.117:4789 group 239.0.0.1:4789
media: Ethernet autoselect (autoselect <full-duplex>)
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Host B configuration:
##### physical interface lagg0 - aggregated interface for wired and wireless interfaces, but wireless is not connected
# ifconfig lagg0
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether f0:de:f1:22:3e:3f
inet 192.168.1.233 netmask 0xffffff00 broadcast 192.168.1.255
laggproto failover lagghash l2,l3,l4
laggport: em0 flags=5<MASTER,ACTIVE>
laggport: wlan0 flags=0<>
groups: lagg
media: Ethernet autoselect
status: active
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
##### VXLAN interface #####
# ifconfig vxlan0
vxlan0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1450
options=80000<LINKSTATE>
ether 00:60:2f:23:21:80
hwaddr 58:9c:fc:10:ff:d1
inet 192.168.20.3 netmask 0xffffff00 broadcast 192.168.20.255
inet6 fe80::260:2fff:fe23:2180%vxlan0 prefixlen 64 scopeid 0x5
groups: vxlan
vxlan vni 1 local 192.168.1.233:4789 group 239.0.0.1:4789
media: Ethernet autoselect (autoselect <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
1. Ping: works
##### ping host A from host B #####
Code:
ping -c 1 192.168.20.2
64 bytes from 192.168.20.2: icmp_seq=0 ttl=64 time=0.445 ms
--- 192.168.20.2 ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.445/0.445/0.445/0.000 ms
Code:
# tcpdump -i vxlan0 -n -vv -X
tcpdump: listening on vxlan0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:46:36.758505 IP (tos 0x0, ttl 64, id 43678, offset 0, flags [none], proto ICMP (1), length 84)
192.168.20.3 > 192.168.20.2: ICMP echo request, id 56401, seq 0, length 64
0x0000: 4500 0054 aa9e 0000 4001 26b5 c0a8 1403 E..T....@.&.....
0x0010: c0a8 1402 0800 1051 dc51 0000 0000 e931 .......Q.Q.....1
0x0020: 1773 1fb5 0809 0a0b 0c0d 0e0f 1011 1213 .s..............
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
19:46:36.758529 IP (tos 0x0, ttl 64, id 55405, offset 0, flags [none], proto ICMP (1), length 84)
192.168.20.2 > 192.168.20.3: ICMP echo reply, id 56401, seq 0, length 64
0x0000: 4500 0054 d86d 0000 4001 f8e5 c0a8 1402 E..T.m..@.......
0x0010: c0a8 1403 0000 1851 dc51 0000 0000 e931 .......Q.Q.....1
0x0020: 1773 1fb5 0809 0a0b 0c0d 0e0f 1011 1213 .s..............
0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"#
0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123
0x0050: 3435 3637 4567
# tcpdump -i bge0 -n -vv -X port 4789
tcpdump: listening on bge0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:46:36.758472 IP (tos 0x0, ttl 64, id 43679, offset 0, flags [none], proto UDP (17), length 134)
192.168.1.233.40121 > 192.168.1.117.4789: [no cksum] VXLAN, flags [I] (0x08), vni 1
IP (tos 0x0, ttl 64, id 43678, offset 0, flags [none], proto ICMP (1), length 84)
192.168.20.3 > 192.168.20.2: ICMP echo request, id 56401, seq 0, length 64
0x0000: 4500 0086 aa9f 0000 4011 4b19 c0a8 01e9 E.......@.K.....
0x0010: c0a8 0175 9cb9 12b5 0072 0000 0800 0000 ...u.....r......
0x0020: 0000 0100 a62b 4931 4779 0060 2f23 2180 .....+I1Gy.`/#!.
0x0030: 0800 4500 0054 aa9e 0000 4001 26b5 c0a8 ..E..T....@.&...
0x0040: 1403 c0a8 1402 0800 1051 dc51 0000 0000 .........Q.Q....
0x0050: e931 1773 1fb5 0809 0a0b 0c0d 0e0f 1011 .1.s............
0x0060: 1213 1415 1617 1819 1a1b 1c1d 1e1f 2021 ...............!
0x0070: 2223 2425 2627 2829 2a2b 2c2d 2e2f 3031 "#$%&'()*+,-./01
0x0080: 3233 3435 3637 234567
19:46:36.758539 IP (tos 0x0, ttl 64, id 55406, offset 0, flags [none], proto UDP (17), length 134, bad cksum 0 (->1d4a)!)
192.168.1.117.11838 > 192.168.1.233.4789: [no cksum] VXLAN, flags [I] (0x08), vni 1
IP (tos 0x0, ttl 64, id 55405, offset 0, flags [none], proto ICMP (1), length 84)
192.168.20.2 > 192.168.20.3: ICMP echo reply, id 56401, seq 0, length 64
0x0000: 4500 0086 d86e 0000 4011 0000 c0a8 0175 E....n..@......u
0x0010: c0a8 01e9 2e3e 12b5 0072 0000 0800 0000 .....>...r......
2. UDP: does not work
Host A: setup a listening UDP socket on VXLAN IP 192.168.20.2 port 9999:
# nc -l -u 192.168.20.2 9999
Verify that the socket is listening:
Code:
# sockstat -L4 | grep nc
root nc 26750 3 udp4 192.168.20.2:9999 *:*
# echo "boo" | nc -u 192.168.20.2 9999
Packet capture from host A shows that the packet arrives, but it is not forwarded to the `nc` process:
Code:
# tcpdump -i bge0 -n -vv -X port 4789
20:21:09.541439 IP (tos 0x0, ttl 64, id 5682, offset 0, flags [none], proto UDP (17), length 82)
192.168.1.233.40121 > 192.168.1.117.4789: [no cksum] VXLAN, flags [I] (0x08), vni 1
IP (tos 0x0, ttl 64, id 5681, offset 0, flags [none], proto UDP (17), length 32)
192.168.20.3.61550 > 192.168.20.2.9999: [udp sum ok] UDP, length 4
0x0000: 4500 0052 1632 0000 4011 dfba c0a8 01e9 E..R.2..@.......
0x0010: c0a8 0175 9cb9 12b5 003e 0000 0800 0000 ...u.....>......
0x0020: 0000 0100 a62b 4931 4779 0060 2f23 2180 .....+I1Gy.`/#!.
0x0030: 0800 4500 0020 1631 0000 4011 bb46 c0a8 ..E....1..@..F..
0x0040: 1403 c0a8 1402 f06e 270f 000c 6d88 626f .......n'...m.bo
0x0050: 6f0a o.
# tcpdump -i vxlan0 -n -vv -X
20:21:09.541452 IP (tos 0x0, ttl 64, id 5681, offset 0, flags [none], proto UDP (17), length 32)
192.168.20.3.61550 > 192.168.20.2.9999: [udp sum ok] UDP, length 4
0x0000: 4500 0020 1631 0000 4011 bb46 c0a8 1403 E....1..@..F....
0x0010: c0a8 1402 f06e 270f 000c 6d88 626f 6f0a .....n'...m.boo.
Host A:
# nc -l 192.168.20.2 9999
Verify that the socket is listening:
Code:
# sockstat -L4 | grep nc
root nc 26927 3 tcp4 192.168.20.2:9999 *:*
Try to establish a TCP handshake:
Code:
# echo "boo" | nc 192.168.20.2 9999
# sockstat -L4 | grep nc
root nc 21933 3 tcp4 192.168.20.3:57818 192.168.20.2:9999
What am I doing wrong?