Solved VXLAN works only on FreeBSD 11.2 but not works on 12.2 and 13 STABLE

alfa · Aug 24, 2021

Hi,
I successfully configured VXLAN tunnel between x64 FreeBSD 11.2 to x64 Linux

But in FreeBSD 12.2 and 13.0 STABLE with below same configuration not works.
So What is the problem with FreeBSD 12.2 and 13.0 what changed is it bug or any other thing?
Any help would be appreciated..

My fully working tested configuration is:

FreeBSD 11.2 side:

Code:

ifconfig vxlan409 create vxlanid 409 vxlanlocal 192.168.99.1 vxlanremote 192.168.99.99 inet 192.168.157.1/24

igb0: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
    ether e4:3a:6e:44:7b:33
    inet 192.168.41.102 netmask 0xffffff00 broadcast 192.168.41.255
    media: Ethernet autoselect (100baseTX <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    
vxlan409: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=80020<JUMBO_MTU,LINKSTATE>
    ether 58:9c:fc:10:d1:3f
    inet 192.168.159.1 netmask 0xffffff00 broadcast 192.168.159.255
    groups: vxlan
    vxlan vni 409 local 192.168.99.1:4789 remote 192.168.99.99:4789
    media: Ethernet autoselect (autoselect <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    
wg0: flags=80c1<UP,RUNNING,NOARP,MULTICAST> metric 0 mtu 1420
    options=80000<LINKSTATE>
    inet 192.168.99.1 netmask 0xffffff00
    groups: wg
    nd6 options=109<PERFORMNUD,IFDISABLED,NO_DAD>

Linux side:

Code:

physical interfaces: eth0,eth1


ip link add name vxlan409 type vxlan id 409 remote 192.168.99.1 local 192.168.99.99
ip link add name vbr0 type bridge
ip link set eth1 master vbr0
ip link set vxlan4095 master vbr0
ip link set vbr0 up

root@linux:~# ifconfig 
br-lan    Link encap:Ethernet  HWaddr E4:3A:6E:41:DC:E9  
          inet6 addr: fe80::e63a:6eff:fe41:dce9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6343 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2121 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:292026 (285.1 KiB)  TX bytes:723502 (706.5 KiB)

eth0      Link encap:Ethernet  HWaddr E4:3A:6E:41:DC:E8  
          inet addr:192.168.20.232  Bcast:192.168.20.255  Mask:255.255.255.0
          inet6 addr: fe80::e63a:6eff:fe41:dce8/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:24969 errors:0 dropped:0 overruns:0 frame:0
          TX packets:14841 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:3747964 (3.5 MiB)  TX bytes:3300227 (3.1 MiB)
          Memory:f7d00000-f7d1ffff 

eth1      Link encap:Ethernet  HWaddr E4:3A:6E:41:DC:E9  
          inet addr:169.254.169.169  Bcast:169.254.169.171  Mask:255.255.255.252
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:9099 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3407 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:555428 (542.4 KiB)  TX bytes:1150362 (1.0 MiB)
          Memory:f7c00000-f7c1ffff 

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:82 errors:0 dropped:0 overruns:0 frame:0
          TX packets:82 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:6389 (6.2 KiB)  TX bytes:6389 (6.2 KiB)

vxlan409  Link encap:Ethernet  HWaddr 4E:00:90:B0:A8:DF  
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8450 errors:15 dropped:0 overruns:0 carrier:15
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:983868 (960.8 KiB)

wg0       Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:192.168.99.99  P-t-P:192.168.99.99  Mask:255.255.255.255
          UP POINTOPOINT RUNNING NOARP  MTU:1420  Metric:1
          RX packets:704 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8561 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:25924 (25.3 KiB)  TX bytes:1699468 (1.6 MiB)

there is a client connected on eth1 and have IP : 192.168.157.100
http https , icmp .. traffic passes through between client and tunnel
eveything works well only FreeBSD 11.2 not 12.2 and 13.0 STABLE

12.2 AND 13.0 STABLE DUMPS:

Code:

root@test13:~ # tcpdump -p port 4789
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 262144 bytes

root@test13:~ # tcpdump -i wg0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg0, link-type NULL (BSD loopback), capture size 262144 bytes
10:47:13.277336 IP 192.168.99.99.54996 > 192.168.99.1.vxlan: VXLAN, flags [I] (0x08), vni 409
ARP, Request who-has 192.168.159.1 tell 192.168.159.100, length 46
10:47:13.633393 IP 192.168.99.99.39365 > 192.168.99.1.vxlan: VXLAN, flags [I] (0x08), vni 409
IP 0.0.0.0.bootpc > 255.255.255.255.bootps: BOOTP/DHCP, Request from e4:3a:6e:41:dc:e9 (oui Unknown), length 300
10:47:14.301605 IP 192.168.99.99.54996 > 192.168.99.1.vxlan: VXLAN, flags [I] (0x08), vni 409
ARP, Request who-has 192.168.159.1 tell 192.168.159.100, length 46


root@linux:~# tcpdump -i wg0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes
07:46:39.424139 IP 192.168.99.99.54996 > 192.168.99.1.4789: VXLAN, flags [I] (0x08), vni 409
ARP, Request who-has 192.168.159.1 tell 192.168.159.100, length 46
07:46:39.680210 IP 192.168.99.99.39365 > 192.168.99.1.4789: VXLAN, flags [I] (0x08), vni 409
IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from e4:3a:6e:41:dc:e9 (oui Unknown), length 300
07:46:40.448129 IP 192.168.99.99.54996 > 192.168.99.1.4789: VXLAN, flags [I] (0x08), vni 409
ARP, Request who-has 192.168.159.1 tell 192.168.159.100, length 46
07:46:41.472093 IP 192.168.99.99.54996 > 192.168.99.1.4789: VXLAN, flags [I] (0x08), vni 409


root@linux:~# tcpdump -i eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
07:52:19.296110 ARP, Request who-has 192.168.159.1 tell 192.168.159.100, length 46
07:52:20.320148 ARP, Request who-has 192.168.159.1 tell 192.168.159.100, length 46
07:52:21.344077 ARP, Request who-has 192.168.159.1 tell 192.168.159.100, length 46
07:52:21.760224 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from e4:3a:6e:41:dc:e9 (oui Unknown), length 300

SirDice · Aug 24, 2021

alfa said:
eveything works well only FreeBSD 11.2

What about 11.4? That might help narrow down the changes.

capra · Aug 25, 2021

I have vxlan working on FreeBSD 12 with the following set in /etc/rc.conf

Code:

cloned_interfaces="vxlan43 vxlan44"
create_args_vxlan43="vxlanid 43 vxlanlocal 192.168.1.5 vxlanremote 192.168.1.1"
ifconfig_vxlan43="inet 10.43.0.5 netmask 255.255.255.0 up"
create_args_vxlan44="vxlanid 44 vxlanlocal 192.168.1.5 vxlanremote 192.168.1.1"
ifconfig_vxlan44="inet 10.44.0.5 netmask 255.255.255.0 up"

alfa · Aug 25, 2021

SirDice said:
SirDice said:

What about 11.4? That might help narrow down the changes.

Click to expand...

I tried. It worked on FreeBSD 11.4 too.
So how can i find changes

SirDice · Aug 25, 2021

If we assume 11-STABLE works and 12-STABLE doesn't we can search if there are any differences between those two with regards to vxlan(4).

Logs for 11-STABLE: https://cgit.freebsd.org/src/log/sys/net/if_vxlan.c?h=stable/11
Logs for 12-STABLE: https://cgit.freebsd.org/src/log/sys/net/if_vxlan.c?h=stable/12
Logs for 13-STABLE: https://cgit.freebsd.org/src/log/sys/net/if_vxlan.c?h=stable/13

If we look at the difference between 11-STABLE and 12-STABLE there are only two commits in 12-STABLE that aren't in 11-STABLE.

alfa · May 13, 2022

Just as normal layer-2 traffic, data in a VXLAN is not encrypted. For security reasons, i have used a VXLAN over Wireguard to encrypt connections.
Now, I have found that the FreeBSD's wireguard kernel implementation is the problem itself. I removed Kernel implementation and installed go user space implementation it worked on FreeBSD 12.2 STABLE

bakul · May 13, 2022

alfa said:
I have found that the FreeBSD's wireguard kernel implementation is the problem itself.

May be file a bug report?

Solved VXLAN works only on FreeBSD 11.2 but not works on 12.2 and 13 STABLE

alfa

SirDice

Administrator

capra

alfa

SirDice

Administrator

alfa

bakul