Solved Xen domU networking - IPv6 works, but no IPv4 routing support

As the subject says, I'm running FreeBSD 10.3 in a Xen domu with a gentoo linux dom0 host. Currently, I can ping my local network (both IPv4 and IPv6), and I can ping and complete TCP transactions via IPv6, but I can neither ping nor get any other kind of throughput via IPv4 outside of my local network.

I'm using the FreeBSD VM image from the website, specifically the qcow2 image (Xen doesn't seem to like the vhd image for some reason). Under such, I get the following from a 'uname -a':
Code:
root@:~ # uname -a
FreeBSD  10.3-RELEASE FreeBSD 10.3-RELEASE #0 r297264: Fri Mar 25 02:10:02 UTC 2016     root@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

Samples:
Code:
root@:~ # ifconfig 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 
       options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> 
       inet6 ::1 prefixlen 128  
       inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1  
       inet 127.0.0.1 netmask 0xff000000  
       nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> 
xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 
       options=3<RXCSUM,TXCSUM> 
       ether 00:16:3e:fe:ce:af 
       inet 10.4.12.9 netmask 0xffffff00 broadcast 10.4.12.255  
       inet6 fe80::216:3eff:fefe:ceaf%xn0 prefixlen 64 scopeid 0x2  
       inet6 <public::ip> prefixlen 64  
       nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> 
       media: Ethernet manual 
       status: active 
root@:~ # netstat -rn 
Routing tables 

Internet: 
Destination        Gateway            Flags      Netif Expire 
default            10.4.12.10         UGS         xn0 
10.4.12.0/24       link#2             U           xn0 
10.4.12.9          link#2             UHS         lo0 
127.0.0.1          link#1             UH          lo0 

Internet6: 
Destination                       Gateway                       Flags      Netif
Expire 
::/96                             ::1                           UGRS        lo0 
default                           2001:470:5:745::1             UGS         xn0 
::1                               link#1                        UH          lo0 
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0 
<public::network>::/64            link#2                        U           xn0 
<public::network>::5              link#2                        UHS         lo0 
fe80::/10                         ::1                           UGRS        lo0 
fe80::%lo0/64                     link#1                        U           lo0 
fe80::1%lo0                       link#1                        UHS         lo0 
fe80::%xn0/64                     link#2                        U           xn0 
fe80::216:3eff:fefe:ceaf%xn0      link#2                        UHS         lo0 
ff01::%lo0/32                     ::1                           U           lo0 
ff01::%xn0/32                     fe80::216:3eff:fefe:ceaf%xn0  U           xn0 
ff02::/16                         ::1                           UGRS        lo0 
ff02::%lo0/32                     ::1                           U           lo0 
ff02::%xn0/32                     fe80::216:3eff:fefe:ceaf%xn0  U           xn0 
root@:~ # ping -c4 10.4.12.10 
PING 10.4.12.10 (10.4.12.10): 56 data bytes 
64 bytes from 10.4.12.10: icmp_seq=0 ttl=64 time=1.330 ms 
64 bytes from 10.4.12.10: icmp_seq=1 ttl=64 time=0.449 ms 
64 bytes from 10.4.12.10: icmp_seq=2 ttl=64 time=0.558 ms 
64 bytes from 10.4.12.10: icmp_seq=3 ttl=64 time=0.169 ms 

--- 10.4.12.10 ping statistics --- 
4 packets transmitted, 4 packets received, 0.0% packet loss 
round-trip min/avg/max/stddev = 0.169/0.627/1.330/0.430 ms 
root@:~ # ping -c4 8.8.8.8 
PING 8.8.8.8 (8.8.8.8): 56 data bytes 

--- 8.8.8.8 ping statistics --- 
4 packets transmitted, 0 packets received, 100.0% packet loss 
root@:~ # ping6 <router::ip> 
PING6(56=40+8+8 bytes) <local:public::ip> --> <router::ip> 
16 bytes from <router::ip>, icmp_seq=0 hlim=64 time=1.734 ms 
16 bytes from <router::ip>, icmp_seq=1 hlim=64 time=0.611 ms 
16 bytes from <router::ip>, icmp_seq=2 hlim=64 time=0.500 ms 
16 bytes from <router::ip>, icmp_seq=3 hlim=64 time=0.636 ms 
^C 
--- <router::ip> ping6 statistics --- 
4 packets transmitted, 4 packets received, 0.0% packet loss 
round-trip min/avg/max/std-dev = 0.500/0.870/1.734/0.501 ms 
root@:~ # ping6 -c4 google.com 
PING6(56=40+8+8 bytes) <local:public::ip> --> 2607:f8b0:4002:c03::65 
16 bytes from 2607:f8b0:4002:c03::65, icmp_seq=0 hlim=48 time=39.873 ms 
16 bytes from 2607:f8b0:4002:c03::65, icmp_seq=1 hlim=48 time=41.131 ms 
16 bytes from 2607:f8b0:4002:c03::65, icmp_seq=2 hlim=48 time=40.161 ms 
16 bytes from 2607:f8b0:4002:c03::65, icmp_seq=3 hlim=48 time=38.208 ms 

--- google.com ping6 statistics --- 
4 packets transmitted, 4 packets received, 0.0% packet loss 
round-trip min/avg/max/std-dev = 38.208/39.843/41.131/1.053 ms
My local rc.conf contents:
Code:
root@:~ # cat /etc/rc.conf 
ifconfig_xn0="inet 10.4.12.9 netmask 255.255.255.0 -lro -tso" 
defaultrouter="10.4.12.10" 
ifconfig_xn0_ipv6="inet6 <local:public::ip> prefixlen 64" 
ipv6_defaultrouter="<router::ip>" 
hostname="myhostname.mydomain.com"
Note that I've disabled TSO and LSO on the FreeBSD (domU) side. My gateway is pointing to a pfsense firewall running under another domu on the same host.

I had an issue a while back that prevented the host (and all real network devices) from using the firewall for TCP traffic, but that was resolved by passing the internet-facing NIC directly to the pfsense firewall rather than using the virtual device.
 
Since you're able to ping the default gateway but not anything beyond it my first guess would be there. It's probably not forwarding the IPv4 traffic and/or not NATing.
 
Other hosts on the network are able to use that gateway, I've been using it every day with various networking appliances including chromecast, a windows 10 steam streaming appliance, and a gentoo laptop. I've disabled IPv6 router advertisements and DHCP6 due to complications with netflix and Hurricane Electric (my tunnel broker), so only the dom0 host at the moment has a functional IPv6 interface... therefore all the other devices are using IPv4... why is just this one not getting through?
 
Here's a packet capture from the pfsense domU:
Code:
15:22:42.802423 00:16:3e:fe:ce:af > 00:16:3e:ae:bd:cc, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 13, offset 0, flags [none], proto ICMP (1), length 84)
    10.4.12.9 > 8.8.8.8: ICMP echo request, id 4, seq 0, length 64
15:22:43.866118 00:16:3e:fe:ce:af > 00:16:3e:ae:bd:cc, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 14, offset 0, flags [none], proto ICMP (1), length 84)
    10.4.12.9 > 8.8.8.8: ICMP echo request, id 4, seq 1, length 64
15:22:44.929676 00:16:3e:fe:ce:af > 00:16:3e:ae:bd:cc, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 15, offset 0, flags [none], proto ICMP (1), length 84)
    10.4.12.9 > 8.8.8.8: ICMP echo request, id 4, seq 2, length 64
15:22:45.993243 00:16:3e:fe:ce:af > 00:16:3e:ae:bd:cc, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 16, offset 0, flags [none], proto ICMP (1), length 84)
    10.4.12.9 > 8.8.8.8: ICMP echo request, id 4, seq 3, length 64
That part's getting through... Here's the outbound side of pfsense:
Code:
15:27:25.996468 d0:50:99:3b:c4:6c > 3c:7a:8a:c6:f7:d7, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 143, offset 0, flags [none], proto ICMP (1), length 84)
    50.88.20.177 > 8.8.8.8: ICMP echo request, id 1796, seq 0, length 64
15:27:27.044001 d0:50:99:3b:c4:6c > 3c:7a:8a:c6:f7:d7, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 144, offset 0, flags [none], proto ICMP (1), length 84)
    50.88.20.177 > 8.8.8.8: ICMP echo request, id 1796, seq 1, length 64
15:27:28.107523 d0:50:99:3b:c4:6c > 3c:7a:8a:c6:f7:d7, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 145, offset 0, flags [none], proto ICMP (1), length 84)
    50.88.20.177 > 8.8.8.8: ICMP echo request, id 1796, seq 2, length 64
15:27:29.171088 d0:50:99:3b:c4:6c > 3c:7a:8a:c6:f7:d7, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 146, offset 0, flags [none], proto ICMP (1), length 84)
    50.88.20.177 > 8.8.8.8: ICMP echo request, id 1796, seq 3, length 64
And yet, it works for other hosts... here's a packet capture from the dom0 host on the LAN side:
Code:
    10.4.12.19 > 8.8.8.8: ICMP echo request, id 23591, seq 3, length 64
15:33:24.322431 00:16:3e:ae:bd:cc > d0:50:99:3b:c4:6d, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 43, id 0, offset 0, flags [none], proto ICMP (1), length 84)
    8.8.8.8 > 10.4.12.19: ICMP echo reply, id 23591, seq 3, length 64
15:33:24.415125 00:16:3e:ae:bd:cc > d0:50:99:3b:c4:6d, ethertype IPv4 (0x0800), length 102: (tos 0x0, ttl 38, id 4692, offset 0, flags [none], proto ICMP (1), length 88)
... and on the outside interface...
Code:
15:26:31.175569 d0:50:99:3b:c4:6c > 3c:7a:8a:c6:f7:d7, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 28460, offset 0, flags [DF], proto ICMP (1), length 84)
    50.88.20.178 > 8.8.8.8: ICMP echo request, id 50944, seq 1, length 64
15:26:31.205813 3c:7a:8a:c6:f7:d7 > d0:50:99:3b:c4:6c, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 44, id 0, offset 0, flags [none], proto ICMP (1), length 84)
    8.8.8.8 > 50.88.20.178: ICMP echo reply, id 50944, seq 1, length 64
15:26:32.177280 d0:50:99:3b:c4:6c > 3c:7a:8a:c6:f7:d7, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 28497, offset 0, flags [DF], proto ICMP (1), length 84)
    50.88.20.178 > 8.8.8.8: ICMP echo request, id 50944, seq 2, length 64
15:26:32.200802 3c:7a:8a:c6:f7:d7 > d0:50:99:3b:c4:6c, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 44, id 0, offset 0, flags [none], proto ICMP (1), length 84)
    8.8.8.8 > 50.88.20.178: ICMP echo reply, id 50944, seq 2, length 64
15:26:33.179304 d0:50:99:3b:c4:6c > 3c:7a:8a:c6:f7:d7, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 28690, offset 0, flags [DF], proto ICMP (1), length 84)
In comparing these, it shows that the requests are going out in both cases, but the only replies are on other hosts... for some reason, the new FBSD domU gets no replies. The firewall insists it isn't blocking anything, btw... but it also insisted as much when I discovered earlier that it was silently dropping packets with bad checksums (as Xen figures it doesn't have to calculate checksums for bridged network connections).
 
Is there a firewall further up the chain? There's a different source address being used on the working (*.178) and non-working (*.177) systems.
 
That's a good catch... I'll have to investigate why it's originating from .177 - that's my modem's address. I suspect the ISP is blocking that traffic as invalid, and well they should.

I think what's going on is that I've enabled a 1:1 NAT on pfsense for my /29 IP allocation (5 public IPs), and my .9 internal address is at the bottom of that mapping. I think I'll have to blacklist the .9 address so nothing occupies it and I'll be fine. Seems like a miscalculation on my part (I suspected this was some form of user error from the start). I'll adjust the internal address to something outside of that range (as I don't really want this to occupy a public address) and see if that fixes it.

[EDIT]Confirmed - that was the problem. Moving to a non-mapped IP resolved the issue.
 
Back
Top