bhyve bhyve vm can't ping or connect to external ips

I have setup srv-io on mce(4) with a VF mce2:

Code:
PF {
    device: "mlx5_core0";
    num_vfs: 8;
}

DEFAULT {
    passthrough: true;
}

VF-0 {
This interface is added to a bridge.
Code:
mce2: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=7ead00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,HWRXTSTMP,NOMAP,TXTLS4,TXTLS6,VXLAN_HWCSUM,VXLAN_HWTSO>
    ether aa:32:77:4a:93:eb
    inet6 fe80::a832:77ff:fe4a:93eb%mce2 prefixlen 64 scopeid 0x6
    media: Ethernet 25GBase-SR <full-duplex,rxpause,txpause>
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether ce:21:ca:4a:87:c7
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000000
    member: mce2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 55
    groups: bridge vm-switch viid-4c918@
    nd6 options=1<PERFORMNUD>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vmnet/freebsd-cloud/0/public
    options=80000<LINKSTATE>
    ether 58:9c:fc:00:41:5d
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 3772
Then I'm launching a vm. The VM get correctly it's IP using DHCP but I can't ping external IPs or the gateway or connect to another site in the CM. Any idea what could be the issue?

On the vm I have :

Code:
root@freebsd:~ # ifconfig -a
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80028<VLAN_MTU,JUMBO_MTU,LINKSTATE>
        ether 58:9c:fc:0f:99:39
        inet6 fe80::5a9c:fcff:fe0f:9939%vtnet0 prefixlen 64 scopeid 0x1
        inet 10.102.1.254 netmask 0xffffff00 broadcast 10.102.1.255
        media: Ethernet autoselect (10Gbase-T <full-duplex>)
        status: active
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
Code:
root@freebsd:~ # netstat -rn4
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            10.102.1.1         UGS      vtnet0
10.102.1.0/24      link#1             U        vtnet0
10.102.1.254       link#1             UHS         lo0
127.0.0.1          link#2             UH          lo0
 
The VM get correctly it's IP using DHCP
Where's the DHCP service located? If it's on the LAN then there's at least some network traffic going out the host, or else DHCP wouldn't work either.
but I can't ping external IPs or the gateway or connect to another site in the CM.
Where's the 10.102.1.1 gateway located? Run tcpdump(1) on the bridge and on mce2, can you actually see the ARP requests?
 
dhcp is running in a mikrotik router with 10.102.1.1 address. On that router it's vlan 102 untagged to the machine.

You're right on the host I can see the arp request and icmp going through the interfacce:

Code:
listening on mce2, link-type EN10MB (Ethernet), capture size 262144 bytes
13:30:46.935509 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 137, length 64
13:30:47.984869 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 138, length 64
13:30:47.984927 ARP, Request who-has 10.102.1.254 tell 10.102.1.1, length 46
13:30:47.985069 ARP, Reply 10.102.1.254 is-at 58:9c:fc:0f:99:39 (oui Unknown), length 28
13:30:49.029463 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 139, length 64
13:30:49.281710 18:fd:74:05:15:d6 (oui Unknown) > Broadcast, ethertype Unknown (0x9003), length 64:
    0x0000:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0030:  0000                                     ..
13:30:50.075015 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 140, length 64
13:30:51.125141 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 141, length 64
13:30:52.175360 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 142, length 64
13:30:53.225375 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 143, length 64

Same on the bridge:

Code:
benoitc@pollen1:~ $ sudo tcpdump -i vm-public
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vm-public, link-type EN10MB (Ethernet), capture size 262144 bytes
13:32:14.284129 18:fd:74:05:15:d6 (oui Unknown) > Broadcast, ethertype Unknown (0x9003), length 64:
    0x0000:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0030:  0000                                     ..
13:32:14.911759 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 221, length 64
13:32:15.955487 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 222, length 64
13:32:17.005462 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 223, length 64
13:32:18.055480 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 224, length 64
13:32:19.095812 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 225, length 64
13:32:19.291246 18:fd:74:05:15:d6 (oui Unknown) > Broadcast, ethertype Unknown (0x9003), length 64:
    0x0000:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0030:  0000                                     ..
13:32:20.129354 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 226, length 64
13:32:21.175249 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 227, length 64
13:32:22.225308 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 228, length 64
13:32:23.272367 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 229, length 64
13:32:24.300667 18:fd:74:05:15:d6 (oui Unknown) > Broadcast, ethertype Unknown (0x9003), length 64:
    0x0000:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0030:  0000                                     ..
13:32:24.320043 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 230, length 64
13:32:25.364831 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 231, length 64
13:32:26.414977 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 232, length 64
13:32:27.465042 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 233, length 64
13:32:28.515003 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 234, length 64
13:32:29.299622 18:fd:74:05:15:d6 (oui Unknown) > Broadcast, ethertype Unknown (0x9003), length 64:
    0x0000:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0030:  0000                                     ..
13:32:29.542165 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 235, length 64
13:32:30.595023 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 236, length 64
13:32:31.644870 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 237, length 64
13:32:32.694856 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 238, length 64
13:32:33.744898 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 239, length 64
13:32:34.297286 18:fd:74:05:15:d6 (oui Unknown) > Broadcast, ethertype Unknown (0x9003), length 64:
    0x0000:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0030:  0000                                     ..
13:32:34.794878 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 240, length 64
13:32:35.844855 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 241, length 64
13:32:36.885347 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 242, length 64
13:32:37.935387 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 243, length 64
13:32:38.985190 IP 10.102.1.254 > 10.102.1.1: ICMP echo request, id 9740, seq 244, length 64
13:32:39.296098 18:fd:74:05:15:d6 (oui Unknown) > Broadcast, ethertype Unknown (0x9003), length 64:
    0x0000:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0010:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
    0x0030:  0000
Though I don't see any return from the router...
 
Code:
13:30:47.984927 ARP, Request who-has 10.102.1.254 tell 10.102.1.1, length 46 
13:30:47.985069 ARP, Reply 10.102.1.254 is-at 58:9c:fc:0f:99:39 (oui Unknown), length 28
There's some layer 2 connectivity. Router wants to know who 10.102.1.254 is and gets a response. Is it possible the router is blocked from responding to ICMP requests from 10.102.1.0/24? Firewall on the router perhaps?
 
When I assign 10.102.1.11 to the server D1on another interface and set the default route to 10.102.1.1 on the router R1, S1 is able to ping R1. Same when I do it for mce2 standaolne. The issue is only happening when mce2 is put in the bridge . Not sure what's going on still
 
When I assign 10.102.1.1 to the server D1
This causes an IP address conflict. Your router already has that address.
dhcp is running in a mikrotik router with 10.102.1.1 address.

On that router it's vlan 102 untagged to the machine.
Are you sure it's untagged? If it's tagged then you should probably add a vlan(4) to mce2 and attach the mce2.102 vlan(4) interface to the vm-public bridge. It's really easy to lose track of which traffic is VLAN encapsulated and which traffic isn't.
 
In such case it wouldn't get the DHCP announcements.
True. But there might be some other things messed up, causing it to respond to tagged/untagged traffic. Or some "dhcp-helper" on a switch or router somewhere picking up the broadcast and relaying it to the DHCP server.

I have tried anyway to use the vlan but it doesn't work. This may be related though...
Check your switch settings. Make absolutely sure everything is untagged and the switch ports themselves are set correctly too.

This was a typo, sorry for that. I fixed it.
Understood. But if the gateway address is on the host, then there's nothing being sent out on the actual interface. Nothing would leave the host.
 
so i have tried to just use the device as a full passthrough device and everything is working as expected (though a little slow). I suspect a bug with the bridge. I will open a ticket. thanks for the help anyway :)
 
Back
Top