bhyve The host and bhyve guest cannot ping each other

I've set up a Windows 10 guest in bhyve. I used to be able to remote desktop to the Windows via freerdp. But it cannot now. The host and the guest are not able to ping each other now. I don't actually realize what I have done. This must be something happening on the host, because I also have Linux guest and there is the same issue between the host and Linux guest.

So I would ask this forum if anybody can shed a light what may be happening.

The problem is that the host and guest cannot ping each other. But both can access the local network and the Internet without any issues.

Here is how I set up the switch for the guests. I don't use NAT.

Bash:
vm switch create public
vm switch add public ue0

Bash:
~ $ vm switch info public
local: _netgraph,: bad variable name
------------------------
Virtual Switch: public
------------------------
  type: standard
  ident: vm-public
  vlan: -
  physical-ports: ue0
  bytes-in: 17404456 (16.598M)
  bytes-out: 26749967 (25.510M)

  virtual-port
    device: tap0
    vm: win

All my network interfaces
Bash:
~ $ ifconfig 
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ue0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=280099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE,RXCSUM_IPV6>
        ether a0:ce:c8:07:29:d6
        inet 192.168.0.4 netmask 0xffffff00 broadcast 192.168.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ue1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 9c:eb:e8:92:16:c2
        inet 192.168.0.5 netmask 0xffffff00 broadcast 192.168.0.255
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether f6:23:81:8c:41:7c
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
        member: ue0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 20000
        groups: bridge vm-switch viid-4c918@
        nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet/win/0/public
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:3a:0c
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 76313

If there is any change I can recall that is remotely connected, that may the change from the on-board NIC to the usb NIC. The on-board NIC stopped working so I started to use the USB nic ue0.

Other than that, I don't recall I change anything about the networking on the host.

If you have any direction I can continue, please let me know. Thanks so much.
 
You have two interfaces (ue0 and ue1) on the same subnet.
 
As far as I know, any modern windows has blocked "ping requests" by default.
You should enable something like ICMP echo request/reply on windows firewall.
Try to ping any freshly-installed non-virtual windows on non-virtual network.
 
As far as I know, any modern windows has blocked "ping requests" by default.
You should enable something like ICMP echo request/reply on windows firewall.
Try to ping any freshly-installed non-virtual windows on non-virtual network.
Though I use Windows here, the point the host and the guest don't connect. As i mentioned in the original post, the same thing happens to a Linux guest.
 
You have two interfaces (ue0 and ue1) on the same subnet.
Can you please elaborate why that is an issue? Sorry that I don't understand that from my little knoeledge about subnet and routing.

Both ue0 and ue1 are using static ip addresses now. I didn't configure ue1 (static or dhcp) before. It's set that way when I was experimenting which one to use (after the onboard nic broke) . Now only ue0 connects to an ethernet cable and ue1 does not. It doesn't seem to affect anything so I didn't revert the change.

The guest also has its own ip address like 192.168.0.140 (dhcp) It can connect to other local network device except the host. So now from my understanding about routing, the guest can connect to other devices in the local network (using dhcp or static ip), the route from the guest to the local network probably is ok.

The host can aslo reach other devices in the same local network too. And other devices in the local network can aslo connect to the host. So the routing from or to the host is probably ok too.

If the same routing mechanism is used in the local network, it shouldn't block the connection between the host and the guest. I do not do that or know how to do it.

That is what I know about the routing. I don't know where to go from it.
 
As far as I know, any modern windows has blocked "ping requests" by default.
You should enable something like ICMP echo request/reply on windows firewall.
Try to ping any freshly-installed non-virtual windows on non-virtual network.
I do not know that Windows blocks ping. Thanks for that. I will not test using ping. My question is not ping Windows. My question is that the host and the guest cannot connect.
 
I remove the configuration for `ue1`, and recreate the vm switch. Here is what it is now.
Bash:
~ $ sudo vm switch destroy public
~ $ sudo vm switch create public
~ $ sudo vm switch add public ue0
~ $ sudo vm switch info public
local: _netgraph,: bad variable name
------------------------
Virtual Switch: public
------------------------
  type: standard
  ident: vm-public
  vlan: -
  physical-ports: ue0
  bytes-in: 11356 (11.089K)
  bytes-out: 105097 (102.633K)

Bash:
~ $ ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ue1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 9c:eb:e8:92:16:c2
        media: Ethernet autoselect
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
ue0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=280099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LINKSTATE,RXCSUM_IPV6>
        ether a0:ce:c8:07:29:d6
        inet 192.168.0.4 netmask 0xffffff00 broadcast 192.168.0.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether fe:4d:4e:61:c3:ea
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
        member: ue0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 20000
        groups: bridge vm-switch viid-4c918@
        nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vmnet/xubuntu/0/public
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:3a:0c
        groups: tap vm-port
        media: Ethernet autoselect
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 30571

In a Linux guest, it has its own ip address
Bash:
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 58:9c:fc:0a:f4:57 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.179/24 brd 192.168.0.255 scope global dynamic noprefixroute enp0s5
       valid_lft 7186sec preferred_lft 7186sec
    inet6 fe80::b44f:4488:c30a:32af/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

The Linux guest can ping/ssh other devices in the local network. Just it cannot ping/ssh to the host. Other devices in the network can ping/ssh to the guest.

The same to the host. It can ping/ssh to other devices in the local network, just not the guest. Other devices in the network can ping/ssh to the host.
 
Can you please elaborate why that is an issue? Sorry that I don't understand that from my little knoeledge about subnet and routing.
Routing will be ambiguous. Both interfaces have the same 'directly connected' network and thus it's not clear which of the interfaces should be used to contact another system within that network. Packets may be received on ue1 and the responses might be sent out on eu0, that's usually not what you want to happen.
 
Routing will be ambiguous. Both interfaces have the same 'directly connected' network and thus it's not clear which of the interfaces should be used to contact another system within that network. Packets may be received on ue1 and the responses might be sent out on eu0, that's usually not what you want to happen.
Thanks for clarification. I replied before yours with an update of my configuration. `ue1` never connects to an Ethernet cable. In my previous reply, it's not configured.

And now, I disconnect the usb for `ue1` from the host so there is no `ue1`. I recreate the vm switch and I still cannot have the host and the guest to communicate.
 
Thanks for clarification. I replied before yours with an update of my configuration. `ue1` never connects to an Ethernet cable. In my previous reply, it's not configured.

And now, I disconnect the usb for `ue1` from the host so there is no `ue1`. I recreate the vm switch and I still cannot have the host and the guest to communicate.
Same issue. Have you resolved that somehow please? Even FreeBSD guest and host cannot ping each other. They both get their DHCP IPv4 address correctly and networking works. It's just that routing issue.
 
what's the output of
Code:
netstat -rnl
on the guest and the host.
Host:
Code:
~> netstat -rnl
Routing tables

Internet:
Destination        Gateway            Flags   Nhop#    Mtu      Netif Expire
default            192.168.0.1        UGS         4   1500        ue0
127.0.0.1          link#1             UH          1  16384        lo0
192.168.0.0/24     link#2             U           2   1500        ue0
192.168.0.11       link#1             UHS         3  16384        lo0

Internet6:
Destination                       Gateway                       Flags   Nhop#    Mtu    Netif Expire
::/96                             link#1                        URS         4  16384      lo0
::1                               link#1                        UHS         1  16384      lo0
::ffff:0.0.0.0/96                 link#1                        URS         4  16384      lo0
fe80::%lo0/10                     link#1                        URS         4  16384      lo0
fe80::%lo0/64                     link#1                        U           3  16384      lo0
fe80::1%lo0                       link#1                        UHS         2  16384      lo0
ff02::/16                         link#1                        URS         4  16384      lo0

Guest:
Code:
[root@ ~]# netstat -rnl
Routing tables

Internet:
Destination        Gateway            Flags   Nhop#    Mtu      Netif Expire
default            192.168.0.1        UGS         4   1500     vtnet0
127.0.0.1          link#2             UH          1  16384        lo0
192.168.0.0/24     link#1             U           2   1500     vtnet0
192.168.0.112      link#2             UHS         3  16384        lo0

Internet6:
Destination                       Gateway                       Flags   Nhop#    Mtu    Netif Expire
::/96                             link#2                        URS         4  16384      lo0
::1                               link#2                        UHS         1  16384      lo0
::ffff:0.0.0.0/96                 link#2                        URS         4  16384      lo0
fe80::%lo0/10                     link#2                        URS         4  16384      lo0
fe80::%lo0/64                     link#2                        U           3  16384      lo0
fe80::1%lo0                       link#2                        UHS         2  16384      lo0
ff02::/16                         link#2                        URS         4  16384      lo0
 
This is my /etc/jail.conf file :
Code:
# Common configs for all jails
path = "/jails/$name";
host.hostname = "$name";
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
persist;
ip4 = inherit;
ip6 = inherit;
mount.devfs;
mount.fdescfs;
allow.mlock;
allow.mount;
allow.mount.devfs;
allow.mount.fdescfs;
allow.mount.nullfs;
allow.mount.tmpfs;
allow.mount.procfs;
allow.mount.zfs;
enforce_statfs=1;
children.max=100;
allow.socket_af;
allow.raw_sockets;
allow.chflags;
allow.sysvipc;
a {
devfs_ruleset="20";
}

ping uses ICMP . For me it works.
 
This is my /etc/jail.conf file :
Thanks Alain, btw, I don't jail them. Could that make any difference? Ofc yes, but any idea? I just have two pretty barebone VMs. I just run them either via vmrun.sh or bhyve directly inside a tmux session. SSH from other machines just works, only host-guest routing is wrong.
 
Moreover, I get random network outages with 2 different Realtek USB ethernet dongles put into my Lenovo P1. Sigh...
 
When I ping 8.8.8.8 from the guest, it works, when I ping the bridged peer -- the host's IP 192.168.0.11, it does not. But interestingly, tcpdump on the host provides the same information in both cases. Also more complex stuff like SSH does not work from the host (192.168.0.11) to the guest (192.168.0.30).

My yet unanswered concern is, whether
Code:
? (192.168.0.30) at 00:a0:98:54:14:18 on ue0 expires in 697 seconds [ethernet]
is a correct ARP.

Guest: ping 8.8.8.8 and ping 192.168.0.11
Code:
root@bsd-dev:~ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=114 time=51.606 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=114 time=38.860 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=114 time=37.610 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=114 time=40.084 ms
^C
--- 8.8.8.8 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 37.610/42.040/51.606/5.592 ms
root@bsd-dev:~ # ping 192.168.0.11
PING 192.168.0.11 (192.168.0.11): 56 data bytes
^C
--- 192.168.0.11 ping statistics ---
4 packets transmitted, 0 packets received, 100.0% packet loss
root@bsd-dev:~ #

Host: tcpdump -i bridge0
Code:
doas tcpdump -i bridge0 host 192.168.0.30 and not arp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on bridge0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
21:34:12.978473 IP 192.168.0.30 > dns.google: ICMP echo request, id 53506, seq 0, length 64
21:34:13.029653 IP dns.google > 192.168.0.30: ICMP echo reply, id 53506, seq 0, length 64
21:34:14.061157 IP 192.168.0.30 > dns.google: ICMP echo request, id 53506, seq 1, length 64
21:34:14.099578 IP dns.google > 192.168.0.30: ICMP echo reply, id 53506, seq 1, length 64
21:34:15.148638 IP 192.168.0.30 > dns.google: ICMP echo request, id 53506, seq 2, length 64
21:34:15.185845 IP dns.google > 192.168.0.30: ICMP echo reply, id 53506, seq 2, length 64
21:34:16.228528 IP 192.168.0.30 > dns.google: ICMP echo request, id 53506, seq 3, length 64
21:34:16.266504 IP dns.google > 192.168.0.30: ICMP echo reply, id 53506, seq 3, length 64
21:34:23.313616 IP 192.168.0.30 > 192.168.0.11: ICMP echo request, id 54530, seq 0, length 64
21:34:23.313640 IP 192.168.0.11 > 192.168.0.30: ICMP echo reply, id 54530, seq 0, length 64
21:34:24.384172 IP 192.168.0.30 > 192.168.0.11: ICMP echo request, id 54530, seq 1, length 64
21:34:24.384176 IP 192.168.0.11 > 192.168.0.30: ICMP echo reply, id 54530, seq 1, length 64
21:34:25.479369 IP 192.168.0.30 > 192.168.0.11: ICMP echo request, id 54530, seq 2, length 64
21:34:25.479391 IP 192.168.0.11 > 192.168.0.30: ICMP echo reply, id 54530, seq 2, length 64
21:34:26.559563 IP 192.168.0.30 > 192.168.0.11: ICMP echo request, id 54530, seq 3, length 64
21:34:26.559580 IP 192.168.0.11 > 192.168.0.30: ICMP echo reply, id 54530, seq 3, length 64
 
This is what I have on the host:
ifconfig_bridge0="192.168.x.y/24" # "DHCP" should also work
cloned_interfaces="bridge0"
autobridge_interfaces="bridge0"
autobridge_bridge0="tap* igb0"


For each guest I have something like this (run on the host):
bhyve ... -s 10,e1000,tap5,mac=aa:bb:cc:dd:ee:ff ... # uniq tap device for each guest


On a guest (FreeBSD)
ifconfig_em0="DHCP"
 
This is what I have on the host:
ifconfig_bridge0="192.168.x.y/24" # "DHCP" should also work
cloned_interfaces="bridge0"
autobridge_interfaces="bridge0"
autobridge_bridge0="tap* igb0"
Now it's getting interesting because I thought that bridges shouldn't have IPs assigned by definition because they operate on Level 2.
 
Back
Top