VM-bhyve no network

teisho · May 15, 2021

I am trying to get the network of my VM to work, but I can't find the error.
The goal is to use a single public IPv6 address for this. No NAT.
It was working until I did service netif restart && service routing restart
What am I missing?

Bash:

root@uruk:~ # vm switch list
NAME    TYPE      IFACE      ADDRESS  PRIVATE  MTU  VLAN  PORTS
public  standard  vm-public  -        no       -    -     em0

Bash:

root@uruk:~ # vm switch info public
------------------------
Virtual Switch: public
------------------------
  type: standard
  ident: vm-public
  vlan: -
  physical-ports: em0
  bytes-in: 6705683049 (6.245G)
  bytes-out: 49415285443 (46.021G)

  virtual-port
    device: tap0
    vm: valheim

Code:

root@uruk:~ # ifconfig 
[…]
vm-public: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 96:3b:b8:50:3b:d4
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000000
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge vm-switch viid-4c918@
    nd6 options=9<PERFORMNUD,IFDISABLED>
    
    tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vmnet-valheim-0-public
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:ff:a6
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 83468

Bash:

root@uruk:~ # vm info valheim
------------------------
Virtual Machine: valheim
------------------------
  state: running (83468)
  datastore: default
  loader: grub
  uuid: 9d130302-8f28-11eb-b855-08606e45db5c
  uefi: default
  cpu: 4
  memory: 4096M
  memory-resident: 27258880 (25.996M)

  console-ports
    com1: /dev/nmdm-valheim.1B

  network-interface
    number: 0
    emulation: virtio-net
    virtual-switch: public
    fixed-mac-address: 58:9c:fc:08:c5:86
    fixed-device: -
    active-device: tap0
    desc: vmnet-valheim-0-public
    mtu: 1500
    bridge: vm-public
    bytes-in: 0 (0.000B)
    bytes-out: 153692 (150.089K)

  virtual-disk
    number: 0
    device-type: file
    emulation: ahci-hd
    options: -
    system-path: /vms/valheim/disk0.img
    bytes-size: 214748364800 (200.000G)
    bytes-used: 1475526656 (1.374G)

teisho · May 15, 2021

Forget it… I recreated the switch and now it's working…
Edit: After half an hour, it stops working...

SirDice · May 15, 2021

teisho said:
It was working until I did service netif restart && service routing restart
What am I missing?

sysutils/vm-bhyve has its own rc(8) script that adds the bridge and sets up various things. By restarting netif you basically removed em0 from the bridges.

teisho · May 16, 2021

Strangely, the VM loses the internet connection after a while.
However, I can reach it from the host. Any ideas?

cmoerz · May 16, 2021

What does ifconfig say about your setup, once internet connectivity is lost? Are both - em0 and the VM's tap interface - still member of the bridge? Is your bridge still UP? Not seeing that in your listing?

You should be able to up your bridge using ifconfig vm-public up.

teisho · May 16, 2021

vm-public is up and em0 and the tap interfaces are members.

Bash:

root@uruk:~ # ifconfig
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
    ether 08:aa:bb:cc:dd
    inet 5.xx.xxx.165 netmask 0xffffffe0 broadcast 5.xx.xxx.191
    inet 5.xx.xxx.155 netmask 0xffffffff broadcast 5.xx.xxx.155
    inet 5.xx.xxx.153 netmask 0xffffffff broadcast 5.xx.xxx.153
    inet 5.xx.xxx.158 netmask 0xffffffff broadcast 5.xx.xxx.158
    inet 5.xx.xxx.175 netmask 0xffffffff broadcast 5.xx.xxx.175
    inet6 fe80::xxx:xxxx:xxx:db5c%em0 prefixlen 64 scopeid 0x1
    inet6 f00:xxx:xxx:xxx::2 prefixlen 64
    inet6 f00:xxx:xxx:xxx::f00 prefixlen 64
    inet6 f00:xxx:xxx:xxx::23 prefixlen 64
    inet6 f00:xxx:xxx:xxx::42 prefixlen 64
    inet6 f00:xxx:xxx:xxx::beef prefixlen 64
    inet6 f00:xxx:xxx:xxx::cafe prefixlen 64
    inet6 f00:xxx:xxx:xxx::10 prefixlen 64
    inet6 f00:xxx:xxx:xxx::1337 prefixlen 64
    inet6 f00:xxx:xxx:xxx::6 prefixlen 64
    inet6 f00:xxx:xxx:xxx::1 prefixlen 64
    inet6 f00:xxx:xxx:xxx::4 prefixlen 64
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo1: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet 172.16.1.2 netmask 0xffffffff
    inet 172.16.1.3 netmask 0xffffffff
    inet 172.16.1.4 netmask 0xffffffff
    groups: lo
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vm-public: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether ce:78:17:36:6e:4f
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 2000000
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge vm-switch viid-4c918@
    nd6 options=1<PERFORMNUD>
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vmnet-valheim-0-public
    options=80000<LINKSTATE>
    ether 58:9c:fc:10:ff:a6
    groups: tap vm-port
    media: Ethernet autoselect
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
    Opened by PID 42251

cmoerz · May 16, 2021

I suppose, this might be some IPv6 related issue then. I've read of cases, where connectivity was lost due to how the packets were switched (i.e. at Hetzner, see a related entry at stackoverflow).

IPv6 connectivity lost on KVM guest after 20 minutes

I have a KVM virtualization server setup at Hetzner. Hetzner provides me with a master IP (95.xxx.xxx.235) and a /29 IPv4 subnet (95.xxx.xxx.184/29) and a /64 IPv6 network (2a01:xxxx:xxxx:xxxx::/64...

serverfault.com

Then again, my IPv6 knowledge is rather limited and I'm afraid any advice from my end would rather send you on a wild goose chase than really fix the underlying issue. I hope some more experienced people can suggest the best approach to troubleshoot this.

VM-bhyve no network

Administrator