Solved Network problem (after doas vm switch create public; doas doas vm switch add public em0)

rsronin · Mar 29, 2022

Lat weekend I was experimenting with bhyve and trying to setup a debian guest. I was not on my home network so I thought the problems where due to this. Although I managed to setup a debian guest once when the guest detected dhcp. I was using churchers/vm-bhyve. The network was only detected during the debian installer phase. Afterward no network.

Yesterday I started from the scratch on my 'own home network' but it seems to getting worse. I gave my thinkpad t430 a fixed ip using ifconfig_em0="inet 192.168.1.43 netmask 255.255.255.0" and defaultrouter="192.168.1.1" in rc.conf. I disabled wifi and removed an usb adapter which seemed to give me an ue0 interface so basicly only ethernet em0 from the dock the t430 is in.

I attached some output from the commands in the terminal.

Thanks in advance.

SirDice · Mar 29, 2022

service netif restart removes any and all routes associated with those interfaces. You need to run service routing restart after it to restore your default gateway.

rsronin · Mar 29, 2022

I tried that too before and just now, which was forcing me to reboot because a lost connection and not being able to perform a service routing restart because of this:

Code:

got only -1 for rlen
route: write to routing socket: Operation not supported
got only -1 for rlen
route: write to routing socket: Operation not supported
got only -1 for rlen
route: write to routing socket: Operation not supported
got only -1 for rlen
route: write to routing socket: Operation not supported

See also at the end of the attached log.txt in the first post.

BTW, this is after a reboot:

Code:

amw@t430 ~> netstat -rn                                                                                                           1
Routing tables

Internet:
Destination        Gateway            Flags     Netif Expire
default            192.168.1.1        UGS         em0
127.0.0.1          link#2             UH          lo0
192.168.1.0/24     link#1             U           em0
192.168.1.43       link#1             UHS         lo0

Internet6:
Destination                       Gateway                       Flags     Netif Expire
::/96                             ::1                           UGRS        lo0
::1                               link#2                        UHS         lo0
::ffff:0.0.0.0/96                 ::1                           UGRS        lo0
fe80::/10                         ::1                           UGRS        lo0
fe80::%lo0/64                     link#2                        U           lo0
fe80::1%lo0                       link#2                        UHS         lo0
ff02::/16                         ::1                           UGRS        lo0
amw@t430 ~> cd /etc
amw@t430 /etc> cat resolv.conf
# Generated by resolvconf
search home
nameserver 192.168.1.1
nameserver 195.121.1.34
nameserver 195.121.1.66

amw@t430 /etc>

rsronin · Mar 29, 2022

Some more observations. After a fresh reboot and:
sudo service netif restart && sudo service routing restart
Network still not availlabe.
After:

doas setfib 0 route add 192.168.1.0/24 -iface em0
doas route add default 192.168.1.1

No difference.

After another reboot I tried an Ubuntu guest in Virtualbox. While this was working until recently, it won't startup now.

SirDice · Mar 29, 2022

Try putting the IP address of the host on the bridge interface instead of em0.

rsronin · Mar 29, 2022

SirDice said:
Try putting the IP address of the host on the bridge interface instead of em0.

Understanding networking in general is difficult for me, so this answer confuses me even more. At the moment the bridge doesn't exist, so probably something wrong with with the network in general.

amw@t430 ~> ifconfig
em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=481249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
    ether 28:d2:44:44:0a:60
    inet 192.168.1.43 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
amw@t430 ~> sudo service netif restart && sudo service routing restart
Password:
Sorry, try again.
Password:
Stopping Network: lo0 em0.
lo0: flags=8048<LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=8c22<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=481249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
    ether 28:d2:44:44:0a:60
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
Starting Network: lo0 em0.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0: flags=8c22<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=481249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
    ether 28:d2:44:44:0a:60
    media: Ethernet autoselect
    status: no carrier
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
delete host 127.0.0.1: gateway lo0
route: route has not been found
delete net default: gateway 192.168.1.1 fib 0: not in table
delete host ::1: gateway lo0
delete net fe80::: gateway ::1
delete net ff02::: gateway ::1
delete net ::ffff:0.0.0.0: gateway ::1
delete net ::0.0.0.0: gateway ::1
add host 127.0.0.1: gateway lo0
route: writing to routing socket: Network is unreachable
add net default: gateway 192.168.1.1 fib 0: Network is unreachable
Additional inet routing options: gateway=YES.
add host ::1: gateway lo0
add net fe80::: gateway ::1
add net ff02::: gateway ::1
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
amw@t430 ~> ifconfig                                                                                                              1
em0: flags=8c22<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=481249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
    ether 28:d2:44:44:0a:60
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
amw@t430 ~>
amw@t430 ~/Documents>

rsronin · Mar 31, 2022

Decided to start from scratch, so not using vm-bhyve. Checked ethernet cables, rebooting de router from de isp again and again, still a lot of troubles figuring out getting a network connection for the guest. Succeeded a few times but could not duplicate so again no network connection(s) available. Today succeeded and could reproduce, but need to reboot the host for a fresh start and bring up the bridge and the tap explicitly with a script. After closing the guest still loosing the em0 connection even after 'sudo service netif restart && sudo service routing restart', sudo ifconfig em0 up, sudo ifconfig em0 192.168.1.43. Learned a lot, but with little understanding. Proof of the pudding atttached.

rsronin · Apr 28, 2022

Last weeks problems with both virtualbox and bhyve.

With respect to bhyve, I cleaned /boot/loader.conf, /etc/rc.conf and /etc/sysctl.conf and made 2 simple scripts which a run before starting starting the guest.

 

hostname="t430"

# AMS

#ifconfig_em0="DHCP"

# bhyve basics

defaultrouter="192.168.1.1"

#cloned_interfaces="bridge0 tap0"

#ifconfig_bridge0_name="em0bridge"

#ifconfig_em0bridge="addm em0 addm tap0"

#ifconfig_bridge0="addm em0 addm tap0"

ifconfig_em0="inet 192.168.1.43 netmask 255.255.255.0"

#ifconfig_em0bridge="inet 192.168.1.33 netmask 255.255.255.0"

#ifconfig_bridge0="inet 192.168.1.33 netmask 255.255.255.0"

#ifconfig_em0="DHCP"

#ifconfig_em0bridge="DHCP"

#ifconfig_bridge0="DHCP"

 

#!/bin/sh



doas kldload if_tap

doas kldload if_bridge

doas kldload bridgestp

doas kldload vmm

doas kldload nmdm



doas sysctl net.link.tap.up_on_open=1

doas sysctl net.inet.ip.forwarding=1



doas ifconfig tap0 create

doas ifconfig bridge0 create

doas ifconfig bridge0 addm em0 addm tap0

doas ifconfig bridge0 up

doas ifconfig bridge0 192.168.1.33/24

doas ifconfig em0 up

doas ifconfig em0 192.168.1.43/24

 

#!/bin/sh

doas arp -d -a

doas service netif restart && doas service routing restart

doas setfib 0 route add 192.168.1.0/24 -iface em0

doas setfib 0 route add default 192.168.1.1

doas ifconfig em0 up

doas ifconfig bridge0 up

doas ifconfig tap0 up

doas ifconfig em0 192.168.1.43/24

doas ifconfig bridge0 192.168.1.33/24

ifconfig

After bhyvectl --destroy --vm=debian I don't need to reboot the ISP router or the Thinkpad t430 and I am able to start the debian guest again.

 

amw@t430 ~/bin> cat sdvm.sh

#!/bin/sh

doas bhyve -c 2 -m 2048M -w -H \

-s 0,hostbridge \

-s 4,virtio-blk,/vm/debian/debian.img \

-s 5,virtio-net,tap0 \

-s 6,hda,play=/dev/dsp2,rec=/dev/dsp0 \

-s 29,fbuf,tcp=0.0.0.0:5900,w=1600,h=900,wait \

-s 30,xhci,tablet \

-s 31,lpc -l com1,stdio \

-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \

debian



amw@t430 ~/bin>

Solved Network problem (after doas vm switch create public; doas doas vm switch add public em0)

rsronin

Attachments

SirDice

Administrator

rsronin

rsronin

SirDice

Administrator

rsronin

rsronin

Attachments

rsronin