bhyve guest (win10) network access depends on startup procedure

Hi there,

I discovered a strange behavior of the client network interface tap and an established bridge to the real networking device (em0). Each configuration (/etc/rc.conf...) is done as described in several howtos.
Client is a win 10 with Red Hat Virtio Eth Adapter, Driver 100.74.104.14100, date 19 jul 2017.
Older client-drivers are recommend (e.g. ...1.118), but has this has no effect.

Host: uname -a
Code:
FreeBSD  11.1-RELEASE FreeBSD 11.1-RELEASE #0 r321309: Fri Jul 21 02:08:28 UTC 2017     root@releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64

Creating the bridge _after_ starting bhyve will succeed, otherwise it fails.

It fails:
1.) creating the bridge with ifconfig bridge0 create ->bridge0 exists
2.) starting bhyve ... -s 5,virtio-net,tap0...
3.) Result: guest can't access network (DHCP is active, IP is class-b! with 255.255.0.0 as netmask!

It works:
1.) starting bhyve ... -s 5,virtio-net,tap0...
2.) now creating the bridge with ifconfig bridge0 create -> bridge0 exists
3.) voilà, everything is fine.

Any hints?
Thank you!
 
I assume you're adding em0 and tap0 to the bridge, and just haven't listed that here.

Do you have the sysctl

net.link.tap.up_on_open=1

set? What about firewalls?
 
Hello,
i took this blog for my configuration:
https://www.ateamsystems.com/tech-blog/howto-windows-10-bhyve-w-freebsd-11/

firewall: nope; connection depends on the sequence, (please see above) - thats why i'm confused

sysctl.conf contains:
Code:
# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
kern.randompid=7152
hw.syscons.bell=0

kern.vt.enable_bell=0
#bhyve networking
net.link.tap.up_on_open=1
net.inet.ip.forwarding=1

rc.conf:
Code:
#hostname="thinkpad"
keymap="de"
ifconfig_em0="DHCP"
ifconfig_em0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable

dumpdev="AUTO"
zfs_enable="YES"

hald_enable="YES"
dbus_enable="YES"
#wlans_iwn0="wlan0"
ifconfig_wlan0="WPA SYNCDHCP"
#Simple Login MGR
#slim_enable=yes
#networking bhyve
cloned_interface="bridge0 tap0"
ifconfig_bridge0="addm em0 addm tap0"

sysctl -a | grep "tap" :
Code:
net.link.tap.debug: 0
net.link.tap.devfs_cloning: 1
net.link.tap.up_on_open: 1
net.link.tap.user_open: 0
debug.if_tap_debug: 0
hw.psm.tap_timeout: 125000
hw.psm.tap_threshold: 25
hw.psm.tap_enabled: -1

Installed, but unused:
vm version
Code:
vm-bhyve: Bhyve virtual machine management v1.1-p8 (build 101073)

It is a fresh install of Freebsd, and i installed only things which seems to be necesary...
But something is missing :-(
Do i have to perform the basic setup for vm?
it's not configured until now.
my next trial: choosing another client-adapter (ne1000); report follows.
 
And try this out:


ifconfig_em0="up promisc"
ifconfig_bridge0_ipv6="inet6 accept_rtadv"
sshd_enable="YES"
powerd_enable="YES"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable

dumpdev="AUTO"
zfs_enable="YES"

hald_enable="YES"
dbus_enable="YES"
#wlans_iwn0="wlan0"
#ifconfig_wlan0="WPA SYNCDHCP"
#Simple Login MGR
#slim_enable=yes
#networking bhyve
cloned_interface="bridge0 tap0"
ifconfig_bridge0="DHCP addm em0 addm tap0 up"


Not sure it will change anything, but worth a shot; I’ve had more consistency (I’m also doing pf filtering, which adds another variable) with the IP assigned to the bridge; I’m using a static IP, but hopefully this config will work, too. ;)

You also do not need the forwarding sysctl set when using a bridge, unless you are forwarding between bridge0 and (commented out here) wlan0, as I recall.
 
Thank you for your assistance.
aragats:
Code:
bhyve -c 2 -m 4G -H -w -s 0,hostbridge -s 3,ahci-cd,null_dvd.iso -s 4,ahci-hd,w10_disk.img -s 5,virtio-net,tap0 -s 29,fbuf,tcp=0.0.0.0:5900,wait -s 30,xhci,tablet -s 31,lpc -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd win10

i figuered out the following:
When you startup the bridge before the tap-interface exits, tap0 won't become a member of the bridge automatically, look here:

bhyve -c 2 -m 4G ....
-----------------
ifconfig
Code:
em0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC> metric 0 mtu 1500
        options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
        ether 00:21:cc:6b:db:d6
        hwaddr 00:21:cc:6b:db:d6
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:80:37:ec:02:00
        hwaddr 02:80:37:ec:02:00
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:ce:1b:cb:b9:00
        nd6 options=3<PERFORMNUD,ACCEPT_RTADV>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 200000
tap0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:0c:ae:f8:00
        hwaddr 00:bd:0c:ae:f8:00
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        groups: tap
        Opened by PID 90733
$
ps -xa
...
34530 0 Is 0:00.01 sh
90733 0 I+ 2:39.03 bhyve: win10 (bhyve)
31669 1 Ss 0:00.01 sh
...
-----------------
tap0 isn't listed, result: no network-access

ifconfig bridge0 addm tap0
makes tap0 to a member of the bridge; also the client interface status changes instantanly from no network to fully functional network access!

ifconfig
Code:
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=42098<VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO>
        ether 00:21:cc:6b:db:d6
        hwaddr 00:21:cc:6b:db:d6
        inet6 fe80::221:ccff:fe6b:dbd6%em0 prefixlen 64 scopeid 0x1
        inet 192.168.178.61 netmask 0xffffff00 broadcast 192.168.178.255
        nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
        media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:80:37:ec:02:00
        hwaddr 02:80:37:ec:02:00
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:ce:1b:cb:b9:00
        nd6 options=9<PERFORMNUD,IFDISABLED>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 200000
tap0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=80000<LINKSTATE>
        ether 00:bd:a1:6b:f8:00
        hwaddr 00:bd:a1:6b:f8:00
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet autoselect
        status: active
        groups: tap
        Opened by PID 96071

is this behavior of the bridge as expected?
 
If you’ve destroyed tap0 after its initial (on boot / init processing) creation, it won’t automatically become a member of bridge0 when it is recreated, if that’s what you are asking. (It should be a member after boot with the above rc.conf settings.) Opening and closing (but not destroying) the tap should leave it as a member.
 
Back
Top