How attach a bhyve guest to a bridge?

[PMc]

I have a bridge running in this way:

other_machine ----- | physif --  BRIDGE  --- HOST
other_machine ----- | physif --          --- jail1
                                         --- jail2
                                         --- guest1

The guest1 is newly added and should connect as shown.

  Name: lanswitch       Type: bridge          ID: 00000009   Num hooks: 7
  Local hook      Peer name       Peer type    Peer ID         Peer hook      
  ----------      ---------       ---------    -------         ---------      
  link16          nge_jail1_1l    eiface       0000003a        ether          
  link17          nge_jail2_1l    eiface       00000032        ether          
  link4           tap1            ether        00000028        lower          
  link3           nge_host_1u     eiface       00000015        ether          
  link2           re0             ether        00000001        lower          
  link1           fxp3            ether        00000005        lower          
  link0           fxp1            ether        00000003        lower

The problem: jail1 and jail2 cannot see each other anymore, and report each other as Host is down.
All the others can see and connect everything (except tap1/guest1, which is not yet configured). But ARP requests from jail1 do not appear at jail2, and vice versa (everybody else does see them).
When I disconnect tap1, everything starts to work correctly again. Also when I connect tap1 to the bridge after the jails, things do work.

What is happening here? Does the bridge walk it's links in the order of creation (from bottom to top in the list) to broadcast the arp, gets some hiccup at link4 (because probably something is wrong with tap1), and then forgets to broadcast the remaining links?
I was thinking I can attach a tap to a bridge just like any physical interface, i.e. connect the lower wire.

 

[PMc]

boah, this is crazy. The outage gets fixed with this:

root@guest1:~ # ifconfig vtnet1 up
root@guest1:~ # ifconfig vtnet1 down

And it comes back after restarting the guest and jails, after a few minutes.

Lets look into this. Here is the guest config:

guest_1_pci_2="virtio-blk,/dev/zvol/im/gvol.conr"
guest_1_pci_3="virtio-blk,/dev/zvol/bm/gvol.conr"
guest_1_pci_4=virtio-net,tap0,mac=06:1d:92:01:03:01
guest_1_pci_5=virtio-net,tap1,mac=06:1d:92:01:03:02
guest_1_ifcreate="tap0 tap1"
guest_1_ifconfig_tap0="inet 192.168.xx.xx netmask 0xfffffffc"
guest_1_ngbridge_tap1="lanswitch"

tap0<=>vtnet0 is inserted in layer3 of the host, there is no bridging, and it gets configured in the guest and then it just works.
But tap1<=>vtnet1 is only attached to the "lanswitch:" ng_bridge(4), and nothing done inside the guest (because I don't need it yet). It is net.link.tap.up_on_open=1 on the host side, and never touched on the vtnet1 guest side. So it obviousely stays down - and that makes the bridge behave crazy!

I think this are at least two bugs - and two liabilities. Technically a tun device is an emulated netif, and should behave like a physical network card. So the other end (which in this case gets software-emulated by bhyve) should be a wire and a network. But this is not how it is done. Instead, bhyve creates another network interface at the other end, which then gets presented as vtnetX to the guest. So what technically should be a network (bus topology), is made into a point-to-point connection. This is ugly and mind-boggling. (From what I have seen, in Rel.13 should be a direct interface from bhyve into netgraph; I hope that allows to get rid of the tap devices.)

The other liability goes back to FastEthernet, when twisted-pair point-to-point wiring became necessary, while the Ethernet logic (that assumes a bus wiring) was kept. That doesn't match, and now switches are needed everywhere, and in the end the natural consequence is that they don't work correctly. Because: what shall we do when the other end goes down? In a point-to-point situation we take our end down as well, but in a bus we just do nothing and pass on. Now in fact we do something in-between that doesn't work.