Bhyve running with Bastille routing bridge issues

vukodlak

vukodlak

having an issue...

Bhyve runs fine by itself.
Bastille runs fine by itself.

When I try to run them at the same time Bhyve messes up my routing completely.
I think it has something to do with Bhyve creating a 2nd bridge and you can't have 2 bridges at once?
or I need to add the bhyve bridge to the bastille bridge group maybe...
Problem is the bhyve page shows you how to set it up and make it work but it doesn't explain anything really.

here is my ifconfig without bhyve: Bastille is running in vnet mode (although yet again I do not know why, I just followed the directions)
I wish I could find a FreeBSD networking class to take...

I just want to know how to setup Bhyve networking to work with my configuration and why.

Thanks!

Code: 
em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether f8:b1:56:b7:fd:cb
        inet 192.168.1.11 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bastille0: flags=8008<LOOPBACK,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em0bridge: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:ff:fb
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: e0a_bastille1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: e0a_bastille0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
e0a_bastille0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vnet host interface for Bastille jail adguard
        options=8<VLAN_MTU>
        ether 02:20:98:b7:fd:cb
        hwaddr 02:3b:1e:6e:ff:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
e0a_bastille1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: vnet host interface for Bastille jail xxxnet
        options=8<VLAN_MTU>
        ether 02:20:99:b7:fd:cb
        hwaddr 02:7a:5f:d2:40:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

Directions I followed for Bhyve:

Code: 
Now that you know your interface name, substitute it if necessary, where we refer to “ix1” below! 

root@fbsd:/# vm init 
root@fbsd:/# vm switch create public 
root@fbsd:/# vm switch add public ix1 
We now have a [I]switch[/I] interface named [I]vm-public [/I]which will show up in future [I]ifconfig[/I] commands; the VMs that we create will automatically get new [I]tap[/I] interfaces created and assigned to them using that [I]vm-public [/I]switch.
 
Last edited by a moderator:
Just a semantic point: you don't use bhyve. You use sysutils/vm-bhyve which uses bhyve.

To answer to your question: you can have several bridges BUT an interface can't be member of more than one bridge.

In your case, I think you need only one bridge and you will use it both for jails and VMs. Given that bridge should have em0 as member if you want those jails and VMs access to your local network. All the epair interfaces from jails must be also members of that bridge as all the tap interfaces from your VMs.

How to do that with sysutils/bastille and/or sysutils/vm-bhyve, I just don't know but maybe this can help you.
 
That helps a lot actually..thank you.
Without a basic understanding of how the routing works I can't fix it.
the vm is indeed creating a seperate bridge from the jails. I just need to figure out now how to add the vm connection to the existing bridge.
 
I think you can use an existing bridge with vm switch create -b.
But, well, read the corresponding man pages and find there the inspiration.
Do your own tests and you will find something that just works. ;)
 
I doubt it's the right way, but I got it to work.
I destroyed all the interfaces from the vm-bhyve.
Created my vm and started it.
Then used this: ifconfig em0bridge addm tap1 to add the tap it creates to the bridge and then just assigned it an ip address in the fedora installer and off it went.
 
vukodlak said:
fconfig em0bridge addm tap1 to add the tap it creates to the bridge and then just assigned it an ip address in the fedora installer and off it went.
Click to expand...

alright cool vukodlak now make sure you add the bridge and tap into the /etc/rc.conf for restart purposes.

Code: 
cloned_interfaces="em0bridge tap1"
ifconfig_tap1="up"
ifconfig_em0bridge="adm tap1"
ifconfig_em0bridge="up"

I like to do everything manually (just in case vm-bhyve aka churchers-bhyve goes down or for what ever reason)

Based on what you explained you can also do something like this for bhyve with vm-bhyve and jail with BastilleBSD....

Code: 
ifconfig create tap1 inet [192.168.1.12/24] up
ifconfig em0bridge create
ifconfig epair create
ifconfig em0bridge addm tap1 addm em0 addm epair0a
ifconfig em0bridge inet 192.168.1.13/24
ifconfig em0bridge up


Above is 1 tap and 1 epair. If you are only using jails is better to do epair.

The Original Post will look something like this instead of Tap like you did.

Code: 
ifconfig em0bridge create
ifconfig epair create
ifconfig -g epair
ifconfig epair0a name e0a_bastille0 up
ifconfig epair0b name e0a_bastille1 up
ifconfig em0bridge addm e0a_bastille0
ifconfig em0bridge inet 192.168.1.13/24
ifconfig em0bridge up
jail -c name=jvnet host.hostname=jvnet persist vnet vnet.interface=e0a_bastille1

Above is essentially a sample what BastilleBSD is doing through shell scripts for you via their commands for jails.
VM-Bhyve required tap instead of epair but is similar with last command instead of jail being bhyve command.

Code: 
bhyve -s 0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 \
-s 2:1,virtio-net,tap1 \
-s 2:2,virtio-net,tap2,mac=00:be:fa:76:45:00 \
-s 3,virtio-blk,/my/image -l com1,stdio \
-A -H -P -m 24G \
bigvm
 
Any idea why i can't stop vm start fedora from creating it's own tap interface?
must have something to do with this line in the conf:

Code: 
network0_type="virtio-net"
network0_switch="public"

There has to be a way to bind it to the em0bridge in the conf file but I have yet to find it.
 
You must log in or register to reply here.
Back
Top