Can BHyve and Jails co-exist without difficulty?

[byrnejb]

I have a FreeBSD-12 development host which supports virtualization (AMD POPCNT feature). Typically I use jails for this. However, I now have a case where for evaluation purposes I have to run some form of Linux. I was following the handbook instructions and ran into this when setting up the virtual nic:

Code:

ifconfig | grep -i igb   
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>

ifconfig bridge0 create
ifconfig: SIOCIFCREATE2: File exists

ifconfig tap1 create

ifconfig bridge0 addm igb0 addm tap1
ifconfig: interface bridge0 does not exist

ifconfig | grep -i bridge
    groups: bridge vm-switch viid-4c918@

I infer that the running jails have something to do with this so I need to know if what I am attempting is possible. And, if so, what is the proper way to configure the vm tap?

 

[SirDice]

They can both use bridge(4) yes. But that's about it. You can add VMs and Jails to the same bridge. Or create separate bridges, it's all up to you.

And, if so, what is the proper way to configure the vm tap?

You don't. Looking at your bridge name I assume you used sysutils/vm-bhyve to create the VM and bridges? Let vm(8) handle it.

 

[byrnejb]

Yes, I did this back in May last and forgot about having done it. Thanks.

 

[Phishfry]

I always avoided taps and bridges. Direct Hardware Passthru is what I always strive for.
Yes you can have VNET Jails and Bhyve on one machine.
Perhaps you have an igb1 interface that is unused. You can use that passed thru to a Bhyve VM or VNET-Jail. No fuss.

As a matter of fact I just learned that you can use the SR-IOV feature of VF on Intel and Chelsio 10G interfaces with VNET Jails.
I am curious to try that out. Shared 10G interfaces with jails. I already tried it with Bhyve.

https://people.freebsd.org/~rstone/BSDCan_SRIOV.pdf

 

[SirDice]

you can use the SR-IOV feature of VF on Intel

I still need to try this, I have an older SuperMicro server at home which is great for running a bunch of test VMs on it. It should support SR-IOV but I've never tried to use it. The current configuration has two interfaces bound with lagg(4) and one single interface for management of the host itself. The lagg(4) interface is bridged with different vlans to my VMs. It's more then enough to play with various applications and trying different configurations.

Code:

root@hosaka:~ # vm switch list
NAME     TYPE      IFACE       ADDRESS  PRIVATE  MTU   VLAN  PORTS
servers  standard  vm-servers  -        no       9000  11    lagg0
public   standard  vm-public   -        no       9000  10    lagg0

Code:

root@hosaka:~ # vm list
NAME            DATASTORE  LOADER     CPU  MEMORY  VNC           AUTOSTART  STATE
case            default    bhyveload  4    4096M   -             Yes [3]    Running (2535)
jenkins         default    bhyveload  4    4096M   -             Yes [5]    Running (3061)
kdc             default    none       2    2048M   0.0.0.0:5901  Yes [2]    Running (2212)
debian          stor10k    grub       2    4096M   -             No         Stopped
fbsd-test       stor10k    bhyveload  2    4096M   -             Yes [13]   Running (5861)
gitlab          stor10k    bhyveload  4    6144M   -             Yes [10]   Running (4915)
gitlab-runner   stor10k    bhyveload  4    4096M   -             Yes [11]   Running (4905)
kibana          stor10k    bhyveload  4    6144M   -             Yes [1]    Running (2012)
lady3jane       stor10k    bhyveload  4    4096M   0.0.0.0:5900  Yes [12]   Running (4842)
plex            stor10k    bhyveload  4    4096M   -             Yes [6]    Running (3581)
riviera         stor10k    bhyveload  2    4096M   -             Yes [9]    Running (4627)
sdgame01        stor10k    bhyveload  4    4096M   -             No         Stopped
tessierashpool  stor10k    bhyveload  4    16384M  -             Yes [4]    Running (10514)
wintermute      stor10k    bhyveload  4    4096M   -             Yes [8]    Running (4360)