bhyve How do I make jailed bhyve VMs talk to each other?

I

ikevin8me

I'm trying to set up "vnet" jailed bhyve VMs which can talk to each other.

Below is my jail .conf file:

Code: 
jailname100 {

 $if = 100;

 exec.prestart = "/sbin/ifconfig epair${if} create up";
 exec.prestart += "/sbin/ifconfig epair${if}a up";
 exec.prestart += "/sbin/ifconfig epair${if}a name ${name}";
 exec.prestart += "/sbin/ifconfig epair${if}b name jail${if}";
 exec.prestart += "/sbin/ifconfig bridge0 addm ${name}";
 exec.prestart += "/sbin/ifconfig tap${if} create up";
 exec.prestart += "/sbin/ifconfig bridge0 addm tap${if}"; 
 exec.prestart += "/sbin/sysctl net.inet.ip.forwarding=1";

 exec.clean;

 host.hostname = "${name}";
 vnet;
 vnet.interface = "tap${if}";
 path = "/myjails/containers/${name}";
 allow.vmm;
 allow.mount;
 mount.devfs;
 devfs_ruleset=100;
 persist;

 exec.start += "/bin/sh /etc/rc";

 exec.stop = "/bin/sh /etc/rc.shutdown";

 exec.poststop += "/sbin/ifconfig ${name} destroy";
 exec.poststop += "/sbin/ifconfig bridge0 deletem tap${if}";
 exec.poststop += "/sbin/ifconfig tap${if} destroy"; 

 exec.consolelog = "/var/log/jail_console_${name}.log";

}

Note: the jails do not have access to the hardware interface card (ie. no access to "em0")

The booted OS inside the jail does indeed have an IP address assigned to tap100, but the problem is it is not reacheable from the (FreeBSD) host or other bhyve VMs. In the (FreeBSD) host, the IP address of the bhyve VM does not appear either.

I suppose the IP address needs to "appear" in the host, in order for the host and VMs to communicate between each other. How can I do that?
 
cmoerz said:
Why don't you use epair devices to connect your vnet jails? Then you could simply bridge your epair with your tap inside the jails?
Click to expand...
If you see my settings above, it has epair. And, they have appeared in the host’s “ifconfig -a”. However, they don’t have any IP addresses attached. So, how do I set the IP addresses?
 
ikevin8me said:
If you see my settings above, it has epair. And, they have appeared in the host’s “ifconfig -a”. However, they don’t have any IP addresses attached. So, how do I set the IP addresses?
Click to expand...
I am not 100% sure, regarding the vnet jails I have a working setup as configured in /etc/jail.conf as below:
Code: 
vnet.interface = "epair${jid}b";
exec.prestart =  "ifconfig epair${jid} create up";
exec.prestart += "ifconfig epair${jid}a up";
exec.prestart += "ifconfig bridge0 addm epair${jid}a up";
exec.start  = "/sbin/ifconfig lo0 127.0.0.1 up";
exec.start += "/sbin/ifconfig epair${jid}b 10.0.0.${jid}/24 up";
exec.start += "/sbin/route add default 10.0.0.254";
...
exec.prestop   = "ifconfig epair${jid}b -vnet ${name}";
exec.poststop  = "ifconfig bridge0 deletem epair${jid}a";
exec.poststop += "ifconfig epair${jid}a destroy";
exec.prestart happens before the jail is started and exec.start when the jail starts. In your configuration epair$b is configured before the jail starts. May be this does not work. In my setup epair$b is configured with the staring jail which works. I am not sure, but may be this makes the difference.
 
forums.freebsd.org

Solved - use a virtual network interface on a host?

Is there a way to use a virtual network interface on a physical FreeBSD host? TL;DR - yes - use epair interfaces. Background: I have this new and quite cheap laptop (Lenovo ideapad 530S-14ARR). It was on outlet sale, therefore cheap. It doesn't have a wired ethernet interface, and the wireless...
forums.freebsd.org forums.freebsd.org

My `/etc/rc.conf`
Code: 
ifconfig_re0="inet 192.168.1.67 netmask 255.255.255.0"
defaultrouter="192.168.1.1"

cloned_interfaces="epair0"
defaultroute_delay="1"
ifconfig_epair0a="inet 10.254.0.1 netmask 255.255.255.0"
ifconfig_epair0b="up"

gateway_enable="YES"

Create virtual switch
Code: 
➜ ~ # vm switch create internal
➜ ~ # vm switch add internal epair0b
➜ ~ # vm switch list
NAME      TYPE      IFACE        ADDRESS  PRIVATE  MTU  VLAN  PORTS
public    standard  vm-public    -        no       -    -     re0
internal  standard  vm-internal  -        no       -    -     epair0b

ifconfig
Code: 
➜ ~ # ifconfig
re0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=202018<VLAN_MTU,VLAN_HWTAGGING,WOL_MAGIC,RXCSUM_IPV6>
        ether 10:2e:15:10:0d:42
        inet 192.168.1.67 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
re1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=60241b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,LRO,WOL_MAGIC,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 10:2e:15:10:0d:43
        media: Ethernet autoselect
        status: no carrier
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair0a: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:e7:94:6f:f9:0a
        inet 10.254.0.1 netmask 0xffffff00 broadcast 10.254.0.255
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair0b: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:e7:94:6f:f9:0b
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vm-public: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether c6:bb:48:73:b8:e5
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
        groups: bridge vm-switch viid-4c918@
        nd6 options=9<PERFORMNUD,IFDISABLED>
vm-internal: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        options=0
        ether ca:41:59:50:b6:b3
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000000
        member: epair0b flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 2000
        groups: bridge vm-switch viid-d1efa@
        nd6 options=9<PERFORMNUD,IFDISABLED>
tap0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: vmnet/artsale-0/0/public
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:ff:c7
        groups: tap vm-port
        media: Ethernet 1000baseT <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 96028
tap1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: vmnet/artsale-0/1/internal
        options=80000<LINKSTATE>
        ether 58:9c:fc:10:ff:dd
        groups: tap vm-port
        media: Ethernet 1000baseT <full-duplex>
        status: active
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        Opened by PID 96028

myvm.conf
Code: 
loader="grub"
cpu="4"
memory="8G"
network0_type="virtio-net"
network0_switch="public"
network1_type="virtio-net"
network1_switch="internal"
disk0_name="disk0"
disk0_dev="sparse-zvol"
disk0_type="virtio-blk"
uuid="9ef59bcd-2084-11ed-92ed-2021088003e6"
network0_mac="58:9c:fc:00:cf:f9"
network1_mac="58:9c:fc:01:43:96"

Code: 
➜ ~ # vm switch info internal
------------------------
Virtual Switch: internal
------------------------
  type: standard
  ident: vm-internal
  vlan: -
  physical-ports: epair0b
  bytes-in: 1815140 (1.731M)
  bytes-out: 1645840 (1.569M)
  virtual-port
    device: tap1
    vm: myvm

Inside the vm
Code: 
root@myvm ~ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s5f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 58:9c:fc:00:cf:f9 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.62/24 brd 192.168.1.255 scope global enp0s5f0
       valid_lft forever preferred_lft forever
    inet6 fe80::5a9c:fcff:fe00:cff9/64 scope link
       valid_lft forever preferred_lft forever
3: enp0s5f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 58:9c:fc:01:43:96 brd ff:ff:ff:ff:ff:ff
    inet 10.254.0.10/24 brd 10.254.0.255 scope global enp0s5f1
       valid_lft forever preferred_lft forever
    inet6 fe80::5a9c:fcff:fe01:4396/64 scope link
       valid_lft forever preferred_lft forever
 
You must log in or register to reply here.
Back
Top