jails Set up iocage jail with two vnet interfaces

On a FreeBSD-12.3p5 host I have an iocage administered jail for which I would like to configure two separate i/f. I have this setup working for a bhyve administered vm on that host but I also need a jail. Here is what I have:

rc.conf:
Code:
cloned_interfaces="bridge0"
ifconfig_bridge0="addm igb0 up"
ifconfig_igb0="inet A.B.71.41/25 up"
defaultrouter="A.B.71.1"

iocage:
Code:
ip4_addr:vnet0|A.B.71.124/25,vnet1|192.168.216.124/16
vnet:1
vnet0_mac:7085c214e667 7085c214e668
vnet1_mac:7085c214e331 7085c214e332
vnet2_mac:none
vnet3_mac:none
vnet_default_interface:auto
vnet_interfaces:none

On the Jail I see this:
Code:
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 70:85:c2:14:e6:68
    hwaddr 02:d3:4b:06:64:0b
    inet 216.185.71.124 netmask 0xffffff80 broadcast 216.185.71.127
    inet6 fe80::7285:c2ff:fe14:e668%epair0b prefixlen 64 tentative scopeid 0x3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>

On the host I see this:
Code:
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
    ether 70:85:c2:da:88:4f
    inet A.B.71.41 netmask 0xffffff80 broadcast A.B.71.127
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
. . .
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: vm-vm_public_net
    ether 02:1a:39:ed:22:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 2000
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 2000000
    groups: bridge vm-switch viid-8b9e1@
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: sshpipe-3 as nic: epair0b
    options=8<VLAN_MTU>
    ether 70:85:c2:14:e6:67
    hwaddr 02:d3:4b:06:64:0a
    inet6 fe80::7285:c2ff:fe14:e667%vnet0.5 prefixlen 64 scopeid 0x5
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Without additional configuration iocage evidently creates and adds epair0b to the jail and vnet0.5 to the host with both belonging to the epair group and using the i gb0 interface. How does one go about adding the second epair1b and vnet1.X to use the bridge0 interface? I have tried discovering what, if any, importance the vnet_interfaces option has on this but nothing that I have found makes things any clearer.

If I try this:
Code:
 iocage set vnet_interfaces="igb0,bridge0" sshpipe-3
vnet_interfaces: none -> igb0,bridge0

Then the jail fails to start:
Code:
iocage start sshpipe-3
* Starting sshpipe-3
  + Start FAILED
ifconfig: interface igb0,bridge0 does not exist
jail: ioc-sshpipe-3: /sbin/ifconfig igb0,bridge0 vnet ioc-sshpipe-3: failed

If I try this:
Code:
iocage set vnet_interfaces="igb0" sshpipe-3
vnet_interfaces: none -> igb0

When I start the jail then the host loses connectivity on igb0.

What is required to specify two independent vnet i/fs in iocage? Do I need to create the epairs in rc.conf and put these in vnet_interfaces?

When I tried this:
Code:
cloned_interfaces="bridge0 epair0 epair1"
ifconfig_bridge0="addm igb0 epair0a epair1a up"
ifconfig_igb0="inet A.B.71.41/25 up"
ifconfig_epair0a="up"
ifconfig_epair1a="up"
defaultrouter="A.B.71.1"

Together with this:
Code:
iocage set vnet_interfaces="epair0b epair1b" sshpipe-3
vnet_interfaces: none -> epair0b epair1b

Starting the jail gives this:
Code:
iocage start sshpipe-3
* Starting sshpipe-3
  + Started OK
  + Using devfs_ruleset: 1005 (iocage generated default)
  + Configuring VNET OK
  + Using IP options: vnet vnet.interface=epair0b vnet.interface=epair1b
  + Starting services OK
  + Executing poststart OK

Which results in this:
Code:
. . .
igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=a520b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6>
    ether 70:85:c2:da:88:4f
    inet 216.185.71.41 netmask 0xffffff80 broadcast 216.185.71.127
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:1a:39:ed:22:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.12 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 8 priority 128 path cost 2000
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
epair0a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:80:11:a4:22:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair0b: flags=8842<BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:80:11:a4:22:0b
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
epair1a: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=8<VLAN_MTU>
    ether 02:e6:d0:56:cb:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
vnet0.12: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: sshpipe-3 as nic: epair0b
    options=8<VLAN_MTU>
    ether 70:85:c2:14:e6:67
    hwaddr 02:7a:32:44:90:0a
    inet6 fe80::7285:c2ff:fe14:e667%vnet0.12 prefixlen 64 scopeid 0x8
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Which provides only one vnet i/f.

It seems to me that iocage is overwriting vnet.interface with the last value given to vnet_interfaces. The examples that I have found use net.interfaces+=<value>. I thought that vnet.interface might take a comma separated list, but using this:
Code:
iocage set vnet_interfaces="epair0b,epair1b" sshpipe-3
vnet_interfaces: epair0b epair1b -> epair0b,epair1b
Results in this:
Code:
iocage start sshpipe-3
* Starting sshpipe-3
  + Start FAILED
ifconfig: interface epair0b,epair1b does not exist
jail: ioc-sshpipe-3: /sbin/ifconfig epair0b,epair1b vnet ioc-sshpipe-3: failed
 
The iocage man page says this:

Multiple interface format:

interface|ip-address/netmask,interface|ip-address/netmask

On shared IP jails, an interface name given before the IP
address adds an alias to that interface.

A netmask in either dotted-quad or CIDR form given after
the IP address is used when adding the IP alias.

In VNET jails, the interface is configured with the IP
addresses listed.

Example:

"vnet0|192.168.0.10/24,vnet1|10.1.1.10/24"

Interfaces vnet0 and vnet1 are configured in a VNET jail.
In this case, no network configuration is necessary in the
jail's rc.conf file.

Default: none
. . .
interfaces=[vnet0:bridge0,vnet1:bridge1 | vnet0:bridge0]
By default, there are two interfaces specified with their
bridge association. Up to four interfaces are supported.
Interface configurations are separated by commas. The
format is interface:bridge, where the left value is the
virtual VNET interface name and the right value is the
bridge name where the virtual interface should be attached.

Default: vnet0:bridge0,vnet1:bridge1

Source: local
Which leaves me rather confused as to how the configuration should be done. If I use
Code:
cloned_interfaces="bridge0 bridge1"
ifconfig_bridge0="addm igb0 up"
ifconfig_bridge1="addm igb0 up"

And set:
Code:
iocage set vnet_interfaces="none" sshpipe-3

Then I still only get one vnet:
Code:
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:1a:39:ed:22:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.14 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 10 priority 128 path cost 2000
    member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 1 priority 128 path cost 20000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:1a:39:ed:22:01
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.14: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: sshpipe-3 as nic: epair0b
    options=8<VLAN_MTU>
    ether 70:85:c2:14:e6:67
    hwaddr 02:f9:ea:a2:20:0a
    inet6 fe80::7285:c2ff:fe14:e667%vnet0.14 prefixlen 64 scopeid 0xa
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 
Top