Solved vnet/vimage (iocage) issues

T

thomas_d

I have 2 internet-facing NIC's with public IP addresses assigned by provider. Goal is to have each jail using one of these each.

Using sysutils/iocage as a jail manager, attempting to use vnet.
Kernel is enabled for options VIMAGE as described here http://iocage.readthedocs.org/en/latest/networking.html
One jail is called thom and one is called dave. They are both set up pretty much identically, but both show different symptoms of failure with ping attempts:

thom:
Code: 
root@43ddfbe5-42b2-11e5-a051-00163ebf5850:~ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host
ping: sendto: No route to host

dave:
Code: 
root@eb77209e-42c0-11e5-8a6d-00163ebf5850:~ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
ping: sendto: Network is unreachable
ping: sendto: Network is unreachable
ping: sendto: Network is unreachable
ping: sendto: Network is unreachable

Pasting how it's all setup.

HOST'S rc.conf file:
Code: 
hostname="HOSTNAME"
# Set dumpdev to "AUTO" to enable crash dumps, "NO" to disable
dumpdev="AUTO"
zfs_enable="YES"
cloned_interfaces="bridge0 bridge1"
ifconfig_bridge0="addm vtnet0 up"
ifconfig_bridge1="addm vtnet1 up"
ifconfig_vtnet0="up"
ifconfig_vtnet1="up"
iocage_enable="YES"

HOST'S sysctl.conf file:
Code: 
net.inet.ip.forwarding=1       # Enable IP forwarding between interfaces
net.link.bridge.pfil_onlyip=0  # Only pass IP packets when pfil is enabled
net.link.bridge.pfil_bridge=0  # Packet filter on the bridge interface
net.link.bridge.pfil_member=0  # Packet filter on the member interface

security.jail.allow_raw_sockets=1

Code: 
# root@HOSTNAME:~ [CMD]# ifconfig[/CMD]
vtnet0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:16:3e:bf:58:50
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
vtnet1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6c00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:16:3e:65:7d:2f
        nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T <full-duplex>
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500                                                                  [1/197]
        ether 02:94:bd:dc:02:00
        nd6 options=9<PERFORMNUD,IFDISABLED>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0:4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: vtnet0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 2000
bridge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:94:bd:dc:02:01
        nd6 options=9<PERFORMNUD,IFDISABLED>
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet1:5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000
        member: vtnet1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 2000
vnet0:4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: 43ddfbe5-42b2-11e5-a051-00163ebf5850
        options=8<VLAN_MTU>
        ether 02:ff:60:43:dd:fb
        inet6 fe80::ff:60ff:fe43:ddfb%vnet0:4 prefixlen 64 scopeid 0x6
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
vnet1:5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: eb77209e-42c0-11e5-8a6d-00163ebf5850
        options=8<VLAN_MTU>
        ether 02:ff:60:eb:77:22
        inet6 fe80::ff:60ff:feeb:7722%vnet1:5 prefixlen 64 scopeid 0x7
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)

Action when starting each jail:
Code: 
# root@HOSTNAME:~ [CMD]# iocage start thom; iocage start dave[/CMD]

* Starting 43ddfbe5-42b2-11e5-a051-00163ebf5850 (thom)
  + Started                  OK
  + Configuring VNETepair0a: Ethernet address: 02:ff:c0:00:06:0a
epair0b: Ethernet address: 02:ff:10:00:07:0b
         OK
  + Starting servicesAug 14 23:55:16 HOSTNAME kernel: nd6_dad_timer: cancel DAD on vnet0 because of ND6_IFF_IFDISABLED.
        OK
* Starting eb77209e-42c0-11e5-8a6d-00163ebf5850 (dave)
  + Started                  OK
  + Configuring VNETepair1a: Ethernet address: 02:ff:c0:00:07:0a
epair1b: Ethernet address: 02:ff:10:00:08:0b
ifconfig: interface 123.45.67.8/24 does not exist
route: writing to routing socket: Network is unreachable
         OK
  + Starting services        OK

Any ideas? D:
 
My config is similar, except that I don't explicitly add the interfaces to the bridges, since iocage will do that. Further I've added my physical NICs to the bridges.

e.g.:
ifconfig bridge0 addm em0
ifconfig bridge1 addm re0

vnet0 is assigned bridge0 automatically and vnet1 is on bridge1.

Hope that helps
 
Oh I believe this may simply be the VPS network devices restricting my vnetX by MAC address.

Code: 
root@0d37d0b4-42f4-11e5-a43b-00163ed10cea:~ # arp -a
? (***.***.***.252) at 84:78:ac:5b:7f:42 on vnet0 expires in 391 seconds [ethernet]
? (***.***.***.251) at 84:78:ac:5b:82:c2 on vnet0 expires in 1033 seconds [ethernet]
? (***.***.***.14) at 02:ff:60:d3:7d:01 on vnet0 permanent [ethernet]

The above is from within the jail. Given that these .252 and .251 addresses are nowhere within my host or jail configuration at all, this surely confirms that the vnet0 interface is reaching other devices outside of the host at level 2. Failure to ping still confirms failure to communicate at layer 3. Does this confirm my suspicion?

Edit: And if so, what can be done about this? MAC spoof?
 
Oh wow, I've got it working on my own :D Pretty proud of this one... Had to do a MAC spoof, first time trying that, I had to change the MAC of my native interface vtnet0 to something else, and then had to:

Code: 
set vnet0_mac="{leave the MAC of vnet0:X},{change the MAC of vnet0 to the MAC given by the VPS provider for the interface}" jailname

If you do a get vnet0_mac jailname before changing it, you'll see what I mean about there being two, one for each epair. Do an ifconfig within the jail to see which one needs changing.
 
You must log in or register to reply here.
Back
Top