Solved DigitalOcean: Starting jails with iocage on boot destroys network on host

marschro

marschro

Hello everyone,

disclaimer first. I run FreeBSD as server for several websites and web applications on digitalocean.com and at the university for over a decade.
I am a total user - and have probably not enough FreeBSD knowledge at all. But FreeBSD just works and I love FreeBSD. So now I registered for this forum, because I want to get better in FreeBSD administration and basic knowledge.

Now to my problem - I want to get deeper into jails and make use of them in the future - because I don't want to switch to linux and use docker:
Setup:
  • System: FreeBSD 13.0-RELEASE-p3
  • Installs: iocage via pkg
  • zfs settings: zfs set mountpoint=/iocage zroot/iocage
rc.conf
  • enabled iocage on start for mounting jails on the host
  • iocage_enable=YES

Created a jail via iocage:
sudo iocage create -r 13.0-RELEASE --name myjail boot=on

After a reboot of host, the whole host is unavaible. The complete network is down. The host can neither pinged nor can I login via ssh.
Has anyone a clue or I hint for me how to troubleshoot that?
I have not yet assigned an ip to the jail. But this should not harm anything, I think.

The network configuration (as ifconfig sees it, before iocage starts jails)
Code: 
vtnet0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
    ether f6:dd:75:8d:fe:56
    inet6 fe80::f4dd:75ff:fe8d:fe56%vtnet0 prefixlen 64 scopeid 0x1
    inet6 2a03:b0c0:3:d0::13ba:4001 prefixlen 64
    inet 68.183.221.232 netmask 0xfffff000 broadcast 68.183.223.255
    inet 10.19.0.6 netmask 0xffff0000 broadcast 10.19.255.255
    media: Ethernet autoselect (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
    ether f2:2a:95:11:7e:6b
    inet6 fe80::f02a:95ff:fe11:7e6b%vtnet1 prefixlen 64 scopeid 0x2
    inet 10.135.78.63 netmask 0xffff0000 broadcast 10.135.255.255
    media: Ethernet autoselect (10Gbase-T <full-duplex>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

AND after iocage did its thing... it seems like it just removed all ip4 addresses from the hosts network stack (sorry screenshot only)
ifconfig_after.png


UPDATE
I did some further debugging. I put
iocage_enable=YES
in rc.conf before the DigitalOcean network scripts.
This results in the inet entries being not removed. What is good.
But ssh connections to the host after a reboot still fail with
kex_exchange_identification: Connection closed by remote host

LAST UPDATE
It seems like iocage somehow destroys the DigitalOcean cloudInit scripts that assign dynamic IP's and the gateway.
The whole IP assignment process of the DigitalOcean rc.d scripts seems broken when using iocage on startup.
It would take me long time to investigate how and why that happens. So I give up at this point and try to learn using the base jail functionalities.
This can be seen as closed without success.
 
I wonder if it's possible to add an include into /etc/rc.conf because DigitalOcean's scripts parses and re-writes rc.conf
 
You can according to rc.conf(5).

Code: 
In addition to /etc/rc.conf.local you can also place smaller
configuration files for each rc(8) script in the /etc/rc.conf.d directory
or ⟨dir⟩/rc.conf.d directories specified in local_startup, which will be
included by the load_rc_config function.  For jail configurations you
could use the file /etc/rc.conf.d/jail to store jail specific
configuration options.  If local_startup contains /usr/local/etc/rc.d and
/opt/conf, /usr/local/rc.conf.d/jail and /opt/conf/rc.conf.d/jail will be
loaded.  If ⟨dir⟩/rc.conf.d/⟨name⟩ is a directory, all of files in the
directory will be loaded.  Also see the rc_conf_files variable below.
 
You must log in or register to reply here.
Back
Top