jails Unable to import an iocage export

[byrnejb]

I used iocage export to transfer a jail from one host to another host. The jail's export files were transferred using rsync. However, I cannot get iocage to successfully complete the import on the destination. This is apparently caused by running out of swap space.

Dec 13 12:20:11 vhost03 kernel: swap_pager: out of swap space
Dec 13 12:20:11 vhost03 kernel: swp_pager_getswapspace(16): failed
Dec 13 12:20:13 vhost03 kernel: pid 26142 (python3.9), jid 0, uid 0, was killed: failed to reclaim memory
Dec 13 16:00:20 vhost03 kernel: swap_pager: out of swap space
Dec 13 16:00:20 vhost03 kernel: swp_pager_getswapspace(24): failed
Dec 13 16:00:23 vhost03 kernel: pid 29796 (python3.9), jid 0, uid 0, was killed: failed to reclaim memory
Dec 13 16:00:24 vhost03 kernel: pid 10070 (sshd), jid 0, uid 0, was killed: failed to reclaim memory
Dec 13 16:00:24 vhost03 kernel: swap_pager: out of swap space
Dec 13 16:00:24 vhost03 kernel: swp_pager_getswapspace(20): failed
Dec 13 16:00:25 vhost03 kernel: swp_pager_getswapspace(17): failed

The system is a remote hot spare and the jail is used only to run an isolated domain name service. The memory allocation when unloaded looks like this:

last pid: 69321;  load averages:  0.12,  0.11,  0.13         up 20+01:24:01  16:16:57
17 processes:  1 running, 16 sleeping
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 8776K Active, 67M Inact, 40K Laundry, 399M Wired, 104K Buf, 3436M Free
ARC: 189M Total, 27M MFU, 149M MRU, 132K Anon, 1330K Header, 11M Other
     149M Compressed, 216M Uncompressed, 1.45:1 Ratio
Swap: 2048M Total, 9456K Used, 2039M Free

As this is iocage the jail has a zfs dataset. I speculate that zfs send/receiveis buffering to the point of memory exhaustion. Is this what is actually causing the swap problem? If so, is there some way to limit the amount of memory iocage import uses?

 

[sko]

As this is iocage the jail has a zfs dataset. I speculate that zfs send/receiveis buffering to the point of memory exhaustion. Is this what is actually causing the swap problem? If so, is there some way to limit the amount of memory iocage import uses?

It seems it's not zfs but python that is exhausting memory and is being killed, so it seems iocage is not simply invoking zfs send/recv but doing something else in between...

With iocell (i.e. the original, shell-based iocage) those export/import jobs work as intended (except for the host-uuid mismatch that has to be resolved manually with the version in ports), as those shell-scripts really just invoke zfs send and zfs recv.
True, 4GB RAM are pretty much on the absolute lowest end of memory for such tasks, but I also zfs recv'd large datasets and regular backups on a NAS with only 3GB of RAM for a while - it was *extremely* slow and tedious, but it always finished the job. So the export/import *should* work *if* iocage is really just invoking zfs send/recv...

 

[byrnejb]

It seems it's not zfs but python that is exhausting memory and is being killed, so it seems iocage is not simply invoking zfs send/recv but doing something else in between...

This problem is with iocage and Python memory management. It has been previously reported and supposedly fixed some time ago. So this may be a regression or the fix may never have actually worked. I stopped investigating this after I discovered the work-around. Which is:

# cd /tmp 
# unzip /zroot/iocage/images/jail_export.zip
Archive:  /zroot/iocage/images/jail_export.zip
  extracting:   zroot/iocage/images/jail_export
  extracting:   zroot/iocage/images/jail_export/root
#
# # before proceeding destroy existing jail if it exists
#
# zfs recv -F root/iocage/jails/jails_export  <  /tmp/zroot/iocage/images/jail_export
# zfs recv -F root/iocage/jails/jails_export/root  <  /tmp/zroot/iocage/images/jail_export_root

 

[sko]

holy crap. I always knew a lot of python code is really bad (hence I avoid anything python-based like the plague), but this must be a new low...

Best advice: use sysutils/iocell; which works and is easily understandable and debuggable, as it is purely a shell-wrapper around the standard tools. TBH I never understood the reasoning of iocage switching to python, other than iXsystems maybe thinking it would better integrate into their mostly python-based intermediate layer of free/trueNAS...

Right after the fork of iocell the migration from an iocage-based install was relatively straightforward (IIRC only putting the options back into files instead of zfs properties and reverting the renaming of datasets); I have no idea though how far they diverged now. But jails are jails - so a manual send|recv restore into a iocell dataset-tree and restoring the jail options should always work.

 

[byrnejb]

This problem in iocage was addressed on the master branch in 2020. The version available from ports does not include the fix.

As of 2023 the fixes in master on github were never placed into the release version. The iocage project has been abandoned by the authors since late 2021. Reported PR 275767