Hi, posting here because this started after the recent 15.0 upgrade, but I'm not convinced it is due to the upgrade.
The problem: at 3am each day something is triggering a bottlenecking of the processes related to mounts for the jails. We have not had this problem before, and have been running this jail setup for about a year now.
Description of our environment: We are currently running Host -> vnet Jail -> 4 child jails on a smaller machine: 8-core, 16gb of memory, with one jail being a reverse proxy, another being a production web app, a third is a development version of the web app, and the 4th is a WIP something else.
ZFS datasets have been created as needed and our approach for mounting datasets into the jail were already following the recommended approach as described in the recent feature for mounting datasets.
When the vnet Jail is started, Host's /usr/local/jails is mounted at vnet Jail's /usr/local/jails. We mount a templated base for each child jail, and then the respective userland, etc.
The symptom: when running
Attempts to solve: I thought maybe there was a conflict between periodic reports being run since high CPU is always starting at 3am, so I disabled the ones in the child jails related to filesystem changes, and for a night that seemed to fix it. Interestingly, there are no errors I can find in any logs, and the jail processes are running fine when checked, but the resources are being hogged by the Host or something else. When running
But yesterday it started showing high CPU again during the day - however, the processes showing the filesystem mounts were not on the list. It seemed to resolve when I experimented by shutting down the development jail, so I figured it might be related to a process on it. I choose that jail because our production jail can call to the development jail but the 4 jails are otherwise not able to talk to each other. However, last night something got triggered and the CPU was high with the filesystem mounts, and this was with the development jail off.
I'm looking for some ideas of what to try or test, thanks.
Host jail.conf which starts the vnet jail:
Vnet jail.conf which starts the child jails:
The problem: at 3am each day something is triggering a bottlenecking of the processes related to mounts for the jails. We have not had this problem before, and have been running this jail setup for about a year now.
Description of our environment: We are currently running Host -> vnet Jail -> 4 child jails on a smaller machine: 8-core, 16gb of memory, with one jail being a reverse proxy, another being a production web app, a third is a development version of the web app, and the 4th is a WIP something else.
ZFS datasets have been created as needed and our approach for mounting datasets into the jail were already following the recommended approach as described in the recent feature for mounting datasets.
When the vnet Jail is started, Host's /usr/local/jails is mounted at vnet Jail's /usr/local/jails. We mount a templated base for each child jail, and then the respective userland, etc.
The symptom: when running
htop from the Host, sorted by CPU high to low, the top two processes show the filesystem mounts. These don't normally appear on the list when things are "working". Although the system responds somewhat to some commands, jails cannot be shutdown with normal commands from the vnet jail. A reboot has been required which returns the loads to normal. The processes - whatever it is - is pegged at the equivalent of one processor core. I haven't found how to trigger the problem, and am not sure how to diagnose what process may be causing it.Attempts to solve: I thought maybe there was a conflict between periodic reports being run since high CPU is always starting at 3am, so I disabled the ones in the child jails related to filesystem changes, and for a night that seemed to fix it. Interestingly, there are no errors I can find in any logs, and the jail processes are running fine when checked, but the resources are being hogged by the Host or something else. When running
htop from a child jail, it's almost all red across all cores. But yesterday it started showing high CPU again during the day - however, the processes showing the filesystem mounts were not on the list. It seemed to resolve when I experimented by shutting down the development jail, so I figured it might be related to a process on it. I choose that jail because our production jail can call to the development jail but the 4 jails are otherwise not able to talk to each other. However, last night something got triggered and the CPU was high with the filesystem mounts, and this was with the development jail off.
I'm looking for some ideas of what to try or test, thanks.
Host jail.conf which starts the vnet jail:
Code:
mount.devfs; # Mount devfs inside the jail
exec.clean; # prevent use of host environment in commands
exec.start = "/bin/sh /etc/rc"; # Start command
exec.stop = "/bin/sh /etc/rc.shutdown"; # Stop command
exec.consolelog = "/var/log/jail_console_${name}.log";
host.hostname = "redacted";
jail_parent {
path = "/usr/local/jails/containers/${name}";
vnet;
vnet.interface="e0b_jail_parent";
devfs_ruleset=70;
exec.created+="zfs jail $name zroot/poudriere";
exec.created+="zfs jail $name zroot/jails/children";
exec.created+="zfs jail $name zroot/jails/root_templates";
exec.created+="zfs jail $name zroot/jails/databases";
exec.created+="/usr/local/jails/scripts/zfs_set_jailed_on";
exec.start+="service jail onestart nginx";
exec.start+="service jail onestart beta";
exec.start+="service jail onestart marketing";
exec.start+="service jail onestart develop";
exec.start+="wg-quick up wg_client";
allow.raw_sockets;
allow.mlock;
allow.mount;
allow.mount.devfs;
allow.mount.fdescfs;
allow.mount.fusefs;
allow.mount.linprocfs;
allow.mount.linsysfs;
allow.mount.nullfs;
allow.mount.procfs;
allow.mount.tmpfs;
allow.mount.zfs;
enforce_statfs = 1;
children.max=20;
exec.prestop+="/sbin/ifconfig e0b_jail_parent -vnet $name";
exec.stop+="service jail onestop develop";
exec.stop+="service jail onestop marketing";
exec.stop+="service jail onestop beta";
exec.stop+="service jail onestop nginx";
exec.stop+="wg-quick down wg_client";
exec.poststop+="zfs unjail $name zroot/poudriere";
exec.poststop+="zfs unjail $name zroot/jails/children";
exec.poststop+="zfs unjail $name zroot/jails/root_templates";
exec.poststop+="zfs unjail $name zroot/jails/databases";
exec.poststop+="/usr/local/jails/scripts/zfs_set_jailed_off";
}Vnet jail.conf which starts the child jails:
Code:
mount.devfs; # Mount devfs inside the jail
devfs_ruleset = "0";
ip6="disable";
exec.clean;
exec.start = "/bin/sh /etc/rc"; # Start command
exec.stop = "/bin/sh /etc/rc.shutdown"; # Stop command
exec.consolelog = "/var/log/jail_console_${name}.log";
path = "/usr/local/jails/$name";
nginx {
# hostname
host.hostname = "redacted";
# network
ip4.addr = "192.168.1.1";
# permissions
enforce_statfs = 1;
allow.raw_sockets;
allow.mount;
allow.mount.fusefs;
mount.fstab = "/usr/local/jails/${name}.fstab";
exec.release += "/usr/local/jails/unmount_nginx_fstab";
}
marketing {
# hostname
host.hostname = "redacted";
# network
ip4.addr = "192.168.2.1";
# permissions
enforce_statfs = 1;
allow.raw_sockets;
allow.mlock;
allow.mount;
allow.mount.devfs;
allow.mount.fdescfs;
allow.mount.fusefs;
allow.mount.linprocfs;
allow.mount.linsysfs;
allow.mount.nullfs;
allow.mount.procfs;
allow.mount.tmpfs;
mount.fstab = "/usr/local/jails/${name}.fstab";
exec.start += "/var/www/sourcefiles/mountsourcefiles";
exec.start += "/var/www/sourcefiles/start_linux_service";
exec.stop += "/var/www/sourcefiles/stop_linux_service";
exec.stop += "/var/www/sourcefiles/unmountsourcefiles";
exec.release += "/usr/local/jails/unmount_marketing_fstab";
}
beta {
# hostname
host.hostname = "redacted";
# network
ip4.addr = "192.168.3.1";
# permissions
enforce_statfs = 1;
allow.raw_sockets;
allow.mlock;
allow.mount;
allow.mount.fusefs;
allow.mount.nullfs;
mount.fstab = "/usr/local/jails/${name}.fstab";
exec.prestart += "sudo -u rsync_user /home/rsync_user/mount_rsync_net_beta_remote_storage";
exec.start += "/var/www/sourcefiles/mountsourcefiles";
exec.prestop += "umount -f /usr/local/jails/beta/s/usr/local/mnt/rsync_remote_storage";
exec.stop += "/var/www/sourcefiles/unmountsourcefiles";
exec.release += "/usr/local/jails/unmount_beta_fstab";
}
develop {
# hostname
host.hostname = "redacted";
# network
ip4.addr = "192.168.4.1";
# permissions
enforce_statfs = 1;
allow.raw_sockets;
allow.mlock;
allow.mount;
allow.mount.fusefs;
allow.mount.nullfs;
mount.fstab = "/usr/local/jails/${name}.fstab";
exec.stop += "/var/www/sourcefiles/mountsourcefiles";
exec.stop += "/var/www/sourcefiles/unmountsourcefiles";
exec.release += "/usr/local/jails/unmount_develop_fstab";
}
