jails Jails: stopping (prolonged deaths), starting, networking et cetera

grahamperrin

grahamperrin

 
Below, what's wrong?

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # service jail onestop
Stopping jails: 13.
root@mowa219-gjp4-8570p-freebsd:~ # cat /etc/jail.conf
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
$j = "/jails";
securelevel = 2;
ip_hostname;

12 {
    path = $j/12;
    host.hostname = "12";
    ip4.addr = 192.168.1.19;
#    ip6.addr = 2001:470:1f1c:a0::2;
    persist;
}

13 {
    path = $j/13;
    host.hostname = "13";
    ip4.addr = 192.168.1.20;
#    ip6.addr = 2001:470:1f1c:a0::2;
    persist;
}
root@mowa219-gjp4-8570p-freebsd:~ # service jail onestart
Starting jails:jail: 12: jail 12 already exists
jail: 13: jail 13 already exists
.
root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ #
 
I restarted the entire system.

Below, is service jail onestop not a proper way to stop a jail?

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # date ; uptime
Sun Feb 20 08:11:48 GMT 2022
 8:11AM  up 59 mins, 5 users, load averages: 0.99, 0.84, 0.74
root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ # service jail onestart
Starting jails: 12 13.
root@mowa219-gjp4-8570p-freebsd:~ # mount | grep jails\/1 | sort
august/jails/12 on /jails/12 (zfs, local, noatime, nfsv4acls)
august/jails/13 on /jails/13 (zfs, local, noatime, nfsv4acls)
devfs on /jails/12/dev (devfs)
devfs on /jails/13/dev (devfs)
root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
    12  192.168.1.19    12                            /jails/12
    13  192.168.1.20    13                            /jails/13
root@mowa219-gjp4-8570p-freebsd:~ # time service jail onestop
Stopping jails: 12 13.
0.096u 0.140s 0:00.27 85.1%     83+128k 7+2io 0pf+0w
root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ # time service jail onestart
Starting jails:jail: 12: jail 12 already exists
jail: 13: jail 13 already exists
.
0.016u 0.032s 0:00.04 100.0%    75+204k 2+0io 0pf+0w
root@mowa219-gjp4-8570p-freebsd:~ #

service(8)
 
From my understanding the service stuff does more than start and stop. And it depends on the content of /etc/rc.conf. How about jal -c THEJAIL and jail -r THEJAIL? These commands just create and remove THEJAIL.
 
thindil said:
… jls -d
Click to expand...

Thank you! Posts crossed paths (moderation for newcomers).

The surprise, to me, is that it takes more than twenty-seven minutes to stop these two jails:

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # uptime ; jls -d
10:49AM  up  3:37, 6 users, load averages: 0.49, 0.74, 1.13
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ # service jail onestart
Starting jails: 12 13.
root@mowa219-gjp4-8570p-freebsd:~ # man jls
root@mowa219-gjp4-8570p-freebsd:~ # man 8 jls
root@mowa219-gjp4-8570p-freebsd:~ # man 8 jail
root@mowa219-gjp4-8570p-freebsd:~ # date ; service jail onestop
Sun Feb 20 10:59:54 GMT 2022
Stopping jails: 12 13.
…
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Sun Feb 20 11:13:00 GMT 2022
   JID  IP Address      Hostname                      Path
    12  192.168.1.19    12                            /jails/12
    13  192.168.1.20    13                            /jails/13
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Sun Feb 20 11:15:07 GMT 2022
   JID  IP Address      Hostname                      Path
    12  192.168.1.19    12                            /jails/12
…
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Sun Feb 20 11:27:02 GMT 2022
   JID  IP Address      Hostname                      Path
    12  192.168.1.19    12                            /jails/12
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Sun Feb 20 11:29:00 GMT 2022
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ # time service jail onestart
Starting jails: 12 13.
0.153u 0.159s 7:35.92 0.0%      179+240k 399+19io 1pf+0w
root@mowa219-gjp4-8570p-freebsd:~ # mkjail update -a
Updating all jails...

Updating 12 jail...

src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 12.3-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Fetching 2 metadata patches.. done.
Applying metadata patches... done.
Fetching 2 metadata files... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 12.3-RELEASE-p2.
No updates are available to install.

Updating 13 jail...

src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 13.0-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Fetching 2 metadata patches.. done.
Applying metadata patches... done.
Fetching 2 metadata files... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 13.0-RELEASE-p7.
No updates are available to install.

root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
    12  192.168.1.19    12                            /jails/12
    13  192.168.1.20    13                            /jails/13
root@mowa219-gjp4-8570p-freebsd:~ # chroot /jails/12
root@mowa219-gjp4-8570p-freebsd:/ # pkg info
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly, please wait...
pkg: Error fetching http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly/Latest/pkg.txz: No address record
Address resolution failed for http://pkg.FreeBSD.org/FreeBSD:12:amd64/quarterly.
Consider changing PACKAGESITE.
root@mowa219-gjp4-8570p-freebsd:/ # exit
exit
root@mowa219-gjp4-8570p-freebsd:~ # chroot /jails/13
root@mowa219-gjp4-8570p-freebsd:/ # pkg upgrade
Updating FreeBSD repository catalogue...
pkg: Repository FreeBSD has a wrong packagesite, need to re-create database
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01   
Fetching packagesite.txz: 100%    6 MiB   3.4MB/s    00:02   
Processing entries: 100%
FreeBSD repository update completed. 31322 packages processed.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pkg: 1.16.3 -> 1.17.5_1

Number of packages to be upgraded: 1

8 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching pkg-1.17.5_1.pkg: 100%    8 MiB   4.4MB/s    00:02   
Checking integrity... done (0 conflicting)
[1/1] Upgrading pkg from 1.16.3 to 1.17.5_1...
[1/1] Extracting pkg-1.17.5_1: 100%
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
Checking for upgrades (1 candidates): 100%
Processing candidates (1 candidates): 100%
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        nano: 5.8 -> 6.0

Number of packages to be upgraded: 1

567 KiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching nano-6.0.pkg: 100%  567 KiB 581.0kB/s    00:01   
Checking integrity... done (0 conflicting)
[1/1] Upgrading nano from 5.8 to 6.0...
[1/1] Extracting nano-6.0: 100%
root@mowa219-gjp4-8570p-freebsd:/ # pkg prime-origins
editors/nano
ports-mgmt/pkg
root@mowa219-gjp4-8570p-freebsd:/ # exit
exit
root@mowa219-gjp4-8570p-freebsd:~ # uname -aKU
FreeBSD mowa219-gjp4-8570p-freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #3 main-n253116-39a36707bd3-dirty: Sat Feb 12 16:47:35 GMT 2022     root@mowa219-gjp4-8570p-freebsd:/usr/obj/usr/src/amd64.amd64/sys/GENERIC-NODEBUG  amd64 1400051 1400051
root@mowa219-gjp4-8570p-freebsd:~ #

The 12 jail was created a few hours ago, there's almost nothing to it.

The 13 jail is older, it seems that I installed only one package (nano).

chrbr said:
… depends on the content of /etc/rc.conf. …
Click to expand...

Nothing jail-related:

Code: 
% grep -i jail /etc/rc.conf  
% ls /etc/rc.conf.d/
%
 
Just for information:
This is my jails etc/rc.conf. I use that template for some time. The source is somewhere in the ezjail documentation. The purpose is to start almost nothing.
Code: 
> cat /usr/jails/fox/etc/rc.conf
network_interfaces=""
rpcbind_enable="NO"
cron_flags="$cron_flgs -J 15"
syslogd_flags="-ss"
sendmail_enable="NO"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
sshd_enable="NO"
dbus_enable="YES"
To start the jail fox I have the following lines in /etc/rc.conf.
Code: 
> grep jail /etc/rc.conf
# Required for jails
# dns and http proxy to the jail
jail_fox_devfs_ruleset=5
jail_enable="yes"
jail_list="fox"
The ruleset is for firefox. I am not sure if jail_enable defaults to "yes" nowadays. I have a different jail for firefox which I start by jail -c THEJAIL. There are almost no delays in handling both jails.
 
My /etc/jail.conf was in what became the spoiler in the opening post.

I'm still taking baby steps, after a first experiment with jails in 2019, so if there's anything obviously wrong with this please let me know:

Code: 
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
$j = "/jails";
securelevel = 2;
ip_hostname;

12 {
    path = $j/12;
    host.hostname = "12";
    ip4.addr = 192.168.1.19;
#    ip6.addr = ⋯not disclosed here⋯;
    persist;
}

13 {
    path = $j/13;
    host.hostname = "13";
    ip4.addr = 192.168.1.20;
#    ip6.addr = ⋯not disclosed here⋯;
    persist;
}
 
My config is as below:
Code: 
allow.raw_sockets = 1;
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
devfs_ruleset = 5;
mount.devfs;

basejail {
        path = "/usr/jails/basejail";
        host.hostname = "basejail";
        ip4.addr = "10.0.0.1";
        interface = "lo1";
        mount.fstab = "/etc/fstab.basejail";
 }
The other jails are configured similar. I do not use ip_hostname and securelevel. I am not sure if persist hurts or not. From jail(8)
Code: 
     persist
             Setting this boolean parameter allows a jail to exist without any
             processes.  Normally, a command is run as part of jail creation,
             and then the jail is destroyed as its last process exits.  A new
             jail must have either the persist parameter or exec.start or
             command pseudo-parameter set.
You have specified exec.start.
But in total there is nothing obviously wrong.
 
chrbr said:
… I do not use ip_hostname …
Click to expand...

I commented out that line, so:

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # date ; cat /etc/jail.conf | grep -v \#
Sun Feb 20 16:18:16 GMT 2022
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
$j = "/jails";
securelevel = 2;

12 {
    path = $j/12;
    host.hostname = "12";
    ip4.addr = 192.168.1.19;
    persist;
}

13 {
    path = $j/13;
    host.hostname = "13";
    ip4.addr = 192.168.1.20;
    persist;
}
root@mowa219-gjp4-8570p-freebsd:~ # jls -d
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ # time service jail onestart
Starting jails: 12 13.
0.096u 0.247s 7:37.35 0.0%      158+229k 733+19io 362pf+0w
root@mowa219-gjp4-8570p-freebsd:~ # date ; service jail onestop
Sun Feb 20 16:34:46 GMT 2022
Stopping jails: 12 13.
root@mowa219-gjp4-8570p-freebsd:~ #

This stop took less than twelve minutes.

A later repeat of the routine resulted in a stop time of more than twenty-six minutes.

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # date ; service jail onestop
Sun Feb 20 17:50:00 GMT 2022
Stopping jails: 12 13.
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Sun Feb 20 18:16:14 GMT 2022
   JID  IP Address      Hostname                      Path
    13  192.168.1.20    13                            /jails/13
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Sun Feb 20 18:20:36 GMT 2022
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ #
 
grahamperrin said:
Below, what's wrong?

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # service jail onestop
Stopping jails: 13.
root@mowa219-gjp4-8570p-freebsd:~ # cat /etc/jail.conf
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
$j = "/jails";
securelevel = 2;
ip_hostname;

12 {
    path = $j/12;
    host.hostname = "12";
    ip4.addr = 192.168.1.19;
#    ip6.addr = 2001:470:1f1c:a0::2;
    persist;
}

13 {
    path = $j/13;
    host.hostname = "13";
    ip4.addr = 192.168.1.20;
#    ip6.addr = 2001:470:1f1c:a0::2;
    persist;
}
root@mowa219-gjp4-8570p-freebsd:~ # service jail onestart
Starting jails:jail: 12: jail 12 already exists
jail: 13: jail 13 already exists
.
root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ #
Click to expand...

Bad naming! You could easily mix up JID and jail name. Don't use integers as jail name.
 
dvl@ that has been at the back of my mind, although I don't recall any change to rc.conf(5) for either of my past experiments with jails.

Gut feeling: with the jail naming/documentation bug worked around, my next step should be another restart of the entire system. Watch this space …
 
grahamperrin said:
Thanks, I received a comparable hint from dvl@ a few hours ago; <https://github.com/mkjail/mkjail/issues/29#issuecomment-1046235357>

A documentation bug, yes? From <https://www.freebsd.org/cgi/man.cgi?query=jail&sektion=8&manpath=FreeBSD#DESCRIPTION> under Jail Parameters:
Click to expand...
Technically they ARE correct. However, as jail allows you to use symbolic names and JIDs, having two different jails - one with JID=12 and one with name=12 makes commands ambiguous. jail will not crash, but the effects could be different from what the user expects. A notice of warning would be a good idea IMHO.
 
grahamperrin said:
dvl@ that has been at the back of my mind, although I don't recall any change to rc.conf(5) for either of my past experiments with jails.

Gut feeling: with the jail naming/documentation bug worked around, my next step should be another restart of the entire system. Watch this space …
Click to expand...
I have experienced this many times. When a jail cannot cleanly stop for some reason, it can stay in dying state and cannot be removed (at least with my level of experience). In my case it mostly happened because of unmounting problems.
A reboot has always helped me in this case.
 
D-FENS said:
I have experienced this many times. When a jail cannot cleanly stop for some reason, it can stay in dying state and cannot be removed (at least with my level of experience). In my case it mostly happened because of unmounting problems. …
Click to expand...

Unfortunately, no better following a restart of the physical computer.

With the naming problem worked around: start times appear to be almost instant (compared to the seven minutes, previously seen).

Stop times remain problematic; slow death.

At a glance, nothing remarkable in /var/log/messages in jail twelve:

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # time service jail onestart twelve
Starting jails: twelve.
0.093u 0.108s 0:00.23 82.6%     133+208k 16+9io 1pf+0w
root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
     5  192.168.1.19                                  /jails/twelve
root@mowa219-gjp4-8570p-freebsd:~ # chroot /jails/twelve/
root@mowa219-gjp4-8570p-freebsd:/ # uname -KU
1400051 1203000
root@mowa219-gjp4-8570p-freebsd:/ # pkg prime-origins
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: n
root@mowa219-gjp4-8570p-freebsd:/ # cat /etc/rc.conf
cat: /etc/rc.conf: No such file or directory
root@mowa219-gjp4-8570p-freebsd:/ # exit
exit
root@mowa219-gjp4-8570p-freebsd:~ # date ; service jail onestop
Mon Feb 21 06:07:15 GMT 2022
Stopping jails: twelve.
root@mowa219-gjp4-8570p-freebsd:~ # chroot /jails/twelve/
root@mowa219-gjp4-8570p-freebsd:/ # tail /var/log/messages
Feb 20 17:24:12 12 sm-msp-queue[50535]: unable to qualify my own domain name (12) -- using short name
Feb 20 17:50:00 12 syslogd: exiting on signal 15
Feb 20 19:13:08  syslogd: kernel boot file is /boot/kernel/kernel
Feb 20 19:14:00  syslogd: exiting on signal 15
Feb 21 03:43:31  syslogd: kernel boot file is /boot/kernel/kernel
Feb 21 03:50:03  syslogd: exiting on signal 15
Feb 21 04:13:38  syslogd: kernel boot file is /boot/kernel/kernel
Feb 21 04:24:50  syslogd: exiting on signal 15
Feb 21 06:03:19  syslogd: kernel boot file is /boot/kernel/kernel
Feb 21 06:07:15  syslogd: exiting on signal 15
root@mowa219-gjp4-8570p-freebsd:/ # exit
exit
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Mon Feb 21 06:08:41 GMT 2022
   JID  IP Address      Hostname                      Path
     5  192.168.1.19                                  /jails/twelve
root@mowa219-gjp4-8570p-freebsd:~ #
 
dvl@ said:
… sudo sysrc jail_enable="YES" first.
Click to expand...

No better, unfortunately (I assume that the one jail started below, thirteen, should take less than five minutes to die):

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # sysrc jail_enable="YES"
jail_enable: NO -> YES
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls
Mon Feb 21 06:14:26 GMT 2022
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Mon Feb 21 06:14:28 GMT 2022
   JID  IP Address      Hostname                      Path
     5  192.168.1.19                                  /jails/twelve
root@mowa219-gjp4-8570p-freebsd:~ # service jail start thirteen
Starting jails: thirteen.
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls
Mon Feb 21 06:14:40 GMT 2022
   JID  IP Address      Hostname                      Path
     6  192.168.1.20                                  /jails/thirteen
root@mowa219-gjp4-8570p-freebsd:~ # tail /jails/thirteen/var/log/messages
Feb 21 06:14:41  sm-mta[16521]: NOQUEUE: SYSERR(root): opendaemonsocket: daemon Daemon0: cannot bind: Can't assign requested address
Feb 21 06:14:41  sm-mta[16521]: daemon Daemon0: problem creating SMTP socket
Feb 21 06:14:46  sm-mta[16521]: NOQUEUE: SYSERR(root): opendaemonsocket: daemon Daemon0: cannot bind: Can't assign requested address
Feb 21 06:14:46  sm-mta[16521]: daemon Daemon0: problem creating SMTP socket
Feb 21 06:14:51  sm-mta[16521]: NOQUEUE: SYSERR(root): opendaemonsocket: daemon Daemon0: cannot bind: Can't assign requested address
Feb 21 06:14:51  sm-mta[16521]: daemon Daemon0: problem creating SMTP socket
Feb 21 06:14:56  sm-mta[16521]: NOQUEUE: SYSERR(root): opendaemonsocket: daemon Daemon0: cannot bind: Can't assign requested address
Feb 21 06:14:56  sm-mta[16521]: daemon Daemon0: problem creating SMTP socket
Feb 21 06:15:01  sm-mta[16521]: NOQUEUE: SYSERR(root): opendaemonsocket: daemon Daemon0: cannot bind: Can't assign requested address
Feb 21 06:15:01  sm-mta[16521]: daemon Daemon0: problem creating SMTP socket
root@mowa219-gjp4-8570p-freebsd:~ # less /jails/thirteen/var/log/messages
root@mowa219-gjp4-8570p-freebsd:~ # less /jails/thirteen/var/log/messages
root@mowa219-gjp4-8570p-freebsd:~ # grep syslogd /jails/thirteen/var/log/messages
Jul 18 23:03:35 13 syslogd: kernel boot file is /boot/kernel/kernel
Jul 18 23:23:59 13 syslogd: exiting on signal 15
Jul 18 23:24:00 13 syslogd: kernel boot file is /boot/kernel/kernel
Jul 20 02:53:42 13 syslogd: exiting on signal 15
Feb 19 10:11:12 13 syslogd: kernel boot file is /boot/kernel/kernel
Feb 19 14:40:15 13 syslogd: exiting on signal 15
Feb 20 01:09:34 13 syslogd: kernel boot file is /boot/kernel/kernel
Feb 20 01:21:24 13 syslogd: exiting on signal 15
Feb 20 01:58:48 13 syslogd: kernel boot file is /boot/kernel/kernel
Feb 20 02:12:06 13 syslogd: exiting on signal 15
Feb 20 08:15:33 13 syslogd: kernel boot file is /boot/kernel/kernel
Feb 20 08:20:28 13 syslogd: exiting on signal 15
Feb 20 10:53:28 13 syslogd: kernel boot file is /boot/kernel/kernel
Feb 20 10:59:54 13 syslogd: exiting on signal 15
Feb 20 11:32:57 13 syslogd: kernel boot file is /boot/kernel/kernel
Feb 20 15:45:41 13 syslogd: exiting on signal 15
Feb 20 16:22:00 13 syslogd: kernel boot file is /boot/kernel/kernel
Feb 20 16:34:47 13 syslogd: exiting on signal 15
Feb 20 16:50:13 13 syslogd: kernel boot file is /boot/kernel/kernel
Feb 20 17:00:01 13 syslogd: exiting on signal 15
Feb 20 17:24:12 13 syslogd: kernel boot file is /boot/kernel/kernel
Feb 20 17:50:01 13 syslogd: exiting on signal 15
Feb 20 19:13:09  syslogd: kernel boot file is /boot/kernel/kernel
Feb 20 19:14:00  syslogd: exiting on signal 15
Feb 21 03:43:41  syslogd: kernel boot file is /boot/kernel/kernel
Feb 21 03:50:03  syslogd: exiting on signal 15
Feb 21 04:13:39  syslogd: kernel boot file is /boot/kernel/kernel
Feb 21 04:24:50  syslogd: exiting on signal 15
Feb 21 06:14:36  syslogd: kernel boot file is /boot/kernel/kernel
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Mon Feb 21 06:17:08 GMT 2022
   JID  IP Address      Hostname                      Path
     5  192.168.1.19                                  /jails/twelve
     6  192.168.1.20                                  /jails/thirteen
root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
     6  192.168.1.20                                  /jails/thirteen
root@mowa219-gjp4-8570p-freebsd:~ # mount | grep jails\/1
august/jails/13 on /jails/thirteen (zfs, local, noatime, nfsv4acls)
august/jails/12 on /jails/twelve (zfs, local, noatime, nfsv4acls)
root@mowa219-gjp4-8570p-freebsd:~ # mount | grep devfs
devfs on /dev (devfs)
devfs on /compat/linux/dev (devfs)
devfs on /compat/ubuntu/dev (devfs)
devfs on /jails/thirteen/dev (devfs)
root@mowa219-gjp4-8570p-freebsd:~ # date ; time service jail stop thirteen
Mon Feb 21 06:17:54 GMT 2022
Stopping jails: thirteen.
0.078u 0.057s 0:00.12 100.0%    54+173k 1+1io 0pf+0w
root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Mon Feb 21 06:23:00 GMT 2022
   JID  IP Address      Hostname                      Path
     5  192.168.1.19                                  /jails/twelve
     6  192.168.1.20                                  /jails/thirteen
root@mowa219-gjp4-8570p-freebsd:~ #
 
Set for host.hostname a FQDN (unset ip_hostname), or set ip_hostname and unset host.hostname.

If that doesn't help run jail(8) with -v option set.
Code: 
# jail -vc 12
# jail -vr 12

The jail name shouldn't be a problem.
 
With ip_hostname set (host.hostname not set):
  1. the first start and stop of a single jail were pleasantly speedy, I imagined success
  2. the stop that followed the second start, of the same jail, is still dying after five hours
Code: 
root@mowa219-gjp4-8570p-freebsd:~ # grep -v \# /etc/jail.conf
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;
securelevel = 2;
ip_hostname;

twelve {
    path = /jails/twelve;
    ip4.addr = 192.168.1.19;
    persist;
}

thirteen {
    path = /jails/thirteen;
    ip4.addr = 192.168.1.20;
    persist;
}
root@mowa219-gjp4-8570p-freebsd:~ # jls -d
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ # date ; uptime ; time service jail onestart thirteen
Mon Feb 21 18:38:49 GMT 2022
 6:38PM  up  8:19, 5 users, load averages: 2.40, 2.03, 1.50
Starting jails: thirteen.
0.070u 0.091s 0:00.20 80.0%     119+202k 17+10io 0pf+0w
root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
     2  192.168.1.20                                  /jails/thirteen
root@mowa219-gjp4-8570p-freebsd:~ # tail -n 2 /jails/thirteen/var/log/messages
Feb 21 18:39:19  sm-mta[35172]: NOQUEUE: SYSERR(root): opendaemonsocket: daemon Daemon0: cannot bind: Can't assign requested address
Feb 21 18:39:19  sm-mta[35172]: daemon Daemon0: problem creating SMTP socket
root@mowa219-gjp4-8570p-freebsd:~ # date ; time service jail onestop thirteen
Mon Feb 21 18:40:00 GMT 2022
Stopping jails: thirteen.
0.050u 0.082s 0:00.11 118.1%    101+182k 0+1io 0pf+0w
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Mon Feb 21 18:40:07 GMT 2022
   JID  IP Address      Hostname                      Path
     2  192.168.1.20                                  /jails/thirteen
…
     root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Mon Feb 21 23:47:21 GMT 2022
   JID  IP Address      Hostname                      Path
     2  192.168.1.20                                  /jails/thirteen
root@mowa219-gjp4-8570p-freebsd:~ #

I'll restart the entire system, update the OS then (maybe Tuesday evening) try the two suggested commands.

I'm very grateful for these examples, and the walk-through.

Before starting this topic, I did try to educate myself with the Michael W. Lucas video, however that was paused after fifteen minutes – when the given example didn't work for me (not his fault). If the problems here can be resolved, I might resume listening next weekend.

The manual page for jail(8) might be amongst the worst that I have encountered.
 
grahamperrin said:
Unfortunately, no better following a restart of the physical computer.
...
Stop times remain problematic; slow death.
At a glance, nothing remarkable in /var/log/messages in jail twelve:
Click to expand...
This happens to my jails (on jail start however!) whenever their DNS does not work properly and the jail . As a quick test: add the jail name (with and without domain name) into jail's /etc/hosts. The jail must be able to resolve its own hostname and the hostnames of any other hosts it depends. Example for /jails/twelve/etc/hosts:
Code: 
127.0.0.1    localhost
192.168.1.19   twelve twelve.my.domain
192.168.1.20   thirteen thirteen.my.domain  # if twelve has a web server and the DB is on another host/jail for example.

Further things to consider:
  • Does the firewall pass DNS traffic correctly?
  • Make sure the resolving works without any delays inside the jail: `ping twelve`, `ping thirteen`, `ping www.test.com`.

If the DNS works properly and the problem persists, then it must be due to a service that runs on `twelve`. Try disabling all services except the syslog and see if the jail starts and stops quickly. We need to debug the service shutdown that causes the delay.

Another idea: I have had problems with stopping jails due to incorrect unmounting order. This is why I decided to do all mounting completely manually as describe here.
After the jail dies, observe the mount situation. Try mounting the devfs manually by removing `mount.devfs` from jail.conf.
 
T-Daemon said:
… run jail(8) with -v option set. …
Click to expand...

Done, thanks. Death after removal took around twenty minutes:

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # date ; uptime ; jls -d
Sun Feb 27 01:50:11 GMT 2022
 1:50AM  up 21:33, 6 users, load averages: 0.11, 0.37, 0.43
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ # jail -vc thirteen
thirteen: run command: /sbin/mount -t devfs -oruleset=4 . /jails/thirteen/dev
thirteen: jail_set(JAIL_CREATE) name=thirteen securelevel=2 path=/jails/thirteen host.hostname=cheekymonkey.bounceme.net ip4.addr=192.168.1.20 persist
thirteen: created
thirteen: run command in jail: /bin/sh /etc/rc
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg
32-bit compatibility ldconfig path: /usr/lib32
Updating motd:.
Creating and/or trimming log files.
Clearing /tmp (X related).
Updating /var/run/os-release done.
Starting syslogd.
Starting sendmail_submit.
Starting sendmail_msp_queue.
Starting cron.

Sun Feb 27 01:51:01 UTC 2022
root@mowa219-gjp4-8570p-freebsd:~ # date ; time jail -vr thirteen
Sun Feb 27 01:59:59 GMT 2022
thirteen: run command in jail: /bin/sh /etc/rc.shutdown
Stopping cron.
Waiting for PIDS: 6581.
.
Terminated
thirteen: sent SIGTERM to: 6510 6577
thirteen: removed
thirteen: run command: /sbin/umount /jails/thirteen/dev
0.040u 0.068s 0:00.25 40.0%     68+166k 174+1io 0pf+0w
root@mowa219-gjp4-8570p-freebsd:~ # mount | grep jails | grep dev
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Sun Feb 27 02:01:00 GMT 2022
   JID  IP Address      Hostname                      Path
     7  192.168.1.20    cheekymonkey.bounceme.net     /jails/thirteen
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Sun Feb 27 02:03:59 GMT 2022
   JID  IP Address      Hostname                      Path
     7  192.168.1.20    cheekymonkey.bounceme.net     /jails/thirteen
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Sun Feb 27 02:19:59 GMT 2022
   JID  IP Address      Hostname                      Path
     7  192.168.1.20    cheekymonkey.bounceme.net     /jails/thirteen
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Sun Feb 27 02:24:23 GMT 2022
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ #

Should I simply ignore the dying times of jails (with death detected by jls(8))?

D-FENS thanks, I'll try some of your suggestions.
 
D-FENS said:
… Try mounting the devfs manually by removing `mount.devfs` from jail.conf.
Click to expand...

Code: 
exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
# mount.devfs;
# $j = "/jails";
# path = /jails/$name;
securelevel = 2;
# ip_hostname;

twelve {
    path = /jails/twelve;
    host.hostname = "twelve";
    ip4.addr = 192.168.1.19;
#    ip6.addr = 2001:470:1f1c:a0::2;
    persist;
}

thirteen {
    path = /jails/thirteen;
    host.hostname = "cheekymonkey.bounceme.net";
    ip4.addr = 192.168.1.20;
#    ip6.addr = 2001:470:1f1c:a0::2;
    persist;
}

– commented out, for the test below.

D-FENS said:
… Make sure the resolving works without any delays inside the jail: `ping twelve`, …
Click to expand...

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # nano /etc/jail.conf
root@mowa219-gjp4-8570p-freebsd:~ # jail -vc twelve
twelve: jail_set(JAIL_CREATE) name=twelve securelevel=2 path=/jails/twelve host.hostname=twelve ip4.addr=192.168.1.19 persist
twelve: created
twelve: run command in jail: /bin/sh /etc/rc
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
Updating motd:.
Updating /var/run/os-release done.
Creating and/or trimming log files.
Starting syslogd.
Clearing /tmp (X related).
Starting sendmail_submit.
Starting sendmail_msp_queue.
Starting cron.

Sun Feb 27 02:37:29 UTC 2022
root@mowa219-gjp4-8570p-freebsd:~ # mount | grep jails | grep dev
root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
     9  192.168.1.19    twelve                        /jails/twelve
root@mowa219-gjp4-8570p-freebsd:~ # chroot /jails/twelve/
root@mowa219-gjp4-8570p-freebsd:/ # ping -4 twelve
ping: illegal option -- 4
usage: ping [-AaDdfnoQqRrv] [-c count] [-G sweepmaxsize] [-g sweepminsize]
            [-h sweepincrsize] [-i wait] [-l preload] [-M mask | time] [-m ttl]
            [-P policy] [-p pattern] [-S src_addr] [-s packetsize] [-t timeout]
            [-W waittime] [-z tos] host
       ping [-AaDdfLnoQqRrv] [-c count] [-I iface] [-i wait] [-l preload]
            [-M mask | time] [-m ttl] [-P policy] [-p pattern] [-S src_addr]
            [-s packetsize] [-T ttl] [-t timeout] [-W waittime]
            [-z tos] mcast-group
root@mowa219-gjp4-8570p-freebsd:/ # ping twelve
ping: cannot resolve twelve: Host name lookup failure
root@mowa219-gjp4-8570p-freebsd:/ # exit
exit
root@mowa219-gjp4-8570p-freebsd:~ # date ; time jail -vr twelve
Sun Feb 27 02:41:00 GMT 2022
twelve: run command in jail: /bin/sh /etc/rc.shutdown
.
twelve: sent SIGTERM to: 10016 10346 10092 10348 10096
twelve: removed
0.026u 0.053s 0:00.07 100.0%    94+181k 0+46io 0pf+0w
root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d
Sun Feb 27 02:41:59 GMT 2022
   JID  IP Address      Hostname                      Path
     9  192.168.1.19    twelve                        /jails/twelve
root@mowa219-gjp4-8570p-freebsd:~ # date ; jail -vc twelve
Sun Feb 27 02:42:10 GMT 2022
twelve: jail_set(JAIL_CREATE) name=twelve securelevel=2 path=/jails/twelve host.hostname=twelve ip4.addr=192.168.1.19 persist
twelve: created
twelve: run command in jail: /bin/sh /etc/rc
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
Updating motd:.
Updating /var/run/os-release done.
Creating and/or trimming log files.
Starting syslogd.
Clearing /tmp (X related).
Starting sendmail_submit.
Starting sendmail_msp_queue.
Starting cron.

Sun Feb 27 02:42:40 UTC 2022
root@mowa219-gjp4-8570p-freebsd:~ # date ; time jail -vr twelve
Sun Feb 27 02:42:49 GMT 2022
twelve: run command in jail: /bin/sh /etc/rc.shutdown
.
twelve: sent SIGTERM to: 10698 10702 10724 10726 10680 10622
twelve: removed
0.012u 0.070s 0:10.08 0.7%      113+210k 0+46io 0pf+0w
root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ # date ; jls -d && jail -vc twelve
Sun Feb 27 02:47:59 GMT 2022
   JID  IP Address      Hostname                      Path
     9  192.168.1.19    twelve                        /jails/twelve
    10  192.168.1.19    twelve                        /jails/twelve
twelve: jail_set(JAIL_CREATE) name=twelve securelevel=2 path=/jails/twelve host.hostname=twelve ip4.addr=192.168.1.19 persist
twelve: created
twelve: run command in jail: /bin/sh /etc/rc
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
Updating motd:.
Updating /var/run/os-release done.
Creating and/or trimming log files.
Starting syslogd.
Clearing /tmp (X related).
Starting sendmail_submit.
Starting sendmail_msp_queue.
Starting cron.

Sun Feb 27 02:48:29 UTC 2022
root@mowa219-gjp4-8570p-freebsd:~ # date ; time jail -vr twelve
Sun Feb 27 02:48:43 GMT 2022
twelve: run command in jail: /bin/sh /etc/rc.shutdown
.
twelve: sent SIGTERM to: 11346 11348 11297 11238 11315 11319
twelve: removed
0.031u 0.061s 0:10.12 0.8%      87+185k 0+46io 0pf+0w
root@mowa219-gjp4-8570p-freebsd:~ #
 
grahamperrin said:
– commented out, for the test below.
Click to expand...

Some thoughts on your test, comparing with the way I do it:
  • When I wrote "comment out mount.devfs in jail.conf" I meant that you need to mount the devfs manually via the exec.prestart hook. I have not tried to start a jail without a devfs at all. An example of my exec.prestart follows below.
  • Instead of using a direct chroot into the jail, I would use jexec twelve /bin/sh. I don't know if there is any difference, but I assume jexec probably does some jail-specific setup chores in addition.
  • I never used jail -c and jail -d, instead I would use service jail start twelve and service jail stop twelve. I don't know if there is any difference.
  • The DNS did not work in your test. Pay attention to what is in /jails/twelve/etc/resolv.conf and make sure the DNS resolver works. You can copy your main host's resolv.conf and ping the nameserver from within the jail to make sure the DNS is accessible.
  • Also, try creating an identical jail with sysutils/iocage and then compare what is different with yours.

Here is what I would use as a prestart script:
Bash: 
#!/bin/sh -x

# mount additional stuff from my jail's fstab manually....

/sbin/mount -t devfs -oruleset=5 . "/jails/twelve/dev"
/sbin/mount -t fdescfs . "/jails/twelve/dev/fd"

And this could serve as a poststop:
Bash: 
#!/bin/sh -x

/sbin/umount "/jails/twelve/dev/fd"  || /sbin/umount -f "/jails/twelve/dev/fd" || true
/sbin/umount "/jails/twelve/dev"     || /sbin/umount -f "/jails/twelve/dev"    || true

# unmount additional stuff from my jail's fstab manually....
 
I'd like to keep things as simple as possible … for now, maybe I should ignore the prolonged deaths and focus on networking. (I did succeed with networking in the past, 2019 maybe, but I don't want to retrace those steps.)

Is what's below a routing problem?

Code: 
root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ # jls -d
   JID  IP Address      Hostname                      Path
    17  192.168.1.19    twelve                        /jails/twelve
    19  192.168.1.19    twelve                        /jails/twelve
    22  192.168.1.19    twelve                        /jails/twelve
    27  192.168.1.19    twelve                        /jails/twelve
root@mowa219-gjp4-8570p-freebsd:~ # jail -vc twelve
twelve: run command: /sbin/mount -t devfs -oruleset=4 . /jails/twelve/dev
twelve: jail_set(JAIL_CREATE) name=twelve allow.raw_sockets=true securelevel=2 path=/jails/twelve host.hostname=twelve ip4.addr=192.168.1.19 persist
twelve: created
twelve: run command in jail: /bin/sh /etc/rc
ELF ldconfig path: /lib /usr/lib /usr/lib/compat
32-bit compatibility ldconfig path: /usr/lib32
Updating motd:.
Updating /var/run/os-release done.
Creating and/or trimming log files.
Starting syslogd.
Clearing /tmp (X related).
Starting cron.

Sun Feb 27 12:33:42 UTC 2022
root@mowa219-gjp4-8570p-freebsd:~ # jexec twelve /bin/csh
root@twelve:/ # grep -v \# /etc/rc.conf
defaultrouter="192.168.1.1"

sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

root@twelve:/ # grep -v \# /etc/resolvconf.conf
name_servers="192.168.1.1"

root@twelve:/ # cat /etc/resolv.conf
# Generated by resolvconf
nameserver 192.168.1.1

root@twelve:/ # route show default
route: route has not been found
root@twelve:/ # route add default
route: writing to routing socket: Operation not permitted
root@twelve:/ # ping freshports.org
load: 0.36  cmd: ping 30581 [select] 6.03r 0.00u 0.00s 0% 2352k
mi_switch+0xc2 sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _cv_wait_sig+0xec seltdwait+0x9d kern_select+0x9a2 sys_select+0x56 amd64_syscall+0x10c fast_syscall_common+0xf8
^C
root@twelve:/ # ping twelve
^C
root@twelve:/ # exit
exit
root@mowa219-gjp4-8570p-freebsd:~ # jail -vr twelve
twelve: run command in jail: /bin/sh /etc/rc.shutdown
Stopping cron.
Waiting for PIDS: 30518.
.
Terminated
twelve: sent SIGTERM to: 30598 30474
twelve: removed
twelve: run command: /sbin/umount /jails/twelve/dev
root@mowa219-gjp4-8570p-freebsd:~ # jls
   JID  IP Address      Hostname                      Path
root@mowa219-gjp4-8570p-freebsd:~ # jls -d
   JID  IP Address      Hostname                      Path
    17  192.168.1.19    twelve                        /jails/twelve
    19  192.168.1.19    twelve                        /jails/twelve
    22  192.168.1.19    twelve                        /jails/twelve
    27  192.168.1.19    twelve                        /jails/twelve
    31  192.168.1.19    twelve                        /jails/twelve
root@mowa219-gjp4-8570p-freebsd:~ #

Could VNET(9) and inherit simplify things for me? I wonder.

Before this topic began, I listened to around fifteen minutes of FreeBSD Fridays: Introduction to Jails (2020), stopped watching when the on-screen example didn't work for me. Resumed playback this morning, stopped again when the next set of on-screen information was ambiguous/confusing. I'm patient, but I might have to clear my mind of jails again for a few days :-/ …
 
I always use VNET with my jails. I am not familiar with the non-VNET jails.
If you have trouble making it work and you don't need to understand all low level specifics, I'd recommend you using a tool like iocage, which is great and does most of the dirty work for you.
Initially we wanted to go with iocage but it had a bug bringing up the network interfaces under certain conditions and we decided going with plain jails. This bug was fixed years ago and it's not an issue now. They also have good documentation.

On the point of using plain jails, I think your problems are caused by a network misconfiguration. This is how I troubleshoot network:
1. Use tcpdump - check if the packets arrive?
2. Network interfaces:
a) are all interfaces up (on jail and host)?
b) do all bridges contain the necessary interfaces as members?
3. IP-address: Are IP addresses set throughout the whole route?
4. IP forwarding: Is it activated if the node is supposed to be a gateway? If not, enable it with sysctl: net.inet.ip.fw.enable=1, net.inet.ip.forwarding=1
5. Firewall/NAT
a) Is NAT configured correctly
b) Does firewall block the packets? Look into /var/log/security on the main host. Jail's logs also land there.
c) The packets must be passed in both directions!
6. routes:
a) On all nodes the routing tables should be correct.
b) Make sure routes are configured in both directions.
7. DNS: Nameserver and domain are set in /etc/resolv.conf
Also, very important: "writing to routing socket: Operation not permitted" - this leads me to think that your jail lacks permissions to access a device file which is controlled by /etc/defaults/devfs.rules on your host. The last one is for VNET jails and the one before that - for all jails. First find out which device is required by your jail and then allow it in devfs.rules.

Can you please post your jail's configuration for more details?
ifconfig, netstat -rn4, sysrc -a, service ipfw status, ipfw list
 
You must log in or register to reply here.
Back
Top