Solved Kernel panic with VNET jail at "service jail onestop"

petlib · Apr 10, 2020

I'm playing around with VNET and Jail on "FreeBSD smaug 12.1-RELEASE-p3 FreeBSD 12.1-RELEASE-p3 GENERIC amd64".
My jail.conf:

Code:

exec.start = "/bin/sh /etc/rc";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.clean;
exec.system_user = "root";
exec.jail_user = "root";
path = "/jails/$name";
allow.mount.devfs;
allow.chflags;
allow.raw_sockets;
mount.devfs;

vpn1 {
        host.hostname = $name;
        exec.consolelog = "/var/log/jail_${name}_console.log";
        devfs_ruleset = "10";
        vnet = "new";
        vnet.interface = "epair15b";
        exec.prestart = "ifconfig bridge15 create up";
        exec.prestart += "ifconfig epair15 create up";
        exec.prestart += "ifconfig bridge15 addm epair15a";
        exec.prestart += "ifconfig bridge15 addm re0";
        exec.start = "ifconfig epair15b inet 10.0.3.4/24";
        exec.start += "route add default 10.0.3.1";
        exec.start += "/bin/sh /etc/rc";
        exec.stop = "/bin/sh /etc/rc.shutdown";
        exec.poststop = "ifconfig epair15a destroy";
        exec.poststop += "sleep 2";
        exec.poststop += "ifconfig bridge15 destroy";
}

Started jail with "service jail onestart"
Jail and networking works as expected. Then when stopping the jail with "service jail onestop" I got a kernel panic:

Code:

Apr 10 07:44:43 smaug kernel: Fatal trap 12: page fault while in kernel mode
Apr 10 07:44:43 smaug kernel: cpuid = 0; apic id = 00
Apr 10 07:44:43 smaug kernel: fault virtual address     = 0x410
Apr 10 07:44:43 smaug kernel: fault code                = supervisor read data, page not present
Apr 10 07:44:43 smaug kernel: instruction pointer       = 0x20:0xffffffff80baff8d
Apr 10 07:44:43 smaug kernel: stack pointer             = 0x28:0xfffffe005a87e800
Apr 10 07:44:43 smaug kernel: frame pointer             = 0x28:0xfffffe005a87e880
Apr 10 07:44:43 smaug kernel: code segment              = base 0x0, limit 0xfffff, type 0x1b
Apr 10 07:44:43 smaug kernel:                   = DPL 0, pres 1, long 1, def32 0, gran 1
Apr 10 07:44:43 smaug kernel: processor eflags  = interrupt enabled, resume, IOPL = 0
Apr 10 07:44:43 smaug kernel: current process           = 0 (thread taskq)
Apr 10 07:44:43 smaug kernel: trap number               = 12
Apr 10 07:44:43 smaug kernel: panic: page fault
Apr 10 07:44:43 smaug kernel: cpuid = 0
Apr 10 07:44:43 smaug kernel: time = 1586497432
Apr 10 07:44:43 smaug kernel: KDB: stack backtrace:
Apr 10 07:44:43 smaug kernel: #0 0xffffffff80c1d2f7 at kdb_backtrace+0x67
Apr 10 07:44:43 smaug kernel: #1 0xffffffff80bd062d at vpanic+0x19d
Apr 10 07:44:43 smaug kernel: #2 0xffffffff80bd0483 at panic+0x43
Apr 10 07:44:43 smaug kernel: #3 0xffffffff810a7dcc at trap_fatal+0x39c
Apr 10 07:44:43 smaug kernel: #4 0xffffffff810a7e19 at trap_pfault+0x49
Apr 10 07:44:43 smaug kernel: #5 0xffffffff810a740f at trap+0x29f
Apr 10 07:44:43 smaug kernel: #6 0xffffffff81081bdc at calltrap+0x8
Apr 10 07:44:43 smaug kernel: #7 0xffffffff80ccd641 at if_detach_internal+0x261
Apr 10 07:44:43 smaug kernel: #8 0xffffffff80cd496c at if_vmove+0x3c
Apr 10 07:44:43 smaug kernel: #9 0xffffffff80cd4918 at vnet_if_return+0x48
Apr 10 07:44:43 smaug kernel: #10 0xffffffff80cfe324 at vnet_destroy+0x124
Apr 10 07:44:43 smaug kernel: #11 0xffffffff80b988d0 at prison_deref+0x2a0
Apr 10 07:44:43 smaug kernel: #12 0xffffffff80c2fad4 at taskqueue_run_locked+0x154
Apr 10 07:44:43 smaug kernel: #13 0xffffffff80c30e08 at taskqueue_thread_loop+0x98
Apr 10 07:44:43 smaug kernel: #14 0xffffffff80b90c43 at fork_exit+0x83
Apr 10 07:44:43 smaug kernel: #15 0xffffffff81082c1e at fork_trampoline+0xe

I have the crash dump. The jail config is based on this guide https://www.nixcraft.com/t/how-to-configure-a-freebsd-jail-with-vnet-and-zfs/952/17

Thanks

genneko · Apr 10, 2020

Hi, I had encoutered the same crashes but the workaround posted on PR 238326 worked like a charm.
With the workaround, I've never exprienced the crash again.

In your case, adding the following line might help.

Code:

exec.prestop = "ifconfig epair15b -vnet vpn1";

petlib · Apr 10, 2020

Thanks genneko! Yet another piece added to the "vnet jail" puzzle

DenisVS · Dec 22, 2021

So where is a clear manual for jail with vnet?

Cath O'Deray · Feb 28, 2022

DenisVS said:
So where is a clear manual for jail with vnet?

How to set up FreeBSD 12 VNET jail with ZFS - nixCraft (2021-01, updated 2021-02-02) was:

posted to Reddit (the comment about panics, later, drew me to this topic)
bookmarked by me to read later
mentioned a few hours ago by D-FENS

– I might find time for the guide at the weekend …

AdrianAli · Apr 11, 2022

I use vnet with Netgraph and the problem is still present on:

Code:

# freebsd-version -k
13.0-RELEASE-p11
# freebsd-version -u
13.0-RELEASE-p11

My Kernel Panic:

Code:

Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x5110000004d8
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80d073e5
stack pointer           = 0x28:0xfffffe00a1acb9d0
frame pointer           = 0x28:0xfffffe00a1acb9d0
code segment            = base rx0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 7470 (ifconfig)
trap number             = 12
panic: page fault
cpuid = 0
time = 1649635952
KDB: stack backtrace:
#0 0xffffffff80c57535 at kdb_backtrace+0x65
#1 0xffffffff80c09f11 at vpanic+0x181
#2 0xffffffff80c09d83 at panic+0x43
#3 0xffffffff8108b1a7 at trap_fatal+0x387
#4 0xffffffff8108b1ff at trap_pfault+0x4f
#5 0xffffffff8108a85d at trap+0x27d
#6 0xffffffff81061f08 at calltrap+0x8
#7 0xffffffff80d1d1a9 at ifunit_ref+0x79
#8 0xffffffff80d1f5fb at ifioctl+0x4eb
#9 0xffffffff80c76edd at kern_ioctl+0x26d
#10 0xffffffff80c76bd6 at sys_ioctl+0xf6
#11 0xffffffff8108baac at amd64_syscall+0x10c
#12 0xffffffff8106282e at fast_syscall_common+0xf8
Uptime: 18m12s
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,
--> or switch off the system now.

My global parameters at jail.conf:

Code:

##########
# GLOBAL #
##########
host.hostname = "$name.mydomain.com";
path = "/home/jails/$name";
exec.system_user = "root";
exec.jail_user = "root";
allow.raw_sockets = 1;
devfs_ruleset="11";
enforce_statfs = 1;
sysvshm = new;
sysvsem = new;
sysvmsg = new;
mount.devfs;

The particular parameters for the Jail:

Code:

myjail {
        vnet;
        vnet.interface = ng0_myjail;
        exec.clean;
        exec.prestart += "jng bridge myjail vtnet1";
        exec.start  = "/sbin/ifconfig ng0_myjail 192.168.1.1/24";
        exec.start += "/sbin/route add default 192.168.1.254";
        exec.start += "/bin/sh /etc/rc";
        exec.stop = "/bin/sh /etc/rc.shutdown jail";
        exec.poststop = "jng shutdown myjail";
}

The following combination of lines:

Code:

        exec.poststop = "/bin/sleep 5";
        exec.poststop += "jng shutdown myjail";

Mitigates the issue, run 50 reboots of jail and no panic kernel is generated. Run host FreeBSD on proxmox/kvm.

Solved Kernel panic with VNET jail at "service jail onestop"

petlib

genneko

petlib

DenisVS

Cath O'Deray

AdrianAli