sysutils/panicmail

[trh411]

Ran across this yesterday when I was perusing the latest freebsd-current@freebsd.org mailing list digest. Thought others on this forum might be interested.

Message: 3
Date: Mon, 04 Nov 2013 02:41:49 -0800
From: Colin Percival <cperciva@freebsd.org>
To: FreeBSD Hackers <freebsd-hackers@freebsd.org>,  FreeBSD current
        <freebsd-current@freebsd.org>
Subject: Automated submission of kernel panic reports:
        sysutils/panicmail
Message-ID: <527779ED.9040303@freebsd.org>
Content-Type: text/plain; charset=ISO-8859-1

Hi all,

After considerable review on freebsd-hackers (thanks dt71 and jilles!) I have
now added sysutils/panicmail to the FreeBSD ports tree.  If you install this
and add
        panicmail_enable="YES"
to your /etc/rc.conf, a panic report will be generated and sent to root@ for
you to review and submit (via email).  You can skip the reviewing step and
submit panics automatically by setting panicmail_autosubmit="YES".

The panics submitted are encrypted to an RSA key which I hold in order to keep
them secure in transit; and I intend to keep the raw panic reports confidential
except to the minimum extent necessary for other developers to help me process
the incoming reports.

If I receive enough panic reports to be useful, I hope to provide developers
with aggregate statistics.  This may include:

* regular email reports listing the "top panics", to help guide developers
towards the most fertile areas for stability improvements;

* email to specific developers alerting them to recurring panics in code they
maintain (especially if it becomes clear that the panic has been recently
introduced); and

* guidance to re@ and secteam@ about how often a particular panic occurs if
an errata notice is being considered

as well as other yet-to-be-imagined reports of a similarly aggregate and
anonymized nature.

So please install the sysutils/panicmail port and enable it in rc.conf!  This
all depends on getting useful data, and I can't do that without your help.

--
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid

 

[jrm@]

Automated FreeBSD panic reporting

Colin Percival has written a blog post describing sysutils/panicmail, a port he created to send automated FreeBSD panic reports.

Here's the short version. Follow the steps below if you want to enable the reports.

1. Install the sysutils/panicmail port.
2. Add the lines
   

   dumpdev="AUTO"
   panicmail_enable="YES"

   to /etc/rc.conf.
3. Make sure that email sent to root goes somewhere (if you're receiving nightly "daily run output" and "security run output" emails, you're good).

 

[cpm@]

I think this information is very helpful for anyone which use an encrypted geli(8)() swap. For further details, read it here.

Following this steps you can use sysutils/panicmail without any problem. But first some rc.d scripts should be patched to work properly, so I add the patch which amending the boot order of scripts.

diff -ur encswap.orig encswap 
--- encswap.orig	2013-11-06 23:31:43.000000000 +0100
+++ encswap	2013-11-07 00:14:03.000000000 +0100
@@ -3,8 +3,8 @@
 # $FreeBSD: release/9.2.0/etc/rc.d/encswap 180563 2008-07-16 19:22:48Z dougb $
 #
 
-# PROVIDE: disks
-# REQUIRE: initrandom
+# PROVIDE: encswap
+# REQUIRE: initrandom disks
 # KEYWORD: nojail
 
 . /etc/rc.subr
diff -ur savecore.orig savecore 
--- savecore.orig	2013-11-06 23:32:21.000000000 +0100
+++ savecore	2013-11-07 00:14:18.000000000 +0100
@@ -4,7 +4,7 @@
 #
 
 # PROVIDE: savecore
-# REQUIRE: dumpon ddb syslogd
+# BEFORE: encswap
 # KEYWORD: nojail
 
 . /etc/rc.subr
diff -ur swap1.orig swap1
--- swap1.orig	2013-11-06 23:32:28.000000000 +0100
+++ swap1	2013-11-07 00:14:29.000000000 +0100
@@ -4,7 +4,7 @@
 #
 
 # PROVIDE: localswap
-# REQUIRE: disks
+# REQUIRE: encswap disks
 # KEYWORD: nojail shutdown
 
 . /etc/rc.subr

According to /usr/ports/sysutils/panicmail/pkg-message is necessary to have enabled some scripts in /etc/rc.conf:

dumpdev="/dev/ada0p3"
dumpdir="/var/crash"
panicmail_enable="YES"

In any case, you should add your /dev/$swap_device which appears listed in /etc/fstab as following:

# Device	Mountpoint	FStype	Options	Dump	Pass#
/dev/ada0p2	/		ufs	rw	1	1
[B]/dev/ada0p3.eli	none		swap	sw	0	0[/B]
fdesc		/dev/fd		fdescfs	rw	0	0
proc		/proc		procfs	rw	0	0
linproc 	/compat/linux/proc	linprocfs	rw	0	0

 

[cpm@]

Well, I realized that savecore(8)() doesn't work as should it. It's clear that after apply the above patch, it changes adequately the order of scripts as follow:

[CMD]% rcorder /etc/rc.d/* | less[/CMD]
/etc/rc.d/sysctl
/etc/rc.d/hostid
/etc/rc.d/zvol
[color="Red"]/etc/rc.d/dumpon[/color]
/etc/rc.d/ddb
/etc/rc.d/initrandom
[color="red"]/etc/rc.d/geli[/color]
/etc/rc.d/gbde
/etc/rc.d/ccd
[color="red"]/etc/rc.d/savecore[/color]
[color="red"]/etc/rc.d/encswap[/color]
[color="red"]/etc/rc.d/swap1[/color]
<snip>

The swap space utilization on my system is the following:

[CMD]% swapinfo[/CMD]
Device          1K-blocks     Used    Avail Capacity
/dev/ada0p3.eli   3907584        0  3907584     0%

Also, /dev/dumpdev is pointing to /dev/ada0p3

[CMD]% ll /dev/dumpdev[/CMD]
lrwxr-xr-x  1 root  wheel  11 13 nov 23:51 /dev/dumpdev@ -> /dev/ada0p3

I created a crash dump using reboot -d but I see these messages related to savecore at boot stage:

savecore: reboot
savecore: unable to write to bounds file: Read-only file system
savecore: info.0: Read-only file system
savecore: unsaved dumps found but not saved
No crash dumps in /var/crash.

Finally, I extracted a kernel dump according to related section of the FreeBSD Developers' Handbook.

Any suggestions?