FreeBSD 9.0-release; crashdump gets 'frozen' when panic occures

[_martin]

Hello,

I made a fresh amd64 9.0-RELEASE install. Then I built a custom kernel (along with world) from cvsup-ed RELENG_9_0 source tree. Everything is hunky-dory but one thing gives me hard time - system won't do a proper crashdump upon panic.

My setup is as follows:

root on ZFS (2nd disk is bigger as the 150GB one was not in stock during failed disk replacement):

# gpart show

=>       34  312581741  ada0  GPT  (149G)
         34        128     1  freebsd-boot  (64k)
        162   16777216     2  freebsd-swap  (8.0G)
   16777378  295804397     3  freebsd-zfs  (141G)

=>       34  488397101  ada1  GPT  (232G)
         34        128     1  freebsd-boot  (64k)
        162   16777216     2  freebsd-swap  (8.0G)
   16777378  295804397     3  freebsd-zfs  (141G)
  312581775  175815360     4  freebsd-swap  (83G)

# zpool status zroot

  pool: zroot
 state: ONLINE
 scan: resilvered 48.9G in 0h11m with 0 errors on Wed Jan 18 14:20:26 2012
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0

errors: No known data errors

Kernel configuration changes against GENERIC kernel:

# diff GENERIC FOXI

22c22
< ident         GENERIC
---
> ident         FOXI
64,65c64,65
< #options      KDTRACE_FRAME           # Ensure frames are compiled in
< #options      KDTRACE_HOOKS           # Kernel DTrace hooks
---
> options       KDTRACE_FRAME           # Ensure frames are compiled in
> options       KDTRACE_HOOKS           # Kernel DTrace hooks
68a69
> options       KDB_UNATTENDED          
338a340,363
>
> #
> # -- custom
> #
> device          coretemp
> device          ichwd
> options         SW_WATCHDOG
>
> options         MROUTING
>
> # PF
> device pf
> device pflog
> device pfsync
> options         ALTQ
> options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
> options         ALTQ_RED        # Random Early Detection (RED)
> options         ALTQ_RIO        # RED In/Out
> options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
> options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
> options         ALTQ_NOPCC      # Required for SMP build
>
> makeoptions WITH_CTF=1                # dtrace prereq
>

And some relevant files:

# grep -vE '^$|^#' /etc/make.conf

CPUTYPE?=nocona
USA_RESIDENT=true
DOC_LANG=en_US.ISO8859-1
NO_PROFILE=true
NO_SENDMAIL=true
WITHOUT_X11=yes
WITHOUT_X=yes
WITH_X=NO
ENABLE_GUI=NO
PERL_VERSION=5.12.4

# grep -vE '^$|^#' /boot/loader.conf

zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"
console="vidconsole,comconsole"
boot_multicons="YES"
boot_serial="YES"
loader_logo="beastiebw"

# grep -iE 'watch|dump|zfs' /etc/rc.conf

watchdogd_enable="YES"
zfs_enable="YES"
dumpdev="AUTO"
dumpdir="/var/crash"

# swapinfo -m

Device          1M-blocks     Used    Avail Capacity
/dev/gpt/swap0       8192        3     8188     0%
/dev/gpt/swap1       8192        3     8188     0%
Total               16384        6    16377     0%

Swap is currently not mirrored, it will be gmirrored later.
Now when I tried to simulate panic with:

# sysctl debug.kdb.panic=1

System starts to dump. But in a matter of seconds watchdog logs that timeout has expired and the whole system gets frozen. I waited long enough (15min) but nothing happened, no change on the screen.

I have KDB_UNATTENDED specified in kernel configuration so when panic occurs system won't escape to debugger prompt put rather gets rebooted.

Watchdog is enabled so system gets rebooted when system freezes, etc.

This combination of options did work on 8.2 though.

 

[ecrist]

We are seeing the same problem, but we're also seeing it on 8.2. We're wondering if it has something to do with mpt or mfi drivers, but there's really no evidence at this juncture point to those. In both 8.2 and 9.0, about 500MB of the dump gets written to disk, then the whole system freezes. A full hardware reset or power cycle is required to get the system restarted.

We are fairly certain we don't have a hardware problem, and that our issue is related to ZFS, but we won't know until we can get a crash dump.

 

[_martin]

A full hardware reset or power cycle is required to get the system restarted.

Yop, sounds about right. I still didn't figure this one out; it's little bit problematic to have downtime for this server and do the testing.

I don't have FC HBAs nor SAS controllers used on this one. I can try to compile the kernel without these modules and test it during next downtime window.

 

[Terry_Kennedy]

We are seeing the same problem, but we're also seeing it on 8.2. We're wondering if it has something to do with mpt or mfi drivers, but there's really no evidence at this juncture point to those. In both 8.2 and 9.0, about 500MB of the dump gets written to disk, then the whole system freezes. A full hardware reset or power cycle is required to get the system restarted.

Crash dumps have been problematic for me since the RELENG6 days. I think that a good part of the problem is that with multiple CPUs, watchdog hardware, and so on it is harder to get the system to stop doing everything it is doing and get a proper crash dump. This leads to long discussions on the mailing lists about particular issues. For example, consider the "Stop scheduler on panic" thread on freebsd-current@.

The one time I really, really needed a crash dump to track down a driver bug for a developer, a number of developers got involved, all contributing some piece to the puzzle to allow me to gather enough info to provide to the driver maintainer. There were quite a few patches to address a number of issues, including a "panic while panicing" problem and the inability to completely write the crash dump file. The eventual workaround was to force the panic routine to execute an already-loaded debugger script. The script completed by the time the 2nd panic came along. Some of those patches may have been integrated into 9.0 - the issue was on 8.2 a year or so ago.

IMHO, this is one of the areas in the FreeBSD system which doesn't get a lot of attention - maybe it isn't as "sexy" as some other parts. Another is the "interspersed kernel message" issue (PRINTF_BUFR_SIZE is just a stopgap workaround and doesn't always fix the problem).

 

[_martin]

Well, few days ago my server crashed. What's worth dump did complete and I have something to analyze .. and learnt not to run smart diagnostics on all disks behind one controller at once.