FreeBSD 10 reboots/crashes every day at 4:00 am

Hello,

I have a problem with my FreeBSD 10 installation.
It reboots or crashes everyday at around 4:00 am.

The last log entry in /var/logs/all.log is:

Code:
Feb 25 04:00:00 rna /usr/sbin/cron[14974]: (root) CMD (/usr/libexec/atrun)
Feb 25 04:01:00 rna /usr/sbin/cron[14996]: (root) CMD (adjkerntz -a)
Feb 25 08:50:12 rna syslogd: restart
Feb 25 08:50:12 rna syslogd: kernel boot file is /boot/kernel/kernel

I already tried manually running adjkerntz -a to see if that is the problem, but that does not seem to be the case as the command runs fine.

The setup uses ZFS on an encrypted geli() device and swap is on a graid() mirror on top of a encrypted geli() device, so the kernel cannot write a kernel crash dump.

Is a reboot the normal behaviour if the kernel crashes?
Can I reconfigure it somehow to stop on a crash and print some debug information to the console?

I’m relatively new to FreeBSD and would be happy for any pointers in the right direction on how to find the cause of the reboot.

Thank you,

Philipp
 
I've heard of this happening because of the scripts that get run in /etc/periodic/*/. Though the daily one is usually run at 3am rather than 4 but it's worth checking /etc/crontab. Several of the scripts do a find which run through the entire filesystem looking for different things. With zfs this can cause memory usage to massively increase until it runs out. I have no experience of zfs myself but I've heard people saying "arc size" needs increasing if that makes any sense?

I've just found this thread: https://forums.freebsd.org/viewtopic.php?&t=42396. Might be similar?
 
schmidp said:
Is a reboot the normal behaviour if the kernel crashes?
Yes, because there's no other way to recover from an error like that.

Can I reconfigure it somehow to stop on a crash and print some debug information to the console?
There should be a switch somewhere but I forgot where. It will keep the message on the screen until you press a key on the console. Mind you, your server will be down until that happens. Is there any way you can set the swap to a non-encrypted device? At least temporarily, a crash dump may provide some extra information as to why it crashed.
 
Hi,

thank you for your replies!

I checked and my kernel has debug support:

Code:
options         KDB                     # Enable kernel debugger support.
options         KDB_TRACE               # Print a stack trace for a panic.

And the sysctrl flag is enabled:

Code:
root@server:~ # sysctl debug.debugger_on_panic
debug.debugger_on_panic: 1

But it just reboots instantly like if someone pressed the reset button.

I also tried disabling periodic by editing /etc/crontab, but again: reboot at 4:03am...

Any other ideas?

Tonight I'll try running a GENERIC kernel and see if that helps...

Thank you, Philipp
 
Morning,

The server does survive the night with a GENERIC kernel :)

I'll now try to find the option that causes the crash/reboot...

- Philipp
 
I have now found the root cause:

If a jail is running and the jail executes

Code:
/etc/periodic/daily/450.status-security

the server crashes. This happens every night at 4:03 CET.

If I run

Code:
/etc/periodic/daily/450.status-security

manually in the jail, the server also crashes immediately.

Now I wanted to try to run each script in

Code:
/etc/periodic/security

manually, but I get:

Code:
ASSERTION FAILED: Unexpected value for  $PERIODIC: ‘'

I then ran:

Code:
setenv PERIODIC "security daily”

which allowed me to run each security script separately.

If I run:

Code:
root@jail:/etc/periodic/security # ./520.pfdenied

the machine immediately reboots.

Looking at 520.pfdenied I tried running the command:

Code:
root@jail:~ # pfctl -sr -v

directly, which also crashes the host immediately.
 
I don't know if this is still true for 10.0... As of 9.1, having

Code:
options         VIMAGE

in the kernel configuration rules out all firewals except ipfw.
IIRC, just thinking abot pf would crash my machine. :)
 
Back
Top