server crash and reboot

CyberCr33p

CyberCr33p

Today I had a server crash. I run FreeBSD 12-STABLE. Any idea what can cause it?

Code: 
Jun 26 19:08:00 server.example.com kernel: pid 6914 (php-fpm), jid 3, uid 1005: exited on signal 11
Jun 26 19:08:00 server.example.com named[90507]: random.c:387: REQUIRE((__builtin_expect(((rng) != ((void *)0)), 1) && __builtin_expect((((const isc__magic_t *)(rng))->magic
== ((('R') << 24 | ('N') << 16 | ('G') << 8 | ('x')))), 1))) failed, back trace
Jun 26 19:08:00 server.example.com kernel: Jun 26 19:08:00 server6 named[90507]: random.c:387: REQUIRE((__builtin_expect(((rng) != ((void *)0)), 1) && __builtin_expect((((const isc__magic_t *)(rng))->magic == ((('R') << 24 | ('N') << 16 | ('G') << 8 | ('x')))), 1))) failed, back trace
Jun 26 19:08:00 server.example.com kernel: pid 90507 (named), jid 0, uid 53: exited on signal 6
Jun 26 19:08:00 server.example.com named[90507]: #0 0x2dcf80 in ??
Jun 26 19:08:00 server.example.com kernel: Jun 26 19:08:00 server6 named[90507]: #0 0x2dcf80 in ??
Jun 26 19:08:00 server.example.com named[90507]: #1 0x4c781a in ??
Jun 26 19:08:00 server.example.com kernel: Jun 26 19:08:00 server6 named[90507]: #1 0x4c781a in ??
Jun 26 19:08:00 server.example.com named[90507]: #2 0x4e356c in ??
Jun 26 19:08:00 server.example.com kernel: Jun 26 19:08:00 server6 named[90507]: #2 0x4e356c in ??
Jun 26 19:08:00 server.example.com named[90507]: #3 0x364335 in ??
Jun 26 19:08:00 server.example.com kernel: Jun 26 19:08:00 server6 named[90507]: #3 0x364335 in ??
Jun 26 19:08:00 server.example.com named[90507]: #4 0x363e1b in ??
Jun 26 19:08:00 server.example.com kernel: Jun 26 19:08:00 server6 named[90507]: #4 0x363e1b in ??
Jun 26 19:08:00 server.example.com named[90507]: #5 0x4252f3 in ??
Jun 26 19:08:00 server.example.com kernel: Jun 26 19:08:00 server6 named[90507]: #5 0x4252f3 in ??
Jun 26 19:08:00 server.example.com named[90507]: #6 0x42379a in ??
Jun 26 19:08:00 server.example.com kernel: Jun 26 19:08:00 server6 named[90507]: #6 0x42379a in ??
Jun 26 19:08:00 server.example.com named[90507]: #7 0x422498 in ??
Jun 26 19:08:00 server.example.com kernel: Jun 26 19:08:00 server6 named[90507]: #7 0x422498 in ??
Jun 26 19:08:00 server.example.com named[90507]: #8 0x4eddab in ??
Jun 26 19:08:00 server.example.com kernel: Jun 26 19:08:00 server6 named[90507]: #8 0x4eddab in ??
Jun 26 19:08:00 server.example.com named[90507]: #9 0x800a82746 in ??
Jun 26 19:08:00 server.example.com kernel: Jun 26 19:08:00 server6 named[90507]: #9 0x800a82746 in ??
Jun 26 19:08:00 server.example.com named[90507]: exiting (due to assertion failure)
Jun 26 19:08:00 server.example.com kernel: Jun 26 19:08:00 server6 named[90507]: exiting (due to assertion failure)
Jun 26 19:11:49 server.example.com syslogd: restart
Jun 26 19:11:49 server.example.com syslogd: kernel boot file is /boot/kernel/kernel
Jun 26 19:11:49 server.example.com kernel:
Jun 26 19:11:49 server.example.com syslogd: last message repeated 1 times
Jun 26 19:11:49 server.example.com kernel: Fatal trap 12: page fault while in kernel mode
Jun 26 19:11:49 server.example.com kernel: cpuid = 2; apic id = 02
Jun 26 19:11:49 server.example.com kernel: fault virtual address#011= 0x31
Jun 26 19:11:49 server.example.com kernel: fault code#011#011= supervisor write data, page not present
Jun 26 19:11:49 server.example.com kernel: instruction pointer#011= 0x20:0xffffffff8109a439
Jun 26 19:11:49 server.example.com kernel: stack pointer#011        = 0x28:0xffffffff827166e0
Jun 26 19:11:49 server.example.com kernel: frame pointer#011        = 0x28:0xffffffff827167c0
Jun 26 19:11:49 server.example.com kernel: code segment#011#011= base 0x0, limit 0xfffff, type 0x1b
Jun 26 19:11:49 server.example.com kernel: #011#011#011= DPL 0, pres 1, long 1, def32 0, gran 1
Jun 26 19:11:49 server.example.com kernel: processor eflags#011= interrupt enabled, resume, IOPL = 0
Jun 26 19:11:49 server.example.com kernel: current process#011#011= 79168 (mysqld)
Jun 26 19:11:49 server.example.com kernel: trap number#011#011= 12
Jun 26 19:11:49 server.example.com kernel: panic: page fault
Jun 26 19:11:49 server.example.com kernel: cpuid = 2
Jun 26 19:11:49 server.example.com kernel: time = 1561565281
Jun 26 19:11:49 server.example.com kernel: KDB: stack backtrace:
Jun 26 19:11:49 server.example.com kernel: #0 0xffffffff80c16e77 at kdb_backtrace+0x67
 
Probably something that happened before, it starts with a php-fpm crashing (signal 11), then named borks and the panic seems to have been triggered by mysqld. Check the logs from before 19:08:00.
 
It doesn't show anything related:

Code: 
Jun 26 19:06:46 server.example.com pure-ftpd[48894]: (?@127.0.0.1) [INFO] Logout.
Jun 26 19:06:46 server.example.com postfix-smtp/smtpd[85002]: connect from localhost[127.0.0.1]
Jun 26 19:06:46 server.example.com postfix-smtp/smtpd[85002]: disconnect from localhost[127.0.0.1] ehlo=1 quit=1 commands=2
Jun 26 19:06:56 server.example.com kernel: ipfw: 56599 Deny TCP 92.118.37.84:41610 5.9.31.xxx:9576 in via em0
Jun 26 19:07:07 server.example.com kernel: ipfw: 56599 Deny TCP 5.113.16.224:49730 5.9.31.xxx:445 in via em0
Jun 26 19:07:08 server.example.com kernel: ipfw: 56599 Deny TCP 5.113.16.224:51189 5.9.31.123:445 in via em0
Jun 26 19:07:09 server.example.com kernel: ipfw: 56599 Deny TCP 119.147.218.37:40919 5.9.172.26:445 in via em0
Jun 26 19:07:11 server.example.com kernel: ipfw: 56599 Deny TCP 92.118.37.84:41610 5.9.172.26:8012 in via em0
Jun 26 19:07:16 server.example.com pure-ftpd[72773]: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
Jun 26 19:07:16 server.example.com pure-ftpd[72773]: (?@127.0.0.1) [INFO] Logout.
Jun 26 19:07:16 server.example.com postfix-smtp/smtpd[89847]: connect from localhost[127.0.0.1]
Jun 26 19:07:16 server.example.com postfix-smtp/smtpd[89847]: disconnect from localhost[127.0.0.1] ehlo=1 quit=1 commands=2
Jun 26 19:07:33 server.example.com kernel: ipfw: 56599 Deny TCP 92.118.37.84:41610 5.9.31.123:21176 in via em0
Jun 26 19:07:38 server.example.com kernel: ipfw: 56599 Deny TCP 92.118.37.84:41610 5.9.172.26:29792 in via em0
Jun 26 19:07:46 server.example.com pure-ftpd[14466]: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
Jun 26 19:07:46 server.example.com pure-ftpd[14466]: (?@127.0.0.1) [INFO] Logout.
Jun 26 19:07:46 server.example.com postfix-smtp/smtpd[78481]: connect from localhost[127.0.0.1]
Jun 26 19:07:46 server.example.com postfix-smtp/smtpd[78481]: disconnect from localhost[127.0.0.1] ehlo=1 quit=1 commands=2
Jun 26 19:07:59 server.example.com kernel: ipfw: 56599 Deny TCP 27.223.200.217:42844 5.9.172.26:22 in via em0
Jun 26 19:08:00 server.example.com kernel: pid 6914 (php-fpm), jid 3, uid 1005: exited on signal 11
 
Is this server-grade hardware too? I mean does it have ECC memory? With seemingly random processes just crashing I'm inclined to blame bad or broken memory. But if it's ECC memory I would have expected to see a bunch of MCA errors about it.
 
Code: 
nm -n /boot/kernel/kernel | grep ffffffff8109a

ffffffff8109a800 t free_pv_chunk
ffffffff8109a8f0 T pmap_is_modified
ffffffff8109a910 t pmap_page_test_mappings
ffffffff8109af70 T pmap_is_prefaultable
 
You might want to check the memory in that case. Also check the status of your disks, a bad sector in your swap partition can lead to some weird and wonderful crashes too.
 
You must log in or register to reply here.
Back
Top