Help find the cause kernel panic

evgenv

evgenv

There are two server on FreeBSD 9.1-RELEASE-p1, yesterday began to go to reboot for no apparent reason, to rebut go with a difference an hour, in /var/messages only
Code: 
reboot after panic: page fault

In FreeBSD does not tell where to look strong. While only did backtrace core dump.

Code: 
root@nas1:/boot/kernel # kgdb kernel.symbols /var/crash/vmcore.1
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff8057d166 at kdb_backtrace+0x66
#1 0xffffffff8054717e at panic+0x1ce
#2 0xffffffff8078d850 at trap_fatal+0x290
#3 0xffffffff8078db8d at trap_pfault+0x1ed
#4 0xffffffff8078e1ae at trap+0x3ce
#5 0xffffffff8077876f at calltrap+0x8
#6 0xffffffff8064f458 at ip_fragment+0x1e8
#7 0xffffffff8064fda6 at ip_output+0x706
#8 0xffffffff8064cc13 at ip_forward+0x303
#9 0xffffffff8064e2c8 at ip_input+0x5c8
#10 0xffffffff80608d6b at netisr_dispatch_src+0x20b
#11 0xffffffff8061c284 at ng_iface_rcvdata+0x114
#12 0xffffffff80617c88 at ng_apply_item+0x208
#13 0xffffffff80616dee at ng_snd_item+0x39e
#14 0xffffffff80617c88 at ng_apply_item+0x208
#15 0xffffffff80616dee at ng_snd_item+0x39e
#16 0xffffffff80619bcc at ng_car_rcvdata+0xbc
#17 0xffffffff80617c88 at ng_apply_item+0x208
Uptime: 5h20m46s
Dumping 697 out of 4075 MB:..3%..12%..21%..33%..42%..51%..62%..72%..81%..92%

Reading symbols from /boot/kernel/ng_vjc.ko...Reading symbols from /boot/kernel/ng_vjc.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/ng_vjc.ko
#0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
224   pcpu.h: No such file or directory.
   in pcpu.h
(kgdb) backtrace 
#0  doadump (textdump=Variable "textdump" is not available.
) at pcpu.h:224
#1  0xffffffff80546c61 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:448
#2  0xffffffff80547157 in panic (fmt=0x1 <Address 0x1 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:636
#3  0xffffffff8078d850 in trap_fatal (frame=0xc, eva=Variable "eva" is not available.
) at /usr/src/sys/amd64/amd64/trap.c:857
#4  0xffffffff8078db8d in trap_pfault (frame=0xffffff800026eb00, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:773
#5  0xffffffff8078e1ae in trap (frame=0xffffff800026eb00) at /usr/src/sys/amd64/amd64/trap.c:456
#6  0xffffffff8077876f in calltrap () at /usr/src/sys/amd64/amd64/exception.S:228
#7  0xffffffff805a8507 in m_copym (m=0x0, off0=1500, len=1480, wait=1) at /usr/src/sys/kern/uipc_mbuf.c:542
#8  0xffffffff8064f458 in ip_fragment (ip=0xfffffe0008be8058, m_frag=0xffffff800026ed28, mtu=Variable "mtu" is not available.
) at /usr/src/sys/netinet/ip_output.c:822
#9  0xffffffff8064fda6 in ip_output (m=0xfffffe0008be8000, opt=Variable "opt" is not available.
) at /usr/src/sys/netinet/ip_output.c:653
#10 0xffffffff8064cc13 in ip_forward (m=0xfffffe011275d400, srcrt=Variable "srcrt" is not available.
) at /usr/src/sys/netinet/ip_input.c:1494
#11 0xffffffff8064e2c8 in ip_input (m=0xfffffe011275d400) at /usr/src/sys/netinet/ip_input.c:702
#12 0xffffffff80608d6b in netisr_dispatch_src (proto=1, source=Variable "source" is not available.
) at /usr/src/sys/net/netisr.c:1013
#13 0xffffffff8061c284 in ng_iface_rcvdata (hook=Variable "hook" is not available.
) at /usr/src/sys/netgraph/ng_iface.c:781
#14 0xffffffff80617c88 in ng_apply_item (node=0xfffffe00165f4200, item=0xfffffe0125fa1080, rw=0) at /usr/src/sys/netgraph/ng_base.c:2391
#15 0xffffffff80616dee in ng_snd_item (item=Variable "item" is not available.
) at /usr/src/sys/netgraph/ng_base.c:2308
#16 0xffffffff80617c88 in ng_apply_item (node=0xfffffe01257da500, item=0xfffffe0125fa1080, rw=0) at /usr/src/sys/netgraph/ng_base.c:2391
#17 0xffffffff80616dee in ng_snd_item (item=Variable "item" is not available.
) at /usr/src/sys/netgraph/ng_base.c:2308
#18 0xffffffff80619bcc in ng_car_rcvdata (hook=Variable "hook" is not available.
) at /usr/src/sys/netgraph/ng_car.c:365
#19 0xffffffff80617c88 in ng_apply_item (node=0xfffffe01256fd600, item=0xfffffe0125fa1080, rw=0) at /usr/src/sys/netgraph/ng_base.c:2391
#20 0xffffffff80616dee in ng_snd_item (item=Variable "item" is not available.
) at /usr/src/sys/netgraph/ng_base.c:2308
#21 0xffffffff80617c88 in ng_apply_item (node=0xfffffe01257da500, item=0xfffffe0125fa1080, rw=0) at /usr/src/sys/netgraph/ng_base.c:2391
#22 0xffffffff80616dee in ng_snd_item (item=Variable "item" is not available.
) at /usr/src/sys/netgraph/ng_base.c:2308
#23 0xffffffff80617c88 in ng_apply_item (node=0xfffffe0125717200, item=0xfffffe0125fa1080, rw=0) at /usr/src/sys/netgraph/ng_base.c:2391
#24 0xffffffff80616dee in ng_snd_item (item=Variable "item" is not available.
) at /usr/src/sys/netgraph/ng_base.c:2308
#25 0xffffffff806255ae in ng_ppp_rcvdata (hook=Variable "hook" is not available.
) at /usr/src/sys/netgraph/ng_ppp.c:1522
#26 0xffffffff80617c88 in ng_apply_item (node=0xfffffe005df00a00, item=0xfffffe0125fa1080, rw=0) at /usr/src/sys/netgraph/ng_base.c:2391
#27 0xffffffff80616dee in ng_snd_item (item=Variable "item" is not available.
) at /usr/src/sys/netgraph/ng_base.c:2308
#28 0xffffffff80617c88 in ng_apply_item (node=0xfffffe001677bc00, item=0xfffffe0125fa1080, rw=0) at /usr/src/sys/netgraph/ng_base.c:2391
#29 0xffffffff80616dee in ng_snd_item (item=Variable "item" is not available.
) at /usr/src/sys/netgraph/ng_base.c:2308
#30 0xffffffff80617c88 in ng_apply_item (node=0xfffffe00161c6800, item=0xfffffe0125fa1080, rw=0) at /usr/src/sys/netgraph/ng_base.c:2391
#31 0xffffffff80616dee in ng_snd_item (item=Variable "item" is not available.
) at /usr/src/sys/netgraph/ng_base.c:2308
#32 0xffffffff8061e90a in ng_ksocket_incoming2 (node=0xfffffe001695f000, hook=Variable "hook" is not available.
) at /usr/src/sys/netgraph/ng_ksocket.c:1150
#33 0xffffffff80617d8b in ng_apply_item (node=0xfffffe001695f000, item=0xfffffe0016ad7080, rw=1) at /usr/src/sys/netgraph/ng_base.c:2462
#34 0xffffffff80619027 in ngthread (arg=Variable "arg" is not available.
) at /usr/src/sys/netgraph/ng_base.c:3426
#35 0xffffffff8051983f in fork_exit (callout=0xffffffff80618e60 <ngthread>, arg=0x0, frame=0xffffff800026fc40) at /usr/src/sys/kern/kern_fork.c:992
#36 0xffffffff80778c9e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:602
#37 0x0000000000000000 in ?? ()
#38 0x0000000000000000 in ?? ()
#39 0x0000000000000001 in ?? ()
#40 0x0000000000000000 in ?? ()
#41 0x0000000000000000 in ?? ()
#42 0x0000000000000000 in ?? ()
#43 0x0000000000000000 in ?? ()
#44 0x0000000000000000 in ?? ()
#45 0x0000000000000000 in ?? ()
#46 0x0000000000000000 in ?? ()
#47 0x0000000000000000 in ?? ()
#48 0x0000000000000000 in ?? ()
#49 0x0000000000000000 in ?? ()
#50 0x0000000000000000 in ?? ()
#51 0x0000000000000000 in ?? ()
#52 0x0000000000000000 in ?? ()
#53 0x0000000000000000 in ?? ()
#54 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit---
 
Thank you!

Prompt yet to what version better upgrade: 9.1-RELEASE-p12 or 9.2-RELEASE? We have Custom Kernel, in both cases will have to update with Generic Kernel, and then recompile the Custom?

Kernel config:
Code: 
#
# GENERIC -- Generic kernel configuration file for FreeBSD/amd64
#
# For more information on this file, please read the config(5) manual page,
# and/or the handbook section on Kernel Configuration Files:
#
#    http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-config.html
#
# The handbook is also available locally in /usr/share/doc/handbook
# if you've installed the doc distribution, otherwise always see the
# FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the
# latest information.
#
# An exhaustive list of options and more detailed explanations of the
# device lines is also present in the ../../conf/NOTES and NOTES files.
# If you are in doubt as to the purpose or necessity of a line, check first
# in NOTES.
#
# $FreeBSD: releng/9.1/sys/amd64/conf/GENERIC 238090 2012-07-04 00:54:16Z delphij $

cpu      HAMMER
ident      nas1

makeoptions   DEBUG=-g      # Build kernel with gdb(1) debug symbols

options    SCHED_ULE      # ULE scheduler
options    PREEMPTION      # Enable kernel thread preemption
options    INET         # InterNETworking
#options    INET6         # IPv6 communications protocols
options    SCTP         # Stream Control Transmission Protocol
options    FFS         # Berkeley Fast Filesystem
options    SOFTUPDATES      # Enable FFS soft updates support
options    UFS_ACL         # Support for access control lists
options    UFS_DIRHASH      # Improve performance on big directories
options    UFS_GJOURNAL      # Enable gjournal-based UFS journaling
options    MD_ROOT         # MD is a potential root device
options    NFSCL         # New Network Filesystem Client
options    NFSD         # New Network Filesystem Server
options    NFSLOCKD      # Network Lock Manager
options    NFS_ROOT      # NFS usable as /, requires NFSCL
options    MSDOSFS         # MSDOS Filesystem
options    CD9660         # ISO 9660 Filesystem
options    PROCFS         # Process filesystem (requires PSEUDOFS)
options    PSEUDOFS      # Pseudo-filesystem framework
options    GEOM_PART_GPT      # GUID Partition Tables.
options    GEOM_RAID      # Soft RAID functionality.
options    GEOM_LABEL      # Provides labelization
options    COMPAT_FREEBSD32   # Compatible with i386 binaries
options    COMPAT_FREEBSD4      # Compatible with FreeBSD4
options    COMPAT_FREEBSD5      # Compatible with FreeBSD5
options    COMPAT_FREEBSD6      # Compatible with FreeBSD6
options    COMPAT_FREEBSD7      # Compatible with FreeBSD7
options    SCSI_DELAY=5000      # Delay (in ms) before probing SCSI
options    KTRACE         # ktrace(1) support
options    STACK         # stack(9) support
options    SYSVSHM         # SYSV-style shared memory
options    SYSVMSG         # SYSV-style message queues
options    SYSVSEM         # SYSV-style semaphores
options    _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options    PRINTF_BUFR_SIZE=128   # Prevent printf output being interspersed.
options    KBD_INSTALL_CDEV   # install a CDEV entry in /dev
options    HWPMC_HOOKS      # Necessary kernel hooks for hwpmc(4)
options    AUDIT         # Security event auditing
options    MAC         # TrustedBSD MAC Framework
#options    KDTRACE_FRAME      # Ensure frames are compiled in
#options    KDTRACE_HOOKS      # Kernel DTrace hooks
options    INCLUDE_CONFIG_FILE     # Include this file in kernel
options    KDB         # Kernel debugger related code
options    KDB_TRACE      # Print a stack trace for a panic

# Make an SMP-capable kernel by default
options    SMP         # Symmetric MultiProcessor Kernel


# IPFW
options      IPFIREWALL
options      IPFIREWALL_VERBOSE
options      IPFIREWALL_VERBOSE_LIMIT=1000
options      IPFIREWALL_FORWARD
options      IPFIREWALL_NAT

options      LIBALIAS
options      IPFILTER
options      IPDIVERT

# NETGRAPH
options      NETGRAPH
options      NETGRAPH_IPFW
options      NETGRAPH_TEE
options      NETGRAPH_NETFLOW
options      NETGRAPH_SPLIT
options      NETGRAPH_BPF
options      NETGRAPH_CAR
options      NETGRAPH_ECHO
options      NETGRAPH_ETHER
options      NETGRAPH_NAT
options      NETGRAPH_PPP
options      NETGRAPH_PPTPGRE
options      NETGRAPH_TCPMSS
options      NETGRAPH_SOCKET
options      NETGRAPH_KSOCKET
options      NETGRAPH_MPPC_ENCRYPTION
options      NETGRAPH_IFACE
options      NETGRAPH_DEVICE

# UTF-8
options      TEKEN_UTF8
#options      TEKEN_XTERM

# CARP
device      carp

# CPU frequency control
device      cpufreq

# Bus support.
device      acpi
device      pci

# ATA controllers
device      ahci      # AHCI-compatible SATA controllers
device      ata      # Legacy ATA/SATA controllers
options    ATA_CAM      # Handle legacy controllers with CAM
options    ATA_STATIC_ID   # Static device numbering
device      mvs      # Marvell 88SX50XX/88SX60XX/88SX70XX/SoC SATA
device      siis      # SiliconImage SiI3124/SiI3132/SiI3531 SATA


# ATA/SCSI peripherals
device      scbus      # SCSI bus (required for ATA/SCSI)
#device      ch      # SCSI media changers
device      da      # Direct Access (disks)
#device      sa      # Sequential Access (tape etc)
device      cd      # CD
device      pass      # Passthrough device (direct ATA/SCSI access)
device      ses      # Enclosure Services (SES and SAF-TE)
device      ctl      # CAM Target Layer

# atkbdc0 controls both the keyboard and the PS/2 mouse
device      atkbdc      # AT keyboard controller
device      atkbd      # AT keyboard
device      psm      # PS/2 mouse

device      kbdmux      # keyboard multiplexer

device      vga      # VGA video card driver
options    VESA      # Add support for VESA BIOS Extensions (VBE)

#device      splash      # Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device      sc
options    SC_PIXEL_MODE   # add support for the raster text mode

device      agp      # support several AGP chipsets

# Serial (COM) ports
device      uart      # Generic UART driver

# PCI Ethernet NICs.
#device      bxe      # Broadcom BCM57710/BCM57711/BCM57711E 10Gb Ethernet
#device      de      # DEC/Intel DC21x4x (``Tulip'')
device      em      # Intel PRO/1000 Gigabit Ethernet Family
device      igb      # Intel PRO/1000 PCIE Server Gigabit Family
device      ixgbe      # Intel PRO/10GbE PCIE Ethernet Family
#device      le      # AMD Am7900 LANCE and Am79C9xx PCnet
#device      ti      # Alteon Networks Tigon I/II gigabit Ethernet
#device      txp      # 3Com 3cR990 (``Typhoon'')
#device      vx      # 3Com 3c590, 3c595 (``Vortex'')

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device      miibus      # MII bus support

# Pseudo devices.
device      loop      # Network loopback
device      random      # Entropy device
device      ether      # Ethernet support
device      vlan      # 802.1Q VLAN support
device      tun      # Packet tunnel.
device      pty      # BSD-style compatibility pseudo ttys
device      md      # Memory "disks"
device      gif      # IPv6 and IPv4 tunneling
device      faith      # IPv6-to-IPv4 relaying (translation)
device      firmware   # firmware assist module

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device      bpf      # Berkeley packet filter

# USB support
#options    USB_DEBUG   # enable debug msgs
device      uhci      # UHCI PCI->USB interface
device      ohci      # OHCI PCI->USB interface
device      ehci      # EHCI PCI->USB interface (USB 2.0)
device      xhci      # XHCI PCI->USB interface (USB 3.0)
device      usb      # USB Bus (required)
device      uhid      # "Human Interface Devices"
device      ukbd      # Keyboard
device      umass      # Disks/Mass storage - Requires scbus and da
device      ums      # Mouse
 
evgenv said:
Prompt yet to what version better upgrade: 9.1-RELEASE-p12 or 9.2-RELEASE?
Click to expand...
Doesn't matter much, 9.1-RELEASE is supported until December 2014, 9.2-RELEASE is supported until September 2014.

We have Custom Kernel, in both cases will have to update with Generic Kernel, and then recompile the Custom?
Click to expand...
If you use freebsd-update(8), yes.
 
I updated FreeBSD, ports and applications, but the server still panics :-( I can not understand why. At the debugger all the same that in the first message.
Code: 
root@nas1:/usr/obj/usr/src/sys/nas1 # uname -a
FreeBSD nas1.domain.net 9.1-RELEASE-p13 FreeBSD 9.1-RELEASE-p13 #1: Tue May 20 16:01:35 MSK 2014     root@nas1.domain.net:/usr/obj/usr/src/sys/nas1  amd64
It began two weeks ago, before the server worked without problems.

If we add a rule to ipfw
Code: 
deny icmp from any to any
the server works without problems, but lost the opportunity to ping hosts.
 
You must log in or register to reply here.
Back
Top