Solved Kernel Panics under high ZFS Load (locate database building, etc.)

epopen

epopen

Hi Everyone.

My PC is FreeBSD 10.1-Release p9 AMD64 @ 4GB RAM
But encounter kernel panic problem in after boot several minutes problem started at last month.
Like under panic message.
Code: 
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x20
fault code     = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff810962d4
stack pointer    = 0x28:0xfffffe006b96a5f0
frame pointer    = 0x28:0xfffffe006b96a640
code segment     = base 0x0, limit 0xfffff, type 0x1b
       = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process     = 55286 (ccache)
trap number     = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff804b8680 at kdb_backtrace+0x60
#1 0xffffffff80480cc3 at panic+0x153
#2 0xffffffff8072debf at trap_fatal+0x38f
#3 0xffffffff8072e16f at trap_pfault+0x29f
#4 0xffffffff8072d875 at trap+0x425
#5 0xffffffff80713ac3 at calltrap+0x8
#6 0xffffffff81094fbe at fzap_cursor_retrieve+0x16e
#7 0xffffffff81099f3f at zap_cursor_retrieve+0x1bf
#8 0xffffffff810d35f1 at zfs_freebsd_readdir+0x3e1
#9 0xffffffff807a2550 at VOP_READDIR_APV+0x80
#10 0xffffffff805306ac at kern_getdirentries+0x21c
#11 0xffffffff80530468 at sys_getdirentries+0x28
#12 0xffffffff8072e67a at amd64_syscall+0x25a
#13 0xffffffff80713dab at Xfast_syscall+0xfb
Uptime: 4m32s
Dumping 373 out of 3920 MB:..5%..13%..22%..31%..43%..52%..61%..73%..82%..91%

The problem occur high ZFS access (ex locate datebase building).
My dmesg -a is http://pastebin.com/K5RYNYrv

And Kernel configuration
Code: 
machine   amd64
cpu   HAMMER
makeoptions   DEBUG=-g
options   ALTQ_NOPCC
options   ALTQ_PRIQ
options   ALTQ_HFSC
options   ALTQ_RIO
options   ALTQ_RED
options   ALTQ_CBQ
options   ALTQ
options   FLOWTABLE
options   DEVICE_POLLING
options   BPF_JITTER
options   ACPI_DEBUG
options   TMPFS
options   VFS_AIO
options   DIRECTIO
options   UFS_EXTATTR_AUTOSTART
options   UFS_EXTATTR
options   SUIDDIR
options   RCTL
options   RACCT
options   P1003_1B_MQUEUE
options   P1003_1B_SEMAPHORES
options   GEOM_CACHE
options   MPTABLE_FORCE_HTT
options   IPI_PREEMPTION
options   USB_DEBUG
options   IEEE80211_AMPDU_AGE
options   IEEE80211_DEBUG
options   SC_PIXEL_MODE
options   VESA
options   ACPI_DMAR
options   SMP
options   KDB_TRACE
options   KDB
options   INCLUDE_CONFIG_FILE
options   MAC
options   PROCDESC
options   CAPABILITIES
options   CAPABILITY_MODE
options   AUDIT
options   HWPMC_HOOKS
options   KBD_INSTALL_CDEV
options   PRINTF_BUFR_SIZE=128
options   _KPOSIX_PRIORITY_SCHEDULING
options   SYSVSEM
options   SYSVMSG
options   SYSVSHM
options   STACK
options   SCSI_DELAY=5000
options   COMPAT_FREEBSD32
options   GEOM_LABEL
options   GEOM_PART_GPT
options   PSEUDOFS
options   PROCFS
options   MD_ROOT
options   QUOTA
options   UFS_DIRHASH
options   UFS_ACL
options   SOFTUPDATES
options   FFS
options   SCTP
options   TCP_OFFLOAD
options   INET6
options   INET
options   PREEMPTION
options   SCHED_ULE
options   NEW_PCIB
options   GEOM_PART_MBR
options   GEOM_PART_EBR_COMPAT
options   GEOM_PART_EBR
options   GEOM_PART_BSD
device   isa
device   mem
device   io
device   uart_ns8250
device   cpufreq
device   acpi
device   pci
device   ahci
device   scbus
device   da
device   cd
device   pass
device   atkbdc
device   atkbd
device   psm
device   vga
device   splash
device   sc
device   agp
device   miibus
device   msk
device   wlan
device   wlan_ccmp
device   wlan_amrr
device   iwn
device   loop
device   random
device   ether
device   firmware
device   bpf
device   uhci
device   ehci
device   usb
device   ukbd
device   umass
device   uhid
device   ulpt
device   ums
device   sound
device   snd_hda
device   snd_ich
device   mmc
device   mmcsd
device   sdhci
device   virtio
device   virtio_pci
device   vtnet
device   virtio_blk
device   virtio_scsi
device   virtio_balloon
device   cpuctl
device   mptable
device   ada
device   acpi_video
device   acpi_wmi
device   smb
device   smbus
device   hwpmc
device   pf
device   pflog
device   pfsync
device   coretemp
device   ichsmb
device   ichwd
device   acpi_toshiba
device   iwn5000fw

I tried kgdb kernel.debug vmcore0 and got under result
Code: 
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x20
fault code     = supervisor read data, page not present
instruction pointer   = 0x20:0xffffffff810962d4
stack pointer    = 0x28:0xfffffe006b96a5f0
frame pointer    = 0x28:0xfffffe006b96a640
code segment     = base 0x0, limit 0xfffff, type 0x1b
       = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags   = interrupt enabled, resume, IOPL = 0
current process     = 55286 (ccache)
trap number     = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0xffffffff804b8680 at kdb_backtrace+0x60
#1 0xffffffff80480cc3 at panic+0x153
#2 0xffffffff8072debf at trap_fatal+0x38f
#3 0xffffffff8072e16f at trap_pfault+0x29f
#4 0xffffffff8072d875 at trap+0x425
#5 0xffffffff80713ac3 at calltrap+0x8
#6 0xffffffff81094fbe at fzap_cursor_retrieve+0x16e
#7 0xffffffff81099f3f at zap_cursor_retrieve+0x1bf
#8 0xffffffff810d35f1 at zfs_freebsd_readdir+0x3e1
#9 0xffffffff807a2550 at VOP_READDIR_APV+0x80
#10 0xffffffff805306ac at kern_getdirentries+0x21c
#11 0xffffffff80530468 at sys_getdirentries+0x28
#12 0xffffffff8072e67a at amd64_syscall+0x25a
#13 0xffffffff80713dab at Xfast_syscall+0xfb
Uptime: 4m32s
Dumping 373 out of 3920 MB:..5%..13%..22%..31%..43%..52%..61%..73%..82%..91%

Reading symbols from /boot/kernel/linux.ko.symbols...done.
Loaded symbols for /boot/kernel/linux.ko.symbols
Reading symbols from /boot/modules/vboxdrv.ko...done.
Loaded symbols for /boot/modules/vboxdrv.ko
Reading symbols from /boot/modules/cuse4bsd.ko...done.
Loaded symbols for /boot/modules/cuse4bsd.ko
Reading symbols from /boot/kernel/zfs.ko.symbols...done.
Loaded symbols for /boot/kernel/zfs.ko.symbols
Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
Loaded symbols for /boot/kernel/opensolaris.ko.symbols
Reading symbols from /boot/kernel/krpc.ko.symbols...done.
Loaded symbols for /boot/kernel/krpc.ko.symbols
Reading symbols from /boot/kernel/linprocfs.ko.symbols...done.
Loaded symbols for /boot/kernel/linprocfs.ko.symbols
Reading symbols from /boot/kernel/wlan_wep.ko.symbols...done.
Loaded symbols for /boot/kernel/wlan_wep.ko.symbols
Reading symbols from /boot/kernel/wlan_tkip.ko.symbols...done.
Loaded symbols for /boot/kernel/wlan_tkip.ko.symbols
Reading symbols from /boot/modules/vboxnetflt.ko...done.
Loaded symbols for /boot/modules/vboxnetflt.ko
Reading symbols from /boot/kernel/netgraph.ko.symbols...done.
Loaded symbols for /boot/kernel/netgraph.ko.symbols
Reading symbols from /boot/kernel/ng_ether.ko.symbols...done.
Loaded symbols for /boot/kernel/ng_ether.ko.symbols
Reading symbols from /boot/modules/vboxnetadp.ko...done.
Loaded symbols for /boot/modules/vboxnetadp.ko
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
219   pcpu.h: No such file or directory.
   in pcpu.h

And (kgdb) backtrace
Code: 
#0  doadump (textdump=<value optimized out>) at pcpu.h:219
#1  0xffffffff80480942 in kern_reboot (howto=260)
  at /usr/src/sys/kern/kern_shutdown.c:452
#2  0xffffffff80480d02 in panic (fmt=<value optimized out>)
  at /usr/src/sys/kern/kern_shutdown.c:759
#3  0xffffffff8072debf in trap_fatal (frame=<value optimized out>,
  eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:865
#4  0xffffffff8072e16f in trap_pfault (frame=0xfffffe006b96a540,
  usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:676
#5  0xffffffff8072d875 in trap (frame=0xfffffe006b96a540)
  at /usr/src/sys/amd64/amd64/trap.c:440
#6  0xffffffff80713ac3 in calltrap ()
  at /usr/src/sys/amd64/amd64/exception.S:232
#7  0xffffffff810962d4 in zap_leaf_lookup_closest (l=0xfffff8012dca9a00, h=0,
  cd=0, zeh=0xfffffe006b96a658)
  at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_leaf.c:458
#8  0xffffffff81094fbe in fzap_cursor_retrieve (zap=0xfffff8012dca9a00,
  zc=0xfffffe006b96a918, za=0xfffffe006b96a800)
  at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap.c:1190
#9  0xffffffff81099f3f in zap_cursor_retrieve (zc=0xfffffe006b96a918,
  za=0xfffffe006b96a800)
  at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs---Type <return> to continue, or q <return> to quit---
/zap_micro.c:1290
#10 0xffffffff810d35f1 in zfs_freebsd_readdir (ap=<value optimized out>)
  at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:2565
#11 0xffffffff807a2550 in VOP_READDIR_APV (vop=<value optimized out>,
  a=<value optimized out>) at vnode_if.c:1821
#12 0xffffffff805306ac in kern_getdirentries (td=0xfffff80005e54920,
  fd=<value optimized out>,
  buf=0x801830000 <Address 0x801830000 out of bounds>,
  count=<value optimized out>, basep=0xfffffe006b96aad0, residp=0x0)
  at vnode_if.h:758
#13 0xffffffff80530468 in sys_getdirentries (td=0xfffff8012dca9a00,
  uap=0xfffffe006b96ab80) at /usr/src/sys/kern/vfs_syscalls.c:4003
#14 0xffffffff8072e67a in amd64_syscall (td=0xfffff80005e54920, traced=0)
  at subr_syscall.c:134
#15 0xffffffff80713dab in Xfast_syscall ()
  at /usr/src/sys/amd64/amd64/exception.S:391
#16 0x0000000800d00dfa in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
Finally , I tried (kgdb)list *0xffffffff810962d4 (instruction pointer)and got under result.
Code: 
0xffffffff810962d4 is in zap_leaf_lookup_closest (/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_leaf.c:458).
453    unit16_t lh;
454    struct zap_leaf_entry *le;
455
456     ASSERT3U(l->l_phys->l_hdr.lh_magic, ==, ZAP_LEAF_MAGIC);
457  
458     for (lh = LEAF_HASH(l, h); lh <= bestlh; lh++) {
459       for (chunk = l->l_phys->l_hash[lh];
460        chunk != CHAIN_END; chunk = le->le_next) {
461         le = ZAP_LEAF_ENTRY(l, chunk);
Look like loop access out of range?

I thinked storage(Intel 540 SSD 240GB) fault started last month.
But I tried smartctl -c -t offline & long /dev/ada0 and haven't any error mesage.
And zpool strub MYPOOL report "No known error"

I can't proper debug and don't know root cause.
Can help me?

Thanks everyone a lot.
Epopen.
 
Are you using ECC memory? Have you tried backing up your data and reinstall FreeBSD to see if the problem has gone away? You did not give us enough information about your hardware configurations including available space on your drives. Using ZFS requires sufficient RAM and disks cannot be maxed more than 80% of the capacity otherwise you may run into problems with ZFS performance or worse... data corruption.
 
Thanks tobik & Remington

To tobik:
I had been test memory by recommend http://www.memtest.org/ in last 7 hours, tested cycle:7 times, test range:ALL
Result: All pass, no any error.

To Remington:
Use no ECC memory, I can't backup any data because kernel panic when access ZFS pool.
Have suggest about get other require information? Please tell me.
Current capacity:90% almost...

Thanks a lot.
 
What is your RAM size? You may want to try increasing your RAM.

If you value your data, I would suggest replacing your memory to ECC type if your motherboard supports it. Your memory may be okay but it won't stop the corrupted data bits from radio or solar emissions. ECC will take care of that.

You should take your drive offline and duplicate your drive or use acronis true image or dd to copy all the data bits on the disk. It will take a long time to copy or image the drive. I highly recommend to do this first before doing anything further. It's very difficult to recover corrupted zpool which could be causing the panic.

Have you tried booting from FreeBSD USB stick and mount your zpool? If so, then you can try to export your zpool to a larger disk.

Are you using mirror or raidz? If raidz, try adding another same type of drive to expand available capacity.

I don't have any other suggestions except try the above but do duplicate or image your drive first.
 
Remington said:
What is your RAM size? You may want to try increasing your RAM.

If you value your data, I would suggest replacing your memory to ECC type if your motherboard supports it. Your memory may be okay but it won't stop the corrupted data bits from radio or solar emissions. ECC will take care of that.

You should take your drive offline and duplicate your drive or use acronis true image or dd to copy all the data bits on the disk. It will take a long time to copy or image the drive. I highly recommend to do this first before doing anything further. It's very difficult to recover corrupted zpool which could be causing the panic.

Have you tried booting from FreeBSD USB stick and mount your zpool? If so, then you can try to export your zpool to a larger disk.

Are you using mirror or raidz? If raidz, try adding another same type of drive to expand available capacity.

I don't have any other suggestions except try the above but do duplicate or image your drive first.
Click to expand...

Thanks Remington.

About memory, I installed 4GB, it is motherboard maximum memory capacity, and it is Laptop, so ECC memory impossible.
On the Laptop, running ZFS over 2 years and work fine.
I don't why problem started at last month even all of hardware work fine.

I plan to backup whole data and suspend the problem until FreeBSD 10.2 Release and try again.
If 10.2 Release same problem, back to UFS2.

Thanks all a lot.
Neko.
 
I had the same problem, starting about a month ago, with two of my FreeBSD servers, both running ZFS. They had been fine up until a freebsd-update was performed around early March. In my case, what I found was the system would crash when the hard drive was heavily accessed (similar circumstances to what the original poster mentioned).

In my case I was able to work around the problem by:

1) Removing some disk/file checks that were running via cron and causing the heavy load.

2) Setting ZFS's ARC memory usage to a lower amount. For example, in your /boot/loader.conf file, add the lines
Code: 
vfs.zfs.arc_max="1024M"
vfs.zfs.vdev.cache.size="100M"

Feel free to raise or lower the numbers to suit your situation. Since I made these changes, both servers have been on-line and running fine. Perhaps if you can find a way to reduce the amount of disk/ARC usage it'll fix the problem.

In addition, since both of us apparently experienced the problem around the same time, I suspect a system update may have broken something relating to ZFS. Have you tried rolling back your OS to an earlier version/update?
 
  • Thanks
Reactions: Oko
NewGuy said:
I had the same problem, starting about a month ago, with two of my FreeBSD servers, both running ZFS. They had been fine up until a freebsd-update was performed around early March. In my case, what I found was the system would crash when the hard drive was heavily accessed (similar circumstances to what the original poster mentioned).

In my case I was able to work around the problem by:

1) Removing some disk/file checks that were running via cron and causing the heavy load.

2) Setting ZFS's ARC memory usage to a lower amount. For example, in your /boot/loader.conf file, add the lines
Code: 
vfs.zfs.arc_max="1024M"
vfs.zfs.vdev.cache.size="100M"

Feel free to raise or lower the numbers to suit your situation. Since I made these changes, both servers have been on-line and running fine. Perhaps if you can find a way to reduce the amount of disk/ARC usage it'll fix the problem.

In addition, since both of us apparently experienced the problem around the same time, I suspect a system update may have broken something relating to ZFS. Have you tried rolling back your OS to an earlier version/update?
Click to expand...
Thanks your solution NewGuy.

I had been test your solution.
Under is my Laptop's original value in /boot/loader.conf.
Code: 
vfs.zfs.arc_max="256M"
vfs.zfs.vdev.cache.size="64M"
Accord your suggest, I tested some value until your solution.
OF course, reboot after every modified.
But same kernel panic.

I think, IF any ZFS's (e.g. parameter, memory) insufficient or incorrect
SHALL NOT kernel panic, issue error/warning/notice message only.
Bug?

Thanks a lot.
 
Did you also try lower values for ARC_MAX in the /boot/loader.conf file? According to the FreeBSD Handbook, laptops with relatively small amounts of RAM might benefit from having an ARC_MAX under "100M".
 
If I'm reading right the pool is already 90% full is that right? That is a situation ZFS wasn't designed for, it was always assumed that the user would expand or recreate the pool with larger disks before the pool gets too full.
 
Hi All!

I am sorry reply late, because I had been business trip overseas and can't touch my problem laptop until yesterday.

To NewGuy and belon_cfy:
About amount of vfs.zfs.arc_max
I tried 100M/64M/32M/8M in /boot/loader.conf, same kernel panic.

To belon_cfy:
1.I got ftp://ftp.freebsd.org/pub/FreeBSD/s....1-STABLE-amd64-20150421-r281836-memstick.img and tried it.

Boot it in Live CD mode, mount ZFS pool at /tmp/ZFSMOUNTPOINT by under command:
zpool import altroot=/tmp/ZFSMOUNTPOINT MYZPOOL

And test high load situation like periodic locate database building by under command...
grep asdfg -rs /tmp/ZFSMOUNTPOINT and
find /tmp/ZFSMOUNTPOINT MYZPOOL -name xxxxxx

Over 2 hours loop tested, didn't kernel panic, WORK FINE!
The problem fixed in 10.1-STABLE possible I think, Import to 10.2-RELEASE maybe.

Note: In live CD mode sysctl -a value is
Code: 
vfs.zfs.arc_max 2888327168
vfs.zfs.vdev.cache.size: 0
2.Yes, I removed zvol as swap partition after the problem occurred at early time, same kernel panic.

To kpa:
Yes, My zpool approx 93% full.
But shall not kernel panic, If cause poor performance @ 100% full, it OK.
The problem is bug and fixed in 10.1-STABLE I think.

I hope fixed the problem at 10.2-RELEASE.
But thread's title keep "unsolved" until the problem fixed.

Thanks every a lot.
 
Last edited by a moderator:
Same problem here, both 9.3 and 10.1 causes kernel panics under load. FreeBSD 10.2-STABLE fixes this.
 
Hi everyone.

Thread & state update.
Yesterday, I saw some ZFS relation file update in new 10.1-RELEASE patch 12.
I tried rebuild & install world & kernel.
Test result: FAIL, same problem.

Thanks all.
 
Hi everyone.

I had been upgrade to 10.2-RELEASE AMD64
Test result: PASS(No panic, but stall when heavy access, ex: rm -rf <huge directory>)

Note: How can I mark thread "SOLVED"?

Thanks all.
 
Given you only have 4GB of RAM and your pool is over 90% full, unless that's changed, this doesn't surprise me. Have a look at this link and see if it may point you in the right direction. There is a newer sysctl(8) tunable in the discussion from that link you can set that may help. There were a bunch of commits made against ZFS since then but it may still be relevant to your issue.

The fact that the machine no longer panics is certainly a good thing. :)
 
For marking the thread solved, click on "Thread Tools" in the upper right of the page when viewing the thread, then click on "Edit" in the drop down. From there you can mark the thread prefix solved.
 
protocelt said:
Given you only have 4GB of RAM and your pool is over 90% full, unless that's changed, this doesn't surprise me. Have a look at this link and see if it may point you in the right direction. There is a newer sysctl(8) tunable in the discussion from that link you can set that may help. There were a bunch of commits made against ZFS since then but it may still be relevant to your issue.

The fact that the machine no longer panics is certainly a good thing. :)
Click to expand...

Thanks you, protocelt.

Sorry, I can't touch my computer and check memory usage now.
Reply when get result. :p

I have under discuss.
Prior 10.1-RELEASE AMD64 (Same machine (4GB RAM), over 90% capacity, Work fine.
In 10.1-RELEASE AMD64, Got the problem.
Now 10.2-RELEASE AMD64, have some problem

1.About Reboot/Shutdown stall at "All buffer synced", under is situation.
a.If shutdown/reboot immediately after boot ready(before daily periodic task start),
Work fine.
b.When daily periodic task started, I saw ]find STATE field by top
1.zio->io / RUN, SSD activity: Access.
2.empty ( Yes, no any character, I first time saw it empty), SSD activity: IDLE.
3.Reboot/Shutdown, stall at "All buffer synced" over night,
Solution: Power bottom only.

2. devel/ccache stall.
I used it to build ports (cache direcotr in ZFS)
Build run 1 minute, later stall.

3. rm hugh directory (ccache's cache direcotry, 5.XGB, 8-layer hash in ZFS)
Run 1 minute approx, later SSD access idle, rm can't return to prompt.

I think, 10.2-RELEASE resolved panic issue, but new problem occur.
Thanks you a lot.
 
You must log in or register to reply here.
Back
Top