ZFS Crash

gkontos

gkontos

HI all,

today I had my first crash after 6 years!!!

The system is an old HP ML110G3 server, recently upgraded to 4GRAM and a Dual Core 64 bit processor. The root pool is on ZFS mirror and I have added a Promise 4 port SATA controller containing 3 X 640 GB pool in Raidz1.
It all started after a 50GB copy from the Raidz1 to the root mirror. I noticed high memory consumption during the copy. After a while the free memory returned to 1G. And then I started copying some GBs from a Windows 7 machine via samba (something I do all the time) to the Raidz1 storage. During the copy I didn't notice anything weird memory wise... But when I returned after 10 minutes the machine would not respond.(headless) I plugged in a monitor and I saw some weird messages like,

bringing down bge0
kern.memory ....
system will reboot in 15 seconds

Did a cold start and everything came back to normal.

Your thoughts are very much appreciated!!!

Code: 
hp# zfs list -r
NAME                        USED  AVAIL  REFER  MOUNTPOINT
tank                        169G   995G  32.0K  /tank
tank/apple                 15.6G   104G  15.6G  /tank/apple
tank/data                   121G   995G   121G  /tank/data
tank/gotos                  242M   995G   242M  /tank/gotos
tank/kitos                  152M   995G   152M  /tank/kitos
tank/video                 32.4G   995G  32.4G  /tank/video
zroot                      5.79G   223G   519M  legacy
zroot/swap                    2G   225G    16K  -
zroot/tmp                  31.5K   223G  31.5K  /tmp
zroot/usr                  3.17G   223G  2.61G  /usr
zroot/usr/home               49K   223G    49K  /usr/home
zroot/usr/ports             279M   223G   247M  /usr/ports
zroot/usr/ports/distfiles  32.6M   223G  32.6M  /usr/ports/distfiles
zroot/usr/ports/packages     18K   223G    18K  /usr/ports/packages
zroot/usr/src               298M   223G   298M  /usr/src
zroot/var                   118M   223G   237K  /var
zroot/var/crash            18.5K   223G  18.5K  /var/crash
zroot/var/db                115M   223G   110M  /var/db
zroot/var/db/pkg           4.46M   223G  4.46M  /var/db/pkg
zroot/var/empty              18K   223G    18K  /var/empty
zroot/var/log              2.74M   223G  2.74M  /var/log
zroot/var/mail               19K   223G    19K  /var/mail
zroot/var/run              54.5K   223G  54.5K  /var/run
zroot/var/tmp                27K   223G    27K  /var/tmp
Code: 
hp# uname -a
FreeBSD hp.aicom.loc 8.0-RELEASE-p1 FreeBSD 8.0-RELEASE-p1 #0: Sun Dec 13 02:24:13 EET 2009     gkontos@hp.aicom.loc:/usr/obj/usr/src/sys/GENERIC  amd64
 
My thoughts:

What makes you thinking about a "crash"?
A crash is not an unknown reason for a reboot.
Did you check logfiles?
 
Also some more info:

The server is mainly running:
samba
local dns
dhcp
apache22 + php + mysql (cacti)
syslog receives for a pix fw

And now during a samba copy:
Code: 
last pid:  1389;  load averages:  0.27,  0.22,  0.20                                                                                                                           up 0+01:02:03  19:02:23
62 processes:  1 running, 61 sleeping
CPU:  0.9% user,  0.0% nice, 12.7% system,  0.4% interrupt, 86.0% idle
Mem: 114M Active, 17M Inact, 657M Wired, 864K Cache, 3156M Free
Swap: 2048M Total, 2048M Free


Code: 
hp# dmesg 
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-RELEASE-p1 #0: Sun Dec 13 02:24:13 EET 2009
    gkontos@hp.aicom.loc:/usr/obj/usr/src/sys/GENERIC
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Pentium(R) D CPU 3.20GHz (3200.13-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0xf64  Stepping = 4
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xe4bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,CNXT-ID,CX16,xTPR,PDCM>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 4294967296 (4096 MB)
avail memory = 4113715200 (3923 MB)
ACPI APIC Table: <HP     OEMAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ioapic0: Changing APIC ID to 2
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <HP OEMXSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
pci7: <ACPI PCI bus> on pcib2
bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x4101> mem 0xfeaf0000-0xfeafffff irq 17 at device 0.0 on pci7
miibus0: <MII bus> on bge0
brgphy0: <BCM5750 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto
bge0: Ethernet address: 00:13:21:cc:39:35
bge0: [ITHREAD]
uhci0: <Intel 82801G (ICH7) USB controller USB-A> port 0xdc00-0xdc1f irq 23 at device 29.0 on pci0
uhci0: [ITHREAD]
uhci0: LegSup = 0x0f30
usbus0: <Intel 82801G (ICH7) USB controller USB-A> on uhci0
uhci1: <Intel 82801G (ICH7) USB controller USB-B> port 0xd880-0xd89f irq 19 at device 29.1 on pci0
uhci1: [ITHREAD]
uhci1: LegSup = 0x0f30
usbus1: <Intel 82801G (ICH7) USB controller USB-B> on uhci1
uhci2: <Intel 82801G (ICH7) USB controller USB-C> port 0xd800-0xd81f irq 18 at device 29.2 on pci0
uhci2: [ITHREAD]
uhci2: LegSup = 0x0f30
usbus2: <Intel 82801G (ICH7) USB controller USB-C> on uhci2
ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem 0xfe9ffc00-0xfe9fffff irq 23 at device 29.7 on pci0
ehci0: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <Intel 82801GB/R (ICH7) USB 2.0 controller> on ehci0
pcib3: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci8: <ACPI PCI bus> on pcib3
atapci0: <Promise PDC40718 SATA300 controller> port 0xec00-0xec7f,0xe800-0xe8ff mem 0xfebff000-0xfebfffff,0xfebc0000-0xfebdffff irq 16 at device 0.0 on pci8
atapci0: [ITHREAD]
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: SIGNATURE: 00000101
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: SIGNATURE: 00000101
ata3: [ITHREAD]
ata4: <ATA channel 2> on atapci0
ata4: [ITHREAD]
ata5: <ATA channel 3> on atapci0
ata5: SIGNATURE: 00000101
ata5: [ITHREAD]
vgapci0: <VGA-compatible display> port 0xe000-0xe0ff mem 0xe8000000-0xefffffff,0xfebb0000-0xfebbffff irq 16 at device 2.0 on pci8
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <Intel ICH7 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 31.1 on pci0
ata0: <ATA channel 0> on atapci1
ata0: [ITHREAD]
atapci2: <Intel ICH7 SATA300 controller> port 0xd480-0xd487,0xd400-0xd403,0xd080-0xd087,0xd000-0xd003,0xcc00-0xcc0f mem 0xfe9ff800-0xfe9ffbff irq 19 at device 31.2 on pci0
atapci2: [ITHREAD]
atapci2: AHCI called from vendor specific driver
atapci2: AHCI v1.10 controller with 4 3Gbps ports, PM not supported
ata6: <ATA channel 0> on atapci2
ata6: [ITHREAD]
ata7: <ATA channel 1> on atapci2
ata7: [ITHREAD]
ata8: <ATA channel 2> on atapci2
ata8: [ITHREAD]
ata9: <ATA channel 3> on atapci2
ata9: [ITHREAD]
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart1: [FILTER]
ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
cpu0: <ACPI CPU> on acpi0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 102400001024
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 102400001024
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc8fff,0xc9000-0xcdfff,0xcf800-0xd47ff,0xd4800-0xd57ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 13
ZFS storage pool version 13
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ad4: 610480MB <WDC WD6401AALS-00J7B1 05.00K05> at ata2-master SATA300
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ad6: 610480MB <WDC WD6401AALS-00J7B1 05.00K05> at ata3-master SATA300
ad10: 610480MB <WDC WD6401AALS-00J7B1 05.00K05> at ata5-master SATA300
ad12: 238475MB <Seagate ST3250410AS 3.AAA> at ata6-master SATA150
ad14: 238475MB <Seagate ST3250410AS 3.AAA> at ata7-master SATA150
SMP: AP CPU #1 Launched!
Root mount waiting for: usbus3 usbus2 usbus1 usbus0
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
Root mount waiting for: usbus3
Root mount waiting for: usbus3
uhub3: 6 ports with 6 removable, self powered
Trying to mount root from zfs:zroot
ugen0.2: <American Power Conversion> at usbus0
bge0: link state changed to UP
 
Erratus said:
My thoughts:

What makes you thinking about a "crash"?
A crash is not an unknown reason for a reboot.
Did you check logfiles?
Click to expand...

Yes nothing there! Like I said the server was not reachable from network. I had to plug in a monitor and then pressed the power switch!
 
Could be a CPU issue x(
I am copying as we speak 500G from a Windows box to the server and I get very high CPU loads from samba. Also when the systems peaks its CPU the fan goes crazy :stud
 
Also,
Code: 
bringing down bge0
kern.memory ....
system will reboot in 15 seconds
is not a usual message after a crash, no panic that i have come across, brings down interfaces gracefully.
Where your file systems unmounted properly?
Did you have an background fsck (any fsck related messages in /var/log/messages) in the very next boot?
 
gkontos said:
Could be a CPU issue x(
I am copying as we speak 500G from a Windows box to the server and I get very high CPU loads from samba. Also when the systems peaks its CPU the fan goes crazy :stud
Click to expand...

did you check for any temperature-related settings (like shutting-off automatically over certain thresholds) in the BIOS?
 
achix said:
did you check for any temperature-related settings (like shutting-off automatically over certain thresholds) in the BIOS?
Click to expand...
No bios settings for those.
Another thing that I noticed is that if I use zfs send/receive for file operations the memory stays the same unlike traditional cp -r
It could be that I'm "hogging" the system with traditional file copy utilities to transfer files between pools.
Nevertheless, samba fires up my cpu to 100% which could be a problem if I run in high temp.

George
 
achix said:
Also,
Code: 
bringing down bge0
kern.memory ....
system will reboot in 15 seconds
is not a usual message after a crash, no panic that i have come across, brings down interfaces gracefully.
Where your file systems unmounted properly?
Did you have an background fsck (any fsck related messages in /var/log/messages) in the very next boot?
Click to expand...
In ZFS there is no fsck...
 
phoenix said:
I'm guessing you're going from memory here, so was the "kern.memory" bit along the lines of "kmem_map too small"?
Click to expand...
Phoenix I really can't recall but you are probably guessing right.
Do you think I should perform any tuning ?
I read about zfs tuning but haven't found anything regarding amd64 with 4G or ram.
If you have any suggestions please let me know!!!

Thanks
 
If the error was "kmem_map too small", then you'll need to do some tuning via /boot/loader.conf. Particularly with vm.kmem_size_max. You shouldn't hit that error with the 64-bit version, though, at least in theory.

Even though the auto-tuning for ZFS in FreeBSD 8.x is greatly improved, you may want to limit the max size of the ARC via vfs.zfs.arc_max. Start by limiting it to 1/2 of RAM, and then move it lower if things continue to lock up.

On our 8 GB boxes, running FreeBSD 7.2, I have ARC limited to 1 GB, in order to keep things running smoothly.
 
Thanks !!!

I will try to reproduce the problem, this time with a monitor plugged in, and then perform some tuning.

Regards,

George
 
Mr. phoenix!!!

Here we go again:
Code: 
panic: kmem_malloc(131072): kmem_map too small 1345368064 total allocated
cpuid = 0
Uptime 22h36m20s
Cannot dump. Device not defined or unavailable.
Automatic reboot in 15 seconds - press a key on the console to abort
This happened during a samba transfer from a Windows 7 to my box.

No tuning yet, I wait for your suggestions!!!

George
 
It looks like you're running the 64-bit version of FreeBSD 8.0, with 4 GB of RAM.

FreeBSD 8.0 has a kmem size limit over 5 GB, so you don't need to worry about limiting the kmem. But you do need to limit the amount of RAM that ZFS uses for the ARC.

Try setting just vfs.zfs.arc_max=2048M. That will limit the ARC to 1/2 of your RAM. If it crashes again, then drop it to 1024M. Then to 512M. If it still crashes with ARC below 512M, there's something else wrong.
 
phoenix said:
It looks like you're running the 64-bit version of FreeBSD 8.0, with 4 GB of RAM.

FreeBSD 8.0 has a kmem size limit over 5 GB, so you don't need to worry about limiting the kmem. But you do need to limit the amount of RAM that ZFS uses for the ARC.

Try setting just vfs.zfs.arc_max=2048M. That will limit the ARC to 1/2 of your RAM. If it crashes again, then drop it to 1024M. Then to 512M. If it still crashes with ARC below 512M, there's something else wrong.
Click to expand...

Panic again and again x(
That happens during large smb copies to the raidz1 pool...
I googled a bit and came up with this settings:
Code: 
vm.kmem_size=1500M
vm.kmem_size_max=4G
vfs.zfs.arc_min: 122880000
vfs.zfs.arc_max: 983040000
After a few gigs of transfer things appear to move smoothly
Code: 
TEXT=15380840, 14.6683 MB
DATA=1183639552, 1128.81 MB
TOTAL=1199020392, 1143.47 MB
Code: 
gkontos@hp>vmstat -m | grep solaris
      solaris 20828 756925K       - 13518339  16,32,64,128,256,512,1024,2048,4096

George
 
You must log in or register to reply here.
Back
Top