Kernel Panic with ZFS

C

cliffboss

Hello,

I am currently experiencing some really nasty stability problems today. It all started with a power outage this morning. When the power came back I booted up the server and ran zpool scrub on the array. Later in the day I check the status on the scrub and could not ssh into the box. I power cycle the machine and ssh back in to the machine. About 10 minute later ssh disconnects and the machine stops responding.

Next I plug my monitor into the box and watch what was going on. Like before around the 10 minute mark I see a kernel panic happen. The message states that it could not write the crash dump to the disk :\

I then powered cycle the machine once it came back up I then rebooted it. The reboot takes a very long time waiting to process to shutdown. It gets to the syncing of the nodes, spiting out a bunch of 4s, 2s and 1s and then says time out. Next the screen says that it is emptying the buffer and a bunch of numbers are written to the screen. After about 2-5min of this I get another kernel panic.

Tried the above again and the same result. :\

Any help trying to diagnosis the issue?

Notes:
  • Up to today my systems has been stable with no crashes what so ever.
  • I am running with on 2GB of ram (which i know is sub optimal, but the machine is really only used to store and watch tv shows)
  • I also read that upgrading to 8.2 stable fixes a lot of issues with ZFS, not sure if this will fix my problem

Thanks,
Cliffboss
 
Hi,

Please describe a bit further your setup.
Is it a root on ZFS installation and how many disks are you using ?

[CMD=""]# zpool status[/CMD]

Also, have a look at /var/log/messages. A power failure can cause problems to disks. Older versions of ZFS behave strange when they can not read or write from a disk.
 
Current setup:
  • Intel Core 2 Duo E6600
  • 2GB of ram
  • 4x 2TB Western Digital Green drive
  • 1x 640GB Western Digital drive
  • ZFS is not on the root
  • The drives are plugged into the on board stat controller
  • The 4 Green drives are in a RAIDZ setup

Code: 
FileServer# zpool status
  pool: storage
 state: ONLINE
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz1    ONLINE       0     0     0
            ad6     ONLINE       0     0     0
            ad8     ONLINE       0     0     0
            ad10    ONLINE       0     0     0
            ad12    ONLINE       0     0     0

errors: No known data errors

See the attachment for the last 24hours of the messages log
 
I had the exact same problem.
What I experienced was, that the kernel panics where caused by extensive swapping etc.
I bootet into single user mode, forced an unmount of the zpool and mounted it again. So far, the problem didn't return.
 
Epikurean said:
I had the exact same problem.
What I experienced was, that the kernel panics where caused by extensive swapping etc.
I bootet into single user mode, forced an unmount of the zpool and mounted it again. So far, the problem didn't return.
Click to expand...

This is done by exporting and importing the zpool right?
 
You can try booting off a 9.0 install CD or an mfsBSD CD (they support ZFSv28), and doing a transaction rollback import:
# zpool import -F <poolname>
Then do an export, and a normal import, and another export. Then boot into the normal OS and try to import the pool.
 
@cliffboss,

Your attachment didn't go through for some reason.

I would first try what phoenix suggested. Upgrading to 8.2-STABLE and ZFS V28 might also be a good step.
 
phoenix said:
You can try booting off a 9.0 install CD or an mfsBSD CD (they support ZFSv28), and doing a transaction rollback import:
# zpool import -F <poolname>
Then do an export, and a normal import, and another export. Then boot into the normal OS and try to import the pool.
Click to expand...

I tried this, but it failed horribly.

The zpool import -f failed complaining that the GPT table has been corrupted for each of the disks :(

Looks like I am going to rebuild the pool from a back up.
 
cliffboss said:
Like before around the 10 minute mark I see a kernel panic happen. The message states that it could not write the crash dump to the disk :\
Click to expand...
Crash dumps have not worked reliably (or at all, actually) for me in years. The last time they worked for me was in non-SMP FreeBSD 5.x.

There are actually a large number of things which work to prevent crash dumps from operating reliably. There's a discussion of one of them (and some tangents) on the freebsd-current mailing list.

I've found that the best way to find out what caused the panic is to have some way of recording the console output (either a serial console logged somewhere, or certain types of KVM / remote console support). The info the developers need has likely scrolled off your screen by the time you get the message about not being able to access the crash dump.

I then powered cycle the machine once it came back up I then rebooted it. The reboot takes a very long time waiting to process to shutdown. It gets to the syncing of the nodes, spiting out a bunch of 4s, 2s and 1s and then says time out.
Click to expand...
That's not good. Those are dirty buffers which need to be written back to disk. If the number doesn't decrease, it can indicate a hardware or driver problem (writes never complete) or some sort of buffer chain corruption (which could also be hardware or software) where it has lost track of what exactly it is doing.

Normally I don't immediately say "go run memtest86+", but in this case I think it might be a good idea, just to rule out any memory issues.

If you are using any type of "smart" disk controller (such as a RAID card, even if you aren't using it in RAID mode), you should check to see if there are any firmware updates for the card.

I am running with on 2GB of ram (which i know is sub optimal, but the machine is really only used to store and watch tv shows)
Click to expand...
ZFS really, really wants to have a lot of RAM. The folks that originally developed it at Sun seem to have had the expectation that it would be running in a 64-bit environment with lots of RAM, which was not an unreasonable assumption given the hardware Sun was selling.

I also read that upgrading to 8.2 stable fixes a lot of issues with ZFS, not sure if this will fix my problem
Click to expand...
8-STABLE has ZFS 28/5, while 8.2-RELEASE has ZFS 15/4, IIRC. ZFS 28 is a much newer code base, and FreeBSD development is being done with ZFS 28. Some patches might be able to be backported to the older ZFS 15 implementation, but the best testing and coverage is likely to be under ZFS 28.

The zpool import -f failed complaining that the GPT table has been corrupted for each of the disks

Looks like I am going to rebuild the pool from a back up.
Click to expand...
That sounds like something ran wild and started writing in inappropriate places on the disk drives. Even once you fix the underlying problem, the pool may not be recoverable. I'd wait for an expert opinion (or go ask on the freebsd-fs mailing list) before clobbering the pool, unless you're sure you have a good backup.

I don't know of any definite hardware issue that could cause this, given the hardware you listed. There was a bug in the 3Ware twa-class firmware which caused erroneous writes to low block numbers (since fixed), and Samsung had a firmware bug on some drive models which could cause data corruption.
 
Just an update, last night before bed I decide to turn off ZFS by removing the line from the rc.conf file.

So far it has been stable for a little over 8 hours. :)

Bad news this seems that ZFS is the problem. :(

When I get home from work I am going to re enable ZFS again and see what happens. Maybe I proper start/shutdown might fix things.
 
cliffboss said:
I tried this, but it failed horribly.

The zpool import -f failed complaining that the GPT table has been corrupted for each of the disks :(

Looks like I am going to rebuild the pool from a back up.
Click to expand...
The good thing is that you have backups.
But don't give up yet. It looks like the power failure messed up somehow your GPT tables.

[CMD=""]# gpart show[/CMD]

Will show you if there are any problems and :

[CMD=""]# gpart recover /dev/adx [/CMD]

Might be able to fix this for you. you might want to have a look at gpart(8)() before.

If you do decide to restore from a backup. I would suggest the use of labels instead. See glabel(8)() for better reference.

George
 
cliffboss said:
I tried this, but it failed horribly.

The # zpool import -f failed complaining that the GPT table has been corrupted for each of the disks :(

Looks like I am going to rebuild the pool from a back up.
Click to expand...

Note: -f and -F do two very different things. You want to use the capital F.
 
gkontos said:
The good thing is that you have backups.
But don't give up yet. It looks like the power failure messed up somehow your GPT tables.

[CMD=""]# gpart show[/CMD]

Will show you if there are any problems and :

[CMD=""]# gpart recover /dev/adx [/CMD]

Might be able to fix this for you. you might want to have a look at gpart(8)() before.

If you do decide to restore from a backup. I would suggest the use of labels instead. See glabel(8)() for better reference.

George
Click to expand...

Just did what you said above and then did a [CMD=""]zpool import storage[/CMD]
and it worked.

I checked the status of the zpool and it said it re silvered 3k.

Now my question is can I trust the data on the drive, even when the GPT were corrupted and restored?
 
cliffboss said:
Just did what you said above and then did a [CMD=""]zpool import storage[/CMD]
and it worked.

I checked the status of the zpool and it said it re silvered 3k.

Now my question is can I trust the data on the drive, even when the GPT were corrupted and restored?
Click to expand...

Good question! I honestly can't advise on that but it doesn't appear to be like a faulty drive.
You many want to use smartmontools just to keep an eye on those drives.

Regards,
George
 
Already had that program running from day 1. It reports all drives as healthy. As for one of the drive being faulty, doubt it since FreeBSD was reporting all 4 drives having corrupted GPT [Chance of all be duds highly unlikely]

What I'll probably do unless anyone else has any insight is destroy the pool, format and recreate the pool from a backup. [Only have had the systems up for a month now so they aren't that out of date]
 
cliffboss said:
-F told me there was not such switch. I used -f thinking you had done a typo...
Click to expand...

-F was added in a later version of ZFS than was shipped in FreeBSD 8.2. It's part of the ZFSv28 that's included with FreeBSD 9.0 and 8-STABLE. Hence why I said you have to use a 9.0 install CD (drop to shell) or an 8.2 mfsbsd CD (that includes ZFSv28).
 
phoenix said:
-F was added in a later version of ZFS than was shipped in FreeBSD 8.2. It's part of the ZFSv28 that's included with FreeBSD 9.0 and 8-STABLE. Hence why I said you have to use a 9.0 install CD (drop to shell) or an 8.2 mfsbsd CD (that includes ZFSv28).
Click to expand...

I did grab a copy of 9.0 rc2
 
You must log in or register to reply here.
Back
Top