ZFS and kernel panic? (FreeBSD 8.2)

[pww]

3 July 2012

(using FreeBSD 8.2 with ZFS raidz1 over 4 drives)
I was dd'ing from /dev/zero to a file on the raid. Before the dd'ing finished, I
unwittingly initiated a cp from that file to another file on the raid. A little while
later, the entire system seemed to "freeze". (i.e., all log activities, crontabs, mail
services, ssh, network, etc. stopped). When I rebooted the system, I discovered that the
second file had the time stamp around the "freeze" time. However, the first file dd
had completed (the expected size was correct) and had a timestamp that was well
after the "freeze" time.

So, it looked like the system partially kernel panic'd but finished the dd.

How would I find out what happened? My system is headless, therefore I don't have console.
There doesn't seem to be any kernel dumps or any logged activity after the "freeze" time.
(I checked messages, auth.log, and maillog so far) Anywhere else?

-phil

 

[throAU]

Sounds like it may have run out of RAM or similar.

If you have a serial port you could set up a serial console for monitoring, but my first questions would be:
- What platform are you running on (x64, x86?)
- How much RAM is in the machine?
- How big is your ZFS pool?
- Do you have any additional ZFS options such as compression or de-duplication turned on?
- Where is your swap file/partition? Is root on ZFS?

If your system is 100% ZFS, and runs into resource contention that interferes with ZFS operation, then it will have no ability to write logs. If it is memory constrained then it may be trying to swap to ZFS which is starved for memory and hang as you describe.

 

[pww]

Sounds like it may have run out of RAM or similar.

If you have a serial port you could set up a serial console for monitoring, but my first questions would be:
- What platform are you running on (x64, x86?)

x86

- How much RAM is in the machine?

1G RAM

- How big is your ZFS pool?

raidz1 4x300G

- Do you have any additional ZFS options such as compression or de-duplication turned on?

nothing

- Where is your swap file/partition? Is root on ZFS?

Booting off a gmirror'd UFS. Root is not on ZFS.

If your system is 100% ZFS, and runs into resource contention that interferes with ZFS operation, then it will have no ability to write logs. If it is memory constrained then it may be trying to swap to ZFS which is starved for memory and hang as you describe.

These are the only options in /boot/loader.conf
vm.kmem_size="512M"
vfs.zfs.arc_max="256M"

-phil

 

[throAU]

ZFS is not recommended for use on x86. There have been many reports of stability problems with ZFS on x86 in the past.

The way ZFS works, it needs a fairly large address space, irrespective of how much physical RAM you have.

I would recommend not running ZFS on x86, and not running it with 1GB of RAM. FreeNAS actually recommend 6GB or more to get the performance benefit of read-ahead caching, but if you don't need that you can probably get by with less. So long as you have a 64 bit kernel.

From the ZFS best practices wiki:

System/Memory/Swap Space
Run ZFS on a system that runs a 64-bit kernel

I suspect you'll need to plug in a console to diagnose/tune it further; but be aware, you are running on a platform that is definitely not recommended for use with ZFS.

If you're determined to try and make it work, I'd suggest having a read of the FreeBSD ZFS Tuning Guide.