ZFS: i/o error - all block copies unavailable

roddi

roddi

Hi

After replacing a drive in my root pool my FreeBSD 9.0 box did not come back but fails to boot with the following message:

Code: 
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS object directory
ZFS: unexpected object set type 0
When I boot the install DVD, the pool imports fine and a scrub shows no errors.

The pool is a raidz2 pool with six drives that booted fine for months until I replaced the one drive.

Any idea what this could be? I found a lot of stuff on the net where people are asking this kind of question but never a good explanation on what the root cause might be. A 512/2048 sector drive problem in the boot loader maybe?

Any help greatly appreciated!
Roddi
 
You'll probably need to reinstall the boot code onto that drive (or maybe just back into the pool)?

A resilver only touches data in the pool, so only the parts of the disk used by ZFS. A resilver does not touch the boot area of the disk.
 
Hi,

Just to let you know: I moved the root Raidz pool to a new mirror pool on two additional 2.5" drives and now it's booting like a charm again.

Seems like booting from a Raidz/Raidz2 is broken in some cases that are hard to recreate reliably. All forum posts I have seen about this topic lead nowhere. (e.g.: http://forums.freebsd.org/showthread.php?t=26036)
A bug is open on this since 8.1-RELEASE: http://www.FreeBSD.org/cgi/query-pr.cgi?pr=kern/151910

This is extremely frustrating. There should be at least some useful information in the messages displayed.

My two cents and thanks for enduring my rant,
Roddi
 
After playing with zpool attach and detach on my pools (moving cache and log vdevs) ZFS displays this at boot:

Code: 
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS object directory

The system boots fine. However, the message is annoying.

I reinstalled the boot code:

Code: 
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada2
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada3

It did not help. I recently upgraded my zpools from v28 to the feature thing, the error is still there. It was there before the upgrade too.

Code: 
mage@martina ~ $ zpool status
  pool: backup
 state: ONLINE
  scan: scrub repaired 0 in 7h34m with 0 errors on Sun Dec  9 02:24:14 2012
config:

        NAME        STATE     READ WRITE CKSUM
        backup      ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0    ONLINE       0     0     0
            ada1    ONLINE       0     0     0
        cache
          ada2p4    ONLINE       0     0     0

errors: No known data errors

  pool: martina
 state: ONLINE
  scan: scrub repaired 0 in 0h2m with 0 errors on Sat Dec  8 18:52:34 2012
config:

        NAME        STATE     READ WRITE CKSUM
        martina     ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada3p2  ONLINE       0     0     0
            ada2p2  ONLINE       0     0     0

errors: No known data errors

  pool: media
 state: ONLINE
  scan: scrub repaired 0 in 17h58m with 0 errors on Sun Dec  9 12:48:03 2012
config:

        NAME        STATE     READ WRITE CKSUM
        media       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada4    ONLINE       0     0     0
            ada5    ONLINE       0     0     0
        logs
          mirror-1  ONLINE       0     0     0
            ada2p3  ONLINE       0     0     0
            ada3p3  ONLINE       0     0     0
        cache
          ada3p4    ONLINE       0     0     0

errors: No known data errors
mage@martina ~ $ gpart show ada2
=>       34  156301421  ada2  GPT  (74G)
         34        128     1  freebsd-boot  (64k)
        162   75497472     2  freebsd-zfs  (36G)
   75497634    2097152     3  freebsd-zfs  (1.0G)
   77594786   41943040     4  freebsd-zfs  (20G)
  119537826   36763629        - free -  (17G)

mage@martina ~ $ gpart show ada3
=>       34  156301421  ada3  GPT  (74G)
         34        128     1  freebsd-boot  (64k)
        162   75497472     2  freebsd-zfs  (36G)
   75497634    2097152     3  freebsd-zfs  (1.0G)
   77594786   41943040     4  freebsd-zfs  (20G)
  119537826   36763629        - free -  (17G)

smartctl says all drives PASSED.
 
I have also been getting this message lately.

Code: 
ZFS: i/o error - all block copies unavailable

None of the other messages others have reported, and then everything proceeds normally.

I tried re-installing the boot code, but it didn't help.

Don't recall exactly when I started seeing it, but it was after I had upgraded from 9.0-RELEASE to 9.1-RELEASE. No custom kernels or such. Though I've fiddled with ZFS settings in loader.conf and sysctl.conf, but don't think any would account for this.

One big change was that I original installed as a single 120 GB SSD with a large swap partition, and upgraded to two 120 GB SSDs....which I did by partitioning new SSD with big zpool and lots of tiny partitions to be used as ZILs for the other zpools in my system. Then zfs send/receive from the old small zroot to the bigger new zroot (initially names something else). Then destroy old zroot, export rpool and import it as zroot. Then repartition old SSD and turn zroot into a mirrored zpool.

Pretty sure everything was fine after the operation, but I might have missed it in later boots when I was having trouble with a different SSD. First swapped the controller, to eventually tracking it down to the hot swap bay being bad.

My zpools are:
Code: 
root@zen:~ 102# zpool status
  pool: lcldsk
 state: ONLINE
  scan: scrub repaired 0 in 23h43m with 0 errors on Wed Jan 23 22:05:20 2013
config:

        NAME            STATE     READ WRITE CKSUM
        lcldsk          ONLINE       0     0     0
          mirror-0      ONLINE       0     0     0
            gpt/disk2   ONLINE       0     0     0
            gpt/disk1   ONLINE       0     0     0
        logs
          mirror-1      ONLINE       0     0     0
            gpt/zil0-0  ONLINE       0     0     0
            gpt/zil0-1  ONLINE       0     0     0
        cache
          gpt/l2-0      ONLINE       0     0     0

errors: No known data errors

  pool: lraidz
 state: ONLINE
  scan: scrub repaired 0 in 13h59m with 0 errors on Sat Mar 23 12:21:53 2013
config:

        NAME            STATE     READ WRITE CKSUM
        lraidz          ONLINE       0     0     0
          raidz1-0      ONLINE       0     0     0
            gpt/stb00   ONLINE       0     0     0
            gpt/stb01   ONLINE       0     0     0
            gpt/stb11   ONLINE       0     0     0
            gpt/stb10   ONLINE       0     0     0
        logs
          mirror-1      ONLINE       0     0     0
            gpt/zil1-0  ONLINE       0     0     0
            gpt/zil1-1  ONLINE       0     0     0
        cache
          gpt/l2-1      ONLINE       0     0     0

errors: No known data errors

  pool: lraidz2
 state: ONLINE
  scan: scrub repaired 0 in 48h50m with 0 errors on Sun Dec 23 09:59:37 2012
config:

        NAME            STATE     READ WRITE CKSUM
        lraidz2         ONLINE       0     0     0
          raidz2-0      ONLINE       0     0     0
            gpt/hds02   ONLINE       0     0     0
            gpt/hds12   ONLINE       0     0     0
            gpt/hds03   ONLINE       0     0     0
            gpt/hds13   ONLINE       0     0     0
            gpt/hds04   ONLINE       0     0     0
            gpt/hds14   ONLINE       0     0     0
        logs
          mirror-1      ONLINE       0     0     0
            gpt/zil2-0  ONLINE       0     0     0
            gpt/zil2-1  ONLINE       0     0     0
        cache
          gpt/l2-2      ONLINE       0     0     0

errors: No known data errors

  pool: vbox
 state: ONLINE
  scan: scrub repaired 0 in 0h1m with 0 errors on Mon Apr 22 21:21:50 2013
config:

        NAME         STATE     READ WRITE CKSUM
        vbox         ONLINE       0     0     0
          gpt/vbox0  ONLINE       0     0     0

errors: No known data errors

  pool: zroot
 state: ONLINE
  scan: scrub repaired 0 in 0h4m with 0 errors on Mon Apr 22 22:26:49 2013
config:

        NAME            STATE     READ WRITE CKSUM
        zroot           ONLINE       0     0     0
          mirror-0      ONLINE       0     0     0
            gpt/zroot1  ONLINE       0     0     0
            gpt/zroot0  ONLINE       0     0     0

errors: No known data errors

and

Code: 
SMART status:
Checking health of /dev/ada14: OK
Checking health of /dev/ada13: OK
Checking health of /dev/ada12: OK
Checking health of /dev/ada11: OK
Checking health of /dev/ada10: OK (but has failed in the past)
Checking health of /dev/ada9: OK (but has failed in the past)
Checking health of /dev/ada8: OK
Checking health of /dev/ada7: OK
Checking health of /dev/ada6: OK
Checking health of /dev/ada5: OK
Checking health of /dev/ada4: OK
Checking health of /dev/ada3: OK
Checking health of /dev/ada2: OK
Checking health of /dev/ada1: OK
Checking health of /dev/ada0: OK

The disks to zpools is basically,

motherboard - zroot - ada0 & ada1, lcldsk - ada2 & ada3
portmultiplier enclosures (sil3132) - lraidz - ada4/5/9/10 & lraidz2 - ada6/7/8/11/12/13
internal SSD on sil3124 - vbox - ada14

lcldsk/lraidz/lraidz have mirrored log devices on ada0/ada1, and cache device from ada14.

I have thought about moving the SSDs to a SATA III card, but its not high on my list of projects.

The Dreamer.
 
You must log in or register to reply here.
Back
Top