Solved Stuck in degraded/resilver endless loop

L

lethe

Hi,

I've had a disk failure in a ZFS pool. I replaced the disk, it resilvered but always stays in the degraded state:
Code: 
  NAME  STATE  READ WRITE CKSUM
   storage2  DEGRADED  0  0  3
    raidz1-0  DEGRADED  0  0  6
    ada2  ONLINE  0  0  0
    replacing-1  DEGRADED  0  0  0
    210848086431766885  UNAVAIL  0  0  0  was /dev/ada3/old
    7484184995992719114  UNAVAIL  0  0  0  was /dev/ada3/old
    ada3  ONLINE  0  0  0
    ada7  ONLINE  0  0  0
If I reboot, it will resilver automatically. I've re-replaced the drive with another brand new one, same thing. Logs show no errors relating to any other disk than ada2 (which I'll have to replace next but can't because the pool won't self-heal) about offline uncorrectable. So it should not prevent the pool from healing.

Replacing the disk and resilvering leaves it degraded. Rebooting causes an automatic resilver and it stays degraded. Clear on the pool doesn't help. I don't have available SATA ports to add a spare to test further.

I'm out of ideas, looking for insight or a push in the right direction.

9.1-RELEASE-p1 x64
This system is currently running ZFS pool version 28.

Thank you.
 
What happens if you remove the missing disks:
Code: 
# zpool remove storage2 210848086431766885
# zpool remove storage2 7484184995992719114
I've had similar situations occur, and removing the missing disks after the resilver completed once fixed it.
 
Thanks, just tried this but it's not working:

Code: 
root@sb1:/root # zpool remove storage2 210848086431766885
cannot remove 210848086431766885: only inactive hot spares, cache, top-level, or log devices can be removed
root@sb1:/root # zpool remove storage2 7484184995992719114
cannot remove 7484184995992719114: only inactive hot spares, cache, top-level, or log devices can be removed

I also updated to 9.1-RELEASE-p22 and upgraded the pools, but no difference.
 
Try issuing an offline to the drive first:
Code: 
# zpool offline storage2 210848086431766885
# zpool remove storage2 210848086431766885
 
lethe I also had this issue recently and managed to fix it by issuing a # zpool clear <pool>. It caused the resilver to start again but after it completed the dead disk was removed and the pool back in an online state.
 
Thanks both of you, I tried both still no go.

Code: 
# zpool offline storage2 210848086431766885
cannot offline 210848086431766885: no valid replicas

After doing a clear and waiting for the resilver to complete, everything's still the same:
Code: 
  pool: storage2
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
   corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
   entire pool from backup.
  see: http://illumos.org/msg/ZFS-8000-8A
  scan: resilvered 884G in 46h22m with 3 errors on Thu Nov 20 12:15:19 2014
config:

   NAME  STATE  READ WRITE CKSUM
   storage2  DEGRADED  0  0  3
    raidz1-0  DEGRADED  0  0  6
    ada2  ONLINE  0  0  0
    replacing-1  DEGRADED  0  0  0
    210848086431766885  UNAVAIL  0  0  0  was /dev/ada3/old
    7484184995992719114  UNAVAIL  0  0  0  was /dev/ada3/old
    ada3  ONLINE  0  0  0
    ada7  ONLINE  0  0  0

errors: 2 data errors, use '-v' for a list
 
Hi,

There is something confusing. Following is the output from zpool status, right?

lethe said:
Code: 
  NAME  STATE  READ WRITE CKSUM
   storage2  DEGRADED  0  0  3
    raidz1-0  DEGRADED  0  0  6
    ada2  ONLINE  0  0  0
    replacing-1  DEGRADED  0  0  0
    210848086431766885  UNAVAIL  0  0  0  was /dev/ada3/old
    7484184995992719114  UNAVAIL  0  0  0  was /dev/ada3/old
    ada3  ONLINE  0  0  0
    ada7  ONLINE  0  0  0
Click to expand...

Is your output really like this? Are some spaces missing? Are ada2, ada3 and ada7 members of storage2?

If I compare with what I have for my zpool, RAIDz-1 is indented once and members of RAIDz-1 are indented twice.

Code: 
NAME STATE READ WRITE CKSUM
storage ONLINE 0 0 0
  raidz1-0 ONLINE 0 0 0
    gpt/hita-disk1 ONLINE 0 0 0
    gpt/hita-disk2 ONLINE 0 0 0
    gpt/hita-disk3 ONLINE 0 0 0
    gpt/hita-disk4 ONLINE 0 0 0

It's certainly because of missing spaces, so can you confirm your output first?
 
Thanks everyone.

I did what gavinjs0 suggested, deleted the corrupted files and cleared storage2. After another long resilver, everything is back in order. Did not have to run another replace.

Code: 
# zpool status storage2
  pool: storage2
state: ONLINE
  scan: resilvered 782G in 45h0m with 0 errors on Mon Dec  1 07:16:06 2014
config:

   NAME  STATE  READ WRITE CKSUM
   storage2  ONLINE  0  0  0
    raidz1-0  ONLINE  0  0  0
    ada2  ONLINE  0  0  0
    ada3  ONLINE  0  0  0
    ada7  ONLINE  0  0  0

errors: No known data errors
 
You must log in or register to reply here.
Back
Top