ZFS Unavailable

D

diss3ntive

Hello,

My storage/ pool doesn't seem to be alive. . . Let me show the stdout's:

Code: 
# zfs list
cannot iterate filesystems: I/O error
NAME         USED  AVAIL  REFER  MOUNTPOINT
rpool       2.41G  91.6G    18K  none
rpool/root  72.2M  91.6G  72.2M  legacy
rpool/usr   1.35G  91.6G  1.35G  legacy
rpool/var    137M  91.6G   137M  legacy
temp        67.5K  1.69T    18K  /temp

Code: 
# zpool status storage 

  pool: storage
 state: FAULTED
status: One or more devices could not be used because the label is missing
        or invalid.  There are insufficient replicas for the pool to continue
        functioning.
action: Destroy and re-create the pool from a backup source.
   see: [url]http://www.sun.com/msg/ZFS-8000-5E[/url]
 scrub: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     FAULTED      0     0     1  corrupted data
          raidz2    DEGRADED     0     0     6
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da9     ONLINE       0     0     0
            da10    ONLINE       0     0     0
            da11    ONLINE       0     0     0
            spare   DEGRADED     0     0     0
              da12  FAULTED      0     0     0  corrupted data
              da22  ONLINE       0     0     0
            da12    ONLINE       0     0     0
            da13    ONLINE       0     0     0
            da14    ONLINE       0     0     0
            da15    ONLINE       0     0     0
            da16    ONLINE       0     0     0
            da17    ONLINE       0     0     0
            da18    ONLINE       0     0     0
            da19    ONLINE       0     0     0
            da20    ONLINE       0     0     0
            da21    ONLINE       0     0     0

Code: 
# zpool replace storage da12 da23
cannot open 'storage': pool is unavailable

Code: 
# zpool scrub storage
cannot scrub 'storage': pool is currently unavailable

As you can see from the first stdout, it complains of an iterative I/O issue with, something, and when doing a status on the pool in question I can see a single faulted drive. I earlier tried replacing the faulted drive with the hotspare, but as you can see from the output above it crapped out after a reboot.

So! A few simple things to mention is the system, obviously, had issues which were "fixed" for the time being, but failed after a reboot. I had to specify the shell path before even booting into the rpool, and manually starting services. What steps can I take to bring this pool alive again so I can fix it?
 
I would also highly suggest labelling your hard-drives as well. A good naming scheme would be using manufacture and last few characters of the hard-drive serial number, such as what I've done below:


Code: 
$ zpool status
  pool: TANK
 state: ONLINE
 scrub: scrub completed after 5h16m with 0 errors on Sat Jul 31 05:28:00 2010
config:

	NAME                STATE     READ WRITE CKSUM
	TANK                ONLINE       0     0     0
	  raidz2            ONLINE       0     0     0
	    label/HI_1NJKE  ONLINE       0     0     0
	    label/WD_28896  ONLINE       0     0     0
	    label/WD_13920  ONLINE       0     0     0
	    label/WD_00658  ONLINE       0     0     0
	    label/WD_01756  ONLINE       0     0     0
	    label/WD_42255  ONLINE       0     0     0
	    label/HI_XGW9F  ONLINE       0     0     0
	    label/HI_XGKNF  ONLINE       0     0     0
	    label/HI_XXUTK  ONLINE       0     0     0
	    label/WD_02281  ONLINE       0     0     0
	    label/HI_0WZ6L  ONLINE       0     0     0

errors: No known data errors
$
 
Can you give an example of how to label them like that? I guess you need to do it before using the disk in a zfs pool?
 
I think you'd use glabel()

Just as a matter of curiosity, if you label a disk in this way and then give the entire device over for ZFS to use, how does ZFS know not to trample all over the GEOM metadata where the label is recorded?
 
Jago said:
You have a single vdev group consisting of over 20 drives. Under no cirumstances should you be using more than 10 drives in a single raidz/raidz2 vdev. Raidz is reasonable for up to 5 drives, raidz2 for up to 8-9, bigger vdevs is just asking for trouble.

See the ZFS Best Practices Guide: http://www.solarisinternals.com/wik...onfiguration_Requirements_and_Recommendations

Googling around will also reveal some other people who have run into similar issues like you.
Click to expand...

nice article, I do wonder why raidz is so much more popular that zfs mirror.
 
chrcol said:
nice article, I do wonder why raidz is so much more popular that zfs mirror.
Click to expand...

Because some people tend to think that "cheap, fast, reliable - pick 2" somehow magically doesn't apply to them ;)
 
Because sometimes you really want more usable disk space than raw throughput. mirror vdevs are faster than raidz vdevs, but raidz vdevs have more usable space than mirror vdevs.

Decide which is more important (speed/size) and use the appropriate vdev type. :)

@OP: My first ZFS storage box used a single raidz2 vdev with 24 drives. Everything worked well for several weeks, until 1 drive died. Physically replacing it was simple, doing the "zfs replace" command was easy. However, the new drive *NEVER* finished re-silvering and would constantly restart. And the throughput dropped to under 1 MB/s, thrashing all the drives constantly.

IOW, don't do that. :) Backup the data, destroy the pool, create a new pool using multiple raidz vdevs, then restore the data. There's no other way around it.
 
You must log in or register to reply here.
Back
Top