Zpool - Incorrecly added a drive to my raidz configuration

Z

ZpoolNoob

I was hoping that someone could help me out with the following problem - I had previously successfully created my ZFS raidz storage pool but I wanted to add a fourth drive to it.

I made the mistake of using the following command:

zpool add -f storage00 /dev/disk/by-uuid/24a92cd8-265a-4bc5-974e-bdb61d927633

I am now unsure how to remove the drive and then add it back correctly. At first my pool was completely nonfunctional. I then preformed a export and and then an import. Next I performed a scrub of the pool. Now when I run a zpool status I get the following:

Code: 
 pool: storage00
 state: UNAVAIL
status: One or more devices could not be used because the label is missing
        or invalid.  There are insufficient replicas for the pool to continue
        functioning.
action: Destroy and re-create the pool from
        a backup source.
   see: [url]http://www.sun.com/msg/ZFS-8000-5E[/url]
 scrub: scrub completed after 4h55m with 0 errors on Tue Aug  9 03:27:21 2011
config:


  pool: storage00
 state: UNAVAIL
status: One or more devices could not be used because the label is missing
        or invalid.  There are insufficient replicas for the pool to continue
        functioning.
action: Destroy and re-create the pool from
        a backup source.
   see: [url]http://www.sun.com/msg/ZFS-8000-5E[/url]
 scrub: scrub completed after 4h55m with 0 errors on Tue Aug  9 03:27:21 2011
config:

        NAME                                                      STATE     READ WRITE CKSUM
        storage00                                                 UNAVAIL      0     0     0  insufficient replicas
          raidz1-0                                                ONLINE       0     0     0
            disk/by-id/ata-WDC_WD10EARS-00Y5B1_WD-WCAV5T006995    ONLINE       0     0     0
            disk/by-id/ata-WDC_WD10EADS-65L5B1_WD-WCAU49103942    ONLINE       0     0     0
            disk/by-id/ata-WDC_WD1001FALS-00J7B0_WD-WMATV0044081  ONLINE       0     0     0
          disk/by-id/ata-WDC_WD3200JB-00KFA0_WD-WCAMR2531970      UNAVAIL      0     0     0  corrupted data

errors: No known data errors

I am still able to read and write to the pool. It appears as ZFS is seeing the drive as a second raid/mirror configuration.
I have tried the following:

Code: 
:~$ sudo zpool detach storage00  /dev/disk/by-id/ata-WDC_WD3200JB-00KFA0_WD-WCAMR2531970
cannot detach /dev/disk/by-id/ata-WDC_WD3200JB-00KFA0_WD-WCAMR2531970: only applicable to mirror and replacing vdevs

:~$ sudo zpool offline -t storage00 /devdisk/by-id/ata-WDC_WD3200JB-00KFA0_WD-WCAMR2531970
cannot offline /devdisk/by-id/ata-WDC_WD3200JB-00KFA0_WD-WCAMR2531970: no such device in pool

:~$ sudo zpool replace -f storage00 /dev/disk/by-uuid/24a92cd8-265a-4bc5-974e-bdb61d927633
cannot open '/dev/disk/by-uuid/24a92cd8-265a-4bc5-974e-bdb61d927633': No such file or directory

I would like to remove the drive with out losing all of my data however if this is not possible I am open to suggestions.
Please let me know if I have omitted any pertinent data or if any other information is needed.

Any help would be much appreciated.
 
Welcome, and congratulations!

Maybe you should have taken closer look at the manual before trying something like that, I sincerely hope you have backups of some sort.
action: Destroy and re-create the pool from a backup source.
Click to expand...
I know how you feel, most of us have been there ourselves. Backups, backups, backups. It cannot be stressed enough.

We live, we learn.

You cannot add drives to an existing raidz vdev. New drives added get their own vdev instead- like you´ve noticed by now.

The only chance I can think of (slim to none) is to buy another hard drive that is big enough to hold everything, or most of what´s inside your current pool. A 3TB drive e.g. Then create another pool (storage01?) on that single drive and then copy what you can from storage00 over to storage01. After that you can destroy storage00 and recreate it again with all the four drives raidz at once.

BIG FAT WARNING!!!
If you mix 1TB drives in with a small 320GB in the same raidz vdev, zfs would have to go by the smallest denominator for parity´s sake, so those 3x1TB drives would only have 320GB useable.
3x1TB+1x320GB raidz = 960GB
I can´t see the point in that.

Furthermore, the whole meaning of labels, as I see them, is to generalize them as much as possible, so even if something changes in the setup of your pool, the names should still be the same:
Code: 
        NAME             STATE     READ WRITE CKSUM
        storage00        ONLiNE       0     0     0
          raidz1-0       ONLINE       0     0     0
            gpt/disk1    ONLINE       0     0     0
            gpt/disk2    ONLINE       0     0     0
            gpt/disk3    ONLINE       0     0     0
I mean, if you have to replace one your current drives, perhaps:
Code: 
disk/by-id/ata-WDC_WD1001FALS-00J7B0_WD-WMATV0044081
that model has gone end of life, and you have to buy a different type/brand altogether. Or if you yank one out and insert it somewhere else, the names won´t match. Keep it simple, always works.

/Sebulon
 
/dev/disk/by-id/ is a Linux thing, and will give you a unique name based on the model and serial number of the disk. It should never change on any given disk so it's an excellent identifier to point ZFS at.

It's a shame FreeBSD doesn't have this feature.
 
Hi,

Firstly is the new disk still connected? Why is zfs telling you it is unavailable? If you can get the disk online then you simply need to copy your data, then as Sebulon says, destroy the pool and start again. Trying firstly with zfs send/receive would be a good idea, if that doesn't work then just a plain copy or tar would be a fallback option...

ta Andy.
 
Thanks everyone for the help!

I ended up doing an export/import and a scrub, followed by a reboot. I was then able to get to the pool back online and retrieve my data.

I then destroyed the pool and recreate is properly.

Thanks again to everyone!


However the original question still remains; how would I remove that extra drive?

1. From what it looked like ZFS was saying that the drive was in a different pool or sub-pool.
a. And after the second scrub my original three drives were back online again and I was able to read and write to them successfully.

It was almost like that new drive didn't matter as ZFS didn't add it to the over all size (or shrink the pool)and my pool just ended up in a Unavailable/Degraded state.

Still puzzled....


Zpool
 
Glad you got your data back, situations like that can be scary as hell.

However it feels like you really should go back and read up on the system you are trying to use. Personally, I spent much time reading and playing on virtual systems a lot so that I got a good feel for how stuff like this works before trying it live.

First off, from the FreeBSD Handbook
Then the FreeBSD Wiki
And ultimately, the ZFS Admin guide. Keep it under your pillow=)

I can however, try to explain the concept of vdev´s. First off, you have a pool. That pool can be made up of just one hard drive, or thousands, depending on the system. Anyhow, those hard drives will be grouped into different types of vdev´s. There are single vdev´s, mirrored vdev´s and- raidz, raidz2 and raidz3 vdev´s. A new vdev gets created every time you add new hard drives to the pool. It could look something like this:
Code: 
# zpool status
  pool: pool1
 state: ONLINE
 scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	pool1        ONLINE       0     0     0
	  mirror-0   ONLINE       0     0     0
	    ada1     ONLINE       0     0     0
	    ada2     ONLINE       0     0     0
          raidz-0    ONLINE       0     0     0
	    ada3     ONLINE       0     0     0
	    ada4     ONLINE       0     0     0
	    ada5     ONLINE       0     0     0
          raidz3-0   ONLINE       0     0     0
	    ada6     ONLINE       0     0     0
	    ada7     ONLINE       0     0     0
	    ada8     ONLINE       0     0     0
	    ada9     ONLINE       0     0     0
	    ada10    ONLINE       0     0     0
	    ada11    ONLINE       0     0     0
        ada12        ONLINE       0     0     0
        ada13        ONLINE       0     0     0
          raidz-1    ONLINE       0     0     0
	    ada14    ONLINE       0     0     0
	    ada15    ONLINE       0     0     0
	    ada16    ONLINE       0     0     0
          raidz3-1   ONLINE       0     0     0
	    ada17    ONLINE       0     0     0
	    ada18    ONLINE       0     0     0
	    ada19    ONLINE       0     0     0
	    ada20    ONLINE       0     0     0
	    ada21    ONLINE       0     0     0
	    ada22    ONLINE       0     0     0
        ada23        ONLINE       0     0     0

errors: No known data errors
Where the vdev´s explained would be:
Code: 
# zpool status
  pool: pool1
 state: ONLINE
 scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	pool1        ONLINE       0     0     0
-------------------------------------------------
         [B]vdev0[/B]
	  mirror-0   ONLINE       0     0     0
	    ada1     ONLINE       0     0     0
	    ada2     ONLINE       0     0     0
-------------------------------------------------
         [B]vdev1[/B]
          raidz-0    ONLINE       0     0     0
	    ada3     ONLINE       0     0     0
	    ada4     ONLINE       0     0     0
	    ada5     ONLINE       0     0     0
-------------------------------------------------
         [B]vdev2[/B]
          raidz3-0   ONLINE       0     0     0
	    ada6     ONLINE       0     0     0
	    ada7     ONLINE       0     0     0
	    ada8     ONLINE       0     0     0
	    ada9     ONLINE       0     0     0
	    ada10    ONLINE       0     0     0
	    ada11    ONLINE       0     0     0
-------------------------------------------------
       [B]vdev3[/B]
        ada12        ONLINE       0     0     0
-------------------------------------------------
       [B]vdev4[/B]
        ada13        ONLINE       0     0     0
-------------------------------------------------
         [B]vdev5[/B]
          raidz-1    ONLINE       0     0     0
	    ada14    ONLINE       0     0     0
	    ada15    ONLINE       0     0     0
	    ada16    ONLINE       0     0     0
-------------------------------------------------
         [B]vdev6[/B]
          raidz3-1   ONLINE       0     0     0
	    ada17    ONLINE       0     0     0
	    ada18    ONLINE       0     0     0
	    ada19    ONLINE       0     0     0
	    ada20    ONLINE       0     0     0
	    ada21    ONLINE       0     0     0
	    ada22    ONLINE       0     0     0
-------------------------------------------------
       [B]vdev7[/B]
        ada23        ONLINE       0     0     0
-------------------------------------------------

errors: No known data errors
Notice how vdev3,4- and 7 has neither mirror or raidz next to them? That´s because they are single disk vdev´s that doesn´t have parity, which means that if anything were to happen to any of those three hard drives, the whole pool would be crap. That´s what happened in your case. You had 3xraidz+1xsingle, and then you pulled the 1xsingle and then whole pool went kaputt. If you want to be able to pull out drives, you have to add, at least, two drives mirrored:
Code: 
# zpool add tank mirror ada3 ada4
# zpool status
  pool: pool1
 state: ONLINE
 scan: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	pool1        ONLINE       0     0     0
          raidz-0    ONLINE       0     0     0
	    ada0     ONLINE       0     0     0
	    ada1     ONLINE       0     0     0
	    ada2     ONLINE       0     0     0
	  mirror-0   ONLINE       0     0     0
	    ada3     ONLINE       0     0     0
	    ada4     ONLINE       0     0     0

errors: No known data errors
I hope that explains it.

/Sebulon
 
Small correction to your table above: vdev3 is only 1 disk (ada13). vdev4 would also be only 1 disk (ada14). Thus renumbering all your vdevs afterward by one.

There's no such thing as a "striped vdev". vdevs can be:
  • a single disk
  • a pair of disks in a mirror configuration
  • X number of disks in a mirror configuration (n-way mirroring)
  • X number of disks in a raidz1 configuration
  • X number of disks in a raidz2 configuration
  • X number of disks in a raidz3 configuration
  • a single disk configured as a log device
  • X number of disks configured as a mirrored log device
  • a single disk configured as a cache (L2ARC) device
  • X number of disks configured as a cache device (ZFS internally stripes across them)
  • X number of disks configured as a mirrored cache device (not really all that useful, but supported)
And, you can replace "disk" above with "any block device", including raw files (only really useful for testing), and network devices (like iSCSI exports).

A ZFS pool can called a "striped pool", meaning it's comprised of at least one non-redundant vdev. Or a "mirrored pool", meaning it's comprised of all mirror vdevs. Or a "raidz pool", meaning it's comprised of all raidz vdevs.

But there's no such thing as a "striped vdev".
 
You must log in or register to reply here.
Back
Top