I accdidentally added a disk to ZFS array

[traustitj]

I am running FreeBSD 9.1.

I have been looking at answers for this question and actually found a few, but all with no solution.

I have a 8 disk raidz2 array. One disk decided to die, no problem, I added a new disk to the computer, and then accidentally did

zpool add mysan da1

so now I have 8 disks, and 1 disk out of the array. As far as I have read, this means if the da1 disk dies, the whole array goes with it, which is frightening to say the least, is that true ?

Is there absolutely no way to do this without recreating the storage from scratch ? We are talking about 7 TB of data, backups and stuff. It takes days to move to another computer and days to put back.

This is how the layout looks like now :

[root@q ~]# zpool status mysan
  pool: mysan
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
	the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: resilvered 29K in 0h0m with 0 errors on Mon May 20 19:22:06 2013
config:

	NAME                     STATE     READ WRITE CKSUM
	mysan                    DEGRADED     0     0     0
	  raidz2-0               DEGRADED     0     0     0
	    da0                  ONLINE       0     0     0
	    6676788984868813488  UNAVAIL      0     0     0  was /dev/da5
	    da2                  ONLINE       0     0     0
	    da3                  ONLINE       0     0     0
	    da4                  ONLINE       0     0     0
	    da5                  ONLINE       0     0     0
	    da6                  ONLINE       0     0     0
	    da7                  ONLINE       0     0     0
	  da1                    ONLINE       0     0     0

errors: No known data errors

 

[da1]

Hi,

zpool offline mysan da1 thought offline-ing myself is not something I would normally do . then:
zpool detach mysan da1
zpool replace mysan 6676788984868813488 da1

That should do the trick I think.

As far as I have read, this means if the da1 disk dies, the whole array goes with it

You have a raidz2, why should the array die if you loose 1 disk (even 2). Because it's raid2, it is supposed to withstand 2x dead disks.

PS: maybe you should take into consideration using labels.

 

[pillai_hfx]

Unfortunatly this is one of the things to be watchful for when replacing a disk as far as I know. This was discussed before a few times in different forums. Here is an example: http://forums.freebsd.org/showthread.php?t=23127.

From your zpool status you can see that da1 got striped with your raidz2 vdev . I don't think you would be able to simply detach da1 and replace your failed disk with it.

 

[kpa]

You can't do detach on devices in a striped pool, once the device is part of a striped pool like in this case it is there forever. The detach operation works only on devices in mirror vdevs.

 

[HarryE]

Don't do anything without testing

You may lose all of your pool data if/when da1 goes bad. You cannot off-line da1, there are insufficient replicas. Just buy another similar disk and # zpool attach mysan da1 da8. You'll get a regular ZFS mirror for da1 and the newly bought da8.

The pool will look like

[root@q ~]# zpool status mysan
  pool: mysan
 state: ONLINE
  scan: resilvered 29K in 0h0m with 0 errors on Mon May 20 19:22:06 2013
config:

	NAME                     STATE     READ WRITE CKSUM
	mysan                    ONLINE       0     0     0
	  raidz2-0               ONLINE       0     0     0
	    da0                  ONLINE       0     0     0
	    da2                  ONLINE       0     0     0
	    da3                  ONLINE       0     0     0
	    da4                  ONLINE       0     0     0
	    da5                  ONLINE       0     0     0
	    da6                  ONLINE       0     0     0
	    da7                  ONLINE       0     0     0
          mirror-1               ONLINE       0     0     0
            da1                  ONLINE       0     0     0
            da8                  ONLINE       0     0     0
errors: No known data errors

As a good practice, create a test zpool using memory-mapped temporary files (md) and test the zpool commands on that pool before submitting them to the real pool, such as ]

mdconfig -a -t malloc -s 100MB
mdconfig -a -t malloc -s 100MB
mdconfig -a -t malloc -s 100MB
mdconfig -a -t malloc -s 100MB
mdconfig -a -t malloc -s 100MB
mdconfig -a -t malloc -s 100MB
zpool create test1 raidz2 md0 md1 md2 md3
zpool add -f test1 md4

Now you have a similar pool to the current one. Try any zpool commands on test1 pool, safely. Don't forget to destroy the test1 pool and md devices before rebooting.