Solved [Solved] Mirror raid error. too many disks in one vdev

D

Dcent

Hello, I have been working with FreeBSD and ZFS for one year now and have never had big problems with it. But now I ran into one, that I can't google my way to solve. I have a big 405 TB server with mirror RAID. All the vdevs are looking fine, but then I have one vdev-65 that is acting weird. instead of 2 disks in a mirror it has 5 all in all, I can't detach any of them or do any thing to them.

Code: 
mirror-64                     ONLINE       0     0     0
    da190                     ONLINE       0     0     0
    da141                     ONLINE       0     0     0
mirror-65                     ONLINE       0     0     0
    da172                     ONLINE       0     0     0
    da315                     ONLINE       0     0     0
 da135                        ONLINE       0     0     0
 da316                        ONLINE       0     0     0
 da176                        ONLINE       0     0     0
mirror-69                     ONLINE       0     0     0
    da181                     ONLINE       0     0     0
    da124                     ONLINE       0     0     0

The devicess look like above, in the pool there are 378 disks, and 183 vdevs. And as you can see above there is a jump in vdevs from 65 to 69; can this have anything to do with my disk problem? Has anyone else ever seen a zpool do this before, having 5 disks in a 2-disk mirror RAID?
 
Re: Mirror raid error. to many disks in one vdev

Can you show the output of the following (it's a lowercase L, not a 1 just to be clear):
It will output 4 identical labels so we only really need to see one of them.

Code: 
zdb -l /dev/da135

There's technically nothing wrong with having 5 disks in one mirror vdev. You should be able to detach all but 1 though.
 
Re: Mirror raid error. to many disks in one vdev

I have the output here for the one da135.

Code: 
--------------------------------------------
LABEL 0
--------------------------------------------
    version: 5000
    name: 'zdata'
    state: 0
    txg: 1130609
    pool_guid: 4295100231021637183
    hostid: 3113250528
    hostname: 'fdrevbackup.danskscanning.dk'
    top_guid: 1689873263577018192
    guid: 1689873263577018192
    vdev_children: 187
    vdev_tree:
        type: 'disk'
        id: 66
        guid: 1689873263577018192
        path: '/dev/da135'
        phys_path: '/dev/da135'
        whole_disk: 1
        metaslab_array: 141
        metaslab_shift: 34
        ashift: 9
        asize: 2000394125312
        is_log: 0
        DTL: 5488
        create_txg: 633
    features_for_read:

No, there is nothing wrong with that, but I built the RAID with 2 disks and it just out of nowhere put in 3 more. And I can't detach them at all, not even one of them.
 
Re: Mirror raid error. to many disks in one vdev

Well I have some bad news.

That disk is a top level 'single disk' vdev. It is not part of any mirror. Obviously I can't confirm for the other 2 but I suspect they are the same.
The loss of that disk will fault the entire pool.

Considering the mirror numbers are not in sequence, I can only assume that someone has detached the other disks from these mirrors at some point. Either that or ZFS has somehow managed to screw up and not only lose the second disks that should be in these mirrors, but also rewrite the labels on their counterparts. (not an issue I've ever heard about or could imagine happening).***

Hopefully you have some spare disks in this server and can re-make these 3 into mirrors by attaching new disks to them.

nothere is nothing wrong with that
Click to expand...

Just to be clear, yes there is a lot wrong with it. Your entire 378 disk pool relies on these 3 disks.
You need to do the following as soon as possible:

Code: 
zpool attach zdata da135 {newdisk}
zpool attach zdata da316 {newdisk}
zpool attach zdata da176 {newdisk}

There is nothing wrong with mirror-65, it has two disks in it.

***Edit: The other possibility is that you did something like this:
The -f would of been required as it would complain about mismatched vdevs otherwise.

Code: 
---loads of previous zpool add commands---
zpool add zdata mirror da172 da315 (--adding mirror number 65--)
zpool add -f zdata da135
zpool add -f zdata da316
zpool add -f zdata da176
zpool add -f zdata mirror da181 da124
 
Re: Mirror raid error. to many disks in one vdev

No, that pool is a non-redundant pool because you have some of the disks as separate disks that are not part of any mirror vdev. There is no way to recover all of the data in this case if there's a problem with one of the disks that are not part of a mirror. You can expand the pool into a fully redundant one by attaching more disks and attach those new disks to the single disk vdevs so that they become 2-disk mirrors.
 
Re: Mirror raid error. to many disks in one vdev

Can you just add a disk to a vdev by zpool add zdata mirror-65 da123 if that disk is not in the pool? Or is there another way of doing it?
 
Re: Mirror raid error. to many disks in one vdev

usdmatt said:
Well I have some bad news.

That disk is a top level 'single disk' vdev. It is not part of any mirror. Obviously I can't confirm for the other 2 but I suspect they are the same.
The loss of that disk will fault the entire pool.

Considering the mirror numbers are not in sequence, I can only assume that someone has detached the other disks from these mirrors at some point. Either that or ZFS has somehow managed to screw up and not only lose the second disks that should be in these mirrors, but also rewrite the labels on their counterparts. (not an issue I've ever heard about or could imagine happening).***

Hopefully you have some spare disks in this server and can re-make these 3 into mirrors by attaching new disks to them.

nothere is nothing wrong with that
Click to expand...

Just to be clear, yes there is a lot wrong with it. Your entire 378 disk pool relies on these 3 disks.
You need to do the following as soon as possible:

Code: 
zpool attach zdata da135 {newdisk}
zpool attach zdata da316 {newdisk}
zpool attach zdata da176 {newdisk}

There is nothing wrong with mirror-65, it has two disks in it.

***Edit: The other possibility is that you did something like this:
The -f would of been required as it would complain about mismatched vdevs otherwise.

Code: 
---loads of previous zpool add commands---
zpool add zdata mirror da172 da315 (--adding mirror number 65--)
zpool add -f zdata da135
zpool add -f zdata da316
zpool add -f zdata da176
zpool add -f zdata mirror da181 da124
Click to expand...

So if I use the commaneds above here, I can get the vdevs 66, 67, 68 back?
 
Re: Mirror raid error. to many disks in one vdev

Assuming you have 3 available disks, the following will make those three single disk vdevs back into mirrors:

Code: 
zpool attach zdata da135 {newdisk}
zpool attach zdata da316 {newdisk}
zpool attach zdata da176 {newdisk}

They might not be called mirror-66,67,68 but the name is fairly irrelevant.
 
Re: Mirror raid error. to many disks in one vdev

zpool attach zdata da135 da121 !!!! It worked :D

Thank you all so much for this good support and for helping me and @decent.
 
Last edited by a moderator:
Re: Mirror raid error. too many disks in one vdev

Dcent said:
Hello, I have been working with FreeBSD and ZFS for one year now and have never had big problems with it. But now I ran into one, that I can't google my way to solve. I have a big 405 TB server with mirror RAID. All the vdevs are looking fine, but then I have one vdev-65 that is acting weird. instead of 2 disks in a mirror it has 5 all in all
Click to expand...

No, it doesn't. mirror-65 vdev has 2 disks in it. Underneath that, you see 3 single-disk vdevs, meaning 3 non-redundant disks, meaning you have a non-redundant pool! If any one of those 3 drives die ... your entire pool is gone!

You can't detach them because there's nothing to detach from. You need to find 3 more drives, and attach them to each of the non-redundant disks to create 3 more mirror vdevs.

Edit: I see others have already helped you to solve this. :)
 
You must log in or register to reply here.
Back
Top