Solved ZFS mirror error. Too many disks, none can be removed.

Hello.
I have a server with 186 mirrors, with 2 disks in every one of them. But I have a problem with mirror 81. It looks wrong and I can't do much.

Code:
 mirror-81                   DEGRADED     0     0     0
            spare-0                   DEGRADED     0     0     0
              replacing-0             DEGRADED     0     0     0
                11341275369875020532  REMOVED      0     0     0  was /dev/da364/old
                da365                 ONLINE       0     0     0  (resilvering)
              da257                   FAULTED      3     0     0  too many errors
            da249                     ONLINE       0     0     0  (resilvering)
          mirror-82                   ONLINE       0     0     0
            da366                     ONLINE       0     0     0
            da241
What is wrong here is that one disk was dead. Then I replaced it with a hot spare, but after 1 hour the server killed itself. When it came up again I found the broken disk and replaced it with a new one but now the hot spare can't be detached. Every time I try it says:
Code:
cannot detach da257: no valid replicas
It is still resilvering on the disk, but it have been doing this for over a week now. I have tried to detach any of the disks even tried to replace the hot spare this morning, because now it it faulted. I can't replace it because it is in use and I can't add a new hot spare to the pool. That just give me this error.
Code:
Assertion failed: (nvlist_lookup_string(cnv, "path", &path) == 0), file /usr/src/cddl/sbin/zpool/../../../cddl/contrib/opensolaris/cmd/zpool/zpool_vdev.c, line 687.
Abort (core dumped)
 
da257 is the good disk in the mirror, you don't want to remove it.
da365 and 11341275369875020532 are the bad disks in the mirror. One is removed, the other resilvering. These are the disks you need to work with.

Try the following:
Code:
# zpool detach <poolname> 11341275369875020532
# zpool detach <poolname> da365
# zpool attach <poolname> da257 da365

That will remove the old device (11341275369875020532), remove the spare device (da365) thus breaking the mirror and making mirror-81 turn into a single-disk vdev, and then re-attach the spare device (da365) to the known-good drive, thus creating a new mirror vdev. Wait for it to resilver and then carry on.
 
Hi Phoenix thanks for your answer.

I am a friend of Decent and helping him with the mirror problem.

I have tried zpool detach zpool 11341275369875020532
Then it returns this: "cannot detach 11341275369875020532: no valid replicas"

and the same goes for da365 returning the same.

Do you have any idea what to do ?
 
Is there another disk available in the pool that you could use temporarily? If so:

Code:
# zpool attach <poolname> da257 da-whatever
<wait for it to resilver>
# zpool detach <poolname> 11341275369875020532
# zpool detach <poolname> da365

That will add the extra disk to the mirror, making it a 3-way mirror vdev. Once the re-silver is complete, you should be able to remove the other two disks.

Oh, shit, crap. Just re-read your post. You may be screwed completely. :( da257 has been marked as FAULTED due to "too many errors" during the resilver. That's why you can't detach the spare or da365. You're going to lose data one way or another, and you might lose the entire pool. :(

You might get lucky and attaching another drive to da257 might work. You can also try to force the FAULTED drive back online using # zpool online <poolname> da257

You've discovered why single-redundancy sucks with very large disks. And why you should never use "zpool replace" when using mirrors. When using mirrors, always attach a new disk (making it 3-way), then detach the old disk (dropping it back to 2-way).
 
If I read this correctly, you're having four problems: The first problem is getting rid of the spares, which for some reason unknown to me has a replacing-operation performing on them (11341275369875020532 and da365). The second problem is a faulted drive in a mirror (da257). The third problem is that da249 is marked as resilvering. The fourth, and most important problem, seems to be that you don't have any healthy drives in the mirror-81 vdev.

I have no idea how you would end up in this situation. May we get relevant output from # zpool history pool?
 
Well don't know what you can use the history for, but here it is:


Code:
2013-11-12.14:27:40 zpool create -f zdata mirror da0 da36
2013-11-12.14:28:12 zpool add -f zdata mirror da1 da37
2013-11-12.14:28:13 zpool add -f zdata mirror da2 da38
2013-11-12.14:28:15 zpool add -f zdata mirror da3 da39
2013-11-12.14:28:16 zpool add -f zdata mirror da4 da40
2013-11-12.14:28:17 zpool add -f zdata mirror da5 da41
2013-11-12.14:28:19 zpool add -f zdata mirror da6 da42
2013-11-12.14:28:20 zpool add -f zdata mirror da7 da43
2013-11-12.14:28:22 zpool add -f zdata mirror da8 da44
2013-11-12.14:28:24 zpool add -f zdata mirror da9 da45
2013-11-12.14:28:26 zpool add -f zdata mirror da10 da46
2013-11-12.14:28:28 zpool add -f zdata mirror da11 da47
2013-11-12.14:28:30 zpool add -f zdata mirror da12 da48
2013-11-12.14:28:32 zpool add -f zdata mirror da13 da49
2013-11-12.14:28:35 zpool add -f zdata mirror da14 da50
2013-11-12.14:28:37 zpool add -f zdata mirror da15 da51
2013-11-12.14:28:39 zpool add -f zdata mirror da16 da52
2013-11-12.14:28:41 zpool add -f zdata mirror da17 da53
2013-11-12.14:28:44 zpool add -f zdata mirror da18 da54
2013-11-12.14:28:46 zpool add -f zdata mirror da19 da55
2013-11-12.14:28:48 zpool add -f zdata mirror da20 da56
2013-11-12.14:28:50 zpool add -f zdata mirror da21 da57
2013-11-12.14:28:52 zpool add -f zdata mirror da22 da58
2013-11-12.14:28:55 zpool add -f zdata mirror da23 da59
2013-11-12.14:28:57 zpool add -f zdata mirror da24 da60
2013-11-12.14:28:59 zpool add -f zdata mirror da25 da61
2013-11-12.14:29:01 zpool add -f zdata mirror da26 da62
2013-11-12.14:29:02 zpool add -f zdata mirror da27 da63
2013-11-12.14:29:04 zpool add -f zdata mirror da28 da64
2013-11-12.14:29:06 zpool add -f zdata mirror da29 da65
2013-11-12.14:29:09 zpool add -f zdata mirror da30 da66
2013-11-12.14:29:11 zpool add -f zdata mirror da31 da67
2013-11-12.14:29:14 zpool add -f zdata mirror da32 da68
2013-11-12.14:29:16 zpool add -f zdata mirror da33 da69
2013-11-12.14:29:19 zpool add -f zdata mirror da34 da70
2013-11-12.14:29:25 zpool add -f zdata mirror da35 da71
2013-11-12.15:05:26 zpool add -f zdata mirror da72 da108
2013-11-12.15:05:28 zpool add -f zdata mirror da73 da109
2013-11-12.15:05:30 zpool add -f zdata mirror da74 da110
2013-11-12.15:05:32 zpool add -f zdata mirror da75 da111
2013-11-12.15:05:34 zpool add -f zdata mirror da76 da112
2013-11-12.15:05:36 zpool add -f zdata mirror da77 da113
2013-11-12.15:05:37 zpool add -f zdata mirror da78 da114
2013-11-12.15:05:39 zpool add -f zdata mirror da79 da115
2013-11-12.15:05:41 zpool add -f zdata mirror da80 da116
2013-11-12.15:05:43 zpool add -f zdata mirror da81 da117
2013-11-12.15:05:45 zpool add -f zdata mirror da82 da118
2013-11-12.15:05:47 zpool add -f zdata mirror da83 da119
2013-11-12.15:05:49 zpool add -f zdata mirror da84 da120
2013-11-12.15:05:51 zpool add -f zdata mirror da85 da121
2013-11-12.15:05:53 zpool add -f zdata mirror da86 da122
2013-11-12.15:05:55 zpool add -f zdata mirror da87 da123
2013-11-12.15:05:57 zpool add -f zdata mirror da88 da124
2013-11-12.15:05:58 zpool add -f zdata mirror da89 da125
2013-11-12.15:06:00 zpool add -f zdata mirror da90 da126
2013-11-12.15:06:02 zpool add -f zdata mirror da91 da127
2013-11-12.15:06:05 zpool add -f zdata mirror da92 da128
2013-11-12.15:06:07 zpool add -f zdata mirror da93 da129
2013-11-12.15:06:09 zpool add -f zdata mirror da94 da130
2013-11-12.15:06:11 zpool add -f zdata mirror da95 da131
2013-11-12.15:06:13 zpool add -f zdata mirror da96 da132
2013-11-12.15:06:15 zpool add -f zdata mirror da97 da133
2013-11-12.15:06:17 zpool add -f zdata mirror da98 da134
2013-11-12.15:06:19 zpool add -f zdata mirror da99 da135
2013-11-12.15:06:21 zpool add -f zdata mirror da100 da136
2013-11-12.15:06:23 zpool add -f zdata mirror da101 da137
2013-11-12.15:06:26 zpool add -f zdata mirror da102 da138
2013-11-12.15:06:28 zpool add -f zdata mirror da103 da139
2013-11-12.15:06:30 zpool add -f zdata mirror da104 da140
2013-11-12.15:06:33 zpool add -f zdata mirror da105 da141
2013-11-12.15:06:35 zpool add -f zdata mirror da106 da142
2013-11-12.15:06:41 zpool add -f zdata mirror da107 da143
2013-11-12.15:58:32 zfs create zdata/F_Drev
2013-11-12.16:01:14 zpool add -f zdata mirror da144 da180
2013-11-12.16:01:16 zpool add -f zdata mirror da145 da181
2013-11-12.16:01:19 zpool add -f zdata mirror da146 da182
2013-11-12.16:01:21 zpool add -f zdata mirror da147 da183
2013-11-12.16:01:23 zpool add -f zdata mirror da148 da184
2013-11-12.16:01:25 zpool add -f zdata mirror da149 da185
2013-11-12.16:01:28 zpool add -f zdata mirror da150 da186
2013-11-12.16:01:30 zpool add -f zdata mirror da151 da187
2013-11-12.16:01:32 zpool add -f zdata mirror da152 da188
2013-11-12.16:01:35 zpool add -f zdata mirror da153 da189
2013-11-12.16:01:37 zpool add -f zdata mirror da154 da190
2013-11-12.16:01:39 zpool add -f zdata mirror da155 da191
2013-11-12.16:01:42 zpool add -f zdata mirror da156 da192
2013-11-12.16:01:44 zpool add -f zdata mirror da157 da193
2013-11-12.16:01:46 zpool add -f zdata mirror da158 da194
2013-11-12.16:01:49 zpool add -f zdata mirror da159 da195
2013-11-12.16:01:51 zpool add -f zdata mirror da160 da196
2013-11-12.16:01:53 zpool add -f zdata mirror da161 da197
2013-11-12.16:01:56 zpool add -f zdata mirror da162 da198
2013-11-12.16:01:58 zpool add -f zdata mirror da163 da199
2013-11-12.16:02:01 zpool add -f zdata mirror da164 da200
2013-11-12.16:02:03 zpool add -f zdata mirror da165 da201
2013-11-12.16:02:06 zpool add -f zdata mirror da166 da202
2013-11-12.16:02:08 zpool add -f zdata mirror da167 da203
2013-11-12.16:02:10 zpool add -f zdata mirror da168 da204
2013-11-12.16:02:13 zpool add -f zdata mirror da169 da205
2013-11-12.16:02:16 zpool add -f zdata mirror da170 da206
2013-11-12.16:02:19 zpool add -f zdata mirror da171 da207
2013-11-12.16:02:22 zpool add -f zdata mirror da172 da208
2013-11-12.16:02:25 zpool add -f zdata mirror da173 da209
2013-11-12.16:02:27 zpool add -f zdata mirror da174 da210
2013-11-12.16:02:30 zpool add -f zdata mirror da175 da211
2013-11-12.16:02:33 zpool add -f zdata mirror da176 da212
2013-11-12.16:02:36 zpool add -f zdata mirror da177 da213
2013-11-12.16:02:43 zpool add -f zdata mirror da178 da214
2013-11-12.16:08:25 zpool add -f zdata mirror da216 da252
2013-11-12.16:08:28 zpool add -f zdata mirror da217 da253
2013-11-12.16:08:31 zpool add -f zdata mirror da218 da254
2013-11-12.16:08:33 zpool add -f zdata mirror da219 da255
2013-11-12.16:08:36 zpool add -f zdata mirror da220 da256
2013-11-12.16:08:39 zpool add -f zdata mirror da221 da257
2013-11-12.16:08:42 zpool add -f zdata mirror da222 da258
2013-11-12.16:08:44 zpool add -f zdata mirror da223 da259
2013-11-12.16:08:47 zpool add -f zdata mirror da224 da260
2013-11-12.16:08:50 zpool add -f zdata mirror da225 da261
2013-11-12.16:08:52 zpool add -f zdata mirror da226 da262
2013-11-12.16:08:55 zpool add -f zdata mirror da227 da263
2013-11-12.16:08:58 zpool add -f zdata mirror da228 da264
2013-11-12.16:09:01 zpool add -f zdata mirror da229 da265
2013-11-12.16:09:04 zpool add -f zdata mirror da230 da266
2013-11-12.16:09:07 zpool add -f zdata mirror da231 da267
2013-11-12.16:09:10 zpool add -f zdata mirror da232 da268
2013-11-12.16:09:13 zpool add -f zdata mirror da233 da269
2013-11-12.16:09:15 zpool add -f zdata mirror da234 da270
2013-11-12.16:09:18 zpool add -f zdata mirror da235 da271
2013-11-12.16:09:21 zpool add -f zdata mirror da236 da272
2013-11-12.16:09:23 zpool add -f zdata mirror da237 da273
2013-11-12.16:09:26 zpool add -f zdata mirror da238 da274
2013-11-12.16:10:30 zpool add -f zdata mirror da239 da275
2013-11-12.16:10:33 zpool add -f zdata mirror da240 da276
2013-11-12.16:10:36 zpool add -f zdata mirror da241 da277
2013-11-12.16:10:39 zpool add -f zdata mirror da242 da278
2013-11-12.16:10:42 zpool add -f zdata mirror da243 da279
2013-11-12.16:10:45 zpool add -f zdata mirror da244 da280
2013-11-12.16:10:47 zpool add -f zdata mirror da245 da281
2013-11-12.16:10:50 zpool add -f zdata mirror da246 da282
2013-11-12.16:10:53 zpool add -f zdata mirror da247 da283
2013-11-12.16:10:56 zpool add -f zdata mirror da248 da284
2013-11-12.16:11:04 zpool add -f zdata mirror da249 da285
2013-11-25.07:59:00 zpool clear zdata
2013-11-25.07:59:10 zpool clear zdata
2013-11-25.08:02:45 zpool clear zdata
2013-11-25.08:04:07 zpool clear zdata
2013-11-26.07:29:19 zpool clear zdata
2013-11-26.07:32:17 zpool clear zdata
2013-11-26.07:33:20 zpool clear zdata
2013-11-26.07:36:57 zpool clear zdata
2013-11-26.07:51:26 zpool replace zdata 3831200204989771795 da104
2013-11-26.07:54:18 zpool clear zdata
2013-11-26.08:57:26 zpool add zdata mirror da248 da254
2013-11-26.11:02:39 zpool replace -f zdata 6580619803073622214 da155
2013-11-26.11:11:33 zpool add zdata spare da145
2013-11-29.07:28:57 zpool replace zdata 974335334163000597 da145
2013-12-02.07:09:09 zpool clear zdata
2013-12-06.13:14:41 zpool scrub zdata
2013-12-10.14:55:47 zpool clear zdata
2013-12-10.14:59:06 zpool detach zdata 11210730724159969895
2013-12-10.15:01:47 zpool online zdata da248
2013-12-10.15:55:33 zpool replace zdata da248 da248
2013-12-16.08:21:04 zpool clear zdata
2013-12-18.10:24:11 zpool clear zdata
2013-12-20.15:30:56 zpool replace zdata da154 da145
2013-12-30.06:41:32 zpool clear zdata
2014-01-03.14:22:11 zpool clear zdata
2014-01-03.14:24:01 zpool detach zdata da154
2014-01-03.14:25:44 zpool detach zdata da145
2014-01-03.14:41:30 zpool replace zdata da167 da154
2014-01-03.15:13:37 zpool clear zdata
2014-01-06.08:00:45 zpool clear zdata
2014-01-06.08:12:43 zpool replace zdata da167 da155
2014-01-09.15:07:43 zpool scrub zdata
2014-01-13.11:46:32 zpool clear zdata
2014-01-13.12:29:02 zpool replace zdata 16258576517851923034 da280
2014-01-15.09:23:25 zpool replace zdata da165 da165
2014-01-20.10:09:36 zpool add zdata spare da167
2014-01-20.10:09:55 zpool add zdata spare da193
2014-01-20.13:14:24 zpool add zdata mirror da287 da332
2014-01-20.13:15:17 zpool add zdata mirror da288 da333
2014-01-20.13:15:53 zpool add zdata mirror da289 da334
2014-01-20.13:20:44 zpool add zdata mirror da290 da335
2014-01-20.13:21:12 zpool add zdata mirror da291 da336
2014-01-20.13:21:40 zpool add zdata mirror da292 da337
2014-01-20.13:22:09 zpool add zdata mirror da293 da338
2014-01-20.13:22:58 zpool add zdata mirror da294 da339
2014-01-20.13:24:12 zpool add zdata mirror da295 da340
2014-01-20.13:24:52 zpool add zdata mirror da296 da341
2014-01-20.13:26:00 zpool add zdata mirror da297 da342
2014-01-20.13:27:45 zpool add zdata mirror da298 da343
2014-01-20.16:12:04 zpool clear zdata
2014-01-20.16:12:26 zpool online zdata da342
2014-01-20.16:13:09 zpool online zdata 1331786078584418327
2014-01-21.07:43:54 zpool replace zdata 1331786078584418327 da234
2014-01-21.14:13:57 zpool online zdata 1331786078584418327
2014-01-21.14:14:11 zpool clear zdata
2014-01-21.14:37:08 zpool detach zdata da234
2014-01-21.14:37:42 zpool online zdata da376
2014-01-21.14:43:00 zpool clear zdata
2014-01-21.15:07:45 zpool online zdata da376
2014-01-21.15:08:16 zpool clear zdata
2014-01-22.10:03:15 zpool clear zdata
2014-01-22.10:13:25 zpool replace zdata da365 da365
2014-01-22.10:14:39 zpool clear zdata
2014-01-22.12:25:54 zpool add zdata mirror da1 da45
2014-01-22.12:26:56 zpool add zdata mirror da2 da48
2014-01-22.12:28:06 zpool add zdata mirror da3 da49
2014-01-22.12:29:08 zpool add zdata mirror da5 da50
2014-01-22.12:30:20 zpool add zdata mirror da6 da51
2014-01-22.12:31:59 zpool add zdata mirror da7 da52
2014-01-22.12:32:55 zpool add zdata mirror da8 da53
2014-01-22.12:33:57 zpool add zdata mirror da10 da55
2014-01-22.12:34:53 zpool add zdata mirror da11 da56
2014-01-22.12:35:53 zpool add zdata mirror da12 da57
2014-01-22.12:37:10 zpool add zdata mirror da15 da58
2014-01-22.12:37:54 zpool add zdata mirror da17 da59
2014-01-22.12:39:24 zpool add zdata mirror da18 da61
2014-01-22.12:40:28 zpool add zdata mirror da19 da62
2014-01-22.12:41:53 zpool add zdata mirror da25 da64
2014-01-22.12:42:42 zpool add zdata mirror da26 da65
2014-01-22.12:43:51 zpool add zdata mirror da27 da66
2014-01-22.12:44:31 zpool add zdata mirror da28 da67
2014-01-22.12:45:30 zpool add zdata mirror da29 da68
2014-01-22.12:46:25 zpool add zdata mirror da31 da69
2014-01-22.12:48:44 zpool add zdata mirror da32 da71
2014-01-22.12:49:50 zpool add zdata mirror da33 da72
2014-01-22.12:50:51 zpool add zdata mirror da34 da73
2014-01-22.12:51:41 zpool add zdata mirror da35 da75
2014-01-22.12:52:59 zpool add zdata mirror da36 da76
2014-01-22.12:54:03 zpool add zdata mirror da37 da78
2014-01-22.12:55:23 zpool add zdata mirror da38 da80
2014-01-22.12:56:18 zpool add zdata mirror da39 da82
2014-01-22.12:57:22 zpool add zdata mirror da40 da83
2014-01-22.13:02:40 zpool add zdata mirror da41 da84
2014-01-22.13:03:38 zpool add zdata mirror da42 da85
2014-01-22.13:04:27 zpool add zdata mirror da43 da88
2014-01-22.13:05:14 zpool add zdata mirror da44 da89
2014-01-22.13:06:31 zpool online zdata 11341275369875020532
2014-01-23.06:59:54 zpool replace zdata 17111771926446099342 da128
2014-01-23.08:55:21 zpool replace zdata 11341275369875020532 da144
2014-01-25.11:55:08 zpool clear zdata
2014-01-25.12:01:50 zpool replace zdata 11341275369875020532 da364
2014-01-25.12:03:40 zpool clear zdata
2014-01-27.08:13:21 zpool detach zdata 11210730724159969895
2014-01-27.08:15:43 zpool replace zdata 9132969045324487606 da234
2014-01-28.08:15:29 zpool detach zdata da182
2014-01-28.08:16:09 zpool clear zdata
2014-01-28.08:56:25 zpool clear zdata
2014-01-28.09:02:20 zpool detach zdata da169
2014-01-28.09:10:17 zpool replace zdata da117 da117
2014-01-30.09:22:34 zpool detach zdata 17111771926446099342
2014-01-30.09:22:51 zpool detach zdata 9132969045324487606
2014-01-30.09:23:33 zpool detach zdata 12366269882642857912
2014-01-30.09:23:53 zpool clear zdata
2014-01-30.14:26:24 zpool clear zdata
2014-01-30.14:32:45 zpool attach zdata da135 da121
2014-01-30.14:38:42 zpool attach zdata da316 da169
2014-01-30.14:44:17 zpool attach zdata da176 da182
2014-01-30.14:52:44 zpool online zdata da193
2014-02-03.08:12:00 zpool clear zdata
2014-02-05.07:50:13 zpool clear zdata
2014-02-05.07:53:39 zpool replace zdata 5765566219276969055 da312
2014-02-07.07:07:24 zpool clear zdata
2014-02-07.08:26:07 zpool online zdata da257
2014-02-07.08:26:42 zpool clear zdata
2014-02-07.08:39:10 zpool attach zdata da249 da256
2014-02-11.10:05:54 zpool clear zdata
2014-02-11.13:25:16 zpool attach zdata da77 da165
2014-02-11.13:26:09 zpool detach zdata 1331786078584418327


*Edit

This is the part where it all began:

Code:
2014-01-22.13:06:31 zpool online zdata 11341275369875020532
2014-01-23.06:59:54 zpool replace zdata 17111771926446099342 da128
2014-01-23.08:55:21 zpool replace zdata 11341275369875020532 da144
2014-01-25.11:55:08 zpool clear zdata
2014-01-25.12:01:50 zpool replace zdata 11341275369875020532 da364
2014-01-25.12:03:40 zpool clear zdata
2014-01-27.08:13:21 zpool detach zdata 11210730724159969895

The vdev also look like this now.

Code:
 mirror-81                   DEGRADED     0     0     0
            spare-0                   DEGRADED     0     0     0
              replacing-0             DEGRADED     0     0     0
                11341275369875020532  REMOVED      0     0     0  was /dev/da364/old
                da364                 ONLINE       0     0     0  (resilvering)
              da256                   ONLINE       0     0     0  (resilvering)
            da248                     ONLINE       0     0     0
            da255                     ONLINE       0     0     0  (resilvering)
 
Back
Top