ZFS Replacing Drive in ZFS RAID10

jasonvp

jasonvp

Hey gang -

I think I goofed, but hope someone can help. I have a ZFS equiv of a RAID10 on my server, and one of the drives /dev/ada3 ate it. This is what it looked like prior to any operations:

Code: 
NAME                          STATE     READ WRITE CKSUM
    local                         DEGRADED     0     0     0
      mirror-0                    ONLINE       0     0     0
        diskid/DISK-WOL240296723  ONLINE       0     0     0
        diskid/DISK-WOL240299242  ONLINE       0     0     0
      mirror-1                    DEGRADED     0     0     0
        ada3                      FAULTED      0   297     0  too many errors
        diskid/DISK-69Q0A0HUF6BG  ONLINE       0     0     0
      mirror-2                    ONLINE       0     0     0
        diskid/DISK-WOL240299299  ONLINE       0     0     0
        diskid/DISK-WOL240299261  ONLINE       0     0     0
      mirror-3                    ONLINE       0     0     0
        ada7                      ONLINE       0     0     0
        ada0                      ONLINE       0     0     0

I did the zpool offline local /dev/ada3 and then zpool detach local /dev/ada3 and I think, perhaps, this is where I mucked up? After swapping the drive out, it came back as /dev/ada3 again. I couldn't zpool replace because the old drive was already gone and the new drive took its same name. I added the new one back hoping that because it was a RAID10, it would mirror it back up to ada2 and start the silvering process. No love.

Code: 
    NAME                          STATE     READ WRITE CKSUM
    local                         ONLINE       0     0     0
      mirror-0                    ONLINE       0     0     0
        diskid/DISK-WOL240296723  ONLINE       0     0     0
        diskid/DISK-WOL240299242  ONLINE       0     0     0
      diskid/DISK-69Q0A0HUF6BG    ONLINE       0     0     0
      mirror-2                    ONLINE       0     0     0
        diskid/DISK-WOL240299299  ONLINE       0     0     0
        diskid/DISK-WOL240299261  ONLINE       0     0     0
      mirror-3                    ONLINE       0     0     0
        ada7                      ONLINE       0     0     0
        ada0                      ONLINE       0     0     0
      ada3                        ONLINE       0     0     0

I'm not sure what to do at this point. There's data on ada2 (diskid/DISK-69Q0A0HUF6BG) and I need to create a mirror out of it and ada3, but without destroying what's on the first one. Is there a way to do that? Or am I in trouble?

Thanks for any help.
 
You didn't need the detach step, offline/ replace would have been enough. Now you have added ada3 as top-level vdev which is obviously wrong looking at the status output, you should have used attach command instead. What you could try doing is use the remove command on ada3 (it has limitations documented in zpool-remove(8) so I'm not sure if it will work) and then use the attach command properly.
 
cyric said:
You didn't need the detach step, offline/ replace would have been enough. Now you have added ada3 as top-level vdev which is obviously wrong looking at the status output, you should have used attach command instead. What you could try doing is use the remove command on ada3 (it has limitations documented in zpool-remove(8) so I'm not sure if it will work) and then use the attach command properly.
Click to expand...

I know how I goofed; I was following poorly written instructions that didn't apply to what I was trying to do. But there's little I can do about that now. Remove doesn't work since it has no replicas. Remove keeps complaining to me about that. What I'm not clear on is: If I shut the system down and pop the drive out (hot-swap), then power on and re-connect the device, will ZFS see that and pop it back into the pool automatically?
 
jasonvp said:
What I'm not clear on is: If I shut the system down and pop the drive out (hot-swap), then power on and re-connect the device, will ZFS see that and pop it back into the pool automatically?
Click to expand...
AFAIK, zfsd(8) should handle this case if it's enabled and autoreplace property is set to on for the pool.
 
cyric said:
Hmm, what release are you on? zpool get feature@device_removal?
Click to expand...
Very old. I'm not in a place where I can upgrade it right now as it's too important and very stable as is. Prior to this little f up, it was up for 800+ days.

FreeBSD 12.2-RELEASE-p4 GENERIC


# zpool get feature@device_removal
NAME PROPERTY VALUE SOURCE
local feature@device_removal enabled local
zroot feature@device_removal enabled local


The reboot without the drive in place caused the entire pool to not come online. It's where my home directories are and I couldn't SSH into the server once booted. So I know the pool was off. Popping the drive back in and rebooting brought the entire thing back online, but again, /dev/ada3 is part of the pool and I can't remove it.
 
jasonvp said:
Very old. I'm not in a place where I can upgrade it right now as it's too important and very stable as is.
Click to expand...
OK, I'm not really sure if it's really that old for device removal, but on a more recent version (running -CURRENT actually, but don't tell anyone) removal seems to work for me in test pool in somewhat similar setup:
Code: 
  pool: tst
 state: ONLINE
config:

        NAME               STATE     READ WRITE CKSUM
        tst                ONLINE       0     0     0
          /home/cyric/e    ONLINE       0     0     0
          mirror-1         ONLINE       0     0     0
            /home/cyric/a  ONLINE       0     0     0
            /home/cyric/b  ONLINE       0     0     0
          mirror-2         ONLINE       0     0     0
            /home/cyric/c  ONLINE       0     0     0
            /home/cyric/d  ONLINE       0     0     0

# zpool remove tst /home/cyric/e
# zpool status tst
  pool: tst
 state: ONLINE
remove: Removal of vdev 0 copied 341M in 0h0m, completed on Sat Jun  7 23:08:23 2025
        816 memory used for removed device mappings
config:

        NAME               STATE     READ WRITE CKSUM
        tst                ONLINE       0     0     0
          mirror-1         ONLINE       0     0     0
            /home/cyric/a  ONLINE       0     0     0
            /home/cyric/b  ONLINE       0     0     0
          mirror-2         ONLINE       0     0     0
            /home/cyric/c  ONLINE       0     0     0
            /home/cyric/d  ONLINE       0     0     0

errors: No known data errors

Not really sure what else you could try outside of upgrading/checking removal on newer version, sorry.
 
You must log in or register to reply here.
Back
Top