ZFS Faulted disk replacement

Hi,
I have this setup:
Code:
# zpool status
  pool: mail
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        mail        ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            ada0    ONLINE       0     0     0
            ada1    ONLINE       0     0     0
            ada2    ONLINE       0     0     0
            ada3    ONLINE       0     0     0
            ada4    ONLINE       0     0     0
            ada5    ONLINE       0     0     0
            ada6    ONLINE       0     0     0
          raidz2-1  ONLINE       0     0     0
            ada7    ONLINE       0     0     0
            ada8    ONLINE       0     0     0
            ada9    ONLINE       0     0     0
            ada10   ONLINE       0     0     0
            ada11   ONLINE       0     0     0
            ada12   ONLINE       0     0     0
            ada13   ONLINE       0     0     0
        spares
          ada14     AVAIL

errors: No known data errors

In case of faulted disk in any of my vdevs (eg. ada4), correct way to solve is:

Code:
# zpool replace mail ada4 ada14

and then physically remove ada4, and replace it with a new spare disk, or should I take offline ada4 first?

In your opinion, there is no reason to physically swap ada4 and ada14 ?
Thank you
 
Does anyone know if zfsd is fully functional? I know it exists in recent versions of FreeBSD but I've never got round to actually trying it. zfsd should automatically handle using ada14 as a hot spare.

I see no reason to physically swap the spare drive. The idea is that the spare can be used immediately when a pool drive fails to minimise the amount of degraded time.

My understanding is that the general method for using spare is as follows:

  1. Drive fails and a replace command is issued (either by hand or by zfsd to replace the disk
  2. At this point the spare should show as INUSE and the failed disk & spare will show as a "spare" vdev in the pool (similar to a mirror vdev)
  3. Once re-silvered, the original disk can be detached, leaving just the spare in the pool.
  4. Remove the old disk and replace, then set this as a spare.
It's also possible to replace the old disk, then resilver onto the replacement and put the spare back, but that means resilvering twice which seems a worse option to me.

Test example below (At the end I could physically replace md1 if it was a real disk and set it as a spare)
Code:
# zpool create test md0 md1 spare md2
# zpool status test
  pool: test
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        test        ONLINE       0     0     0
          md0       ONLINE       0     0     0
          md1       ONLINE       0     0     0
        spares
          md2       AVAIL

errors: No known data errors
#
#
#
# zpool replace test md1 md2
# zpool status test
  pool: test
 state: ONLINE
  scan: resilvered 49.5K in 0h0m with 0 errors on Tue Feb 26 10:45:26 2019
config:

        NAME                   STATE     READ WRITE CKSUM
        test                   ONLINE       0     0     0
          md0                  ONLINE       0     0     0
          spare-1              ONLINE       0     0     0
            md1                ONLINE       0     0     0
            md2                ONLINE       0     0     0
        spares
          3658298727657793991  INUSE     was /dev/md2

errors: No known data errors
#
#
#
# zpool detach test md1
# zpool status test
  pool: test
 state: ONLINE
  scan: resilvered 49.5K in 0h0m with 0 errors on Tue Feb 26 10:45:26 2019
config:

        NAME        STATE     READ WRITE CKSUM
        test        ONLINE       0     0     0
          md0       ONLINE       0     0     0
          md2       ONLINE       0     0     0

errors: No known data errors
 
Hi usdmatt,
thank you, I confirm your steps :)

I did basically same operations:

Scenario:

Code:
# zpool status
  pool: mail
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        mail        ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            ada1    ONLINE       0     0     0
            ada2    ONLINE       0     0     0
            ada3    ONLINE       0     0     0
            ada4    ONLINE       0     0     0
            ada5    ONLINE       0     0     0
            ada6    ONLINE       0     0     0
            ada7    ONLINE       0     0     0
          raidz2-1  ONLINE       0     0     0
            ada8    ONLINE       0     0     0
            ada9    ONLINE       0     0     0
            ada10   ONLINE       0     0     0
            ada11   ONLINE       0     0     0
            ada12   ONLINE       0     0     0
            ada13   ONLINE       0     0     0
            ada14   ONLINE       0     0     0
        spares
          ada15     AVAIL

errors: No known data errors

I removed one disk from first VDEV (actually is VirtualBox virtual machine):

Code:
# zpool status
  pool: mail
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: none requested
config:

        NAME                      STATE     READ WRITE CKSUM
        mail                      DEGRADED     0     0     0
          raidz2-0                DEGRADED     0     0     0
            ada1                  ONLINE       0     0     0
            ada2                  ONLINE       0     0     0
            ada3                  ONLINE       0     0     0
            18226958454424104776  UNAVAIL      0     0     0  was /dev/ada4
            ada4                  ONLINE       0     0     0
            ada5                  ONLINE       0     0     0
            ada6                  ONLINE       0     0     0
          raidz2-1                ONLINE       0     0     0
            ada7                  ONLINE       0     0     0
            ada8                  ONLINE       0     0     0
            ada9                  ONLINE       0     0     0
            ada10                 ONLINE       0     0     0
            ada11                 ONLINE       0     0     0
            ada12                 ONLINE       0     0     0
            ada13                 ONLINE       0     0     0
        spares
          ada14                   AVAIL

At this point, I replace missing disk with ada14:

Code:
# zpool replace mail 18226958454424104776 ada14
# zpool status
  pool: mail
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: resilvered 23K in 0h0m with 0 errors on Tue Feb 26 21:37:38 2019
config:

        NAME                        STATE     READ WRITE CKSUM
        mail                        DEGRADED     0     0     0
          raidz2-0                  DEGRADED     0     0     0
            ada1                    ONLINE       0     0     0
            ada2                    ONLINE       0     0     0
            ada3                    ONLINE       0     0     0
            spare-3                 UNAVAIL      0     0     0
              18226958454424104776  UNAVAIL      0     0     0  was /dev/ada4
              ada14                 ONLINE       0     0     0
            ada4                    ONLINE       0     0     0
            ada5                    ONLINE       0     0     0
            ada6                    ONLINE       0     0     0
          raidz2-1                  ONLINE       0     0     0
            ada7                    ONLINE       0     0     0
            ada8                    ONLINE       0     0     0
            ada9                    ONLINE       0     0     0
            ada10                   ONLINE       0     0     0
            ada11                   ONLINE       0     0     0
            ada12                   ONLINE       0     0     0
            ada13                   ONLINE       0     0     0
        spares
          8678491041146001141       INUSE     was /dev/ada14

Honestly, spare-3 with the two disks "inside" is a bit confusing.

After this, I remove the unavailable entry:

Code:
# zpool detach mail 18226958454424104776
# zpool status
  pool: mail
 state: ONLINE
  scan: resilvered 23K in 0h0m with 0 errors on Tue Feb 26 21:37:38 2019
config:

        NAME        STATE     READ WRITE CKSUM
        mail        ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            ada1    ONLINE       0     0     0
            ada2    ONLINE       0     0     0
            ada3    ONLINE       0     0     0
            ada14   ONLINE       0     0     0
            ada4    ONLINE       0     0     0
            ada5    ONLINE       0     0     0
            ada6    ONLINE       0     0     0
          raidz2-1  ONLINE       0     0     0
            ada7    ONLINE       0     0     0
            ada8    ONLINE       0     0     0
            ada9    ONLINE       0     0     0
            ada10   ONLINE       0     0     0
            ada11   ONLINE       0     0     0
            ada12   ONLINE       0     0     0
            ada13   ONLINE       0     0     0

errors: No known data errors

Last point should be: physically replace ada4 with a new disk, and assign spare role, right?

A couple of question:

  • Is there a way to manually set a disk in faulted state with ZFS?
  • Actually it is not really crucial to assign spare role to a disk, since resilvering process is not automatic. It is more like "ehy, remember that you have THAT disk available as spare", or is there any other reason beyond this?

Thank you!
 
Yes, I would replace the failed disk physically and use the new one as a new spare.

Don't think you can mark a disk as faulty, but you can offline a disk which stops it being used. You could then physically replace that disk and do a resilver, or replace it with a hot spare.

All the spare feature in ZFS really does is mark a disk as a spare. On Solaris there is a component of the OS called the fault management daemon, which handles faults throughout Solaris, including performing a "zpool replace" if a pool disk fails and a spare has been marked. This daemon is separate from ZFS, and heavily tied to Solaris so is not present in OpenZFS. As such, the spares didn't really serve any purpose on FreeBSD other than let you know they're there.

In recent versions of FreeBSD there is zfsd though (disabled by default as far as I'm aware). This should handle automatically pulling in a spare is a disk goes faulted. I haven't used this so have no idea whether it works but as it's in RELEASE you'd assume it does...
 
Back
Top