Solved ZFS confused by renamed devices

L

listentoreason

Nutshell: After replacing a dead disk with a new spare, the devices renumbered causing confusion, first in the human and then in FreeBSD. I now have my pool mounted in a degraded state, wondering where /dev/da10.eli is, and unwilling to accept /dev/da9.eli as the same device.

Edit: Solution: I Used zfs unmount POOLNAME, zpool export POOLNAME, then zpool import POOLNAME. This causes ZFS to re-inspect all devices in /dev/ and deduce which ones are part of the pool.

Tried, Failed: zpool replace (it knows that da9 is supposed to be part of the pool already), zpool detach (it knows it's not in the pool at the moment), zpool online (generally confused)

Next? I think I should zpool import, per a prior post by @phoenix?

Details: I recently had to replace a failed disk in a RAID-Z3 (thread). I don't have hot-swap hardware so I silvered in a cold spare, and left the dead disk in place. I powered off the system before going on vacation. When I returned and booted it hung (apparently infinitely) while repeatedly throwing "Error 5" messages about the dead disk. So I powered off, removed the drive, added a new cold spare, and rebooted. My geli attach-and-mount script then choked when it came time to attach the new device, /dev/da10 ("geli: Cannot read metadata from /dev/da10: Invalid argument."). Much confusion followed (more details below) before I realized that removing a drive and adding its replacement had caused renumbering of the devices. What was now called "da10" used to be "da11", and was already in the pool. The new replacement device was now "da9". I was able to successfully attach it with geli. But the pool is doggedly looking for da10.eli, when what I have is da9.eli.

As mentioned above, I think I need to do an import and let ZFS work out the new nomenclature itself; I'd appreciate confirmation of that! I'd expect I'd need to zfs unmount first?

In that same thread, wblock@ warns that data disks should not be labeled. Why is that? It would seem like a mechanism to avoid this problem. Is there a way to consistently reference devices in a pool such that the system's renumbering of the underlying devices doesn't cause this problem? Or is there a way to control how the system assigns device numbers, so it does not "recycle" and/or rearrange previously used device names?

Somewhat related: Is there any way to get FreeBSD to boot past a dead disk? The old dead device required manual intervention to "do" anything (the zpool is just data, no system files), so I was surprised it locked up the boot.

For completeness here are the dead-end alleys I stumbled through:
Code: 
citadel 332 [ROOT] zpool replace abyss /dev/da10.eli /dev/da9.eli
invalid vdev specification
use '-f' to override the following errors:
/dev/da9.eli is part of active pool 'abyss'

citadel 337 [ROOT] zpool detach abyss /dev/da9.eli
cannot detach /dev/da9.eli: no such device in pool
citadel 338 [ROOT] zpool detach abyss /dev/da10.eli
cannot detach /dev/da10.eli: only applicable to mirror and replacing vdevs

citadel 341 [ROOT] zpool online abyss /dev/da9.eli
cannot online /dev/da9.eli: no such device in pool
citadel 342 [ROOT] zpool online abyss /dev/da10.eli
warning: device '/dev/da10.eli' onlined, but remains in faulted state
use 'zpool replace' to replace devices that are no longer present
 
Oko said:
Start by reading this.
http://wiki.nas4free.org/doku.php?id=faq:0149
Then lets talk more.
Click to expand...

Sorry, I wasn't clear on what was transpiring. The order of events is:

  1. A disk in the pool (da2) dies hard, is offlined by system from pool
  2. I provision a ready-to-go cold spare (da10) with geli, and zpool replace it into the pool - Pool resilvers in a few hours and is now fully operational
  3. When I power cycle the system later, the dead disk prevents a boot
  4. I remove the dead disk (da2), adding in a new cold-spare drive.
  5. System boots fine, but the system now has renamed some/all devices:
    1. This seems to occur because the "roster" of drives in the system has changed (one removed, one new one added).
    2. Several of the renamed devices simply "swap" names with other pool members, which seems to transpire without complaint from ZFS.
    3. However, what was called /dev/da10 is now called /dev/da9, and there was no /dev/da9 previously (the previous da9 device has a gpart layer on top of it resulting in a different device name, gpt/GELI-da9.eli). ZFS is aware that:
      1. /dev/da10.eli is missing
      2. /dev/da9.eli is part of the pool
      3. ... but cannot realize that they are the same device, just with different names.
I appear to have resolved the issue; I used zpool export POOLNAME followed immediately by zpool import POOLNAME (all devices were "rooted" in /dev/, so I did not use the -d option). The pool is now online with all members and no errors (I'll scrub tonight).

So my immediate problem is addressed, but I'd still be interested in advice to mitigate the problem in the future:
  1. Can I control how FreeBSD names/numbers devices, so it never renames drives after they're registered for the first time?
  2. wblock@ said that data disks should not be given labels. Naively I'd think that labels would be an elegant way to consistently track a device; I'd be interested in knowing why labels should be avoided in this situation.

Thanks!
 
Use labels and you'll never run into this issue again. :)

My current setup is to use GPT to partition the disk with a single partition that starts at 1 MB (-a 1M), and runs for "disk size - 1MB", just in case. And then I use GPT labels to name the partition (-l labelname), usually based on where the disk is physically located.

And then use the label when adding disks to the pool ( zpool add raidz gpt/label1 gpt/label2 gpt/label3 ...).

That way, it doesn't matter what the physical device node is, ZFS will read the GPT labels from /dev/gpt/ and use those.
 
Last edited:
Wow, sorry, it looks like this thread fell off my radar! That's exactly what I needed to know. I've been moving (very) slowly on rotating out unlabeled devices with labeled ones. Slowly, because if I mess up an encrypted zpool I'll lose all the data. I also realized that without labels it would be hard (nearly impossible?) to correlate the .eli files from /var/backups with a reconstituted pool. That is, if I needed to move the drives over to a new machine, without labels I would have a hard time knowing what device gpt_GELI-da3.eli went with. However, label_geliDisk01.eli should be trivial to match up.

Thanks again, sorry for not replying earlier!
 
You must log in or register to reply here.
Back
Top