Solved Hot swap drive behind hardware RAID 0 (JBOD)

Hi. Could someone please teach me how to hot swap a drive inside a hot-swap enclosure. Specifically, Dell PowerEdge R720, H710 card hardware raid but all drives running JBOD style, so each is its own RAID-0. Note that this isn't a case of replacing a failed drive, rather I wrote data I needed and now want to move that physical drive elsewhere, insert a fresh one to work with, etc.

Is there some `mfiutil` incantation that would let me take it offline somehow so that I can eject it, replace, bring the new online etc. Appears that `camcontrol` would let me do such things in a straightforward manner but sadly I am behind entirely unnecessary hardware RAID.

Thank you

*EDIT* running 12.2 release in case it matters
 
Could someone please teach me how to hot swap a drive inside a hot-swap enclosure.
ZFS? Just yank out the drive, put a new one in and use zfs replace to replace it. It's hot-swap, the whole point of it is that you can just yank out the drive without having to power-down or do anything special. You typically remove drives that are broken, so they're usually not active anyway. If it's a normal, active drive, then you can use zfs offline to 'offline' the disk before removing it.

Specifically, Dell PowerEdge R720, H710 card hardware raid but all drives running JBOD style, so each is its own RAID-0.
No, JBOD and a single disk RAID0 are two different things. It's very important to understand this distinction. A RAID-0 has meta-data on the disk that tells the controller it's part of a RAID-0, even if it's just a single disk. A JBOD doesn't have this meta-data. JBOD and single disk RAID0 are not interchangeable.
 
Since this thread is marked “UFS”, I assume there are UFS partitions on the drive, not ZFS.

Basically, make sure that no file systems are mounted from that drive (umount all of them if necessary). If there’s an active swap partition on that drive, use swapoff(8). If this was on a “normal” connection (SATA / SCSI / SAS), you could use camcontrol standby to power-down the drive, but I think this might not work if the drive is hidden behind a hardware RAID. Still might be worth a try, though.

Afterwards you can just remove the drive. However – as SirDice has mentioned above – if this disk was really configured for RAID (no matter what RAID level), note that it contains meta data that probably prevents the drive from being used normally. That is, you cannot simply connect it to a different controller, or put it in an external USB case. It will only be recognized by the same kind of RAID controller.

By the way, what kind of RAID controller is this exactly? If it’s aac(4) (Dell PERC), there are management tools that can be used with FreeBSD (via Linux compatibility). I’m not familiar with these, but maybe they provide a way to power-down (and thus spin-down) a drive prior to removing it. Also, some management software have a feature to “export” a drive, removing the meta data, so it can be used elsewhere. I don’t know if Dell supports this, though.
 
Since this thread is marked “UFS”, I assume there are UFS partitions on the drive, not ZFS.
correct. UFS only.

By the way, what kind of RAID controller is this exactly? If it’s aac(4) (Dell PERC), there are management tools that can be used with FreeBSD (via Linux compatibility). I’m not familiar with these, but maybe they provide a way to power-down (and thus spin-down) a drive prior to removing it. Also, some management software have a feature to “export” a drive, removing the meta data, so it can be used elsewhere. I don’t know if Dell supports this, though.
Like I mentioned in the original it is H710. That would be PERC for which my system used mfi(4) driver, that's why I mentioned mfiutil(8)
 
No, JBOD and a single disk RAID0 are two different things. It's very important to understand this distinction. A RAID-0 has meta-data on the disk that tells the controller it's part of a RAID-0, even if it's just a single disk. A JBOD doesn't have this meta-data. JBOD and single disk RAID0 are not interchangeable.
Darn. That's such a pain :( Strange mfiutil offers mfiutil create jbod which IIUC is exactly every disk behind RAID-0. But maybe if you create such concoction via on-board controller software at boot time, which was my case, it writes meta on disk? Anyway. Thank you. Back to headscratching ... I don't have 10G network here to move the files. Not sure what to do

I'll be reflashing that stupid PERC card to IT mode soon enough
 
That's such a pain :( Strange mfiutil offers mfiutil create jbod which IIUC is exactly every disk behind RAID-0.
No, this just signals to the controller the disk should be used as-is (the controller keeps track of this by storing settings in an EEPROM).
 
Dear SirDice and olli@ and everyone else.

I just thought of a different approach to achieve the same goal. I'll be reflashing these controllers to do direct passthrough IT mode (or whatever its called) anyway. Is there a way to read data off these drives after that controller is gone. As long as I can get access to that data its all good. I'm neither attached to that particular controller, nor to any RAID setup. All I want is to get that data. So e.g. can I do the following. Physically remove those drives, reflash the controller, insert those drives and manage to get the data out e.g. mount read only and move to another drive or smth. That'll be perfect!

So yeah. End goal the same, approach different and I no longer care about being able to hot swap from behind RAID. I know I'll be able once I can talk to them over scsi directly via e.g. camcontrol.
 
Is there a way to read data off these drives after that controller is gone. As long as I can get access to that data its all good.
They're JBODs, so any controller would be able to access them.
 
Well, I just tried an experiment. Backed up data on drive 7, unmounted then pulled it. I then plugged it back in. Console report attached. But the gist is that the drive is seen but marked as unconfigured good. That volume 7 however no longer exists it seems, so there isn't anything for me to mount. I've tried to present the drive as syspd to the host with mfiutil syspd 7 but got back:
Code:
mfiutil: Command failed: Wrong firmware or drive state
mfiutil: Failed to set drive 7 to JBOD: Input/output error

But you're saying once I get that controller into direct mode or switch to some LSI HBA without RAID I should be able to see data on that drive probably via /dev/da? since? Those are SAS drives.
 

Attachments

  • mfiutil_8_.png
    mfiutil_8_.png
    356.5 KB · Views: 171
But you're saying once I get that controller into direct mode or switch to some LSI HBA without RAID I should be able to see data on that drive probably via /dev/da?
No, that depends on the controller. It's probably still going to be detected as an mfi(4) controller, only as one without RAID capabilities. So the drives are still going to be /dev/mfid*. The mfi(4) driver supports multiple chipsets, with and without RAID. Depending on the chipset that's used you might want to switch to mrsas(4) though. Then your disks will show up as /dev/da*. But you won't be able to use mfiutil(8) any more. You can still use sysutils/megacli though. Even an HBA has features you can turn on or off, like a BBU or cache settings.
 
Happy to report a small success - data are safe. Here's the experiment I ran:
- disconnect PERC H710,
- turn off "embedded RAID" in BIOS or System Setup or whatever Dell calls it,
- plug old LSI HBA that managed my SuperMicro into PCIe slot,
- run two SAS SFF-8087 cables to the backplane,
- boot from live FreeBSD or mfsbsd,
- LSI controller shows up and manages to insert its ROM code or whatever,
- both LSI and later dmesg show the drives,
- drives show up as expected da0, da1, ...
- mount /dev/da7 /mnt; ls /mnt ... hooray

So. Guess I'm reflashing that H710 mini card to IT mode. Discovered this step by step with screenshots for anyone who cares - even installs FreeBSD in the end :)
 
Back
Top