ZFS Drive Bay Caught On Fire

I have/had a server with 16 hot-swap drive bays. I had 10 2TB drives installed in a RAIDZ2 config (12TB). The backplane on the drive bays literally caught on fire and burned a American nickel sized hole in the pc board. The drives appear to be ok but that drive bay is out of commission.

I have a second machine that has an 8 drive bay - already populated with 3TB drives making a 16 TB RAIDZ2. I do not have ready access to another 10+ drive bay system. Can I dd copy these 2TB ZFS drives into files on the the 16TB system - use a loopback driver to mount them and copy the files off? Or do I need to go get another 10+ drive bay?

Thanks,
-- Chris

IMG_6464.JPG
 
I have/had a server with 16 hot-swap drive bays. I had 10 2TB drives installed in a RAIDZ2 config (12TB). The backplane on the drive bays literally caught on fire and burned a American nickel sized hole in the pc board. The drives appear to be ok but that drive bay is out of commission.

I have a second machine that has an 8 drive bay - already populated with 3TB drives making a 16 TB RAIDZ2. I do not have ready access to another 10+ drive bay system. Can I dd copy these 2TB ZFS drives into files on the the 16TB system - use a loopback driver to mount them and copy the files off? Or do I need to go get another 10+ drive bay?

Thanks,
-- Chris

IMG_6464.JPG

You should be able to copy the drive or the partition (whichever you used to create the pool) into raw image files and import the pool from those with the -d option pointing to the directory where the files are.

EDIT: You will need to use mdconfig to create a loop mount of the image file and then import from resulting mdX devices, not with the -d option I suggested above. Also see ralphbsz's caveats below re: nested zpools. (zpool-from-file-on-zpool).
 
Can you share some details?
What brand is the server? If self-built who makes the backplane. I don't know if I have ever seen anything so nasty.
Did the event power off the box or did you have to yank the cord?
 
Can you share some details?
What brand is the server? If self-built who makes the backplane. I don't know if I have ever seen anything so nasty.
Did the event power off the box or did you have to yank the cord?

I've been around servers for 30 years and have never seen this. I can't tell what did the damage either. There are no exposed traces or power wires. There are some scattered surface mount capacitors and I wonder if it's possible one of those went pyrotechnic. There are some larger electrolytic caps on the back side of the board but it looks like the flames were from inside. The drive that was in that bay has some soot on it but no obvious damage so the flame/heat was confined to inner surface in the picture.

It's a SuperMicro case and backplane.
 
First, contact SuperMicro. Not because you want to beat them up, not because you want to harass them, not even because you want a refund. But because you want to help them fix this problem. I'm sure their engineering department is very interested in this. They are located in the San Francisco bay area, so that might be a sensible time zone for you. I've dealt with their sales and engineering folks a few times, and they are really friendly and competent.

My theory would be a short in the power traces of the backplane, inside the PC board. Given that PC power supplies can easily make 1/2 or 3/4 kiloWatt (and run at currents that are awfully close to what an arc-welder uses), this is not surprising.

But I've never seen this kind of backplane damage caused by the backplane itself. I've seen a 60-disk enclosure (name withheld, don't want to hurt anyone's feelings) where one power supply caught fire. Fortunately, no disk drives were hurt (the power crowbar circuit worked), and the data was fine afterwards. According to the local service person, the smoke and fire damage was spectacular.

Getting back to your original question: Copying the raw drives into files, making loop devices, and then trying to import those into ZFS *might* work. It might also fail rather badly. You might want to ask on the kernel developer mailing list, perhaps there is a ZFS or VFS developer there who can answer that question. Here is some explanation why it might fail: If you do that, you end up going through ZFS twice for a single IO request: User space IO request goes into ZFS (through the VFS layer at the kernel's top interface), gets decoded into disk sectors and blocks, and goes to the low-level block device drivers. Since that is a loop device driver, it goes back into VFS layer as a file system request, and then back into ZFS, and finally to a real block device. But by this point, if there are any data structures that are *per IO request* in either the VFS layer or in ZFS itself, they have been used twice. The most popular example for such data structures are locks (which protect data structure): Doing something like this has the potential for deadlocking (I've learned this the hard way).

If your suggestion doesn't work, my first suggestion would be a hardware hack. You still have functioning HBAs or SATA controllers; maybe you can quickly borrow enough adapter cables (octopus cables) and power splitters to run all disks. Disks work fine sitting on a work bench or office table. The second suggestion: To prevent having to make two trips through the *same* file system, put a network in between: Make the loopback devices on the second node, make them into iSCSI targets (using the iSCSI server whose name I can't remember), then on the original node set up the iSCSI initiator (a.k.a. client), and mount them this way. Will run slower (extra trip across the network), but higher probability of success.
 
If you import the pools readonly (zpool import -o readonly=on [...]) from the image-backed md devices, I'd wager you'll have a better chance of not deadlocking, as you won't have transactions trying to settle at two levels, which seems, well, wobbly. You can enforce it at the md level, as well, which might help by letting the md device know that it will never be modified.

Good luck!
 
Back
Top