getting rid of a RAID controller

patpro · Jun 12, 2013

Hello,

I have a server, far away in a datacenter, running FreeBSD 8.1. The server has a PCIe 3Ware 9650SE-4LPML RAID controller. The configuration is this one:

RAID 1, 2 disks: system, binaries, and some user data
JBOD/Passthrough, 1 disk, user data
JBOD/Passthrough, 1 disk, user data
/, /tmp, and /var are UFS
/usr/local, /usr/ports, /home are ZFS.

I want to get rid of the 3Ware and plug my HDDs on the motherboard, so I'm facing few challenges:

how can I make sure HDDs will be recognized natively when they are not "served" by the RAID controller?
how can I make sure I can boot from one of my mirrored HDD?
how can I recreate a RAID 1 array from the two mirrored HDD?

I know I'll have to edit /etc/fstab, and export/import ZFS.
I know the 3ware controller reserves the first MB on each HDD to write info/config about arrays.

I can spend only limited time in the datacenter, and have to plan a long time ahead for each visit, so the more I can do remotely, the better it is. I'm thinking about this scenario:

export ZFS pools
break mirror: plug HDD "A" to motherboard, leave HDD "B" on RAID controller, boot from "B"
format and partition "A" to prepare for a software RAID (gpart?), rsync data from "B" to "A", tweak fstab on "A"
remove RAID controller, plug HDDs "B", "C", "D" to motherboard, boot from "A"
format "B", add "B" to software RAID, reconstruct mirror
reboot from A+B RAID 1, import ZFS pools

Any piece of advice is welcome, anybody who has some experience in that field is welcome. I've used gpart only once, copy-pasting from a tutorial, and I don't know how to deal with the creation of a "one HDD RAID 1 array" (step 3).

I don't know if my HDDs will be recognized without formatting ("C" and "D"). If it's absolutely sure they are recognized without a format step, I guess I can just export ZFS, change fstab, plug all HDDs on motherboard, reboot from one half of the mirror and do everything else remotely.

Thanks!

wblock@ · Jun 12, 2013

One-disk RAID 1 array creation is shown in the Handbook RAID 1 section: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/GEOM-mirror.html, the "Creating a Mirror with an Existing Drive" part. Actually, it's not that big a deal: when creating the array, just give the gmirror(8) command one disk instead of two.

One problem with gmirror(8) is that the first disk can die, and the machine will not be able to boot. Unless you have the second mirror disk set up as the next boot device in the BIOS, anyway. graid(8) is another option for the boot mirror, if the motherboard supports it.

Consider just adding new drives. That makes big changes less stressful: set up the new drives, sync the data, only remove the old drives once everything is tested.

patpro · Jun 12, 2013

Thanks for the pointers, I was just reading the gmirror part, actually.

wblock@ said:
Consider just adding new drives. That makes big changes less stressful: set up the new drives, sync the data, only remove the old drives once everything is tested.

I wish I could. That's a 1U server, holding only four slots for HDDs: it's full, I can't add any other HDD.

phoenix · Jun 12, 2013

If you created "Single Disk" arrays for the last two drives, you're screwed.

That puts metadata on the disk that can only be read by 3Ware controllers, and makes the drives unusable by anything not 3Ware.

The best solution would be to attach a drive via USB/eSATA and copy the data off. Then move that drive inside the server. Then repeat for the other data drive.

Then break the RAID 1 on the controller, and move one disk to the motherboard controller, and do the gmirror re-jigging.

Otherwise, nuke the system, move the drives to the motherboard controller, and restore from backups.

patpro · Jun 13, 2013

phoenix said:
If you created "Single Disk" arrays for the last two drives, you're screwed. That puts metadata on the disk that can only be read by 3Ware controllers, and makes the drives unusable by anything not 3Ware.

The best solution would be to attach a drive via USB/eSATA and copy the data off. Then move that drive inside the server. Then repeat for the other data drive.

Well, that's no big deal, sum of data for both "solo" HDDs is smaller than available storage left on the RAID pool. So the fast path is to copy data from those drives to the RAID pool, and restore after wiring everything on motherboard. But I'm glad to know I have to copy those data before trying to plug drives on the motherboard. Rewiring will not be that easy, and the fewer steps I have to do, the better it is.

phoenix said:
Otherwise, nuke the system, move the drives to the motherboard controller, and restore from backups.

Not a very good option in my case, the backup is far away, hosted on my DSL, I'm not ready to upload ~600 GB in a 100~120 KB/s pipe

And I would lose my snapshots, because I don't back up the whole ZFS, I just do rsync.

throAU · Jun 18, 2013

patpro said:
Not a very good option in my case, the backup is far away, hosted on my DSL, I'm not ready to upload ~600 GB in a 100~120 KB/s pipe And I would lose my snapshots, because I don't back up the whole ZFS, I just do rsync.

Would you prefer the option of "lose your data"?

patpro · Jun 21, 2013

throAU said:
Would you prefer the option of "lose your data"?

Well, frankly, most of these data can be lost. It's about 70 GB system/users/production and 500+ GB backups. Backups could be flushed, even if I would rather keep them available. I have another problem: the server does not behave very nicely those days when I'm doing only medium IO.
Example:

I'm starting a remote backup through SCP of an HDD -> after about 2-3 GB downloaded, the connection times out, it won't ping any longer, and 10-35 minutes later it comes back online with a huge time shift (10-35 minutes) as if time froze during server blackout. The server does not panic, nor reboot, nothing in log files.
I'm starting a local backup through cp -r or rsync from one HDD to another -> same behavior.
If I put a bandwidth limit (say 300 or 500 KBps) on my remote backup process, then the server won't blackout

I suspect the RAID card, hence my question about getting rid of it. I've discovered the HDD-to-HDD symptoms after my posts here, it makes me fear about the step when I'm supposed to break the mirror, and copy data from old half on RAID controller to new half on motherboard.

About the possibility of booting from a drive without reformat, here is the LSI support answer:

There is a 1024k sector offset at the beginning of the drive where most on-board motherboard controllers cannot boot from it if moved off the raid controller. This does vary from motherboard-to-motherboard though. I have seen some boot the RAID 1 member just fine whereas in most cases it cannot.

patpro · Aug 14, 2013

Finally, here what I've done:
- I've bought a brand new server with IPMI, and an SSD.
- I've installed FreeBSD 9.1 on SSD (zfs root), configured IPMI.
- I've sent the server to the data center with few instructions (remove HDDs from current server, put them in new one, boot).

Every HDD was readable on the new server (plugged directly on motherboard, no RAID controller), so I was able to move data from one HDD to another, reinstall a fresh full ZFS system, and get my data back to where it belongs.

The new server runs FreeBSD 9.1 from a zmirror of HDDs. The SSD is not mounted and I keep it inside the server as a disaster-recovery system I could boot from, just in case. IPMI allows me to boot in BIOS and manage boot order as if I'm sitting in front of the computer. Very handy.

Note: both solo HDD (JBOD/Passthrough) were fully operational out of the box. I've erased them to get rid of 3Ware metadata, but that was not compulsory.