ZFS and MSAs

T

tanked

Hello, at my company we currently have a Windows 2008 box attached to an MSA with two 2TB volumes; the file system and data has become corrupt so we are currently moving the data elsewhere but I was wondering about replacing the OS on the box with the latest 64-bit FreeBSD release. The box has 12GB of memory in but could you tell me what are the implications of having ZFS volumes on the MSA? I know ZFS prefers to have direct access to the physical drives and apart from setting copies=2 or something like that, I don't see how ZFS would replace corrupt files etc.

Any thoughts or experiences would be appreciated.
 
Apparently it is an external disk array from HP. And yeah, when these things do not export the individual drives, then the self-healing and redundancy capabilities of ZFS cannot be fully trusted, because ZFS then does not have control over where specific things are stored physically.

E.g. you could set copies=2, but who then guarantees that not both copies are lost when one particular component fails or misbehaves in the external enclosure? These copies are spread around a vdev, assuming that they are then also spread around on the physical storage, but whether this is true or not really depends on how the external disk array treats the data.

The basic rule is to never put ZFS on top of another volume management or RAID implementation, but to let ZFS handle this. If you have the chance to export all those drives individually, so that FreeBSD sees them as separate devices, then you should do that and create your pools with those devices.
 
As long as the storage array handles sync writes properly which it should do with enterprise level kit I can't really see a problem. I'd rather run ZFS on it than NTFS. We've had the same issue with a 4TB FC array that started getting files that couldn't be opened or deleted. At least ZFS would of found the errors on a scrub rather than years later when the files are needed.

Yes the advice is to give ZFS the raw disks rather than creating a HW array and putting ZFS on top but you don't really have that choice here. You'll have to make sure the hardware you are using is supported by FreeBSD of course. We had huge trouble getting an FC card to connect to the above mentioned disk array a few years back and ended up using Linux.

If you can manage with 2TB I would suggest to mirror the LUNs so at least if ZFS finds an error it should be able to fix it. If not, you can 'stripe' the LUNs givings ~4TB but obviously while you'll still be able to scrub the file system, ZFS will only tell you that data is corrupt (and hopefully give the file path). The only option would be to delete the corrupt files and recover from backup (You do runs backups right... :/). That's better than not knowing though.

Yes, you can set copies >1 on a per filesystem basis to give some redundancy for corrupt data. I'm not a big fan of this setting though as people seem to think it's better than it is. Sure it'll try and put the copies on separate vdevs, but it won't guarantee it. And even if it does, if you lose a vdev the pool's faulted anyway so that other copy is useless unless you're a ZFS dev and your life depends on getting it back (and you were stupid enough not to back it up).

As I see it copies is only really of use when you have a single disk (i.e a laptop or a LUN from some centralised storage box like yours). It allows the system to automatically sort out the odd checksum failure but won't help you if have a disk fail that can't be handled by pool-level redundancy. If you have a small amount of really important data, it will be worth setting this on the file systems containing that data. If it's all important, there's no point setting copies=2 on everything, you may as well have just mirrored the LUNs in the first place...

In your situation you may also get some extra safety and performance by putting a power-safe SSD as ZIL in the head unit and maybe an SSD as read cache (L2ARC).

And make sure you have backups.
 
Sorry for the late reply, yes its a HP 2000 MSA (Modular Smart Array) and thanks for the advice.
 
You must log in or register to reply here.
Back
Top