ab2k,
ANOKNUSA,
Phishfry,
gofer_touch,
Oko and
wblock@, I appreciate your fabulous replies so much; thank you.
Best turn here is to get BBU for it. Else, if you have uninterruptable power supply, it works good and connected via USB interface to this computer to signal it to make proper shutdown when you get power loss - then you can turn on Write-Through cache through CLI (even if you don't have BBU). It will have even better performance.
Thanks for this. Adding a BBU might be an option, though I'd feel like investing the effort only if non-BBU, write-through performance doesn't turn out to be good enough. I haven't collected any performance information yet.
There's no backup power where this computer's installed, which I know is very bad, so I'm inclined to lean towards safety at the expense of performance when there's a choice.
About creating RAID0 disks from every disk you have to use it with ZFS - well you can experiment here. But I think you will get corrupted ZFS after some reboots of the server, just because RAID controller will write some technical information to disks. ZFS needs dedicated disks and nothing else. But you are up to try
I know what you mean. I appreciate that with some RAID controllers and configurations thereof, ZFS checksums and scrubs may duplicate things happening in the RAID controller, and that there are too many potential combinations of hardware and settings for anybody to know for sure what each outcome will be long-term.
Besides not storing checksums, 'seems that UFS has the profound benefit of being a
known, having already been put on top of every imaginable RAID controller for twenty years or more. Yet it sure is hard to give up ZFS replication.
While you can often interact with RAID controllers through the operating system, they still do their own thing independent of the operating system. They typically abstract the storage array so that the operating system sees a single block device; hence, ZFS can only create a single stripe vdev out of that whole stack of disks. There's no data parity or disk redundancy.
Thank you. Yeah, I know what you mean about ZFS not being able to use its own RAID implementation when it's given a single RAID virtual disk to deal with. I'd definitely rather use ZFS RAID than the black-box MegaRAID controller's RAID, if I had a choice.
Some RAID controllers have JBOD-mode that allows you to format all the disks independently. ZFS can create multi-device vdevs in such an instance; however, the RAID controller will still try to manage that storage on its own at a level below ZFS. The controller likely also has a write cache, and possibly basic error detection and correction.
Making a single-disk RAID 0 virtual disk for each physical hard disk's as close to JBOD mode as this controller gets. According to the MegaRAID firmware setup utility (<Ctrl><H> during POST), its cache is configured in write-through mode.
I did see some MegaRAID configuration options regarding its error detection; it had patrol options (its own scrub mechanism, I guess). Yet I can appreciate that even if these things were disabled, the MegaRAID's innards would remain black-box; there's no way to know for sure what it's doing (not that I have the skills to dive in to ZFS internals--it's just that I know that various other people do, and have).
I suppose the only path towards trusting a controller like this has got to be through hard-earned experience. Though our RAID controller must be one of the most common models, it could be that an insufficient number of people have used FreeBSD with ZFS on it, to have built up a good enough level of trust.
ZFS might not be aware of the write cache and error correction, and might not know what the controller is doing below the ZFS pool layer, so the operations performed by ZFS and the controller can interfere with each other and impact both performance and data integrity.
Thanks; I know what you mean. It's data integrity that I'm most concerned about here.
If you're using a RAID controller that doesn't even offer JBOD mode, you're pretty much stuck with UFS. Even with JBOD, you probably shouldn't use ZFS with a dedicated RAID controller. There's just no way to really know how they might interact. Your data is still better off on a decent hardware RAID setup with UFS than a single UFS disk.
Thank you!
Very well said. I would also like to chime in on the BBU. They charge so much money for the replacement battery I really wonder its value compared to a system wide approach of a APC Battery Backup. The life of a BBU is around 18-24 months so it is a considerable expense which should be considered. Maybe I got duds. Replacement was $180 from reputable seller.
I also experienced a shorter than expected BBU life span once; 'seems that the replacements aren't always superb.
The BBU option for the
9268-8i 9267-8i is actually a "Super-Cap," which (I think) uses capacitors to keep the write cache charged during outages, instead of using the battery pack typical of BBUs. A friend of mine mentioned a few months ago that these capacitor-based BBUs last a lot longer, though I'm not sure what his opinion was based on. At any rate, I think I'll avoid adding a BBU unless the write-through performance turns out too doggy. These are pretty fast SAS disks, and most I/O will be choked by GigE networking before it even reaches the computer, but we'll have to try it to find out if it's good enough.
One thing I found years ago that made me weary of ZFS is the need to have a very similar version of ZFS running on your recovery machine. This has changed over the years though. I still feel there is a certain bulk to ZFS. I know this comment might be unpopular.
If you consider 16GB memory OK for an fileserving appliance then you probably disagree.
I know what you mean. In our case, I think we'll have the ability to match the FreeBSD version and ZFS feature flags on the recovery machine exactly, though expecting to keep them synchronized in the long haul may be naive.
I guess it also depends on what you intend to use the storage pool for as well. If you just need a bunch of bulk storage to offload something temporarily, I would think that using ZFS should be ok. For long term or critical storage, very likely a bad idea. My vote would be for the UFS setup.
Thanks. Ironically, if we could use ZFS on this storage hardware, the intended use would indeed be long term, critical storage. But if we're not able to use ZFS replication, our intended use actually changes; the storage would then be used only as the computer's boot device and root filesystem, and to store backups on.
As a volume manager I prefer hardware RAID cards over any software RAID (Linux, BSD, ZFS you name it). High end LSI cards like the one you have are easy to monitor, alerting software is great and replacing failed HDD is a breeze.
Yeah, I was actually impressed by everything
mfiutil(8) seemed to allow.
Using HAMMER with your RAID card you will get the best of both worlds, the best file system and the best volume manager.
Thanks; I sure appreciate your thoughts here, and I think HAMMER's really interesting.
Yet the scenario we're dealing with demands something that's more of a "known." Believe me, even using FreeBSD and ZFS for this in the first place, is stretching the limits of how far off the beaten path we're willing to venture, in the context we're working in. To use HAMMER, I'd be starting out with ~zero DragonFly BSD and HAMMER knowledge, and it'd probably take me a year to learn enough to use these things confidently--I've already been studying and experimenting with ZFS for two years, and I feel that my knowledge is barely adequate even there. I enjoyed your post though; thank you all the same.
wblock@ said:
ZFS is cross-platform and has better than a decade of deployment to a large group of diverse users. Hammer only runs on Dragonfly and so is limited to a small userbase. In turn, that means a lot more opportunity for bugs that would have been discovered with more users.
Thanks Warren. Yeah, I think that FreeBSD with ZFS is the least beaten-path of all the possible paths, for my context.
I appreciate the ZFS vs HAMMER discussion, yet I don't have the brain power to explore HAMMER quickly enough to even consider it for this.
I'll now
add a new dimension to my question change my question's context, to include recent advice from
Allan Jude on something similar.
If we stick with UFS on the MegaRAID, it doesn't mean that we won't use ZFS at all--it means that we'll use ZFS only on LUs visible over the computer's QLogic QLE2462 FC HBA with
isp(4). We have access to various nice bunches of disks here and there on Hitachi Modular and Enterprise storage arrays, so we'll make LUs backed by RAID groups on these and present them to the QLE2462. The plan's that they'll pass through the GEOM multipath software before being seen by ZFS, though I haven't ever configured this GEOM feature before, so I'm not confident about this part yet.
It
has crossed my mind that avoiding ZFS on our MegaRAID controller only to
use ZFS with these storage arrays might be illogical. Like the MegaRAID virtual disk, the way these arrays handle LUs internally is black-box.
On July 6th, I mailed Kris Moore and
Allan Jude, hosts of
BSDNow, to ask them about the risks of using ZFS with these kinds of FC SAN storage arrays.
Allan Jude answered my question in
Episode 151: Fuzzy Auditing; his answer begins 54:52 in to the show.
His answer was really interesting. My understanding of it's that should the array's write cache content be lost, the worst case scenario would be ZFS back-tracking to an older uberblock to import the pool at its previous transaction group, which would cause up to five seconds' worth of writes to be lost. He also offered the opinion that I shouldn't let this risk stop me from using ZFS, because the other ZFS benefits outweigh it.
Yet, there may be other risks associated with this, that
Allan Jude didn't think to mention, since I asked specifically about write cache problems.
New Question: with
Allan Jude's sage BSDNow advice in mind, does anybody believe that using ZFS on our
9268-8i 9267-8i MegaRAID virtual disk, is
more risky than using ZFS with these FC SAN storage arrays?
Thank you again
ab2k,
ANOKNUSA,
Phishfry,
gofer_touch,
Oko and
wblock@.
