After days of downtime, because of corrupted 3ware RAID arrays, I decided to move all these setups to ZFS. All for good! Especially with regards to storage management. The only downside is the need for more RAM. But this is trouble only with old servers using now exotic RAM types. RAM is very cheap these days.
In my opinion, the only significant benefit of using battery backed RAID controller for ZFS is the added layer of management. You may, for example verify each drive separately. But you may do the same with smartmontools with any controller. The IOP benefit is something I did not observe --- but plan to test more extensively soon.
Of course, there is nothing wrong to use 3ware-type RAID controlers with ZFS, even if you don't use any of their RAID functionality. If you attach more disks, this is the way to go.
For the ZIL, probably the best solution is battery backed RAM disk. You don't need huge capacity here and small capacity, fast SSDs are probably not easy to find anymore. For the lower end/cheap systems -- any flash device is good for this purpose. Trouble is, you cannot remove the ZIL after you start using it.
Same for L2ARC. Here, an SSD is wonderful! But for lower end systems, you may use USB flash tokens, even many of these. Commodity motherboards come with 6 or more USB ports. If you get say 20 MBps read from an $10 4GB flash stick, with 6 of these you will get 120 MBps for $60 at 24GB. Not bad, eh? Still, for larger sizes SSDs win.
I would like to strongly second the statement that Phoenix made "use glabel"! This has saved my day several times and I now routinely label any storage media. I was wondering, if this breaks the ZFS philosophy "use entire disk devices for better performance" -- but haven't seen any degradation so far.
My boot devices of late are .. USB sticks
in gmirror configuration. For several reasons:
- recent motherboards don't even have PATA anymore, especially the desktop models. This makes CF cards useless.
- I happen to use lots of Supermicro systems recently and these have almost always two internal USB slots. Just find small size USB sticks that fit there -- no danger to disconnect these, as when they hang out of the case.
- these are disposable in principle, although I have yet to see one fall on me.
There are also USB flash modules, that directly attach on the motherboard pins (for front panel USB ports for example). These too stay inside the case, are somewhat faster and of course more expensive.
The only trouble with USB flash for boot devices is that FreeBSD 8.0 doesn't play nicely with them at boot time. There is of course an easy fix.