3ware 9500 vs ZFS

oliver@

Developer
Hi,

I plan to upgrade my fileserver from an Atom 390 to a Atom C2750 system. This also involves the fact that I won't have any PCI socket where I could plug in my good-old 3ware 9500S-4LP. I thought of getting a PCIe -> PCI bridge where I could still continue to use the 3ware.
I was wondering what the difference would be between a "3ware RAID-5" and a "ZFS RAID 5".
I'm still not sure if I really want to continue to use UFS2 but beside the benefits ZFS would give me I'm for now just comparing I/O performance.
- My 3ware has a BBU installed so write cache is enabled.
- There are four WD Caviar Green (WD10EADS) attached - so no big performance on the disk side

Did anyone made a performance comparision between a 3ware-RAID-5+writecache@UFS vs. a ZFS-RAID-5? I would be interested in some results (including CPU, memory, ZIL-setup information as those are important factors for ZFS)

I only have 3ware-RAID-5+writecache@UFS stats for my current system (Atom 390, 4GB of RAM)

Code:
Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
nudel.salatschue 6G   104  99 68873  30 29166  20   256  99 118504  31 237.9  13
Latency             99691us     135ms    6204ms   47224us   76205us     285ms
Version  1.96       ------Sequential Create------ --------Random Create--------
nudel.salatschuesse -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16   554   6 +++++ +++  1069   8   554   6 +++++ +++   278   2
Latency               239ms      44us     202ms     153ms      59us     180ms
 
Hi!

I'm not an expert on ZFS, but in a building, so I would like to share my opinions below.

I'm currently building a NAS for a customer, with FreeNAS, and - of course - ZFS on it. As a first remark: the 3ware is a true HW RAID card (a good one), but this is opposite to ZFS's philosophy. HW RAID not recommended for ZFS, because it's a filesystem+volume manager+software RAID combination, thus it wants to manage the disks at lowest levels - HW RAID cards do not enable this even with no RAID volume configured. Use only plain SATA or SAS HBA (in IT mode) cards. With ZFS, your CPU will be the RAID card brain and your RAM will be the cache. In most cases, you don't need L2ARC and/or ZIL drives. Those are only useful for very high IOPS, cacheable, multi-client random operations. Above this, every ZFS guru says that max out RAM first, the consider L2ARC and ZIL.

So, I recommend to use the SATA ports probably you will find on the Atom 2750 board (those MBs come with 6-8-12 SATA ports, as I see). If you have enough RAM and a good CPU (both recommended for ZFS), ZFS will perform as, or better than this HW RAID card. Note that ECC RAM is a must for ZFS, not an option. The Atom 2750 will be powerful enough, and for 4x1TB you can start with 8GB of ECC RAM (1GB recommended for every 1TB, but 8GB minimum recommended, too).

The system I currently build is an Intel R2000 series server with a single Xeon E5-2603, 32GB ECC and 12x 2TB Seagate Constellation CS SATA disks. The disks configured as 2x(6 disk RaidZ2) pool. Comparable to a 12 disk RAID60 array. Linear read and write goes about 950-970MB/s on uncompressed datasets (higher on compressed ones), it easily maxes out 1GbE NIC link over SMB and NFS, too, at 100-110MB/s. More (random and multi client) test will come in the next days.
I know this is not comparable to a home server, but I think you see that this performs as a good HW RAID solution would.
 
Yeah, the general concepts of ZFS and its difference comparing to a hardware RAID are clear to me. My current system already uses also a zmirror setup for its system disks (2xSAMSUNG Spinpoint F3R 1TB).

If I wouldn't have this 3ware since 2005 (which costed me around 460 EUR including the BBU back then) I would definitly NOT invest that money nowadays where we have ZFS available which is working like a charm. But as I own the 3ware I wonder if I should continue using it and delegating the RAID-logic to it while saving host CPU and RAM resources - remember it ist just an atom which has not "unlimited" system ressources available. This are the concerns I have...
If I would only consider filesystem consistency and user handling I would definitly choose ZFS over UFS@3ware without any discussion at all.... but on the other hand it is the money I invested in the past and the computing power I can still delegate to the controller.
 
Sorry about witing ZFS basics :) Most people considering ZFS do not know anything about it.

My opinion that if you keep the 3ware, then use UFS on the disks it manages. I think (ans read so many times) mixing a HW RAID with ZFS on top of it is a bad idea.

On the other hand the new "server grade" Atoms (C2xxx series) have much more power than previous ones. Some articles show test results that these CPUs compare to AMD's top-of-the-line FX9xxx in performace. At least on integer and general tasks. And they compare to base line Xeon X3's in some parameters, too. They support 64GB ECC and have 8 cores @2.4-2.6GHz. Not so bad or weak, I think.

The price of the 3ware lost when you paid it, I think. HW's price loss over time greater than car's. Only you can decide, that losing this money (or the remaining of it) or losing ZFS capabilities is the lesser compromise for you.
 
You need to decide what you want: better performance or better consistency.

RAID controllers are very fast. They will usually saturate the bus (600MB/s are common for 4-disk setups). My rough estimate is that ZFS will only do about half the speed, but it performs better at keeping data consistent (if deployed correctly). RAID controllers are really dumb pieces of hardware that don't do much about consistency and ZFS does a lot there.

I still decided to use ZFS on my (quite expensive) RAID controller, because consistency is most important for me and that's what I expect from a reasonable filesystem layer. But this decision is up to you, because everyone has got their own preferences.
 
oliver@ said:
Did anyone made a performance comparision between a 3ware-RAID-5+writecache@UFS vs. a ZFS-RAID-5? I would be interested in some results (including CPU, memory, ZIL-setup information as those are important factors for ZFS)

I only have 3ware-RAID-5+writecache@UFS stats for my current system (Atom 390, 4GB of RAM)
I'm using 3ware 9650SE-16ML cards on my RAIDzilla II boxes. And I'm using ZFS. Let me summarize for folks who don't want to read the linked article: I'm exporting the 16 * 2TB drives as individual volumes. This provides all the advantages of the 3Ware card (non-volatile BBU write cache, nice management utilities that work on FreeBSD, etc.) but lets me build a ZFS filesystem using those drives.

If I were doing it over from scratch, I'd use a SAS controller so I could use either SATA or SAS drives. I'd probably still use a 3Ware card (9750?) due to the nice management tools. The alternative would be a LSI SAS2008 or similar card.
 
I'll probably go with ZFS and plain SATA only.
For continuing using the 3ware I would need to get a PCIe-to-PCI-Adapter first as the board in question only offers PCIe slots
a) it is not clear if such an adapter will work with FreeBSD
b) I would need to invest ~40 EUR for such an adapter...
 
OK, my Supermicro C2750 Board is here and running. I Decided to use the 4 internal SATA-2 ports for my 1TB WD-Green disks. ZFS is setup as raidz with atime=off,exec=off,setuid=off and this are the bonnie stats:

Code:
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
reis.salatschue 16G    48  99 156003  55 80534  38   124  99 256490  38 244.4  15
Latency               217ms   24869us    1104ms   93065us     435ms     232ms
Version  1.97       ------Sequential Create------ --------Random Create--------
reis.salatschuessel -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 12799  97 +++++ +++ 11286  98 11933  97 +++++ +++ 10617  94
Latency             15729us     586us     695us   38027us     635us     691us

Kinda..... nice :)
 
Back
Top