Good 8 ports SATA Controller

I want to build a FreeBSD 9.0 server (my first) with 4 Seagate Barracuda ST3000DM001 harddisks (expandable to 8) and use them with ZFS. The onboard controller is for a SSD and serving the FreeBSD system.

After a lot of reading in this forum, using Google for hours, I do not feel comfortable about picking a good 8 ports SATA (II/III) controller. Maybe 2 * 4 ports controller?

Please share your thoughts on this with me.

Thanks in advance.
 
Ok, this is a reply to my own post to add some extra information

It's for home use and I want to get the maximum out of ZFS, so I intent to put in 4 striped vdev's of 2 mirrored disks.
SATA II should be enough for conventional disks (NON-SSD), so I am looking for a stabel well supported SATA II/III controller supporting up to 8 disks.

Please help me out here
 
Thank you for your answer.

your suggested controller is from 2007, so 5 years old and still expensive (for home use).

After some more searching I found the LSI SAS3081E-R-SGL. Less expensive and from my point of view comparable to the older card (no hardware expert). I found a post where it seems supported also (not sure about the -SGL addition on the end). Do you think it is a good controller to go for?
 
M_Devil said:
After a lot of reading in this forum, using Google for hours, I do not feel comfortable about picking a good 8 ports SATA (II/III) controller. Maybe 2 * 4 ports controller?
I agree with the post recommending the 3Ware controllers. I have built many systems with them and they are very well-supported on FreeBSD, with both CLI and web-based management.

The various LSI SASxxxx controllers are supported, but will be a bit more difficult to set up and manage. The newest ones may require FreeBSD 9-STABLE or newer as the drivers aren't in 9.0-RELEASE or older. There is a bewildering array of firmware, much of it vendor-specific (Dell, IBM, Supermicro, etc.) and there are different firmware images for RAID vs. "dumb" mode, with the FreeBSD drivers only supporting some of them (generally the "dumb" mode, which fortunately is what you want for ZFS). Lastly, at least some of these controllers have limited driver support for decoding events detected by the card, logging things like "mpt0: mpt_cam_event: 0x0" instead of useful messages. This is matched by limited management utilities for manipulating the card while the FreeBSD is running. You can always reboot and use the BIOS menus, but I really like the convenience of being able to use the 3Ware web utility to delete a unit, swap a drive, and recreate the unit, all while FreeBSD is running.

When considering multiple controllers instead of a single one (as in your 2 x 4-port example), carefully check your system to make sure you can install multiple controllers. You may be limited by the location and type of expansion slots, as well as clearance issues with card overhang interfering with other motherboard or chassis components. There is also the issue of adequate airflow if you're forced to install the cards in adjacent slots. An issue which won't affect you in "dumb" controller mode is the limitation of some cards, which can't build RAID volumes across multiple cards.
 
Wow, thank your for your detail information.

Support
If I understand you correct, the most save solution is the 3Ware 9650SE-8LPML for a 8 port SATA. This controller is a little old, but wel supported. Correct?

Speed
Can I expect a 300 MB/sec liniair write and read throughput in ZFS setup I mention in my previous post with the 3Ware 9650SE-8LPML controller?

Hardware RAID or dumb mode
I read a lot about the 'dumb' mode of controllers (IT firmware?). In your post you mention the web interface for maintenance on the disk, but how does this relate to ZFS? Should ik use the 3Ware 9650SE-8LPML in 'dumb' mode and should I flash the IT firmware for that?
In other words: should I use ZFS or de web interface from the 3Ware 9650SE-8LPML for maintenance?
 
With ZFS you should use the "dumb" mode, also known as JBOD where every individual disk is visible to the operating system as one single disk. The 3ware controller I linked seems to support JBOD out of the box so I don't think you'll have to do anything to the firmware to use it with ZFS.
 
M_Devil said:
If I understand you correct, the most save solution is the 3Ware 9650SE-8LPML for a 8 port SATA. This controller is a little old, but wel supported. Correct?
It is the family I have the most experience with. Depending on how your drives are mounted, you'll need 1 of 3 different types of cables. The controller has the most modern multi-lane variant. Your drive backplane(s) may either have that same connector, need a fan-out from a multi-lane connector to 4 "classic" SATA cables, or have an older "thumbscrew" multi-lane connector.

Can I expect a 300 MB/sec liniair write and read throughput in ZFS setup I mention in my previous post with the 3Ware 9650SE-8LPML controller?
Assuming your drives are up to it, I don't see why not. My setup with 3 raidz1 vdev's in the pool gives me > 500MB/sec read or write.

I read a lot about the 'dumb' mode of controllers (IT firmware?). In your post you mention the web interface for maintenance on the disk, but how does this relate to ZFS? Should ik use the 3Ware 9650SE-8LPML in 'dumb' mode and should I flash the IT firmware for that?
In other words: should I use ZFS or de web interface from the 3Ware 9650SE-8LPML for maintenance?
There's only one firmware type for the 3Ware cards, and it does everything. You won't be using the on-board RAID capability, but you will be creating single volumes (one drive each) and exporting them to the OS. That gets you the advantage of the controller's cache (particularly if you install the optional battery backup card).

You use the ZFS commands to create and manage the pool as usual. What the 3Ware CLI / web interface gets you is the ability to do things normally only found in the BIOS menu, while FreeBSD is running. If a drive fails, you use the 3Ware utility to delete its associated unit. ZFS sees this as "lost device entry" and operates in degraded mode. You swap the new drive in, create a unit on it with the 3Ware utility, which causes a drive with the same name (dax) to appear in FreeBSD (see attached console output)

Code:
[i][Use 3Ware controller to delete logical unit on port 7][/i]
Jul  7 04:27:02 rz1m kernel: (da8:twa0:0:7:0): lost device - 0 outstanding, 1 refs
Jul  7 04:27:06 rz1m kernel: (da8:twa0:0:7:0): removing device entry
[i][Physically remove drive from  port 7][/i]
Jul  7 04:27:25 rz1m kernel: twa0: WARNING: (0x04: 0x0019): Drive removed: port=7
[i][Physically insert new drive in port 7][/i]
Jul  7 04:31:06 rz1m kernel: twa0: INFO: (0x04: 0x001A): Drive inserted: port=7
[i][Use 3Ware utility to create a logical volume on the drive you just inserted][/i]
Jul  7 04:32:27 rz1m kernel: da8 at twa0 bus 0 scbus9 target 7 lun 0
Jul  7 04:32:27 rz1m kernel: da8: <AMCC 9650SE-16M DISK 4.10> Fixed Direct Access SCSI-5 device 
Jul  7 04:32:27 rz1m kernel: da8: 100.000MB/s transfers
Jul  7 04:32:27 rz1m kernel: da8: 1907338MB (3906228224 512 byte sectors: 255H 63S/T 243151C)
Since I use glabel on my drives, I needed to do:
# glabel label twd7 da8
followed by:
# zpool replace data label/twd7 label/twd7
 
kpa said:
With ZFS you should use the "dumb" mode, also known as JBOD where every individual disk is visible to the operating system as one single disk. The 3ware controller I linked seems to support JBOD out of the box so I don't think you'll have to do anything to the firmware to use it with ZFS.
On the 3Ware cards, you generally want to export each drive as a single volume if you're using ZFS. If you export them "raw" (also called JBOD or "dumb", you don't get any of the management features of the 3Ware card, nor writeback caching (which should have a BBU attached to the 3Ware card). The exporting of raw drives has been deprecated by 3Ware over time. It still exists, but is not recommended for new installs.
 
Terry_Kennedy said:
On the 3Ware cards, you generally want to export each drive as a single volume if you're using ZFS. If you export them "raw" (also called JBOD or "dumb", you don't get any of the management features of the 3Ware card, nor writeback caching (which should have a BBU attached to the 3Ware card). The exporting of raw drives has been deprecated by 3Ware over time. It still exists, but is not recommended for new installs.

Ok, so the best way to go is export each drive as single volume. Sounds like instructing the controller to do so. How do I instruct the controller? Install freeBSD (9.0 stable) and use some CLI commands? Or using the Webinterface, but how can I get there when freeBSD is running?
 
M_Devil said:
Ok, so the best way to go is export each drive as single volume. Sounds like instructing the controller to do so. How do I instruct the controller? Install freeBSD (9.0 stable) and use some CLI commands? Or using the Webinterface, but how can I get there when freeBSD is running?
You can add the drives in the 3Ware BIOS menu (necessary if you're planning on booting from your ZFS pool) or afterwards with the CLI (called tw_cli) or web interface (called 3DM). The web interface is a lot easier, in my opinion. Plus, you're not having to type into the BIOS menu on a system that's down, with users complaining. Even if you don't plan on using the CLI version, install it anyway as it provides a 407.status-3ware-raid script for nightly processing which reports on the health of the controller and drives.

The relevant ports are sysutils/tw_cli and sysutils/3dm. The web interface runs its own private web server which (by default) speaks HTTPS on port 888. Thus, it doesn't interfere with having a regular web server on the system.

I forgot to mention that the sysutils/smartmontools works with drives behind 3Ware controllers, regardless of how they are configured. As of a few versions back on the twa(4) driver, smartd started logging a bunch of "Couldn't do [thing] because controller was busy" messages on busy systems. I created a patch for this, but it wasn't accepted upstream. Let me know if you run into the issue and I'll send it to you.

Sample output from the 3ware nightly script:
Code:
(0:1) host:/sysprog/terry# /usr/local/etc/periodic/daily/407.status-3ware-raid 

Checking status of 3ware RAID controllers:

Controller c0:

Unit  UnitType  Status         %RCmpl  %V/I/M  Stripe  Size(GB)  Cache  AVrfy
------------------------------------------------------------------------------
u0    SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u1    SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u2    SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u3    SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u4    SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u5    SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u6    SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u7    SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u8    SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u9    SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u10   SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u11   SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u12   SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u13   SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u14   SINGLE    OK             -       -       -       1862.63   RiW    OFF    
u15   SINGLE    OK             -       -       -       1862.63   RiW    OFF    

VPort Status         Unit Size      Type  Phy Encl-Slot    Model
------------------------------------------------------------------------------
p0    OK             u0   1.82 TB   SATA  0   -            WDC WD2003FYYS-02W0 
p1    OK             u1   1.82 TB   SATA  1   -            WDC WD2003FYYS-02W0 
p2    OK             u2   1.82 TB   SATA  2   -            WDC WD2003FYYS-02W0 
p3    OK             u3   1.82 TB   SATA  3   -            WDC WD2003FYYS-02W0 
p4    OK             u4   1.82 TB   SATA  4   -            WDC WD2003FYYS-02W0 
p5    OK             u5   1.82 TB   SATA  5   -            WDC WD2003FYYS-02W0 
p6    OK             u6   1.82 TB   SATA  6   -            WDC WD2003FYYS-02W0 
p7    OK             u7   1.82 TB   SATA  7   -            WDC WD2003FYYS-02W0 
p8    OK             u8   1.82 TB   SATA  8   -            WDC WD2003FYYS-02W0 
p9    OK             u9   1.82 TB   SATA  9   -            WDC WD2003FYYS-02W0 
p10   OK             u10  1.82 TB   SATA  10  -            WDC WD2003FYYS-02W0 
p11   OK             u11  1.82 TB   SATA  11  -            WDC WD2003FYYS-02W0 
p12   OK             u12  1.82 TB   SATA  12  -            WDC WD2003FYYS-02W0 
p13   OK             u13  1.82 TB   SATA  13  -            WDC WD2003FYYS-02W0 
p14   OK             u14  1.82 TB   SATA  14  -            WDC WD2003FYYS-02W0 
p15   OK             u15  1.82 TB   SATA  15  -            WDC WD2003FYYS-02W0 

Name  OnlineState  BBUReady  Status    Volt     Temp     Hours  LastCapTest
---------------------------------------------------------------------------
bbu   On           Yes       OK        OK       OK       255    08-Nov-2010  

Alarms (most recent first):
  No new alarms.

Sample output from smartmontools nightly script:
Code:
(0:2) host:/sysprog/terry# /usr/local/etc/periodic/daily/smart 

SMART status:
Checking health of /dev/ada0: OK
Checking health of /dev/ada1: OK
Checking health of /dev/twa0 (-d3ware,0): OK
Checking health of /dev/twa0 (-d3ware,1): OK
Checking health of /dev/twa0 (-d3ware,2): OK
Checking health of /dev/twa0 (-d3ware,3): OK
Checking health of /dev/twa0 (-d3ware,4): OK
Checking health of /dev/twa0 (-d3ware,5): OK
Checking health of /dev/twa0 (-d3ware,6): OK
Checking health of /dev/twa0 (-d3ware,7): OK
Checking health of /dev/twa0 (-d3ware,8): OK
Checking health of /dev/twa0 (-d3ware,9): OK
Checking health of /dev/twa0 (-d3ware,10): OK
Checking health of /dev/twa0 (-d3ware,11): OK
Checking health of /dev/twa0 (-d3ware,12): OK
Checking health of /dev/twa0 (-d3ware,13): OK
Checking health of /dev/twa0 (-d3ware,14): OK
Checking health of /dev/twa0 (-d3ware,15): OK
 
I am relative new to freeBSD (some weeks ago I finished my first pfSense installation and I am running freeBSD 9.0 Stable in an Virtualbox environment for gaining knowledge about installing en using freeBSD). Your answers gives me fast and high quality information. Now I feel confident enough to go forward with my first freeBSD server. When I run into trouble, I will for sure ask for your assistent, thank you for offering.
 
Hi Knarf, how does the Supermicro perform and what is you setup in terms of number of disks, ZFS layout?
 
knarf said:
I use this one: Supermicro AOC-SAT2-MV8 with mvs(4).

The card have a 64-bit PCI-X133MHz interface. You need a server mainboard for that kind of interface, because it is not like on every mainboard like PCI or PCIe now. A card with PCIe interface should be a more compatible choice.
 
If you want something with pci-e interface, you can go with a supermicro AOC-USASLP-L8i (or AOC-USAS-L8i which is the same but not low profile). Intel and lsi have cards with the same chipset ( Intel SASUC8I and LSI SAS3081E-R but you need to flash the firmware).

These cards work very well with FreeBSD and are not too expensive.
 
Due my lack of knowlage (sorry for that): How can I connect 8 SATA disks to a 2 ports SAS connector in case of the AOC-USASLP-L8i?
 
Did not know it is possible with just a cable. Does it have negative impact on performance, 4 SATA disks on one SAS connection?
 
The sas connector used on these cards (sff-8087 for internal, sff-8088 for external) actually contains 4 completely separate channels. The sas to sata breakout cable basically just splits out the channels.

It's used on most multi-port cards these days as it saves space and simplifies cabling, especially if you have to same connector on the drive backplanes.
 
kisscool-fr said:
If you want something with pci-e interface, you can go with a supermicro AOC-USASLP-L8i (or AOC-USAS-L8i which is the same but not low profile). Intel and lsi have cards with the same chipset ( Intel SASUC8I and LSI SAS3081E-R but you need to flash the firmware).

These cards work very well with FreeBSD and are not too expensive.
One thing to watch out for is that many of the Supermicro cards use the UIO format, which is unique to Supermicro. According to this Supermicro page, those are the ones with model numbers starting with "AOC-U" (U for UIO).

The UIO cards will work in regular (non-UIO) slots, but the bracket won't be in the expected position for a normal expansion card. People have come up with various ways to deal with this, ranging from removing the bracket and hoping the card doesn't fall out all the way to making a whole new bracket.
 
Is there a valid/good reason to go for the more expensive 3Ware 9650SE-8LPML over the less expensive Supermicro AOC-USASLP-L8i in terms of support, quality or speed?
 
It just depends on your needs. For a home system with 4 HDDs i see no reason to buy the expensive 3ware controller.

For a 24/7 business solution, i would personally think twice about.
 
For a 24/7 business, buy two of those controllers. Keep one in a safe place, because when the time comes that you need a replacement it will no longer be available or will take weeks to arrive.
 
If you are using it for ZFS, just get a 'dumb' SATA controller, and let ZFS handle the rest. There's no point in buying a $500 RAID controller, only to turn around and disable all the RAID features.

The LSI1068 chipset (supported by mpt(4)) works beautifully, and is available in many different controllers. The driver only supports 2 TB disks and 3.0 Gbps SATA2, though.

The LSI2008 chipset (supported by mps(4)) also works beautifully, and supports disks over 2 TB, along with 6.0 Gbps SATA3. These are also available in many controllers from LSI, SuperMicro, Intel, etc.
 
Back
Top