Building up a 16TB system

dvl@

Developer
Hello folks,

I'm looking at putting together a new box, mostly for storage. But it will probably run some virtual machines, and be used for some regression testing of backup software and for developing various website. The box will be sitting in my basement, not in a data center.

I've been looking at the components, I'm looking to order within the next week or two. The system will be running FreeBSD 9. The main OS will be installed on two SSDs and the main storage will be spread across 8x2T HDDs clustered together into a raidz2 ZFS array.

Here are the components I'm considering. The links below are all Newegg, and I like dealing with them.

* SUPERMICRO MBD-H8SGL-O ATX Server Motherboard : $224.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182240

* AMD Opteron 6128 Magny-Cours 2.0GHz 8 x 512KB L2 Cache 12MB L3 Cache Socket G34 115W 8-Core Server : $259.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16819105266

* Kingston 8GB 240-Pin DDR3 SDRAM ECC Registered DDR3 1600 Server Memory : 4 x $59.99 = $239.96
http://www.newegg.com/Product/Product.aspx?Item=N82E16820239230

* PC Power and Cooling Silencer MK III 600W power supply : $99.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16817703036

* Seagate Barracuda ST2000DM001 2TB 7200 RPM 64MB : 8 x $114.99 = $919.92
http://www.newegg.com/Product/Product.aspx?Item=N82E16822148834

* Mushkin Enhanced Chronos MKNSSDCR60GB 2.5″ 60GB SATA III Asynchronous MLC Internal Solid State Drive (SSD) : $59.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16820226247

* Corsair Force Series GT CSSD-F60GBGT-BK 2.5″ 60GB SATA III Internal Solid State Drive (SSD) :$89.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16820233193

* LIAN LI PC-A70F USB3.0 Black Aluminum ATX Full Tower Computer Case : $189.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16811112342



Not listed above is the disk controller. The controller will be used in pass-through mode (i.e. JBOD), not RAID. I've considered these cards:


* SYBA SD-PEX40054 PCI-Express 2.0 x4 SATA III (6.0Gb/s) HyperDuo 4-port RAID Controller Card $41.99
http://www.newegg.com/Product/Product.aspx?Item=N82E16816124056

* SUPERMICRO AOC-USAS2-L8i PCI Express SATA / SAS Eight-Port Internal RAID Adapter
http://www.newegg.com/Product/Product.aspx?Item=N82E16816101334

* HighPoint Rocket 640L PCI-Express 2.0 x4 Low Profile SATA III (6.0Gb/s) RAID Controller Card : 2 x $69.99= $149.98
http://www.newegg.com/Product/Product.aspx?Item=N82E16816115114

* Intel RAID Controller Card 6G SAS PCI-E x8 8 internal ports (RS2WC080)
http://www.newegg.com/Product/Product.aspx?Item=N82E16816117171&Tpk=rs2wc080 $257.99


Any comments on the selection? Thanks. :)
 
I think it would be more cost effective to get a Supermicro board with the 8-port 6 Gbps LSI SAS built right into the board. Then you have the board and disk controllers for a couple hundred bucks. E.g. an X8SI6-F (or -O if you don't care about IPMI), or some AMD equivalent.
 
That's a good idea.

But even with 8 ports onboard, I still need two more ports for the gmirror I plan for the base OS.

I did find this 'gamer' m/b: ASUS Crosshair V (http://www.newegg.com/Product/Product.aspx?Item=N82E16813131876) for $230

I also tried searching for AMD boards by Supermicro, with more SATA III, but failed.

Key here is support for ECC RAM. I want that. It is important.
 
The Supermicro boards I've seen normally have the SAS controller on top of a standaard set of 6 SATA ports, so that makes 14 ports :) Haven't found a single CPU AMD board with this though, I have a Xeon board myself.
All server boards I came across have support for ECC RAM, some pro workstation boards support it also, regular consumer stuff generally doesn't. And yeah I agree, ECC RAM is mandatory for ZFS based servers imo, an unlucky bitflip corrupting a fsck-less storage system is not cool :p
 
Hi,
Recently I bought the same HDD you are aiming for: ST2000DM001. I had big problems with it (probably servo), weird noises coming out from time to time causing little response delay. I did put it through RMA process 3x times. Each time I got new disk. Each time the same issues. All of them had firmware version CC4C.

I never had problems with Seagate disks before, but this model drove me crazy.
 
matoatlantis said:
Hi,
Recently I bought the same HDD you are aiming for: ST2000DM001. ..... All of them had firmware version CC4C.

How recent?

Did you try upgrading the firmware? That was the first thing to do, based on a recommendation elsewhere.
 
Sfynx said:
The Supermicro boards I've seen normally have the SAS controller on top of a standaard set of 6 SATA ports, so that makes 14 ports :) Haven't found a single CPU AMD board with this though, I have a Xeon board myself.

I'm keen on AMD, but would go Intel if required....
 
I would prefer the Seasonic power supplies to anything else. The M12II 620 is $10 less, the X650 is $20 more.

Mixing brands on the SSDs is a good idea, but you may also want to mix internal controller types. Those are both Sandforce. Plextor, Crucial, some Intel drives, and Samsung use non-Sandforce controllers. I've had good results with Plextor and Intel.

And then the hard drives... other than WD Black, I just don't know. Seagate has given me a bad feeling lately, and their drives usually only have a one-year warranty. The WD Red drives are well-regarded and made for RAID, but are too new to have much of a track record. They are overpriced at Newegg. If possible, mixing brands on the 2T drives could be a good idea also.
 
dvl@ said:
How recent?

Did you try upgrading the firmware? That was the first thing to do, based on a recommendation elsewhere.

For the first one, yes. Sorry I can't give you more detailed info, I don't remember. I pushed it to the latest firmware available. The other two, I just tested and returned when the same issue occurred.

Dates (all year 2012):
Code:
1st: Mar 12
2nd: Apr 25
3rd: Jun 11
 
btw, RAID-Z2 is not very strong on small random I/O especially when you're going to mix all kinds of different drives (good for preventing bad batch problems though), so if you expect such a workload it could be beneficial to go for a bit bigger SSD option so you can reserve some space for L2ARC (does not need to be mirrored).
 
wblock@ said:
Mixing brands on the SSDs is a good idea, but you may also want to mix internal controller types. Those are both Sandforce. Plextor, Crucial, some Intel drives, and Samsung use non-Sandforce controllers. I've had good results with Plextor and Intel.

Good point. Interesting. I hadn't considered that.
 
wblock@ said:
And then the hard drives... other than WD Black, I just don't know. Seagate has given me a bad feeling lately, and their drives usually only have a one-year warranty. The WD Red drives are well-regarded and made for RAID, but are too new to have much of a track record. They are overpriced at Newegg. If possible, mixing brands on the 2T drives could be a good idea also.

I recognize the theory behind mixing HDD brands, but with 8 drives, if I used 4 different brands, then two failures with raidz2... I'm still OK. With 2 or three brands... no so much.
 
wblock@ said:
I would prefer the Seasonic power supplies to anything else. The M12II 620 is $10 less, the X650 is $20 more.

Why do you like this brand?
 
Based on 4 brands:

TOSHIBA DT01ACA200 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s 2x $110
http://www.newegg.com/Product/Product.aspx?Item=N82E16822149407

Seagate Barracuda ST2000DM001 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s 2x $115
http://www.newegg.com/Product/Product.aspx?Item=N82E16822148834

Western Digital WD Black WD2002FAEX 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s 2x $179
http://www.newegg.com/Product/Product.aspx?Item=N82E16822136792

Hitachi GST Ultrastar 7K3000 HUA723020ALA640 (0F12455) 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s 2x $240
http://www.newegg.com/Product/Product.aspx?Item=N82E16822145476


Total (without shipping/tax): $1288



Based on two brands (the first two listed above):

TOSHIBA DT01ACA200 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s 2x $110
http://www.newegg.com/Product/Product.aspx?Item=N82E16822149407

Seagate Barracuda ST2000DM001 2TB 7200 RPM 64MB Cache SATA 6.0Gb/s 2x $115
http://www.newegg.com/Product/Product.aspx?Item=N82E16822148834


Total (without shipping/tax): $900

I'm not so sure how much having two brands will help. I think any risk avoidance is minimal.
 
If I had to buy two brands of drives, I might get the Seagate "D-moo" drives and the WD Reds. Toshiba I avoid, not that they've failed on me, just that every one I've ever had was so slow that I wished for it to fail. Hitachi seems to think their questionable reliability is worth more than others.

For an array for size rather than speed, I might consider the WD Greens. The newest ones with 64M cache are supposed to, well, suck less than the earlier ones.
 
gkontos said:
For an HBA controller you might want to consider:

http://www.lsi.com/products/storagecomponents/Pages/LSISAS9207-8i.aspx
http://www.lsi.com/products/storagecomponents/Pages/LSISAS9211-8i.aspx

They both run fantastic under FreeBSD 9-STABLE and 9.1-RCX

My reading shows that the 9207arrives in IT mode (meaning pass-through/JBOD mode), which is just what is needed for a ZFS array.

I can't tell yet about the 9211, but this post seems to indicate that it does.

NOTE: But this post indicates that you need to replace the 9211 firmware with an IT version

Any knowledge on these points?
 
wblock@ said:
If I had to buy two brands of drives, I might get the Seagate "D-moo" drives and the WD Reds. Toshiba I avoid, not that they've failed on me, just that every one I've ever had was so slow that I wished for it to fail. Hitachi seems to think their questionable reliability is worth more than others.

For an array for size rather than speed, I might consider the WD Greens. The newest ones with 64M cache are supposed to, well, suck less than the earlier ones.


I cannot find D-moo drives...

WD Reds are about +50% over other similar drives...

I've had 8x2TB Hitachi drives in another system for about 2 years. Fine so far.

re WD Green: Yeah, to be fair, I'm trying for speed, but I'm not sure I'll get there with this combination. There are so many variables, that I'm considering going with the lower end components and just let it be...
 
dvl@ said:
My reading shows that the 9207arrives in IT mode (meaning pass-through/JBOD mode), which is just what is needed for a ZFS array.

I can't tell yet about the 9211, but this post seems to indicate that it does.

NOTE: But this post indicates that you need to replace the 9211 firmware with an IT version

Any knowledge on these points?

You can always flash the firmware. So far I have only done that on a LSI MegaRaid controller that was having issues with JBOD.
 
dvl@ said:
I cannot find D-moo drives...

Seagate STnnnnDM001. :)

WD Reds are about +50% over other similar drives...

At Newegg, yes. At other places, they're fair.

re WD Green: Yeah, to be fair, I'm trying for speed, but I'm not sure I'll get there with this combination. There are so many variables, that I'm considering going with the lower end components and just let it be...

It would be fun to build two arrays, one higher-end, and one the absolute cheapest, do benchmarks, and run them for a while.
 
wblock@ said:
It would be fun to build two arrays, one higher-end, and one the absolute cheapest, do benchmarks, and run them for a while.

I will accept donations to accept this challenge.
 
Please ignore the above post. I am comparing a 1TB SAS with a 2TB SATA. Sorry.
 
Back
Top