Any advice on choice of additional drives on the ZFS fileserver?

[rusma]

Hello.

First, some background information: I am considering buying more harddrives for my fileserver. It has space enough for 16 harddrives (Cooler Master Stacker), probably 20 or even 24. I have 14 sata ports. I am currently using 6 drives (2 pata for freebsd (zfs gpt mirror), and 4 sata for data (zfs gpt raidz2)). The data drives are each 500G large Seagate Barracuda ES.2, which makes a 1.8T large pool.

I have nearly used up all of the space in the data pool. So, I was wondering if someone could give me some advice on buying new drives. To this point, I have considered 'Samsung SpinPoint F3 HD103SJ 32MB 1TB'. Although it would be really nice to have 2T large drives, I have read some other threads on this forum about choosing drives, and understood that it is important to not choose any 'green/eco' drives with spindown and 4K drives. I think seven drives in a raidz2 (or raidz3?) would be sufficent for a long time.

 

[gpw928]

Hi Rusma,

The Samsung Spinpoint F3 is a very well regarded SATA 2 drive. I have several on my TV server (not with ZFS).

Please rate for your new requirements, in order of priority:

• price,
• seek time,
• transfer rate,
• capacity,
• warranty, and
• MTBF.

Other points which must be addressed:

• Do you have technology limitations? e.g. SATA 2 or SATA 3?
  
• Can your firmware handle disks greater than 2 TB?
  
• What spare capacity does your power supply have to spin up additional disks at power on (peak load)?
  
• Are you likely to need to investigate staggered spin up?

Cheers,


--
Phil

 

[zennybsd]

Hds723020bla642

After much research online about the benchmarks, I chose the following HDD:

Hitachi Deskstar 7K3000 HDS723020BLA642 - HDD - 2TB -
intern - 8,9 cm (3,5") - SATA-600 - 7200 Cache: 64MB
(HDS723020BLA642)

Cheap, fast, robust and large capacity. Paid like 85€ each, bought 12pcs.


*** If any of my posts helped you, please contribute to either http://www.freebsdfoundation.org/donate/ or http://www.thehumanape.org/. Appreciate it! Thanks for your understanding! ***

 

[rusma]

Please rate for your new requirements, in order of priority:

• price,
• seek time,
• transfer rate,
• capacity,
• warranty, and
• MTBF.

Hello. I believe my list would look like this:

• MTBF.
• price,
• capacity,
• seek time,
• warranty, and
• transfer rate,

I am not quite sure. Price is important.

Other points which must be addressed:

• Do you have technology limitations? e.g. SATA 2 or SATA 3?
  
• Can your firmware handle disks greater than 2 TB?
  
• What spare capacity does your power supply have to spin up additional disks at power on (peak load)?
  
• Are you likely to need to investigate staggered spin up?

• Yes, one can say I have technology limitations, since my system only supports sata2 at the moment, but I have read that sata3 drives fall back to sata2.
  
• The firmware on the harddrives? My BIOS? My motherboard uses the ICH10R chipset and latest bios (Asus P5Q Deluxe), in addition I use some generic sata2 extension cards (Promise TX4 and Lycom SIL3201 *something, something*.
  
• My powersupply is rated 750W, and under idle load the system uses ~100W. So, I have much power to spare
  
• I believe that I am not likely to invesigate a staggered spinup. I do not want spin up, spin down drives as I have read that it is bad for the drive health to behave in that way.

Also, the drive zennybsd recommended (HDS723020BLA642), seems quite good. Sadly, it looks quite noisy (~30dB), though. The Barracuda drives I have already is, in my opinion, quite noisy when I am accessing stuff. Samsung drives is also very well regarded noise-wise.

 

[gpw928]

Hi rusma,

There is not much point in looking at your drive options since you have already purchased SATA 3 drives with a capacity of 2 TB.

The firmware limitations for drives > 2 TB are therefore not an issue. If you are interested, read this for background.

The drive you chose requires 1.2 W on the +5 V rail, and 2.0 W on the +12 V rail to spin up. The maximum load occurs at spin up. This is shown on the "specifications" tab here.

To spin up your 12 new disk drives, your power supply will have to be able to supply an additional total 14.4 W on the +5 V rail, and 24.0 W on the +12 V rail.

I don't think that's going to trouble a 750 W power supply unless you have some heavy hitting graphics card(s). But note that it's the +5 V rail where you will run out of power first, and the +5 V rail limit is likely to be in the vicinity of 145 W on a good 750 W PSU.

To be on the safe side you could google "power supply calculator", and use one of the on-line calculators. But be aware that there are large differences in the capabilities between cheap power supplies and quality ones (quality 750 W PSUs start around US$150).

Staggered spinup is used to reduce peak load on the power supply. i.e. different drives spin up at slightly different times so as to demand peak power at different times, and thus reduce the peak power demand when the system is powered on.

Cheers,

--
Phil

 

[phoenix]

I am currently using 6 drives (2 pata for freebsd (zfs gpt mirror), and 4 sata for data (zfs gpt raidz2)). The data drives are each 500G large Seagate Barracuda ES.2, which makes a 1.8T large pool.

Ouch! You are using 4 drives in a raidz2 vdev? Why not 2x mirror vdev? You get the same amount of usable disk space (2 drives worth), but much better performance, and way easier management when replacing drives or adding new drives.

I think seven drives in a raidz2 (or raidz3?) would be sufficent for a long time.

If you created the pool with a 4-drive raidz2 vdev, then you should expand the pool using the same (add more 4-drive raidz2 vdevs). While you can forcibly add vdevs of different types and sizes, it's not recommended for a whole host of reasons.

If you want to change the size/type of vdevs in the pool, you need to:

• backup data in pool
• destroy pool
• create new pool using new vdev layout
• restore data to new pool

 

[rusma]

Hello. First I would like to apologize for answering this late.

There is not much point in looking at your drive options since you have already purchased SATA 3 drives with a capacity of 2 TB.

Hey! I still have my old drives (i.e. 4 500GB Barracudas in a raidz2), i.e. I have not bought any new ones.

To spin up your 12 new disk drives, your power supply will have to be able to supply an additional total 14.4 W on the +5 V rail, and 24.0 W on the +12 V rail.

I don't think that's going to trouble a 750 W power supply unless you have some heavy hitting graphics card(s). But note that it's the +5 V rail where you will run out of power first, and the +5 V rail limit is likely to be in the vicinity of 145 W on a good 750 W PSU.

To be on the safe side you could google "power supply calculator", and use one of the on-line calculators. But be aware that there are large differences in the capabilities between cheap power supplies and quality ones (quality 750 W PSUs start around US$150).

The name of my power supply is 'Thermaltake Toughpower 750W', and it did cost more than 150$ (950NOK if I remember correctly). I do not know if it is any good; I bought it because it was popular at the time I bought the initial parts of the server.

Staggered spinup is used to reduce peak load on the power supply. i.e. different drives spin up at slightly different times so as to demand peak power at different times, and thus reduce the peak power demand when the system is powered on.

This seems interesting to enable just in case. How do I enable it? By reading this article[1], I am guessing it has something to do with S.M.A.R.T.

Why not 2x mirror vdev? You get the same amount of usable disk space (2 drives worth), but much better performance, and way easier management when replacing drives or adding new drives.

Nice tip, I will probably switch to mirrored. I did it this way because I read a thread somewhere saying raidz2 was generally the best solution for every disk setup, perfomance- and security-wise. Also because Raid-Z is one of the components that makes ZFS special to other solutions. I guess I did not consider it as a serious option -- raidz just seems like a generally better word than mirror. By the way, the root filesystem is, as mentioned earlier, in a 2x mirror vdev.

If you created the pool with a 4-drive raidz2 vdev, then you should expand the pool using the same (add more 4-drive raidz2 vdevs). While you can forcibly add vdevs of different types and sizes, it's not recommended for a whole host of reasons.

I will probably change to this setup you are referring to by the time I get the drives. In which case would a raidz2 setup be appropriate? Seems weird to have it as a option if mirrored always would be the better choice.

----
[1] http://en.wikipedia.org/wiki/Spin-up

 

[Anonymous]

.. To this point, I have considered 'Samsung SpinPoint F3 HD103SJ 32MB 1TB'.

This is a really good and stable one. I have this continuously running for more than a year now, and the main advantage of it is, that according to smartctl of the sysutils/smartmontools it is running really cool (19 to 32 °C) while a WD 2 TB disk in another machine is operating at 40 °C, and a Hitachi 3 TB disk goes up to almost 50 °C.

The Samsung and the Hitachi disk don't show the Load_Cycle_Count problem.

Don't buy WD Green Drives for FreeBSD, the Load_Cycle_Count grows very fast and renders the drive unusable within in less than a year. I guess that Samsung Green Drives would show the same problem, but the Load Cycling (Clicking) can be disabled on the Samsung drives I know of with sysutils/ataidle.

 

[bbzz]

Why did you say not to use 4K disks? I use 6 x SAMSUNG Spinpoint F4 HD204UI Ecogreen 2TB, and they are performing well so far. They are also using 4K sectors, but you can align. Also F4 > F3 series.

From all the reviews I did these were topping similar Darkstar and WD (especially WD Green) series. My next choice would be WD Black series, but they cost much more, and would probably not get them just for storage system (price, noise, heat, extra power consumption).

Not sure if there's 3TB series already.

4 disks are definably better off in 2 x mirrored vdev (aka Raid10). However, with mirrored vdev you can't loose both disks in a single mirrored vdev. With Raid60 (raidz2) you can.

If you switch to mirrored setup, you will always only get 50% of total space for storage. With 12 disks, for home storage solution, that's a lot of wasted $$. Two raidz2 pools of 6 disks is better. YMMV!

 

[bbzz]

The Samsung and the Hitachi disk don't show the Load_Cycle_Count problem.

Don't buy WD Green Drives for FreeBSD, the Load_Cycle_Count grows very fast and renders the drive unusable within in less than a year. I guess that Samsung Green Drives would show the same problem, but the Load Cycling (Clicking) can be disabled on the Samsung drives I know of with sysutils/ataidle.

How do you disable the count? I'm looking at ataidle but can't see proper command.

smartctl -d auto -a /dev/ada1 | grep Load_Cycle_Count

gives me

 225 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       101

on all drives. Then again I haven't payed much attention to this...

Thanks

 

[hedgehog]

How do you disable the count? I'm looking at ataidle but can't see proper command.

sysutils/ataidle would help you only if your drive supports APM (Auto-power management). You could see that using the following command:

$ ataidle /dev/ad4
Model:                  WDC WD5000AADS-00S9B0
Serial:                 WD-WCAV91372288
Firmware Rev:           01.00A01
ATA revision:           ATA-8
LBA 48:                 yes
Geometry:               16383 cyls, 16 heads, 63 spt
Capacity:               465GB
SMART Supported:        yes
SMART Enabled:          yes
Write Cache Supported:  yes
Write Cache Enabled:            yes
APM Supported:          no
AAM Supported:          yes
AAM Enabled:            yes
AAM Value:              127
Vendor Recommended AAM: 1

As you can see, my drive doesn't support APM, otherwise I would adjust idle mode like that (note that you have to add atidle to /etc/rc.conf if you want to keep these changes after reboot):
# ataidle -S 120 -I 60 /dev/ad4

If your drive doesn't support APM either, then you should try wdidle3 by WD team. At your own risk! I suggest you to use google to check if anybody else succeed with wdidle3 on the specific drive! I managed to successfully disable this feature with wdidle3 and LCC has stopped growing. I used bootable usb stick with freedos+wdidle3, so if you need an image...

 

[bbzz]

Isn't the LOAD_CYCLE_COUNT referring to number of times disk has been powered up and down? What does standby mode has to do with that?
I've tried this before (drives support APM) but as soon as they spin down they spin back up. I think it has to do with zfs. I haven't been able to successfully spin them down.
Also spinning them up and down reduces their lifetime so I'm not sure how useful that is in the long run. ?

EDIT: Actually LOAD_CYCLE_COUNT is incremented every time disk is spin down.

ataidle -s ada1

But it spins up in an instant. I guess APM is useless in this case.

 

[hedgehog]

Greens can park their head after 8 seconds of inactivity (IntelliPower feature), increasing LCC. On linux/unix it happens too often descreasing performance and killing your drive. Actually I had 950k LCC over 1.5 year on one drive, it's 50k less than vendor limit.

 

[bbzz]

So... you suggest not to spin them down after all? Sorry if I'm not following...
All my drives have APM feature turned off, but like I said I don't think it matters since I wasn't able to ever spin them down successfully.