Building a small, personal NAS

wblock@ said:
There's not much point to looking for a list of bad blocks with drives made in the last two decades, which handle block remapping in firmware. Filling the drive with zeros with dd(1) has about the same effect. Of course, all of this is just an attempt to catch drives that are going to fail early, and it doesn't always work. I had a WD Red 1T that showed no bad blocks, yet had a severe error that prevented it from being used in a mirror and failed the short or long SMART tests from sysutils/smartmontools instantly.

Well, I ran a long SMART test on each drive before the badblocks tests with no errors reported. I figure writing a bunch of random data and reading it back might be better than just zeros. It's probably all the same, but I'm fine with this taking longer.


wblock@ said:
That doesn't sound good. Don't know what would cause it, though.
From what I've seen so far, it might be a hostname thing, but it was giving me problems even before I started using bash (which puts the hostname in the prompt) and it doesn't explain why it would hang after user name before password during a login attempt. I'm still going to keep an eye out, but it hasn't been a problem since the first day. It's possible that it has to do with running from a USB flash drive, but so does the previous server which never displayed this issue.


wblock@ said:
benchmarks/bonnie++ for a hard to read but mostly-realistic result, diskinfo -tv for a quick and absolute best-case result.
Thanks! I'll look into both.


wblock@ said:
It looks like that's either for ZFS-to-ZFS (zfs send/receive), or just uses net/rsync. Which can be done manually, but maybe it knows the best combination of options.

It looks like a wrapper for rsync with all the relevant options pre-selected. I suppose a ZFS send/receive would be the proper method, but I am more familiar with rsync, which gave it some appeal.

The nice thing about FreeBSD and ZFS is that you can pretty much set things up once and with a little automation you don't have to do any of these tasks for a while. But then I forget how to do them and need to relearn each time.
 
So here's something slightly odd. As I'm doing the badblocks passes (just to test the drives, not for marking bad sectors) I can see that one drive finished hours ahead of the others, three are pretty closely timed and then one is hours behind the pack. Seems strange. The only correlation I have is that the fast drive is alone on a breakout cable from the second SFF-8087 port. I would figure that this wouldn't matter and each of the 8 ports on the controller card would have separate data lanes. When they all finish, I'll be able to do some speed tests to get some hard numbers.
 
So in order to get a baseline of my hard drive performance before putting the drives in a RAIDZ, I tested each drive individually using benchmarks/bonnie++. Before running the tests, I formatted and mounted them as such:
Code:
# dd if=/dev/zero of=/dev/da0 count=1k
# gpart create -s gpt da0
# gpart add -a 1m -t freebsd-ufs -l disk0 da0
# newfs -U -f 4096 /dev/da0p1
# mount -t ufs /dev/da0p1 /mnt.0

And here's the output of bonnie++ with each command being in the form of $ bonnie++ -m disk0 -d /mnt.0
bonnie-individual_drives.jpg


So it looks like disk performance is on the order of 135-145MB/s. That seems okay to me. I've seen some reviews with HDTune getting just over 150MB/s out of these, but that might not be as realistic of a result. Unless I'm missing something.
 
bonnie++ is going through the filesystem. Don't know about HDTune, but a lot of benchmarks just read or write blocks on the drive. Try diskinfo -tv ada0.
 
wblock@ said:
bonnie++ is going through the filesystem. Don't know about HDTune, but a lot of benchmarks just read or write blocks on the drive. Try diskinfo -tv ada0.

I ran diskinfo first, actually, just to get a rough idea. And the results are on par with what bonnie++ showed.

Here's some trimmed output:
Code:
#  diskinfo -vt /dev/da0
/dev/da0
        512             # sectorsize
        3000592982016   # mediasize in bytes (2.7T)
        5860533168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        364801          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
             WD-WMC1T0989515    # Disk ident.

Seek times:
        Full stroke:      250 iter in   8.106308 sec =   32.425 msec
        Half stroke:      250 iter in   5.677708 sec =   22.711 msec
        Quarter stroke:   500 iter in   9.435993 sec =   18.872 msec
        Short forward:    400 iter in   3.139615 sec =    7.849 msec
        Short backward:   400 iter in   2.137566 sec =    5.344 msec
        Seq outer:       2048 iter in   0.168892 sec =    0.082 msec
        Seq inner:       2048 iter in   0.159148 sec =    0.078 msec
Transfer rates:
        outside:       102400 kbytes in   0.797209 sec =   128448 kbytes/sec
        middle:        102400 kbytes in   0.820908 sec =   124740 kbytes/sec
        inside:        102400 kbytes in   1.487073 sec =    68860 kbytes/sec

# diskinfo -vt /dev/da1
...
Transfer rates:
        outside:       102400 kbytes in   0.769086 sec =   133145 kbytes/sec
        middle:        102400 kbytes in   0.871308 sec =   117524 kbytes/sec
        inside:        102400 kbytes in   1.414511 sec =    72393 kbytes/sec


# diskinfo -vt /dev/da2
...
Transfer rates:
        outside:       102400 kbytes in   0.692566 sec =   147856 kbytes/sec
        middle:        102400 kbytes in   0.893075 sec =   114660 kbytes/sec
        inside:        102400 kbytes in   1.587866 sec =    64489 kbytes/sec

# diskinfo -vt /dev/da3
...
Transfer rates:
        outside:       102400 kbytes in   0.802362 sec =   127623 kbytes/sec
        middle:        102400 kbytes in   0.843734 sec =   121365 kbytes/sec
        inside:        102400 kbytes in   1.547642 sec =    66165 kbytes/sec


# diskinfo -vt /dev/da4
...
Transfer rates:
        outside:       102400 kbytes in   0.715039 sec =   143209 kbytes/sec
        middle:        102400 kbytes in   0.816365 sec =   125434 kbytes/sec
        inside:        102400 kbytes in   1.467947 sec =    69757 kbytes/sec

Any idea what might explain the 20MB/s difference between some of the drives?
 
cbunn said:
Any idea what might explain the 20MB/s difference between some of the drives?

I remember seeing some variations between the last few drives I set up, but can't remember what magnitude. Some of the WD drives claim to be variable speed, some are 5400, some are 7200. But those may be nominal speeds, maybe the motors aren't locked to an accurate speed. So a slightly faster motor would get the data past the heads faster. The only other thing I can think of is a variation in actuator arm mass or magnet strength, which would make for different seek times. Seems like the manufacturing tolerances would be tight enough to minimize that.

It would be nice to have a contact at one of the drive manufacturers to ask about this type of thing.
 
wblock@ said:
Some of the WD drives claim to be variable speed, some are 5400, some are 7200.
I don't think WD ever claimed variable spindle speed - that's something that some early reviewers came up with when they saw "IntelliPower" in the marketing literature. I don't know why WD persists in trying to hide the spindle speed - it can be inferred from benchmarks, and can be determined exactly by putting a frequency counter on the spindle motor leads, which are exposed on the underside of the drive.

But those may be nominal speeds, maybe the motors aren't locked to an accurate speed. So a slightly faster motor would get the data past the heads faster. The only other thing I can think of is a variation in actuator arm mass or magnet strength, which would make for different seek times. Seems like the manufacturing tolerances would be tight enough to minimize that.
Both the spindle and actuator motors are multi-phase, so they operate at exactly the speed the logic tells them to. Having more than a faction of a percent of "slop" would make the data separator design and servo stuff incredibly complex, not to mention data corruption from write splice errors.

It would be nice to have a contact at one of the drive manufacturers to ask about this type of thing.
If there's a specific WD model / firmware that exhibits large variation between drives, I can ask my contact at WD about it. My initial guess is that the drives are not identical - they either have a different model suffix (on WD drives, the -xxxxxx part of WDmmmmoooo-xxxxxx) or different firmware. At various times, WD has changed the bit density on a drive model in order to reduce the number of platters / heads. There have also been cases where the zone layout was changed to improve performance.

It's also possible that the drives are performing some sort of internal error recovery. Some vendors (not to name names, but I'm talking about you, N****g) feel that putting a naked drive with only an anti-static bag for protection in a large box and throwing in a little packing material is adequate "protection" during shipping. This should show up on something like HD Tune's access time graph.
 
Terry_Kennedy said:
If there's a specific WD model / firmware that exhibits large variation between drives, I can ask my contact at WD about it. My initial guess is that the drives are not identical - they either have a different model suffix (on WD drives, the -xxxxxx part of WDmmmmoooo-xxxxxx) or different firmware. At various times, WD has changed the bit density on a drive model in order to reduce the number of platters / heads. There have also been cases where the zone layout was changed to improve performance.

Well, it would seem that all five of my drives have exactly the same model suffix:

Code:
$ for i in 0 1 2 3 4; do sudo smartctl -a /dev/da$i |grep "Device Model"; done
Device Model:     WDC WD30EFRX-68AX9N0
Device Model:     WDC WD30EFRX-68AX9N0
Device Model:     WDC WD30EFRX-68AX9N0
Device Model:     WDC WD30EFRX-68AX9N0
Device Model:     WDC WD30EFRX-68AX9N0

Terry_Kennedy said:
It's also possible that the drives are performing some sort of internal error recovery. Some vendors (not to name names, but I'm talking about you, N****g) feel that putting a naked drive with only an anti-static bag for protection in a large box and throwing in a little packing material is adequate "protection" during shipping. This should show up on something like HD Tune's access time graph.

I ordered my drives from Amazon and they were properly packed. Each drive was protected by the molded plastic end caps within individual cardboard boxes, lined up inside a larger box with airbags to pad any extra space.

HD Tune is a Windows program only, right? I figured diskinfo and bonnie++ would have been the equivalent of that. Is access time the same as seek time? diskinfo reports that data. Here's the full output of diskinfo for each of my drives, in case it helps:

Code:
$ sudo diskinfo -vt /dev/da0
/dev/da0
        512             # sectorsize
        3000592982016   # mediasize in bytes (2.7T)
        5860533168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        364801          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
             WD-WMC1T0989515    # Disk ident.

Seek times:
        Full stroke:      250 iter in   8.106308 sec =   32.425 msec
        Half stroke:      250 iter in   5.677708 sec =   22.711 msec
        Quarter stroke:   500 iter in   9.435993 sec =   18.872 msec
        Short forward:    400 iter in   3.139615 sec =    7.849 msec
        Short backward:   400 iter in   2.137566 sec =    5.344 msec
        Seq outer:       2048 iter in   0.168892 sec =    0.082 msec
        Seq inner:       2048 iter in   0.159148 sec =    0.078 msec
Transfer rates:
        outside:       102400 kbytes in   0.797209 sec =   128448 kbytes/sec
        middle:        102400 kbytes in   0.820908 sec =   124740 kbytes/sec
        inside:        102400 kbytes in   1.487073 sec =    68860 kbytes/sec

$ sudo diskinfo -vt /dev/da1
/dev/da1
        512             # sectorsize
        3000592982016   # mediasize in bytes (2.7T)
        5860533168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        364801          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
             WD-WMC1T0988655    # Disk ident.

Seek times:
        Full stroke:      250 iter in   8.121221 sec =   32.485 msec
        Half stroke:      250 iter in   5.774391 sec =   23.098 msec
        Quarter stroke:   500 iter in   9.527913 sec =   19.056 msec
        Short forward:    400 iter in   2.549826 sec =    6.375 msec
        Short backward:   400 iter in   2.405381 sec =    6.013 msec
        Seq outer:       2048 iter in   0.203302 sec =    0.099 msec
        Seq inner:       2048 iter in   0.155567 sec =    0.076 msec
Transfer rates:
        outside:       102400 kbytes in   0.769086 sec =   133145 kbytes/sec
        middle:        102400 kbytes in   0.871308 sec =   117524 kbytes/sec
        inside:        102400 kbytes in   1.414511 sec =    72393 kbytes/sec

$ sudo diskinfo -vt /dev/da2
/dev/da2
        512             # sectorsize
        3000592982016   # mediasize in bytes (2.7T)
        5860533168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        364801          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
             WD-WMC1T1013896    # Disk ident.

Seek times:
        Full stroke:      250 iter in   8.238028 sec =   32.952 msec
        Half stroke:      250 iter in   5.857010 sec =   23.428 msec

Seek times:
        Full stroke:      250 iter in   8.238028 sec =   32.952 msec
Seek times:
        Full stroke:      250 iter in   8.238028 sec =   32.952 msec
        Half stroke:      250 iter in   5.857010 sec =   23.428 msec
        Quarter stroke:   500 iter in   9.475112 sec =   18.950 msec
        Short forward:    400 iter in   3.251636 sec =    8.129 msec
        Short backward:   400 iter in   2.026576 sec =    5.066 msec
        Seq outer:       2048 iter in   0.196214 sec =    0.096 msec
        Seq inner:       2048 iter in   0.151343 sec =    0.074 msec
Transfer rates:
        outside:       102400 kbytes in   0.692566 sec =   147856 kbytes/sec
        middle:        102400 kbytes in   0.893075 sec =   114660 kbytes/sec
        inside:        102400 kbytes in   1.587866 sec =    64489 kbytes/sec

$ sudo diskinfo -vt /dev/da3
/dev/da3
        512             # sectorsize
        3000592982016   # mediasize in bytes (2.7T)
        5860533168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        364801          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
             WD-WMC1T1025747    # Disk ident.

Seek times:
        Full stroke:      250 iter in   8.027712 sec =   32.111 msec
        Half stroke:      250 iter in   5.888577 sec =   23.554 msec
        Quarter stroke:   500 iter in   9.404656 sec =   18.809 msec
        Short forward:    400 iter in   3.107148 sec =    7.768 msec
        Short backward:   400 iter in   3.314666 sec =    8.287 msec
        Seq outer:       2048 iter in   0.179241 sec =    0.088 msec
        Seq inner:       2048 iter in   0.154015 sec =    0.075 msec
Transfer rates:
        outside:       102400 kbytes in   0.802362 sec =   127623 kbytes/sec
        middle:        102400 kbytes in   0.843734 sec =   121365 kbytes/sec
        inside:        102400 kbytes in   1.547642 sec =    66165 kbytes/sec

$ sudo diskinfo -vt /dev/da4
/dev/da4
        512             # sectorsize
        3000592982016   # mediasize in bytes (2.7T)
        5860533168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        364801          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
             WD-WMC1T1096918    # Disk ident.

Seek times:
        Full stroke:      250 iter in   8.036516 sec =   32.146 msec
        Half stroke:      250 iter in   5.829626 sec =   23.319 msec
        Quarter stroke:   500 iter in   9.222203 sec =   18.444 msec
        Short forward:    400 iter in   3.612478 sec =    9.031 msec
        Short backward:   400 iter in   2.545658 sec =    6.364 msec
        Seq outer:       2048 iter in   0.199791 sec =    0.098 msec
        Seq inner:       2048 iter in   0.147488 sec =    0.072 msec
Transfer rates:
        outside:       102400 kbytes in   0.715039 sec =   143209 kbytes/sec
        middle:        102400 kbytes in   0.816365 sec =   125434 kbytes/sec
        inside:        102400 kbytes in   1.467947 sec =    69757 kbytes/sec
 
cbunn said:
Well, it would seem that all five of my drives have exactly the same model suffix:
Agreed. What does smartctl show for the firmware version? You have a slightly larger spread of serial numbers than I'd normally expect for a bunch of drives ordered at the same time. If you have the drives out of the enclosure for any reason, you might want to also check the manufacture dates. I get 11/29/2012 for 4 of them and 12/05/2012 for one of them, but that may not be entirely accurate (if you registered them on the WD site, that's the date of registration, not the date of manufacture).

I ordered my drives from Amazon and they were properly packed. Each drive was protected by the molded plastic end caps within individual cardboard boxes, lined up inside a larger box with airbags to pad any extra space.
That's standard retail packaging, which is what I'd expect from Amazon. The problem happens when some other vendors buy 20-pack cartons (which look like this) and then resell the individual drives. In that case, there's no unit piece factory packaging.

HD Tune is a Windows program only, right? I figured diskinfo and bonnie++ would have been the equivalent of that.
Yes, it is Windows only (unfortunately). The advantage it has is that it times how long it takes to read each sector, and can thus provide a nice graph which will indicate any "slow" spots on the drive.
 
Terry_Kennedy said:
What does smartctl show for the firmware version?

Code:
$ for i in 0 1 2 3 4; do sudo smartctl -a /dev/da$i |grep "Firmware Version"; done
Firmware Version: 80.00A80
Firmware Version: 80.00A80
Firmware Version: 80.00A80
Firmware Version: 80.00A80
Firmware Version: 80.00A80

Terry_Kennedy said:
You have a slightly larger spread of serial numbers than I'd normally expect for a bunch of drives ordered at the same time. If you have the drives out of the enclosure for any reason, you might want to also check the manufacture dates. I get 11/29/2012 for 4 of them and 12/05/2012 for one of them, but that may not be entirely accurate (if you registered them on the WD site, that's the date of registration, not the date of manufacture).
I'll give a look next time I'm in there. How did you get those dates, though? Some kind of code hidden in the serial numbers?


Terry_Kennedy said:
That's standard retail packaging, which is what I'd expect from Amazon. The problem happens when some other vendors buy 20-pack cartons (which look like this) and then resell the individual drives. In that case, there's no unit piece factory packaging.
I've definitely been down that road. I love the Egg, but they really drop the ball when it comes to packing some things.


Terry_Kennedy said:
Yes, it is Windows only (unfortunately). The advantage it has is that it times how long it takes to read each sector, and can thus provide a nice graph which will indicate any "slow" spots on the drive.
Nifty. But I don't see any way to pull that off with my set-up, I'm sad to say.
 
cbunn said:
Code:
$ for i in 0 1 2 3 4; do sudo smartctl -a /dev/da$i |grep "Firmware Version"; done
Firmware Version: 80.00A80
Firmware Version: 80.00A80
Firmware Version: 80.00A80
Firmware Version: 80.00A80
Firmware Version: 80.00A80
Well, they're all the same - I don't have any idea why they would perform differently, unless some of them were behind a port expander (which could be on the motherboard).

I'll give a look next time I'm in there. How did you get those dates, though? Some kind of code hidden in the serial numbers?
Nope, just looked up the warranty end dates on wdc.com and subtracted 3 years.
 
Terry_Kennedy said:
Well, they're all the same - I don't have any idea why they would perform differently, unless some of them were behind a port expander (which could be on the motherboard).
They're all on a Supermicro AOC-USAS2-L8i card. No port expanders.

Terry_Kennedy said:
Nope, just looked up the warranty end dates on wdc.com and subtracted 3 years.
Oh. Well know I that one can do that.
 
So I've setup the RAIDZ zpool on the five drives using the instructions found on these forums and elsewhere using the temporary gnop definitions to force ZFS into using 4k sectors. To get an idea of speeds, I transferred the 9.1-Release memstick image (733MB) from the gmirrored USB flash drives to the new zpool within the time command.
Code:
$ time cp FreeBSD-9.1-RELEASE-amd64-memstick.img /storage/tmp/

real    0m0.585s
user    0m0.000s
sys     0m0.467s

But that implies that the transfer was at 1.2GB/s. That can't be right. So I did a few runs of bonnie++ to get a better idea of performance.

bonnie-raidz.jpg


This shows an average write speed of 352MB/s and an average read of 326MB/s. Over the three instances, the writes were very close together, but the three reads had a lot of variation. I'm not sure what that's about.
 
Motherboard seems to be out of stock in the couple of places I've looked. Is there another ITX C20* server option you folks know of?
 
Rubel said:
Motherboard seems to be out of stock in the couple of places I've looked. Is there another ITX C20* server option you folks know of?

Well, there is an update to the S1200KP named the S1200KPR. The main difference is that the KPR board supports Ivy Bridge CPUs out of the box (there are reports that upgrading the BIOS on the KP can allow the same). Provantage has it for $142.33, which is a pretty good price. I'm not sure what they charge for shipping.

Apart from those two boards, I've not seen any others that support an Intel CPU with ECC memory. And I searched quite a bit.
 
Thanks for the info! After some further reading, I've ordered a S1200KRP board and three WD Red drives for my own smaller version of what you've built. I wonder if I can skip the SAS controller and just use the on-board SATA ports plus a Rocker 620A 2 port card I have laying around. Although I don't see that card listed in the hardware compatibility section of the handbook, but google does list it working with some people's installations. If it develops problems and I have to move to another controller, I can just zfs export then zfs import my pool, right?
 
Rubel said:
Thanks for the info! After some further reading, I've ordered a S1200KRP board and three WD Red drives for my own smaller version of what you've built. I wonder if I can skip the SAS controller and just use the on-board SATA ports plus a Rocker 620A 2 port card I have laying around. Although I don't see that card listed in the hardware compatibility section of the handbook, but google does list it working with some people's installations. If it develops problems and I have to move to another controller, I can just zfs export then zfs import my pool, right?

The motherboard has four SATA ports. Other than the three Red drives, what else will you be attaching?

I don't know a whole lot about that controller card, so I can't comment on its compatibility. But if you do get it set up and then you decide to replace it, a ZFS export/import should work fine.
 
cbunn said:
The motherboard has four SATA ports. Other than the three Red drives, what else will you be attaching?
Oh, I just wanted everything to be on SATA III--two of the ports are SATA II, right? But wait, didn't I read somewhere that only SSDs even run fast enough to keep up with SATA III? Maybe I can forget the card altogether and just leave a SATA III port free for a potential future ZIL SSD.

Anyhow, once it's put together I can post some bonnie++ tests to compare to yours if that's interesting to anyone.
 
Rubel said:
Oh, I just wanted everything to be on SATA III--two of the ports are SATA II, right? But wait, didn't I read somewhere that only SSDs even run fast enough to keep up with SATA III? Maybe I can forget the card altogether and just leave a SATA III port free for a potential future ZIL SSD.

SATA II is good for a theoretical 300M/second. That's far above anything a standard hard drive does.

Anyhow, once it's put together I can post some bonnie++ tests to compare to yours if that's interesting to anyone.

Please do.
 
Oops, looks like I ordered registered RAM instead of unbuffered. So much for getting it running this weekend.
 
cbunn said:
Well, there is an update to the S1200KP named the S1200KPR. The main difference is that the KPR board supports Ivy Bridge CPUs out of the box (there are reports that upgrading the BIOS on the KP can allow the same). Provantage has it for $142.33, which is a pretty good price. I'm not sure what they charge for shipping.

Apart from those two boards, I've not seen any others that support an Intel CPU with ECC memory. And I searched quite a bit.

Supermicro boards like the X9SAE support current Intel (non Xeon) CPUs and ECC Ram. While they are a bit more expensive, they have 4 DIMM sockets, 6 SATA ports and more PCIe and PCI slots.
 
c_geier said:
Supermicro boards like the X9SAE support current Intel (non Xeon) CPUs and ECC Ram. While they are a bit more expensive, they have 4 DIMM sockets, 6 SATA ports and more PCIe and PCI slots.

Rubel was asking about Mini-ITX boards, which narrows the selection down considerably.

c_geier said:
@cbunn: Do you have a power meter and can measure idle power usage? I would be really interested in that.

Using my trusty Kill-A-Watt, I get about 50W at the outlet when idle. When fully loaded, it gets close to 70W. Not bad, I think.
 
Back
Top