A FreeBSD NAS is born

[aragon]

I've been slowly getting a FreeBSD NAS up and running at home and thought I'd share all that I've done so far. The spec:

• Chenbro ES34169
• Zotac H55-ITX-C-E
• Intel Core i3 550
• Thermaltake CLP0534 Slim X3
• 2x 4 GB Mushkin 1333 MHz DDR3
• 120 GB Mushkin Callisto Deluxe
• Icy Dock MB559US-1SB
• 5x 2 TB Samsung HD204UI (F4EG)
• Eaton Evolution 650 VA

This Zotac motherboard is pretty awesome for building a NAS. In total it has 7 SATA ports, including the eSATA port. The internal 6 ports are connected to Intel's ICH. The eSATA port comes in to a JMicro JMB363/360 controller which supports Port Multipliers. All ports are AHCI compatible. So far everything on the board works well in FreeBSD, including important things for me like ACPI and CPU C-states (nice story about that later).

I have not tested the included wireless card. I don't use it. I might try plug a flash disk into my vacant mini PCIe slot in future.

And what would geek pride be without pics?

 

[aragon]

It took me a bit of time to spec everything that would fit nicely together in a small case. The Thermaltake heat sink was one of those components that would make or break the spec because there is very little height available in the case for common heat sinks. In the end it looks like I'm going to be running with the case's side door removed anyway because this motherboard and CPU really do need good airflow to keep temperatures low.

 

[aragon]

Needless to say, I would be using ZFS, and after reading many discussions about RAIDZ it became clear that an odd number of drives was optimal for vdev performance. The Chenbro case only has 4 3.5" bays and 1 2.5" bay. I decided to add a 5th 3.5" bay by connecting an Icy Dock hot plug bay externally. I wanted to keep my on board eSATA port free because it's the only SATA port with port multiplier support. I made a small case mod that extends one of the internal SATA connectors to another eSATA port and extends a Molex power plug to the 4 pin DIN socket that the Icy Dock bay uses for power. Why is powered eSATA so under implemented, I wonder?

 

[aragon]

Here are all the internal bits assembled and connected up. All internal SATA ports are populated. I boot off the SSD that is split into 4 MBR partitions on 1 MiB boundaries:

• 8 GB boot
• 8 GB boot (spare, for upgrades, testing, etc.)
• 8 GB swap
• 88 GB L2ARC (overkill?)

Boot partitions are UFS.

 

[graudeejs]

Isn't Intel Core i3 550 power hungry overkill for NAS?
I'd use Atom or Via C3 or something like that...

 

[aragon]

Some more shots from the outside, and a final shot of my baby in her home in the garage - a 12U wall mount cabinet with the Eaton UPS powering her. This UPS is very cool btw - fully supported via misc/nut and interfaced via USB. The garage is the only part of the house that remains cool in summer as it's partly under ground. No aircon needed.

Notice the big gap to the right in the cabinet? In future I might add another 5 drives via the onboard eSATA port and a Lian Li EX50.

 

[aragon]

Oh, that nice story about CPU C-states on this board...

So those don't officially work unfortunately. However, Zotac were extremely helpful in making them work for me. After some mail exchanges with their support they sent me a dev BIOS that enables C-state support, and FreeBSD sees and uses up to C3. Combined with powerd(8) and mav@'s power tuning guide, this system draws very little power (10% load on my 650 VA UPS). At some point I will try measure total AC power draw with a meter and post the results if I do.

Hopefully Zotac won't mind me sharing the BIOS they sent. I don't think they'll be making it public as it has bugs that affect Core i7 and some Core i5 CPUs. Consider this totally unofficial, unsupported, and to be used at your own risk. I will host it for as long as possible.

A146PA09.ROM.gz2

Don't forget to gunzip it before flashing.

SHA256 (A146PA09.ROM) = 2975b8e662ed656b128a7356b43540fc239d8acc20c9dfa0141f0a67a84361ad

 

[aragon]

Isn't Intel Core i3 550 power hungry overkill for NAS?
I'd use Atom or Via C3 or something like that...

If C-states are supported and you set up power saving, not IMHO. This system uses close to, if not the same amount of power at idle. It also buys you a huge head room in performance when you need it. I guess it might go wasted on a dedicated NAS, but if you're using FreeBSD for your NAS, is it really going to be a dedicated NAS? Mine will be serving other functions too.

Remember an Atom doesn't do EIST. A Core i3 can scale its clock speed and core voltage by huge amounts with EIST. According to dev.cpu.0.freq_levels, this CPU draws 73 Watts at 3.2 GHz, and 16 Watts at 1.2 GHz.

 

[Terri_Kennedy]

Here are all the internal bits assembled and connected up.

Here's mine - something at the other end of the spectrum. 32TB data disks, 320GB system disks (mirrored), 48GB RAM, dual E5520 CPUs, 256GB PCI-E SSD, 8 lanes of external SAS to the tape library.

Inside the case
2 of these servers rack-mounted

 

[phoenix]

Nice. What kind of cases are those? We're currently using Chenbro cases, which take up 5U of rack space. We're looking at SuperMicro cases for the future as they only take 4U. These ones look nice.

How much space is there for redundant power in there?

 

[Terri_Kennedy]

Nice. What kind of cases are those?

They're a slightly-custom version of the CI Design NSR316. The motherboard is a SuperMicro X8DTH-iF.

How much space is there for redundant power in there?

These have redundant 820W supplies from 3Y Power (PS-YH5821-1ABR). They normally operate in load-sharing mode. It is possible to get a triple-redundant configuration from 3Y, but it is a semi-custom part and you'd need to order a minimum quantity to get them.

The power supplies support PMBus and there's a cable which plugs into a connector on the SuperMicro motherboard. After a lot of work, I've managed to poll the power supplies under FreeBSD:

PS A Status

 Item                           |                Value 
 ----                           |                ----- 
 DC 12V Output Voltage          |              12.05 V 
 DC 12V Output Current          |               16.5 A 
 Temperature 1                  |             46C/115F 
 Temperature 2                  |              28C/82F 
 Fan 1                          |             4800 RPM 
 Fan 2                          |             4800 RPM 
 DC 12V Output Power            |                195 W 
 AC Input Power                 |                235 W 
 PMBus Revision                 |               0x0011 

PS B Status

 Item                           |                Value 
 ----                           |                ----- 
 DC 12V Output Voltage          |               12.1 V 
 DC 12V Output Current          |               13.5 A 
 Temperature 1                  |             69C/156F 
 Temperature 2                  |              28C/82F 
 Fan 1                          |             4400 RPM 
 Fan 2                          |             4400 RPM 
 DC 12V Output Power            |                163 W 
 AC Input Power                 |                201 W 
 PMBus Revision                 |               0x0011

 

[phoenix]

Nice. Thanks for the info. I'll add that to our list of possible vendors going forward.