SuperMicro Storage Server

P

power

Hey everybody,

I am planning to set up a big storage server for streaming operations.

I searched around the web and found a good configuration for my needs:

Supermicro SC846TQ
  • 2 x Intel Quad-Core Xeon E5620
  • 48GB DDR3 RAM
  • 24x 2TB SATA2
  • 1x 120GB SSD

I want to run as little as possible on the server, thus I want to install the 64-Bit version of FreeBSD 9.2-RELEASE and the NGINX 1.2.9 LEGACY together with a MP4 Streaming plugin.

The average size of the stored files would be about 300MB to 2GB and the server has a 10Gbit internet connection.

My plan is the following:
  1. Installing FreeBSD on the SSD and setting up NGINX with my configuration.
  2. Using the ZFS file system for the 24x 2TB SATA2 disks with a RAID-System.

There is an uploading process (writing) to the server and a lot of streaming (random reading).

And finally the big question: which type of RAIDZ would you use to get a good balance between writing and random reading and how would you configure the ZFS pool?

Do you maybe have some tips with this type of system, on what should I look?
I’m especially focused on getting a good disks performance.

I have good knowledge of Debian and Ubuntu and how to use XFS for storage servers, but I only have basic knowledge of FreeBSD.

Thanks in advance!

Best regards,
power
 
Maybe you should think about L2ARC and ZIL SSDs for this server, but this depends on the workload.
We use the same chassis, and it got room (optional parts) for 4x 2,5" drives internal (edit: the build in optical drive could block that) , and its possible to use 2,5" adaptor for the hot swap bays.

The best random read / mixed workload performance is provided by zfs mirror, followed by raidz2 and raidz1.
But with SSD caching (meta data and data), it should be much better. Maybe you can test that scenario before production use.
 
Hey,

A ZFS mirror pool is like a RAID 1 system, right?

With that I would get the best possible I/O performance and data redundancy for reliability.

Do you think that a 128GB SSD as cache for 24TB usable storage is sufficient? And what do you think about outsourcing the operating system to a dedicated SSD and partition? Would that be a good or bad idea?

Best regards,
power
 
Hi,

Yes, ZFS mirror is ZFS RAID1, with 4 or more disks ZFS will stripe data across mirrored pairs of disks to maximise performance.

Normally on a server you don't need to put the OS on an SSD drive as there shouldn't be much I/O on a server OS disk. Exceptions might be were you expect swap to be heavily used or where you expect exceptionally high I/O to some log files.

Whether 120GB is enough for your requirements completely depends on the actual load that the system will be under, so unless you have a comparable system already built its pretty much impossible to know for sure. 120GB should at least provide good caching of metadata. If possible build the system with the possibility to add more SSDs later if required.

One thing to mention regarding using ZFS mirroring in this configuration, if you are really paranoid you would only use RAIDZ2 or RAIDZ3 as in the event of a disk failure replacing a 2TB drive will take several hours to rebuild the mirror and there is a possibility for the remaining healthy drive to fail which will take down your entire pool and you will lose all your data.

If you were to choose not to use mirroring based on this but still wanted to maximise performance then you might want to create 2 or 3 RAIDZ2 vdevs in the pool. This improves I/O performance vs having a single RAIDZ2 vdev of 24 disks as ZFS will stripe data across the different vdevs.

thanks, Andy.
 
Get 4 SSDs:
  • 2x Intel 530 (or similar) SLC-style SSD (write-optimised)
  • 2x Intel 520 (or similar) MLC-style SSD (read-optimised)

Create 2 partitions on the 320:
  1. FreeBSD install
  2. SLOG

Create 2 partitions on the 530:
  1. Swap
  2. L2ARC

Create a mirror pool using the first partition on each of the smaller SSDs, install FreeBSD onto it using ZFS-on-root. Use the other 2 SSDs for swap space (won't get used much, so doesn't need to be a lot).

Create the storage pool on the SATA drives. Add the second partition of the smaller SSDs as a mirrored SLOG, and the second partition of the larger SSDs as L2ARC.

Voila!

We do this on our storage servers. The biggest ones have 4 SSDs set up like above. The smaller ones have 2 SSDs setup similar to above, but without the SLOG.
 
You must log in or register to reply here.
Back
Top