Advice on hardware for a Personal Desktop Build with a focus on Data Storage (that I can also use as a server for home)

Z

zostj

Goal:
- A decently powerful Personal Desktop with a focus on datahoarding/data storage.
- Case capable of holding 14-18 HDDs (something like Fractal Design Define 7 XL perhaps?)
- ECC RAM for ZFS (with lots of drives - 18 if maxed out somewhere down the road. starting with 6)
- Good enough GPU to drive two monitors: LG DualUp 2560 x 2880 with a 34" 1440p 3440×1440 ultrawide that will also be useful for photo editing via software such as darktable, rawtherapee, etc.
- Power Supply that can supply a system with a max capacity of 18 HDDs
- Fast SSD
- MOBO: ?
- CPU: ?
- HDD: Are Seagate Exos ok? Or will they be too loud in a personal desktop?
- Not really sure what CPU or the amount of ram. But I would like to be able to use the personal desktop also as a server, running some VMs, self-hosted services (just for home use). If I want NAS/Media Server functionalities then I will also need a good GPU? (to be honest, plex isn't really a necessity at the moment as I consume most of my media from my desktop).

I don't really have any experience in regards to server, nas, media server (while at the same time, being relatively new to FreeBSD) so I apologize for not being 100% clear on what I want in regards to that department. Basically a decent personal desktop with a focus on data storage with lots of drive (14-18) that is powerful and flexible enough to be used as a personal homeserver for file sharing, running syncthing, linux VMs, and some other homelab features down the road.
 
For similar purposes I use a dual Xeon v4. One advantage is that it takes registered ECC DDR44, which is dirt cheap. And of course it has plenty of PCIe slots and the lanes to feed it.

I also recommend preparing for SAS drives as those can often be had cheaper than SATA.
 
A large number of drives requires a quality disk controller, as most system boards stop at six SATA connections.
My first NAS is a 6x4 tb Z2 configuration, and I'm out of space.
I boot from an embedded USB stick so I have all six SATA available.

My next NAS is three 18tb Ironwolf as Z1.

I was gifted with several Supermicro E5540 Xeon throw-aways, so I also use ECC
 
As long as FBSD fully supports those controllers, you are golden.
I had a big stock of SATA-III disks at the time.
 
Just buy a brand new or used server/workstation that meets most of your requirements.
There are servers in a tower-type chassis.
By defaults servers will have ECC memory, a lot of HDD-racks, proper cooling, SATA/SAS backplane to avoid huge amount of cables, premium level of quality, long-time support, etc.
In my opinion, homemade storage system with 6-18 HDDs will make a lot of issues because of a lack of your experience.
 
I would also look at motherboard specs as well. i have E5-2690v4 and Asus Z10PE-D16 WS biggest issue i had is PCIE slots and they are not bifurcated so i had to do some tweaking to bios to bifurcate them and have x4x4x4x4 instead of x16 as i use Asus raid card with 4 nvmes and if left x16 i could not have 4 separate drives. Maybe its important for you so just a warning as not all mb`s have this option to choose.
I
 
A personal desktop with 14-18 HDDs is going to require serious attention to cooling. You must have a fan blowing directly onto each disk. And the combination of disks and fans is going to be noisy, regardless of the disks you choose.

Fractal Design make well designed cases, with truly excellent cooling options and good sound deadening ("bituminous" lining). I know because I have a Fractal Design Define R5 for my ZFS server. It' not nearly as bad as its predecessors, but still too noisy for my liking. You do get used to it...

I use Noctua fans, and I'm half way through replacing my old (pre-shingle) 3 TB WD Reds with 4 TB Seagate EXOS drives. I have one of each in mirror formation (a size and performance mis-match, but the Reds are old enough to fail soon).

I have found both types of drive to be satisfactory, even though their market niche and price points are very different. All my EXOS drives have genuine 512 byte sectors. Not sure if the new ones do.

What ever sort of drives you choose, I suggest that you get multiple controllers. I have two 4-port on-board SATA controllers, and two 8-port PCIe SATA controllers. I use only (striped) mirrors. Each side of a mirror is connected to a different controller. I think that's safer (and certainly faster at writing) than any RAID-Z option using a single controller.

I suggest that you think about your ZFS design. In particular, consider intent logs (ZIL), Cache (L2ARC), and special VDEVs. These (mostly) need to be placed on fast reliable media (i.e. "enterprise class" SSDs with genuine power loss protection).

The time to think about your backup strategy is now, because you are going to have a lot to backup. I use high capacity SATA disks rotated through a hot swap bay (which occupies the space required for two 5" DVD players, and uses one of the on-board SATA controllers).

A system of that size justifies a UPS.
 
richardtoohey2 said:
If money is no object, why not two systems - one for the serious storage, and one for the serious desktop?
Click to expand...
I've been struggling and going back and forth between various options. 😂

- From the initial idea of building a computer and using it as both a Personal Desktop and Server. (But worried about buying/using decade old enterprise hardware and somewhat feeling it's not much of an upgrade since my current old desktop's hardware are also from around 2012).

- To just building a modern Personal Desktop and forgetting about the server and just using a USB3 Docker for external HDD and dumping new contents onto it from the internal drive and removing the external HDD for safe storage; wiping the internal drive clean to make more room for stuff and dumping it again to a new external HDD.
(Which is really safe for the data and I get to enjoy a new modern PC, but the tedious nature and not being able to constantly organize and conveniently access the data is a pain.)

- To my final decision of keeping this old Personal Desktop and just building the data storage/server.
 
cracauer@ said:
For similar purposes I use a dual Xeon v4. One advantage is that it takes registered ECC DDR44, which is dirt cheap. And of course it has plenty of PCIe slots and the lanes to feed it.

I also recommend preparing for SAS drives as those can often be had cheaper than SATA.
Click to expand...
THIS.

I also run a v4 server at home (supermicro 6029-TR4T /w 2x E5-2660 v4), although I completely switched from 12 SAS&SATA HDDs to only SSD storage (6xSAS and 6xNVMe) ~2 years ago, almost halving its power consumption.
Not sure about desktop/tower variants, but as 2U servers those systems are available dirt-cheap nowadays. Although I'd always go for a SFF variant today instead of LFF; but if one still relies primarily on spinning rust, LFF might still be interesting (and cheapest by the TB, yet by several magnitudes slower and usually more power hungry).
Most platforms will take v3 and v4 CPUs and v3 CPUs are much cheaper, but one should always go for v4 as they have better power efficiency and C-states support.

LSI SAS9300 based HBAs are available very cheaply and well supported - if NVMe is of interest (and the backplane supports it) the 9400 are the way to go, as they also have become dirt cheap. Those 'white box' reference design controllers from china are perfectly fine for a home server - never had any issues with those and I've used them since the SAS2008/9200 generation. Same goes for X520/710 or mellanox CX3 NICs - the 10GBps cards can be found for as low as 20EUR/$ nowadays and even 100G NICs can be found at around 100EUR/$.

Regarding the usage as desktop AND server, I'd strictly advice against that. Those use cases have vastly different implications for the general setup, security and user permissions etc pp. Usually you also don't want to run the latest release and/or 'latest' pkg repositories on a server; yet on a desktop it might make sense to use them.

Also with 18HDDs you will need proper PSUs and power distribution (i.e. a backplane) - a typical multi-rail ATX PSU as found in a desktop PC won't be able to handle the load of all those drives spinning up at boot, let alone the flimsy molex plugs/adapters (possibly even cascaded to make things worse...). You definitely want a proper server platform with a backplane for such a system. This also opens up the option to use a backplane with an expander, so a 8-channel HBA will be sufficient to connect all disks.
 
I dunno. I run 15 HDs + tape + optical drive from a high quality 850W desktop PSU. No problems apart from cooling all the drives.

Depending on how ugly it can look you can also put drive cages on the floor or shelf next to a small case that holds the mainboard. Google used to run off shelves that way.

I use these cages, but with additional fanning. They are SAS but work for SATA.
 
I have similar drive cages. Have used a whole stack of different ones over the years. I generally try to buy cages that have a fan on the outside of the case. That makes it easier to replace the fan, which you will inevitably have to do after some time.

I've got a 8 x 2.5 inch hotswap in a 2 x 5.24 bay, a 4 x 3.5 inch in 3 x 5.25 bay, a 3 x 3.5 inch in 2 x 5.25 bay. And recently bought a 16 x 2.5 inch in 2 x 5.25 bay. Still need to put it in, haven't tested it yet.
 
I've replaced mine with proper 3 pin ones, and connected to a fan controller. The builtin fan controller on these cages often only have two modes, loud and louder.
 
W.r.t. mainboards, if you get a consumer "desktop" mainboard, get a board with onboard video and 2 or more PCIe x16 slots. Those two PCIe x16 slots can easily handle PCIe x8 cards, most of the LSI SAS/SATA HBAs will need a PCIe x8 slot, as do some 10Gbit network cards.
 
Downside of most (all) consumer/desktop boards: no PCIe bifurcation. I.e. with those boards that only come with 2 x16 slots you will waste all available expansion slots with only 2 cards, no matter if they are only x8 or even x4...
Using NVMe on M.2 carriers or U.2 adapter cards is also limited to either a single drive per slot or you have to get (very expensive) retimer-cards or fully-fledged HBAs.

Regarding 5.25" drive cages:
I also used some icy-dock 4x/5x 3.5" cages once. They might even still lay around on the attic somewhere... With SATA they mostly worked, but with SAS I had more and more incompatibilities over time and retired the whole system because it became annoying (and old...). HBAs were LSI SAS2008 back then, cabling was a mess with all those fanout-cables and caused problems more than once (despite good quality cables with locking taps).
And yes, the PSU could handle the drives (it was a single-rail FSP, IIRC something around 600W), but the cages with only 2 molex connectors seemed to struggle at times, so I had to use staggered spinup to bring up all drives reliably.
TBH I would never want to go back to such a hacky solution - especially once one got accustomed to sesutil(8) and the simplicity of having a single backplane (/w expander) and just 2 cables running from the HBA to that. This also increases reliability *a lot*.

I still use a 4x2.5" SATA cage and dual M.2 NVMe cage in my workstation @work; but those are not used for critical drives and usually unpopulated...
 
I rewire MOLEX connector fans and switch the 12v and 5v pairs, usually in a separate adapter for easy fan replacement.
Very, very few fans fail to start and run at 5v.
Been doing this for decades.

This cuts way back on the fan noise, as well as radically increases fan life.
Sleeve bearing fans will prematurely fail when the fan axis is vertical.
If you cannot run horizontal, then dual-ball bearing fans are required.

In my opinion, combining a data server with a GUI desktop is a fools errand.
The last thing you ever want is your data server used as a web surfer.

On your data server, create backup directories with a unique security ID that cannot be reached by the desktop admin account.
This will keep the malware encrypters out of your backups and may be the only safe guard you have for your online data backups.
 
zostj said:
- Case capable of holding 14-18 HDDs (something like Fractal Design Define 7 XL perhaps?)
Click to expand...
Given modern disk sizes, we're talking about 200-300 TB after applying RAID overheads (for example two groups of 7-9 disks each with a RaidZ2). Given that data volume, I suspect that the purpose of this is archival storage. Do you think you will actually be accessing these disks at high bandwidth? With the correct hardware (lots of PCI lanes, lots of good HBAs) you could probably get over a GByte/s out of these disks, but most people are not capable of processing that much data, without a compute cluster. Also, if you really run with 14-18 disk drives, you have to come up with a solution for monitoring drive health and temperatures, disk replacement, and disk identification and labeling: when one disk fails (which in your system will be somewhat frequently), how will you pull it out and replace it without endangering the rest of the system?

I also wonder whether the amount of data storage you are talking about is realistic. I'm a pretty bad pack rat, and my "archival" file system has a quarter million files on it: photos, scanned PDF documents (I try to be paperless). That is a little less than 1TB; you are proposing to build a 200TB system. Are you sure you need that? How are you planning to back it up (because storage that is not backed up is just temporary). If in reality your space requirements are more reasonable (like a few TB or a dozen TB), I would instead buy two disk drives, both 16-20TB each, and run them as a mirror. Take all the money you save (especially the money used for electricity) to pay for a good backup strategy.

In my area (California), each disk drive costs about $25-50 per year to operate if you are buying electricity at consumer rates. There is a reason most data centers are in regions where electricity is very cheap, and they buy electricity in giant contracts with steep discounts. Anything you can do to reduce the number of disks will pay off.

If you really need this many disks, and they are 90% or 99% for archival storage with low bandwidth requirements, may I suggest physically separating this into two systems, one on the desktop (with just a boot SSD or two, and perhaps a disk or two for data that is actually intensely), and another that is kept separate, in a basement or similar cool room where noise doesn't matter? Here's why: as others already said, a system that can run about a dozen or two dozen disks will be very very noisy, with lots of cooling fans, and will generate significant heat. Not something most people want under their desk, unless they happen to be deaf and feel cold.

HDD: Are Seagate Exos ok? Or will they be too loud in a personal desktop?
Click to expand...
One or two would be fine. 14 or 18 will not be fine. Seagate is a perfectly acceptable brand, although I personally would always choose Hitachi (now a division of WD) for reliable enterprise-grade disks.
 
Over the years, the most disk failures I get in consumer machines are Seagates.
Very few WD disks fail.
YMMV.

In the commercial world, backblaze.com rates the Seagates above the WD.
I assume these are the Ironwolf and not the Exos (desktop) series.

I picked up a pair of 18tb Ironwolf on a firesale deal, along with an 18tb WD Red.
Seagate puts a 5 year warranty behind the Ironwolf, which gives me much more confidence over their Exos line.
The Ironwolf also has a higher duty cycle than the WD.
 
cracauer@ said:
For similar purposes I use a dual Xeon v4. One advantage is that it takes registered ECC DDR44, which is dirt cheap. And of course it has plenty of PCIe slots and the lanes to feed it.

I also recommend preparing for SAS drives as those can often be had cheaper than SATA.
Click to expand...

Being a desktop guy with no prior experience with enterprise hardware and servers, I was initially a little hesitant. However, after having done some research, it seems the old Intel Xeon E5 v4 cpu chips are still very favored by the home lab enthusiasts and considered a real workhorse in their homeserver arsenal.

- Which dual chip combo would you recommend?
- Which motherboard would you recommend? (what brands to go for)?
- Still trying to work out what size drives and the drive amount, but what are some reputable brands to shoot for?

https://www.cpu-monkey.com/en/cpu_group-intel_xeon_e5_v4
The Intel Xeon E5-2667 v4 seems to be pretty popular and has moderate amount of cores/threads, with a TDP of 135W. And with a dual cpu setup, that's 270W power consumption or the amount of heat that will dissipate from the cpu (under maxmium workload). I'm not really going to be doing anything too intensive on the machine, some torrenting and using the server for file sharin (maybe setting up vms down the road), so while the combined cpu TDP is at around 270W, under normal operation it'll probably be around 90-120 maybe? I think a lot of the modern CPU have a TDP of about 90-95W right? So the downside to using the old Intel Xeon E5 v4 is that it is a bit more power hungry compared to modern cpu? W
- Will I need some kind of cpu cooler?

But ya like you mentioned, I checked the prices on ECC DDR4 Registered Memory and they are indeed ridiculously cheap for the amount of memory I can add to the system.
- Generally I should have 1 GB of ECC RAM for every TB of data in ZFS?

You mentioned Registered memory (RDIMMs), however there seems to be another kind called Long (LDRIMMs). Do I just go for the DDR4 'Registered' (RDIMMs) memory?
---------------------------------------------------------------------------------------------

In regards to others' comments. Thank you all very much for the thoughtful replies in regards to cost, temperature, noise concerns.

I think I will still get a big 18 HDD case (but will leave the bottom area empty indefinitely). Honestly, I don't think I will ever fully utilize all 18 HDD. For now, I'm just starting with 6-8 HDD and the most will probably be around 12-14 maybe. And, I will place the server in an adjacent room instead of having it in the same room as my desktop. Initially I was thinking that perhaps I could put a cheap gpu on it (since the cpu has no iGPU) and just have it by my desk and haver a monitor connected to it so I can more conveniently perform admin duties, but perhaps that's a bit of a wishful thinking due to the noise concerns. 😄
 
The 2667, just as all other *7 variants with lower core count but higher max frequencies basically only exist because of stupid per-core licensing. Their premium price tag are never worth it except you absolutely have to have high single-core frequency/performance for your workload. Otherwise variants with more cores on lower frequencies are the much better option.

Back when I upgraded to v4 from v3 I considered the E5-2660 to be the best sweet-spot of cost/core-count/performance, yet still being at a "moderate" TDP of 110W. I ran a "L" variant before, but due to their much lower Tjmax they actually need *more* cooling and hence are probably noisier and with fans being a major factor for power draw the actual power savings for the whole system are questionable...

The current system with 2x E5-2660v4, 256GB RAM (8x32GB Micron), 6x SAS SSD and 6x NVME runs at 165-168W throughout the day. Apart from being a backup storage and general data grave it also runs a bunch of jails e.g. with nextcloud, jellyfin, gitea and my zabbix monitoring and a postgre database for anything that needs a DB. It also serves as a poudriere package builder. Peak power consumption during package builds is ~370W
Noise is absolutely bearable at normal loads - the open frame rack in which this server sits stands directly behind me in my home office. During package builds it gets quite noisy - hence I usually run them while I'm at work.

Your biggest concern regarding noise will be the 18 spinning disks - they will be quite noisy themseleves AND their cooling requirements will add up considerably, causing higher fan noise (and fan power draw). I'd *really* reconsider using that many drives. As others already suggested: Usually if you need a lot of storage space, it's for relatively cold data that doesn't require high bandwidth - i.e. you will be perfectly fine using fewer but larger disks. For anything that really requires high bandwidth and/or IOPS it makes absolutely zero sense using spinning rust anyways - just put that data on SSDs (SAS or NVMe). They will outperform any HDD by order of several magnitudes while using *much* less power and hence requiring also less cooling.
 
As a number point, a dual Xeon v4 takes 100 Watt idle for me in Linux and 150 in FreeBSD. That is with E5-2697A CPUs.

As for boards, any of the Supermicro offerings should be fine as long as they fit your case and support the right TDP. Watch whether the board supports LRDIMMs before you buy the memory modules.

I use the Noctua heatsinks on mine. They are even relatively cheap.
 
My latest addition:
Code: 
CPU: Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz (2893.21-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206d7  Family=0x6  Model=0x2d  Stepping=7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x1fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 206158430208 (196608 MB)
avail memory = 200700841984 (191403 MB)

On a SuperMicro X9DRD-7LN4F-JBOD: https://www.supermicro.com/QuickRefs/motherboard/C606_602/MNL-1336.pdf

This is a big board though, didn't fit in any of the 19" rackmount cases I had. Had to buy a new case, fits perfectly in a SilverStone RM42-502. Haven't really looked at its power consumption. But it chews through a poudriere-bulk(8) very rapidly (USE=TMPFS=all). Lots of PCIe x8 slots for future expansion.
 
You must log in or register to reply here.
Back
Top