I need to upgrade my server hardware. How do I know if it will work?

T

thedude22

My server was initially set up as a simple file server and so hardware performance was a low priority. Now I need to change it into something of a workstation.

I'm looking at the following:

Supermicro X11Spi-Tf motherboard
SK-HYNIX HMAA8GL7MMR4N-TF EDD LRDIMM DDR4
Intel Xeon W-3223 CPU

On the OS support page for Supermicro they show this motherboard is compatible with FreeBSD11, but not necessarily 12 (2nd column scroll down under Linux OS). At least it is left unchecked.

I would like to keep my system up to date. I'm wondering if their list is not exhaustive simply due to insufficient testing. Is there any reason this hardware might not work with FreeBSD 13 even though it is compatible with FreeBSD 11 according to the manufacturer?
 
Thank you. I don't care about graphics because it will be headless.

Do you have a better suggestion for CPU? I need high clock speeds and good single thread performance at an affordable price.

Also, if I just swap out the hardware and turn it on can I expect FreeBSD to boot with no problems? All hdds are in a single zpool which holds the root filesystem, and I think the boot partition is replicated on all drives if I'm not mistaken.
 
Yes ECC. Yes UEFI.

Thanks for the help.

I hope setting up the network interface isn't too bad. I remember that took me a while when I initially set up the thing. And I'll have to drag out the video card, keyboard, etc. to set it up if it doesn't work.
 
Lookout for the Aquantis 10G interfaces that SM uses on some boards. Not supported.
I agree with diizzy that CPU runs hot @160W I am more of a LV guy.
I am still stuck on LGA2011. I did not like the E16xx series either which is similar.
So I missed LGA3647 and now we have LGA40xx ??
If your spending a boatload make sure you get the newest socket.
The W-3223 chip does not look attractive to me.
I would forgo the workstation Xeons and get a regular LGA3647 CPU or newer.
The board is fine. My preferred LGA3647 is the 6138T but i like my cores.
I have no beef buying used CPU's. The motherboard sockets are what are fragile.

thedude22 said:
I'm wondering if their list is not exhaustive simply due to insufficient testing.
Click to expand...
No they just are like that. They 'certify' it to work with one board/cpu and don't update the manuals much.
 
Not trying to spend too much. I got the CPU for under $400 off ebay. The newer ones are all over $1k. Maybe I can undervolt it a bit.

All together with 256gb of ram and the mobo it was around $1700, more than I wanted to spend.

Still need a good cooler than will fit in my 4U case.
 
thedude22 said:
Thank you. I don't care about graphics because it will be headless.

Do you have a better suggestion for CPU? I need high clock speeds and good single thread performance at an affordable price.

Also, if I just swap out the hardware and turn it on can I expect FreeBSD to boot with no problems? All hdds are in a single zpool which holds the root filesystem, and I think the boot partition is replicated on all drives if I'm not mistaken.
Click to expand...
Another option that might be worth considering is Ryzen or even Threadripper?
Asus RoG B550-F or B550-E (one of the few boards that comes with Intel LAN (2.5G however so you need to pull in the driver from https://reviews.freebsd.org/D30668 for now) or go with something like Asus Prime X570-Pro (older , uses the 1G Intel NIC).
The 5000 series does perform very well even single threaded.

​

 
I considered Ryzen 5900x for a while, but the ECC support seems pretty dubious. It seems many testers found issues, especially with multi-bit correction.

I decided I would just go with proper server hardware, rather than enthusiast/gaming stuff.

Those epyc chips are crazy though, but I'm not going to spend 5k on a cpu. I don't care how many cores it has.

Hardware has already been ordered. Hopefully I made a good choice. Spending so much money stresses me out.
 
diizzy said:
Asus RoG B550-F or B550-E (one of the few boards that comes with Intel LAN (2.5G however so you need to pull in the driver from https://reviews.freebsd.org/D30668 for now) or go with something like Asus Prime X570-Pro (older , uses the 1G Intel NIC).
Click to expand...
There is absolutely no point in fooling around with 2.5GBit on serves and I don't even understand why this completely outdated and long-forgotten crap now seems to become the 'new rage' for desktop... 10G is dirt cheap nowadays (switches and transceivers) and about to become somewhat of the low-end standard. Pushing for 2.5G now when practically no switches are available (b/c nobody cared and all went straight to 10G+) is complete nonsense. Don't buy that, it has already died a long time ago...


BOT:
Haven't had any issues yet with Supermicro hardware, even if it was quite recent. It seems they are more conservative with their selection of onboard components, so most of it just works out of the box...
Be sure to use either certified RAM or at least a brand that is known to be compatible with nearly everything on the planet, like kingston or samsung. That's what we're usually using on our Supermicro servers and never had issues with them. Even if that specific module wasn't listed by supermicro, those two brands usually make up the biggest part of their compatibility lists.

When it comes to home-servers I'd never go for brand new hardware - just as with cars, that stuff practically loses nearly half of its value after the first year and even more after the next generation has been announced or hits the shelves. Therefore I'd always go for a previous-generation server as they are available dirt-cheap and still offer more than enough performance and features for a home server.
As a home server I'm currently still running a Supermicro 6029U-TR4T with 2x Xeon E5-2630L v3 (6C/12T) and 128GB RAM, a Supermicro AOC-STGN-i2S dual 10G-SFP, LSI SAS3008 HBA. The whole system (without disks, but with 2x 128G NVMe) has cost me under 1000EUR and has plenty enough headroom for a homeserver while still being silent enough to be run in an open rack in the home office. With ~200W power consumption its also not that hard on the power bill.
 
thedude22 said:
I'm looking at the following:
Supermicro X11Spi-Tf motherboard
Intel Xeon W-3223 CPU
Click to expand...

I got an X11SPA-TF with a W-3245 and lots of bells and whistles last month. It is very nice, a good combined server and workstation.

But the 10G Aquantia is not usable. There is a driver in the ports which works, a bit, not well enough.

The MB needed to have bios updated before it recognised Samsung NVMEs.

Apart from that, very good with Freebsd.
 
A Threadripper can run you around $5k these days (the 64-core one). My personal research into ECC led me to conclude that it's not worth the extra money for a home setup.
 
Geezer said:
I got an X11SPA-TF with a W-3245 and lots of bells and whistles last month. It is very nice, a good combined server and workstation.

But the 10G Aquantia is not usable. There is a driver in the ports which works, a bit, not well enough.

The MB needed to have bios updated before it recognised Samsung NVMEs.

Apart from that, very good with Freebsd.
Click to expand...
Are you saying the onboard network adapter did not work? Did you have to buy a PCI NIC to get on your network?

Network Controllers
  • Dual LAN with 10GBase-T with Intel® X722 + X557

Looks like Intel chip, so hopefully I'm good.

That's an interesting board you got there, pretty expensive though.

Newegg says
  • AI optimized, deep learning, simulation, and content creation
I'm curious what that means because I am going to be doing ML/AI stuff on mine. I wonder if it just means it's built for a bunch of GPUs. I was planning to do everything on the CPU. Didn't even think about using CUDA, but now I feel kinda dumb for not thinking about it. But I don't even know if you can find GPUs anymore anyway at a reasonable price at least. It's something I can look at in the future I guess, if the GPU market ever goes back to normal. I don't know if they have enough memory though. I'm going to be using pretty large datasets.
 
Phishfry said:
Are those Hynix memory modules on SuperMicros supported RAM list for your board?
Click to expand...
Don't think so, but I took the chance because the price was right. For all the talk about supported ram, I've never had an issue in the past, provided it was the right type of ram.
 
astyle said:
A Threadripper can run you around $5k these days (the 64-core one). My personal research into ECC led me to conclude that it's not worth the extra money for a home setup.
Click to expand...
Depends on your tolerance for data errors, and your willingness to spend the money. I don't want to deal with data errors at all. Already have to deal with all the errors in my code, lol.
 
thedude22 said:
Are you saying the onboard network adapter did not work? Did you have to buy a PCI NIC to get on your network?

Network Controllers
  • Dual LAN with 10GBase-T with Intel® X722 + X557

Looks like Intel chip, so hopefully I'm good.
Click to expand...

The MB comes with three RJ45s
  1. Intel 1G - works very well with igb.
  2. IPMI port - very good, one of the reasons I got Supermicro. Their IPMI seems to have more features than other brands, such as SOL and HTML5 KVM.
  3. Aquantia 10G - there is no kernel driver for this. There is a module driver in the ports. It seemed to work reasonably, until I used NAT, and then it was bloody useless. There is a thread in here about it, and how I found out what was wrong. Unfortunately, the driver is unlikely to be improved and Aquantia was a bad choice for both me and maybe Supermicro.
One of the purposes of the machine is NAT so I do need to interfaces.

So I bought a cheap Realtek pci-e. Seems a pity for a nice new, and expensive machine. Still wondering if I should have got an Intel pci-e.
 
thedude22 said:
Depends on your tolerance for data errors, and your willingness to spend the money. I don't want to deal with data errors at all. Already have to deal with all the errors in my code, lol.
Click to expand...
Well, if I did enough troubleshooting to narrow my problems down to data errors that could have been easily avoided with ECC, and nothing else works, then yeah, I would invest in ECC RAM and compatible hardware. A bit like finally realizing why the Threadripper makes a difference when compiling the kernel. If you compile the kernel often enough, then yeah, investing in a Threadripper makes sense.
 
Geezer said:
If you are going for a nice Supermicro MB, then simply get ECC.
Click to expand...
Already did of course.

Does anyone know if I'm going to have issues booting after swapping motherboards out? All my drives are in a single zpool. I'm using root on ZFS. Is it all going to be correctly imported automagically or am I going to need to boot into a USB drive to get everything set up?
 
unless you've built a custom kernel with only specific drivers it will boot just fine.
For legacy/MBR-boot you might need to choose the correct boot drive, with UEFI it should 'just work'™
 
Everything booted up fine. Just had to rename the network interface.

Unfortunately I had to go out and buy a VGA cable because only the onboard video would send a signal. Don't know what the deal with that is, but it's a motherboard issue. IDK how to use the IPMI thing so I had to hook up an actual monitor.

This CPU does not run that hot guys. I've got multiple cores at 100% and they go up to 51C and that's it. That's the highest I've seen. Got a dynatron B13 cooler. Keeps things nice and cool. Pretty quiet too. Server case gets good airflow of course. No complaints about heat though.
 
Is there a way to check cpu clock speed to make sure I'm getting the full turbo boost speed? Maybe my temps are low because I'm stuck at 3.5 GHz.

NVM I found sysctl dev.cpu.0.freq

Okay I enabled powerd and the sysctl reported frequency went up from 3500 to... 3501. Is this for real? I was expecting 4000.
 
Looks like powerd has been optimized to maximize instructions per cycle.. as in, for every processor cycle, powerd tries to get done as much as practically possible. :P
 
thedude22 said:
Unfortunately I had to go out and buy a VGA cable because only the onboard video would send a signal. Don't know what the deal with that is, but it's a motherboard issue. IDK how to use the IPMI thing so I had to hook up an actual monitor.
Click to expand...

Usually you just change the default video output from "Aspeed" to "add-in card" at the BIOS. But because the IPMI can't grab the video signal from that, you won't get anything but a blank screen after POST from the SOL or iKVM console, which you can easily access via "Remote Control -> iKVM/HTML5" in the webinterface.
BIOS/Firmware upgrades are also completely painless over the Supermicro IPMI and don't require any fiddling around with USB sticks or some DOS/Windows flashing software.
 
You must log in or register to reply here.
Back
Top