Well, hellllo there, nurse...err, I mean Gen13

[loveydovey]

How about a half a kilowatt CPU? Does anyone own a Gen11/12/13? What kind of naughty things do you do to your Gen11/12/13 that you can talk about?

Inside Gen 13- how we built our most powerful server yet

Cloudflare's Gen 13 servers introduce AMD EPYC™ Turin 9965 processors and a transition to 100 GbE networking to meet growing traffic demands. In this technical deep dive, we explain the engineering rationale behind each major component selection.

blog.cloudflare.com

 

[loveydovey]

Is it hot?

 

[drhowarddrfine]

The chip runs at 500F so, yes, it's hot. But all those stats are apparently good for them and meaningless for subscribers.

 

[blackbird9]

Cerebras is pretty interesting too, in massively parallel systems. They use wafer scale integration.

Cerebras Systems - Wikipedia

en.wikipedia.org

"In November 2022, Cerebras unveiled the Andromeda supercomputer, which combines 16 WSE-2 chips into one cluster with 13.5 million AI-optimized cores, delivering up to 1 exaflop of AI computing horsepower, or at least one quintillion (1018) operations per second.["

View: https://www.youtube.com/watch?v=9EayHWd8FqI
(Disclaimer - I don't have any connection with the guy in this video).

I don't know what their power consumption or running temperature but i expect it's pretty high. It seems to be targeted at AI, apparently openai uses their kit. It must need a pretty good cooler.

Wafer scale is an idea that has been kicking around for decades, even Clive Sinclair was investigating it in the UK, with his company Anamartic, back in the mid-80's. https://www.computinghistory.org.uk/sec/3042/Anamartic/

 

[ralphbsz]

500W is pretty toasty. Interesting that they cool it with just air, with heat pipes from the socket to a heat sink. At these power levels, liquid cooling would be more efficient (in the sense of power overhead used for the cooling infrastructure), but maybe they're not big enough to worry about the last 1-2% of cooling efficiency.

Another interesting thing: Their rack-mount box has the standard 19" form factor, with mounting ears. Most other giant companies use larger form factors today, since it improves efficiency significantly. The OCP standard is 21" wide boxes, but some others go all the way to about 1m (3 foot) wide units.

And the ratio of 10 disk slots (for 2-1/2" drives) to one CPU is also interesting. As is the choice of small form factor drives. But then, this is Cloudflare not Amazon/Google/Microsoft, so they are not a large storage service provider.

 

[blackbird9]

I would guess this is probably spec''ed by cloudflare and built for them by a whitebox manufacturer like lenovo or celestica. Air cooling is mature tech, doesn't need any special parts in the server room when it goes wrong, just drop a new hot-plug fan in. It says one of their design goals was operational simplicity. That's still quite a lot of heat to dissipate, but I assume they have stress-tested it to death! The SCM apears to be made by Tyan, so perhaps the whole motherboard plus daughter card set is custom built for them by Tyan, or is a customisation of an existing Tyan oem board, although the article doesn't say who manufactuers the motherboard for them, but the Tyan branding is visible on the SCM.