Compile system hardware orientation

MG

MG

While working about 3 years with a i9-13900K as "engineering" system that I use to compile a large FreeBSD system with ports and settings. Time for a new production PC. It needs to be good at SMP compiling and able to handle many millions of small files fast.
I kind of lost track of the expected PC performance today for a price. To get closer, what can I expect in Europe from a new 2000 euro PC, system unit only with 4TB storage and 64GB RAM compared to my current i3900K with the same?
 
*headscratch* After a quick search anything you're looking for (next larger CPU (i9-14900KF), 64G RAM + 4T Storage, no fancy GPU) starts at least at 3k - so 50% above your target.

That's already a very big machine you have. Particulary while you said
MG said:
able to handle many millions of small files fast.
Click to expand...
maybe you better look for another bottleneck and improve the machine/system you have instead of just focusing on a larger CPU, only, since I doubt compared to the 13900K you already have a 14900KF would bring you significantly more performance.
If I were you I'd first look for faster drives and maybe faster RAM. A fat, fast CPU cannot compensate slow RAM and drives.
Maybe a drive's mirror could speed things up, organizing your data otherwise, like placing the things need to be fast on smaller but faster drives. Anyway, I would examine such things first - searching for optimization potential on the existing machine, before I start thinking of buying a whole new machine.
 
Maturin said:
maybe you better look for another bottleneck and improve the machine/system you have instead of just focusing on a larger CPU, only, since I doubt compared to the 13900K you already have a 14900KF would bring you significantly more performance.
If I were you I'd first look for faster drives and maybe faster RAM. A fat, fast CPU cannot compensate slow RAM and drives.
Maybe a drive's mirror could speed things up, organizing your data otherwise, like placing the things need to be fast on smaller but faster drives. Anyway, I would examine such things first - searching for optimization potential on the existing machine, before I start thinking of buying a whole new machine.
Click to expand...
What can still be improved if the compiling performance on SSD storage equals compiling with all needed executables, libraries and the output placed inside a malloc ramdrive?
Never noticed any significant RAM speed improvement within a board chipset generation...
Not sure: can an amount of SSD's overtake the RAM r/w actions? Those PCI lanes and such are beyond my arch understanding...
 
cracauer@ said:
A 14900K is faster than a 13900k, but hardly significant.
Click to expand...
That's what we both agree on: Even with more modern architecture 24 cores @ 6GHz vs 24 cores @ 5.8GHz won't gain that much more performance win, app. 4%. Even if it was 15% one hardly recognizes it. If a job takes eight hours or seven and a half hours does not stick out among the common fluctuation.
So while 6GHz is already pretty much the top max speed to gain really significant performance wins due more CPU power one needs to chose a significant number of more cores aka a board/machine which supports more than one CPU:
cracauer@ said:
SMP server
Click to expand...
cracauer@ already said that.
This is buyable. But not for 2k€. Well, maybe a single CPU, but not a whole machine.
To rent was another option.

Anyway:
MG said:
What can still be improved if the compiling performance on SSD storage equals compiling with all needed executables, libraries and the output placed inside a malloc ramdrive?
Never noticed any significant RAM speed improvement within a board chipset generation...
Not sure: can an amount of SSD's overtake the RAM r/w actions? Those PCI lanes and such are beyond my arch understanding...
Click to expand...
Sure, if your drives were faster than your RAM your machine was extremely poorly configurated, no question. But there are significant differences in drive's speed (Don't trust on the data sellers name, only!) Most compromise most people chose is either a fast or a large drive, while large and fast drives are pretty expensive.
For example I have two 256G PCIe NMVes on my mainboard as a mirror for the system, while /home is on a 1T SSD mirror, and everything else I don't need constantly, or frequently is outsourced to a large HDD ZFS pool, slow, but fast enough for my personal purposes. And for the price I saved what faster SSDs would have cost I got a couple more T instead.
Since my whole system is smaller than 50G and anything else is on other drives, I can chose a small system's drive, and the money I saved by a smaller size I spent on faster drives instead.
That's not only what I meant with
Maturin said:
organizing your data
Click to expand...

Frankly, I don't know what you could optimize. One needs to see the exact specs of your machine, and above all a detailed list of the most workload tasks you're doing to give you any concrete answer.
But since there is no significant performance win to be expected by a better CPU, only, one needs to look for alternatives:
  • Check out your RAM: While you said 64G RAM I suppose you already checked the usage is significantly smaller than 100%, so more RAM wouldn't bring no performance win. But, what speed does it have? Can you replace it by faster RAM? Does your mainboard support faster RAM? (Watch out that RAM, board and CPU match!) If you already checked that, and got no performance win by RAM, then:
  • Organization: What can be done else, what otherwise? Where is optimization potential? Can you free performance, e.g. by using a more lightweight WM instead of a heavy DE, etc.? Can you reschedule tasks? For example let the time consuming jobs run at night, when you don't need to use the machine for other things? Can you outsource tasks on another machine? etc. Such questions often can lead to better solutions due optimization instead of always looking for a bigger gun, only.
  • Learn to live with the situation.
Most of the times in life it comes to compromise all three points.
 
MG said:
What can still be improved if the compiling performance on SSD storage equals compiling with all needed executables, libraries and the output placed inside a malloc ramdrive?
Click to expand...
I think a malloc ramdrive will simply waste memory. In my experience ZFS does pretty good caching. I dd'ed 10GB from a file on a HDD to /dev/null. The first time took 51.8s. The second time 1.67s. binaries/libraries that get used get cached pretty quickly.

You can use multiple SSDs to improve disk throughput. One thing you can try is to take a detailed look at IO throughput under various conditions. Another thing you can do is use caching and METAMODE. Yet another thing you can do is cut out unnecessary builds. Don't build stuff you're never going to use or haven't used in a long long time.

You can use the present exorbitant memory (& SSD) prices as an incentive to reduce waste! [IMHO people who focus on optimizing performance should use the slowest machines!]
 
bakul said:
I think a malloc ramdrive will simply waste memory. In my experience ZFS does pretty good caching. I dd'ed 10GB from a file on a HDD to /dev/null. The first time took 51.8s. The second time 1.67s. binaries/libraries that get used get cached pretty quickly.

You can use multiple SSDs to improve disk throughput. One thing you can try is to take a detailed look at IO throughput under various conditions. Another thing you can do is use caching and METAMODE. Yet another thing you can do is cut out unnecessary builds. Don't build stuff you're never going to use or haven't used in a long long time.

You can use the present exorbitant memory (& SSD) prices as an incentive to reduce waste! [IMHO people who focus on optimizing performance should use the slowest machines!]
Click to expand...
A malloc-based ramdrive means almost no lag by real storage electronics. I use this as comparison witn any actual storage hardware. This is the best we can get. It has a tiny performance overhead because it increases the system memory management load of which part is on diskcontrollers when real devices are used..

My situation: I work with 1 large source tree to build an entire complete FreeBSD system with various programs. It's mostly fun development but I do make some profit with it in local businesses. It now takes +/- 6 hours to create a bootable USB image. Because it's a 1-person project, I can't spend too much time on only experimenting and planning. I want to decrease the total compile time to avoid side-tracking and massive todo lists. My experience is that a regular workstation upgrade improves progress a lot. If I delay it with 3 years or so, I will be going to waste time on working around current hardware limitations while it's likely that it requires nothing more on a recent system because bottlenecks always get the most attention.

Still, the increasing RAM and storage pricing is extremely frustrating. Not only price vs performance and capacity got worse but also the ability to upgrade without paying a lot for things that only have a new connector and don't really improve things. Are they trying to destroy the PC standard? There's already nothing left of the old IBM concept. The backward and forward compatibility of a recent PC is minimized to almost nothing. You can replace a few components and that's it. Time for everything new in the next gen PC... I remember having x86 computers with hardware variating from 286 to the last AT era in 1 case.
 
SMP server gets my vote. For example https://www.lenovo.com/gb/en/p/servers-storage/servers/racks/thinksystem-sr665-v3/len21ts0009. They have intel and 'large memory' variants too.
You get 2 AMD epyc's per node, so lots of cores. Fill it with high speed server-grade SSD's. Max out the RAM. Make a build server out of one of those, put it in a rack somewhere
and run your builds over the network. If you can afford it put 4 of them in a rack so you can run multiple builds in parallel. If you can afford it you can add a separate storage controller, build a SAN to hold your data.
The key is parallelism, hardware stability and storage performance/integrity. The problem is the cost. This kind of thing is really only viable for a company, unless you happen to be Elon. :-)
 
Have a look at what Linus was using, Ryzen threadripper (though this is probably a bit out of date now). An updated version of one of these should be pretty good. This might actually be realistic for a home system, unlike the server gear.
www.zdnet.com

You can build Linus Torvalds' PC: Here's all the hardware and where to buy it - ZDNET

Normally, Torvalds would pop into his local Fry's. But in these pandemic times, he ordered everything from Amazon. Here's the complete list of parts.
www.zdnet.com www.zdnet.com
 
cracauer@ said:
Financially it is hopeless. The problem comes in when you want to add enough RAM to feed all the cores when you get e.g. a dual EPYC server with 2x 64 cores. Even for DDR4.
Click to expand...
Yes, unless you're making some serious money its out of reach. See, you should have taken that hedge fund real-time trading job. Of course buying it through your company helps if you are able to write some of it off against tax, or even get investment grants towards the cost.
 
Here's the updated version of that threadripper box. An entertaining recent video that actually has some pretty good suggestions. Skip the first 2 or 3 sections introducing him though, I suggest start with the choice of cpu section. You could do worse than to follow this, it looks like being a very nice, modern, high performance home developers workstation. I like that he specifies ECC memory, it's going to be very reliable. I'm sure it runs FreeBSD too. 😁

I wonder what some of the freebsd core team developers are using; it would be interesting to get one or two of those guys to tell us what they use at home. I would guess probably something similar to Torvalds box, unless they have a rack in the garage.

View: https://www.youtube.com/watch?v=mfv0V1SxbNA
 
blackbird9 said:
SMP server gets my vote. For example https://www.lenovo.com/gb/en/p/servers-storage/servers/racks/thinksystem-sr665-v3/len21ts0009. They have intel and 'large memory' variants too.
You get 2 AMD epyc's per node, so lots of cores. Fill it with high speed server-grade SSD's. Max out the RAM. Make a build server out of one of those, put it in a rack somewhere
and run your builds over the network. If you can afford it put 4 of them in a rack so you can run multiple builds in parallel. If you can afford it you can add a separate storage controller, build a SAN to hold your data.
The key is parallelism. The problem is the cost. This kind of thing is really only viable for a company, unless you happen to be rich.
Click to expand...
Going off-topic, but I have been trying distributed compiling with distcc years ago. It worked but I found it too complex to keep on using permanently. It would turn my build-from-source script framework to a separate project that targets its own hardware setup. I now compile the system on its own last version.
It's interesting, though. What can be done with 8 HP Elite minis on a Gbit switch? I expect the network will clog first and you have to streamline the communication to get your available bandwith to a point compilers on different machines can actually gain time with a central load schedule.
 
Yes distributed is another interesting thing. Or using a cluster. But nowadays single nodes with massively multi-core cpu's and large memory, and some good fast storage, seems to be the way to go. Another thing I remember that was hot for a while (a few years ago) was pcie ram drives, cards with stacks of battery-backed ram on them that emulated a drive, obviously with very high throughput. I thought of that when I read your comments about using a ram disk. The pcie memory drives seem to have fallen by the wayside now though, I guess now you can put 4TB of ram on a mainboard, they don't count for so much.

I was just watching that video from the two linus's... I need to upgrade my home system! 😁 😂
 
Interesting, looks like there's still at least one company making dram drives https://www.ddrdrive.com/ . It's probably not cheap, it's quite specialised gear.
1793270832040.png

I remember gigabyte used to make them years ago, not sure about now. Of course that thing is going to be fast. It looks like they can sell you a NAS full of those dram drives. Again, it probably doesn't make sense financially.
 
my arm vm beats the crap out of all my shitty intel boxes and vms.
out of curiosity how much time it takes to build graphics/spirv-tools (has no deps besides the build ones ie cmake, python)
make fetch
time make -DNO_DIALOG all
the arm vm built it in 1.43mins with 4g ram and 4 cores
the proliant took a minute more with 100g ram and 12/24 cores. but it is kind of ancient though
a netcup rented vm with 6 cores and 8gb took 7 minutes
 
covacat said:
my arm vm beats the crap out of all my shitty intel boxes and vms.
out of curiosity how much time it takes to build graphics/spirv-tools (has no deps besides the build ones ie cmake, python)
make fetch
time make -DNO_DIALOG all
the arm vm built it in 1.43mins with 4g ram and 4 cores
the proliant took a minute more with 100g ram and 12/24 cores. but it is kind of ancient though
a netcup rented vm with 6 cores and 8gb took 7 minutes
Click to expand...
What arm hardware?
 
MG said:
My situation: I work with 1 large source tree to build an entire complete FreeBSD system with various programs. It's mostly fun development but I do make some profit with it in local businesses. It now takes +/- 6 hours to create a bootable USB image. Because it's a 1-person project, I can't spend too much time on only experimenting and planning. I want to decrease the total compile time to avoid side-tracking and massive todo lists.
Click to expand...
I build 4 combinations of cpu arch + freebsd version at least once a day. Usually takes a couple of minutes each (unless llvm rebuild is triggered). Less time than make installkernel installworld & testing on VMs and real machines. But I don’t build packages. The VMs run on the same 8 year old ryzen machine with 64GB RAM & Mirrored ZFS on HDDs. Now I imagine you are building far more than just the freebsd bits but with ccache + METAMODE you can cut out a lot of unnecessary rebuilds. At least something worth considering!
 
Another option jumps to my mind was a second machine for under 2K similar to the one you already have. Can be headless, use it via ssh. This could double your build capacity for way less than you pay for a machine twice as fast: One machine does the 6h build, while you can work on the other, until you let it start the second job.
Apart from no keyboard, monitor and no GPU, this machine also doesn't need the full storage capacity - it mostly needs to provide a CPU and the RAM. So even it was a bit slower (maybe 90% of your current one) you may get such for 1K. Buy two, and you have 280% faster compiling time.
...just as an idea....
 
You must log in or register to reply here.
Back
Top