Influence (new) hardware on strenuous tasks

Erichans

Member

Reaction score: 80
Messages: 92

I intend to use FreeBSD also on a desktop/server CPU, NVME SSD & DDR4 (and graphics card). Currently using FreeBSD REL-12.2 (and XFCE) (binaries, packages) on an old Samsung laptop: Intel Core2 Duo P7350 @ 2.00GHz, 3GB DDR3, SATA SSD.

What is the effect of the amount of memory, Intel (amd64 architecture) CPU (because I have some experience with those) and core count on heavy tasks such as more well defined tasks: (cross) compiling world or Firefox as an upper bound.
  • At what amount of ram will I get (vastly) diminishing returns on more RAM?
  • What is the influence of core count (i.e. parallelization) on compiling?
I'm looking for ballpark influence.

I have no idea what amount of time compiling world on a modern CPU takes; if you have a (ballpark) figure, perhaps with a specific CPU and ram, I would appreciate that.
 

Geezer

Aspiring Daemon

Reaction score: 454
Messages: 778

In my opinion, ram is most important, not cores. Apache, zfs, DEs and browsers. All these can really do with loads of ram.

  • At what amount of ram will I get (vastly) diminishing returns on more RAM?

When you run out of money.
 
OP
E

Erichans

Member

Reaction score: 80
Messages: 92

Thanks, perhaps my old laptop gave the wrong impression as to the desktop/server hardware intended. Thinking along the lines of an Intel Xeon E3-1280 v5 @ 3.70GHz and up (or perhaps even something like an Intel i7-6700HQ in my P50).

As to "loads of ram", this is a very broad ballpark amount; a little more detailed amount would be helpfull. Thinking along the lines of 32, 64, 96 & 128GB.
 

ct85711

Member

Reaction score: 63
Messages: 95

From my experience ram is going to give you a significant boost up to a point. I'd recommend at least 16-32GB of ram to get the best performance, 8GB minimum. Once you get past 16-32GB of ram, the returns you gain drops off. The reasoning that I've seen on this is that, frankly outside of ZFS, and DB's; few packages will use that much ram ever. As far as compiling, more core/thread counts will speed up compiling for most packages (some packages are not designed to be compiled in parallel and cause troubles). I don't have personal experience with core counts from like the Ryzen/Threadripper amounts of cores to ever see if there is a drop on core amounts. Though the part you will see, is that few packages will use too many of the cores outside of compiling.

The one rule of thumb I've encountered quite commonly on Gentoo (and I am assuming it would be similar to FreeBSD), is expect on average about 2GB per thread when compiling.

Now one thing, that is bonus on compiling when you have plenty of spare ram, is to use the unused ram for as a ram disk. While the performance gain isn't as high as when compared to just using the spinning rust buckets; the ram disk also gives a side bonus in that you get less writes onto the SSD (as you setup the temporary compiled stuff can be set to compile onto the ram disk and not the SSD). The downside on RAM disk, is that it's hogs the portion of ram that you gave it. The other thing to keep in mind (commonly encountered on the browsers), is some packages can be too big to fit in the ram disk and cause it to fail to compile.
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,744
Messages: 39,332

As far as compiling, more core/thread counts will speed up compiling for most packages (some packages are not designed to be compiled in parallel and cause troubles). I don't have personal experience with core counts from like the Ryzen/Threadripper amounts of cores to ever see if there is a drop on core amounts. Though the part you will see, is that few packages will use too many of the cores outside of compiling.
Don't have any exact figures but at some point you're going to hit a bottleneck on your I/O. In order for all those compiler process to work they need to be able to quickly read and write various (intermediate) files. So you're going to need some fast storage too. Poudriere can be configured to use RAM (tmpfs(5) [1]) for this, which is the fastest option you could get it to work. But this obviously consumes copious amounts of memory besides the amount of memory all those compilers are going to need.

[1]:
Code:
# Use tmpfs(5)
# This can be a space-separated list of options:
# wrkdir    - Use tmpfs(5) for port building WRKDIRPREFIX
# data      - Use tmpfs(5) for poudriere cache/temp build data
# localbase - Use tmpfs(5) for LOCALBASE (installing ports for packaging/testing)
# all       - Run the entire build in memory, including builder jails.
# yes       - Enables tmpfs(5) for wrkdir and data
# no        - Disable use of tmpfs(5)
# EXAMPLE: USE_TMPFS="wrkdir data"
 

astyle

Daemon

Reaction score: 656
Messages: 1,469

Might as well weigh in here... My rig has 32 GB of RAM, and a Ryzen 5 1400 (quad-core, 3.4 GHz). And I'm noticing that only one core is really used for compiling - unless I specify the -j4 flag for make. However, specifying the -j4 flag seems to be useful only when recompiling the kernel or compiling a port that has ALL deps (both build-time and run-time) completely satisfied. So basically, very limited variety of scenarios. However, without that flag, my Ryzen just churns without complaining (and it does have a stock Wraith cooler it came with, which is absolutely essential), it can go for hours, even overnight.

Some lessons I learned: be very careful with the make flags that enable parallellism - make is not smart enough to manage different cores, so it might start a job (on another core) that is actually not ready to start yet.

Without the -j4 flag, my Ryzen 5 would take about 3 hours to compile Firefox (after all deps have been satisfied for www/firefox).
 

SirDice

Administrator
Staff member
Administrator
Moderator

Reaction score: 12,744
Messages: 39,332

However, specifying the -j4 flag seems to be useful only when recompiling the kernel
You can use it for buildworld too. Speeds things up tremendously. Only the installkernel/installworld phases should be done without.
 

Alain De Vos

Son of Beastie

Reaction score: 796
Messages: 2,589

Offcourse number of cores for the same cpu-architecture.
I think its a good idea to assign a few cores for compiling and a few cores for realtime stuff like playing youtube.
 

Sevendogsbsd

Daemon

Reaction score: 692
Messages: 1,142

When I built packages on my old HP z800 (96 GB ram), I used as much as 60GB building rust, chromium, Firefox and llvm (?), libreoffice, etc. I had my settings as far as concurrency set pretty high to take advantage of the machine's capabilities. I had 2 hexacore xeon 5550 (I think). I may be misstating some of the terms here because it has been a long time and I no longer have that machine. 100% of my builds were using tmpfs because of the amount of ram I had so that may account for the high ram usage in my case.
 

grahamperrin

Son of Beastie

Reaction score: 832
Messages: 2,687

… Poudriere can be configured to use RAM (tmpfs(5) [1]) for this, which is the fastest option you could get it to work. But this obviously consumes copious amounts of memory besides …

Thanks, I stepped mine down from USE_TMPFS=all to USE_TMPFS=yes. 16 G real memory here. I'll see how things go.

Previously – with all – just rarely, I'd find swap space (16.0 G) nearly exhausted, as in the first of these four shots:

2021-09-19 03:35:16.png2021-09-19 03:48:49.png2021-09-19 03:51:14.png2021-09-19 03:54:38.png

Incidentally, there was gmake -f Makefile -j4 all (four jobs) whilst, if I recall correctly, poudriere was set to PARALLEL_JOBS=3.
 

grahamperrin

Son of Beastie

Reaction score: 832
Messages: 2,687

I stepped mine down from USE_TMPFS=all to USE_TMPFS=yes. 16 G real memory here. I'll see how things go.

On second thoughts, I might stick with all for a while.

I've been using the system with 16.0 of 16.0 G swap used for a few hours, whilst building electron12. Performance has been fine.

2021-09-26 19:24:05 tmpfs.png2021-09-26 20:03:24 tmpfs.png

Postscript (note to self): not too strenuous, but maybe too adventurous for electron12 build purposes.

Code:
…
[ 69% 26723/38463] python "../../build/toolchain/gcc_link_wrapper.py" --output="./chromedriver" -- c++ -fuse-ld=lld -Wl,--build-id=sha1 -fPIC -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,--icf=all -Wl,--color-diagnostics -Wl,--no-call-graph-profile-sort -m64 -Wl,-O2 -Wl,--gc-sections -rdynamic -pie -Wl,--disable-new-dtags -L/usr/local/lib  -fstack-protector-strong -L/usr/local/lib  -o "./chromedriver" -Wl,--start-group @"./chromedriver.rsp"  -Wl,--end-group  -lpthread -lgmodule-2.0 -lglib-2.0 -lgobject-2.0 -lgthread-2.0 -lintl -lnss3 -lsmime3 -lnssutil3 -lplds4 -lplc4 -lnspr4 -ldl -lexecinfo -lkvm -lutil -lgio-2.0 -lrt -ljpeg -lm -lopus -lavcodec -lavformat -lavutil -lopenh264 -lX11 -lXcomposite -lXdamage -lXext -lXfixes -lXrender -lXrandr -lXtst -lfontconfig -lpng16 -lz -lwebpdemux -lwebpmux -lwebp -lfreetype -lexpat -lharfbuzz-subset -lharfbuzz -lxcb -ldrm -lxkbcommon -ldbus-1
FAILED: chromedriver
python "../../build/toolchain/gcc_link_wrapper.py" --output="./chromedriver" -- c++ -fuse-ld=lld -Wl,--build-id=sha1 -fPIC -Wl,-z,noexecstack -Wl,-z,relro -Wl,-z,now -Wl,--icf=all -Wl,--color-diagnostics -Wl,--no-call-graph-profile-sort -m64 -Wl,-O2 -Wl,--gc-sections -rdynamic -pie -Wl,--disable-new-dtags -L/usr/local/lib  -fstack-protector-strong -L/usr/local/lib  -o "./chromedriver" -Wl,--start-group @"./chromedriver.rsp"  -Wl,--end-group  -lpthread -lgmodule-2.0 -lglib-2.0 -lgobject-2.0 -lgthread-2.0 -lintl -lnss3 -lsmime3 -lnssutil3 -lplds4 -lplc4 -lnspr4 -ldl -lexecinfo -lkvm -lutil -lgio-2.0 -lrt -ljpeg -lm -lopus -lavcodec -lavformat -lavutil -lopenh264 -lX11 -lXcomposite -lXdamage -lXext -lXfixes -lXrender -lXrandr -lXtst -lfontconfig -lpng16 -lz -lwebpdemux -lwebpmux -lwebp -lfreetype -lexpat -lharfbuzz-subset -lharfbuzz -lxcb -ldrm -lxkbcommon -ldbus-1
ld.lld: error: failed to open ./chromedriver: No space left on device
c++: error: linker command failed with exit code 1 (use -v to see invocation)
ninja: build stopped: subcommand failed.
*** Error code 1

Stop.
make: stopped in /usr/ports/devel/electron12
=>> Cleaning up wrkdir
===>  Cleaning for electron12-12.0.9_2
build of devel/electron12 | electron12-12.0.9_2 ended at Mon Sep 27 03:11:50 BST 2021
build time: 16:19:48
!!! build failure encountered !!!
 
Top