Benchmarks: FreeBSD 13 vs. NetBSD 9.2 vs. OpenBSD 7 vs. DragonFlyBSD 6 vs. Linux - Phoronix

grahamperrin

grahamperrin

 
February 2021 (FreeBSD 13 BETA):

forums.freebsd.org

Phoronix: FreeBSD 13 BETA Benchmarks - Performance Is Much Better

Article: https://www.phoronix.com/scan.php?page=article&item=freebsd-13-beta1 Straight to the benchmark graphs (scroll down): https://openbenchmarking.org/result/2102103-HA-FREEBSD0515&imw=1&sgm=1&imw=1
forums.freebsd.org forums.freebsd.org

December 2021 (FreeBSD 13.0-RELEASE):

Benchmarks: FreeBSD 13 vs. NetBSD 9.2 vs. OpenBSD 7 vs. DragonFlyBSD 6 vs. Linux - Phoronix

www.phoronix.com www.phoronix.com
Discussions
Zstandard compression

FreeBSD 13.0 (from the article): <https://openbenchmarking.org/result/2112039-TJ-BSDLINUX675&sgm=1&sor&&sgm=1&rmm=CentOS+Linux+8,Clear+Linux+35320,DragonFlyBSD+6.0.1,NetBSD+9.2,OpenBSD+7.0,Ubuntu+20.04.3+LTS,Ubuntu+21.10&ppt=D#r-3384cfbd5b4a7dac9491a6763fa08dddaa48409f>

At least for compression level 8, superior results with vastly inferior hardware: <https://openbenchmarking.org/result/2112127-TJ-2112123TJ48>

Cross-reference <https://www.phoronix.com/forums/for...s-x-hurd-others/1295700?p=1296018#post1296018>
 
Alain De Vos said:
-Perform high intensive database operations
-Pull the power cord
-Check how much data is LOST.
Click to expand...

I'm always interested in this type of thing, however it's probably unreasonably difficult to quantify/qualify losses – reproducibly – in a way that can be meaningful for benchmark purposes.

An extreme example of consequences of untimely interruption during an intense operation (not database-focused):

 
I remember after a power outage, the ufs fsck run, and put the length of some files to zero.
Only the filename in directory listing was "conserved".
 
Sorry, one just can't do a scientific analysis of a non-scientific benchmark.

Sometimes i think Phoronix test considers only computers with redundant power supply, redundant UPS-sytem, and unlimited ram.
Which are nowhere real conditions.
Sometimes Phoronix is measuring disk-acces-time (ie.syncing) , sometimes he is measuring ram-memory-access-time (ie caching) and this proves nothing about the underlying operating system and file-system.
 
Alain De Vos said:
What a pile of crap.
Click to expand...
Exactly. These benchmarks are disorganized, sloppy, probably unreproducible, and don't give anyone guidance for real world installations. Phoronix/Larabel is nothing but a bad journalist who generates his own news.

Anecdote: A friend and their spouse were for many years an independent reporter in a far away country that is very hot and often has disastrous wildfires. The joke they used to make was that one of them would be in the front of someone's house, interviewing the homeowner about how they feel that their house is about to burn down, while the other would be in the back of the house pouring gasoline on the walls.
[/QUOTE]

(About data loss tests)
grahamperrin said:
I'm always interested in this type of thing, however it's probably unreasonably difficult to quantify/qualify losses – reproducibly – in a way that can be meaningful for benchmark purposes.
Click to expand...
It is a bit difficult to measure. The problem is that with modern OSes and file systems, most of the time exactly nothing happens. So you need to try a lot of times. The standard technique is to set up a bank of systems (10 or 40 is reasonable, as it fits in a rack), equip them with remote-controllable power supplies, and then crash the regularly (every few minutes, barely long enough to reboot, check for data loss, and start a synthetic workload that stresses the storage devices and the write path). With such a system, you'll get several thousand samples per day. If the system survives a few weeks without any incidents, that is statistically quite meaningful.

There are two problems with this attempt. First, consumer-grade hardware will probably show problems in unrelated components, in particular power supplies and fans, and sometimes even motherboards (the DC/DC converters in the motherboard don't like the surges). This means that just babysitting and diagnosing problems turns into a full-time job. Second, the problem is more often than not the storage device itself. A friend in the industry was testing SSDs early on (in the 2000s, when SSDs were still hot and exciting), and found that just power-cycling SSDs during writes would cause completely bizarre behavior, such as timeouts, data-dependent behavior, and so on. That means that once a "data loss" event happens, diagnosing the root cause will take a lot of effort. You can't just say "the system failed", and even less can you blame the file system code, when in reality many failures are caused by disk hardware, CPUs, and so on.

Reminder: Memory and CPUs are far from perfect either, in particular if extensively power cycled.

Alain De Vos said:
Sorry, one just can't do a scientific analysis of a non-scientific benchmark.
Click to expand...
That's a really good summary of Phoronix. The GIGO principle applies here.
 
Why rant about Phoronix when something can be patched in FreeBSD to work around the issue that's documented by Facebook?
 
There used to be posts like this on Phoronix and people would do the same tests on their hardware and see that FreeBSD was usually faster than Linux in the same tests. I don't know how to explain it. Perhaps Phoronix uses super recent hardware that performs worse on FreeBSD. Maybe he's cherry-picking his results a bit to put Linux in a better light. I did some benchmarks today on FreeBSD and Clear Linux, on the following hardware: i3-3240, 4GB RAM @ 1600Mhz, 850 EVO 500GB, Nvidia GTX 650. For FreeBSD I benchmarked the version 12.3 install which I have been using daily for almost 4 years usage. For Clear Linux I used an installation on a USB stick, so FreeBSD uses the proprietary Nvidia Linux driver, and Clear Linux uses nouveau. There is also a slight difference in Firefox version but I don't think these things had a big impact on the results. So these are my results:

FreeBSD 12.3 + Firefox 102
WebXPRT 4 112
Kraken 1.1 1774.6
Speedometer 2.0 62.5
Octane 2.0 16317
Basemark Web 3.0 141.44
SilverBench P3425
JetStream 2 48368
Gimp start time 4.5s the first time
LibreOffice start time very slow, maybe because I installed thousands of fonts
Google ping latency 13.4 ms
RAM XFCE and ZFS 810MB

Clear Linux + Firefox 97
WebXPRT 4 111
Kraken 1.1 1799.9
Speedometer 2.0 74
Octane 2.0 16625
Basemark Web 3.0 272
SilverBench P2216
JetStream 2 62751
Gimp start time 7s the first time
LibreOffice start time 3s or 4s the first time
Google ping latency 13.55 ms
RAM Gnome 910MB

FreeBSD opens the game 0 A.D. and the spreadsheet app Gnumeric both in exactly 1 second on my FreeBSD system. But I couldn't test the launch times in Clear Linux because I was using a Live USB stick with no persistence. Clear Linux apps start faster if you already opened them, I don't see this with FreeBSD.

1. FreeBSD is much slower in Basemark Web 3.0 and JetStream 2
2. Clear Linux is much slower in SilverBench

Clear Linux may indeed be faster on average, but not as much as you would expect by phoronix's benchmark and in not all things Clear Linux is effectively faster. Clear Linux is the fastest of all Linux systems according to Phoronix.

I think my benchmarks say that FreeBSD will usually perform similarly or slightly better than Ubuntu/Fedora/Arch Linux/Debian/Alpine Linux/NixOS.
 
Yesterday I compared Clear Linux's performance against FreeBSD in WASM, JavaScript, HTML5 and CSS.

These were my findings for JavaScript, CSS, HTML5, WASM:

JavaScript

Octane 2.0 JavaScript Benchmark

Octane 2.0 JavaScript Benchmark - the Javascript benchmark for the modern web
chromium.github.io chromium.github.io
Conclusion: The difference between FreeBSD and Clear Linux in these benchmarks is less than 5% on average in Chromium.

CSS

CSS Layout Performance Test

IE9 Test Drive Demo
testdrive-archive.azurewebsites.net testdrive-archive.azurewebsites.net
Conclusion: FreeBSD wins in Maze Solver in Chromium, Clear Linux wins in StyleBench in Firefox

HTML5

Chalkboard HTML5 Benchmark

HTML5 and SVG Performance Benchmark
testdrive-archive.azurewebsites.net testdrive-archive.azurewebsites.net
Conclusion: FreeBSD is 1% faster than Clear Linux in Firefox in this test.

WASM
Conclusion: FreeBSD is 3% faster than Clear Linux in Chromium in this test.

My general idea would be that FreeBSD has similar CPU performance to Clear Linux.
Clear Linux doesn't have a decent ZFS implementation and also seems to have lower IOPS because many apps start slower.

Doesn't this result mean that FreeBSD has slightly higher CPU performance than most Linux distros and more IOPS than most Linux systems?
In my test Clear Linux is much faster than AlmaLinux in StyleBench and Speedometer in Firefox.
 
Voltaire said:
Clear Linux doesn't have a decent ZFS implementation and also seems to have lower IOPS because many apps start slower.

Doesn't this result mean that FreeBSD has slightly higher CPU performance than most Linux distros and more IOPS than most Linux systems?
In my test Clear Linux is much faster than AlmaLinux in StyleBench and Speedometer in Firefox.
Click to expand...

Application start performance on identical hardware should be dominated by filesystem readahead. But more aggressive readahead costs memory elsewhere.

Speed of malloc(3) also plays a role. It it actually not difficult to beat the malloc in glibc. But you can fix it by using LD_PRELOAD to use Google's tcmalloc. Another factor is how many pre-zeroed VM pages are available. Always having many is also a tradeoff with memory usage.

Benchmarking this repeatedly is slightly annoying since FreeBSD doesn't have drop_caches for VM pages.
 
cracauer@ said:
Benchmarking this repeatedly is slightly annoying since FreeBSD doesn't have drop_caches for VM pages.
Click to expand...
It is not with all apps that I notice it, but with GIMP, for example, which is installed on Clear Linux by default. GIMP opens almost twice faster on FreeBSD than on Clear Linux, you don't need an exact timer to observe it.

On the other hand, Clear Linux is also snappy on my old hardware and Wayland is nice to have, although I think Compton works well too.

I think Clear Linux uses a lot of Flatpak for apps. 0 A.D. boots as fast as in FreeBSD, it takes one second on a decent SSD. It could also be Flatpak causing some Clear Linux apps to start a little slower. It's hard to say for sure, but I found that Emacs also started faster on FreeBSD.
 
I ran some other tests and these are my results:

WebXPRT 3 on Firefox.
Clear Linux: 179
GhostBSD: 180

AnTuTu HTML5 Test on Chromium.
Clear Linux: 41599
GhostBSD: 41763

RoboHornet Pro on Chromium.
Clear Linux: 8.055 seconds
GhostBSD: 7.459 seconds

My general conclusion would be that FreeBSD and Clear Linux are, on average, almost equally fast in Chromium and Firefox.
I've done quite a bit of general testing to say this with certainty.
Browsers are one of the most popular types of software out there.
I don't think I'll blindly trust Phoronix benchmarks from now on.
 
The tests i performed speed of linux & freebsd are almost the same.
One did better in one test, another did better in another test.
So speed should not be determinend if you chose one or another.
 
One could factor in the time needed to keep the installation running, adapt the setup to new wasy of doing things, etc.
I found that I spend very little time doing this for my FreeBSD boxes.
 
Crivens said:
One could factor in the time needed to keep the installation running, adapt the setup to new wasy of doing things, etc.
I found that I spend very little time doing this for my FreeBSD boxes.
Click to expand...
A server must also have quite high performance. If you read the Phoronix article he makes it seem like Linux is on average 2x faster than FreeBSD. Although in reality FreeBSD is about exactly as fast as Clear Linux in most of the popular apps. Then you should also know that Clear Linux is a lot faster than, for example, AlmaLinux. Although Phoronix pretends that AlmaLinux is 'lightning fast' in its benchmarks. AlmaLinux isn't nearly as fast as Clear Linux, I'd guess it's 27% slower than Clear Linux on average. Which means that overall, FreeBSD is going to be just a little bit faster than most of the popular Linux distros. Think of Debian, OpenSUSE, Ubuntu and AlmaLinux. They will have slightly lower performance than FreeBSD in most apps.

What bothers me about Linux is its stability. The FreeBSD system has never completely crashed in over 4 years. If the desktop crashes, which I have hardly seen, I can always go to a console and shut down the PC from there. I've been testing Clear Linux for a few days now, and running FlightGear in Wayland completely crashed the system. By that I mean I had to cut power to the PC by holding down the power button. The system stopped responding at all. I can still understand that Wayland on nouveau does not work perfectly as it should. But then on xorg I also randomly saw two complete system crashes. I'm going to argue that not every Linux is going to crash several times in a few days, but I've found that complete system crashes are a common phenomenon with Linux systems. I'm kind of a 'distrohopper' by nature, so I've used hundreds of Linux systems in the meantime. But with none of those systems have I ever felt as reliable in terms of stability as FreeBSD.

Another issue for me is the audio. I use some custom options on FreeBSD which makes the audio very good, although the default audio quality is very good as well. The audio quality of Clear Linux was nothing short of a horror. It literally sometimes makes popping noises that I never heard on FreeBSD. And I struggled to understand people in Zoom. Clear Linux's sound in audio also gives me the feeling that it has many times less detail and that the frequencies are not correct or are partially missing completely.
 
When I benchmark straight applications (as in one at a time) I rarely detect differences between Linux (Debian) and FreeBSD.

Things get interesting, however, if you run different benchmarks competing for different resources at the same time. For example, load the CPUs to the max and then also benchmark NFS serving. You can see different tradeoff decisions made in Linux and FreeBSD pretty clearly.
 
You must log in or register to reply here.
Back
Top