Is this benchmark fair?

W

Weinter

A new benchmark have emerge testing FreeBSD vs OpenSolaris vs Ubuntu and FreeBSD doesn't seem to be pwning others (in the benchmark) any opinions?
See the benchmark here
 
Sure it's fair. Default settings are not perfect for everything and RELENG_7 still has GIANT in the TTY and VFS layers.
 
That and most differences are marginal. The 2GB file encryption for instance, could be attributed to pth (threading library) being more optimized for linux threading under this specific workload.
It's finding out where the slowdowns are that is hard.
 
Benchmarks were done using GENERIC kernel which as mentioned above is not optimized for speed. GENERIC kernel is optimized to run on as many hardware as possible and to be as stable as possible. This means it has no optimization tweaks and it's bloated with every possible driver that FreeBSD supports.

Another things that shines through this benchmarks is UFS2's poor performance compared to ZFS and Ext3. I think UFS2+gjournal created with "-o time" option would yield a lot better results.

Personally even if FreeBSD was incurably slower then Linux/Solaris I would still take stability and ports over speed any day.
 
ArtemD said:
Another things that shines through this benchmarks is UFS2's poor performance compared to ZFS and Ext3. I think UFS2+gjournal created with "-o time" option would yield a lot better results.
Click to expand...

Sort of. The sequential write test, ext3 seems to do very bad. However, it's hard to translate this to real workloads as the network stack will almost always be involved.

It is kind of weird, they chose to create UFS partitions and yes, I agree I would love to see those results with gjournal and ZFS.
 
Were the tests performed several times? Was the first run of each test discarded ('preheating')? What was the standard deviation? How can we tell if the measurement error is not bigger than the measured differences between systems? Without knowing that, the tests are not really relevant or useful.
 
ArtemD said:
Benchmarks were done using GENERIC kernel which as mentioned above is not optimized for speed. GENERIC kernel is optimized to run on as many hardware as possible and to be as stable as possible. This means it has no optimization tweaks and it's bloated with every possible driver that FreeBSD supports.
Click to expand...

I think they didn't optimize Linux kernel or Solaris kernel as well so that point should be invalid
 
ArtemD said:
Another things that shines through this benchmarks is UFS2's poor performance compared to ZFS and Ext3. I think UFS2+gjournal created with "-o time" option would yield a lot better results.
Click to expand...

Sorry i don't understand this part "-o time" option
Could you elaborate?
 
Weinter said:
Sorry i don't understand this part "-o time" option
Could you elaborate?
Click to expand...
From "man newfs":
Code: 
-o optimization
             (space or time).  The file system can either be instructed to try
             to minimize the time spent allocating blocks, or to try to mini-
             mize the space fragmentation on the disk.  If the value of min-
             free (see above) is less than 8%, the default is to optimize for
             space; if the value of minfree is greater than or equal to 8%,
             the default is to optimize for time.  See tunefs(8) for more
             details on how to set this option.

I haven't done any benchmarks to test effect of this option in real life, but I assume it does help.
 
Weinter said:
I think they didn't optimize Linux kernel or Solaris kernel as well so that point should be invalid
Click to expand...

That is correct. I do think thou that placing too much value on this makes no sense since I assume most of FreeBSD users run custom kernels.

P.S. I personally think FreeBSD is "fast enough" for everything I needed it for. I choose FreeBSD for stability, ports, good documentation etc.
 
The odd thing is, ext3 in stock options is defenetly slower than UFS on certain applications (see http://shaver.off.net/diary/2008/05/25/fsyncers-and-curveballs/).

For me ext3 is simply slower overall, and this is coming from someone who went from Ubuntu 8.1/AMD64 to FreeBSD 7.1-BETA2/AMD64 (still GENERIC too!)

Perhaps they didn't have softupdates on, otherwise I don't know. In my opinion FreeBSD 7.1 is much faster compared to Ubuntu 8.10 (just personal feeling).
 
Kernel debug does not slow the system down. The debug data is in separate files that only get loaded during debugging.
 
Are you certain; whenever someone complains in the mailing list about slow performance on Beta / CURRENT versions the standard reply seems to be "turn the debugging off"... or am I thinking of something else?
 
kamikaze said:
Kernel debug does not slow the system down. The debug data is in separate files that only get loaded during debugging.
Click to expand...

You really dont know what are you talking about.

kernel on CURRENT have many debug options turned on, and it is always slower than one without such options.

Code: 
NOTE TO PEOPLE WHO THINK THAT FreeBSD 8.x IS SLOW:
        FreeBSD 8.x has many debugging features turned on, in
        both the kernel and userland.  These features attempt to detect
        incorrect use of system primitives, and encourage loud failure
        through extra sanity checking and fail stop semantics.  They
        also substantially impact system performance.  If you want to
        do performance measurement, benchmarking, and optimization,
        you'll want to turn them off.  This includes various WITNESS-
        related kernel options, INVARIANTS, malloc debugging flags
        in userland, and various verbose features in the kernel.  Many
        developers choose to disable these features on build machines
        to maximize performance.  (To disable malloc debugging, run
        ln -s aj /etc/malloc.conf.)
 
kamikaze said:
Kernel debug does not slow the system down. The debug data is in separate files that only get loaded during debugging.
Click to expand...

Not true. System with debugging enabled _is_ slower, because it runs various checks (debugging code) that system with debugging disabled does not run, because they get compiled out. This affects both kernel (WITNESS, which makes locking slower by checking lock order, and INVARIANTS, which enables KASSERTs) and userland (malloc debugging).

Debug data you're talking about are so called "symbols". That's true, symbols don't slow things down. But other debugging options, that are enabled by default in -CURRENT (RELENG_8) definitely do.
 
In the article it says they are using FreeBSD 7.1-BETA 2, so I assume all the debugging/WITNESS stuff is not turned on in this case.
 
Well the article didn't convince me to switch our Glassfish server to Solaris (which I was considering), so I think FreeBSD came out pretty okay. Especially since there's some headroom for tuning.
 
trasz@ said:
Were the tests performed several times? Was the first run of each test discarded ('preheating')? What was the standard deviation? How can we tell if the measurement error is not bigger than the measured differences between systems? Without knowing that, the tests are not really relevant or useful.
Click to expand...

Not to mention that:
- systems are not optimized for a single specific workload
- systems are tested one by one for a single specific workload
- No combined tests are done
- systems are evaluated on "nr. of 1st positions", difference between contestants isn't processed/weighed in ranking.

So, for this test to have any merit, you know only that for a specific workload os foo comes out best without it being optimized for it.
If you dedicate a machine for one specific workload (or a workload that is highly dominant for a specific worklad) you always optimize for it and it can't be assumed that "because without optimizing os foo comes out best, it will still be best once optimized".
All in all, it's good for stirring up discussions in forums, but not much else.
 
My thoughts are that these benchmark tests are complete bunk. You can use any test you feel like using to prove so-and-so system is better than so-and-so other system. Big corporations do it all the time.
 
The only debugging options active by default in RELENG_7 are debugging symbols for the kernel. And they only use disk space.
 
ArtemD said:
That is correct. I do think thou that placing too much value on this makes no sense since I assume most of FreeBSD users run custom kernels.
Click to expand...

I'm not sure that's true. Currently all our FreeBSD servers (4.x, 5.x and 6.x) run GENERIC. With 7.0, the only change I'm making is to choose SCHED_ULE. In the distant past I did some more kernel tuning, but there simply hasn't been a need in recent years.

Given how well the GENERIC kernel works, how many users will actually go through the work of building a custom kernel?
 
Well, there's always the subgroup of desktop users running CURRENT with the extra debugging disabled - though I suspect there's not that many of us.
 
tomh009 said:
Given how well the GENERIC kernel works, how many users will actually go through the work of building a custom kernel?
Click to expand...

I agree. I actually had to reread the original quote because I thought it said, "I assume most of FreeBSD users DO NOT run custom kernels.", which I would have expected.
 
You must log in or register to reply here.
Back
Top