First Look: FreeBSD 9.0 On Intel Sandy Bridge

I just read the first page and looked at the pictures on the rest...

I've got to ask, it seems like that graphics card is the ONLY hardware that is "equal" for both OS's. So how can this be a fair test if you are testing OS's on different hardware? Shouldn't the hardware be the same if you are testing the OS?

I'm confused... the author couldn't have just neglected this? right? :\
 
Bobbla said:
I just read the first page and looked at the pictures on the rest...

I've got to ask, it seems like that graphics card is the ONLY hardware that is "equal" for both OS's. So how can this be a fair test if you are testing OS's on different hardware? Shouldn't the hardware be the same if you are testing the OS?

I'm confused... the author couldn't have just neglected this? right? :\
Click to expand...

It's Phoronix. :( They put the bare minimum amount of thought into benchmarking, and don't do anything remotely close to "scientific" or "mathematically sound" or even "statistically relevant" thought into it.

The only way to make this benchmark relevant would be to:
  • Install FreeBSD onto the laptop, without any GUI. Do the benchmarks from single-user mode (or at last with as many services disabled as possible) several times. Graph the results.
  • Install Ubuntu onto the laptop, without any GUI. Do the benchmarks from single-user mode (or at last with as many services disabled as possible) several times. Graph the results.

But, Phoronix isn't about statistically relevant benchmarks. Just crazy headlines and pretty images that bring in page views.
 
phoenix said:
The only way to make this benchmark relevant would be to:
  • Install FreeBSD onto the laptop, without any GUI. Do the benchmarks from single-user mode (or at last with as many services disabled as possible) several times. Graph the results.
  • Install Ubuntu onto the laptop, without any GUI. Do the benchmarks from single-user mode (or at last with as many services disabled as possible) several times. Graph the results.
Click to expand...
I don't understand how benchmarking an operating system in a state that it would never normally be used is of any relevance. Why do we care how operating systems perform in single user mode?
 
aragon said:
I don't understand how benchmarking an operating system in a state that it would never normally be used is of any relevance. Why do we care how operating systems perform in single user mode?
Click to expand...

We don't. However, if you want to benchmark ZFS vs EXT4, why have strigi and cron waste fs load? And more importantly, if strigi is slowing in Ubuntu, but not running in FreeBSD, what's that benchmark worth? Therefore we try to run the minimal required software while benchmarking, and this can be achieved by booting into single user mode.

It's interresting to see that in "theory" FreeBSD is running slower then Ubuntu, because to me FreeBSD "feels" faster. And I can add my music collection to vlc in a second with FFS2 while it takes nearly a minute with EXT4 (or ZFS).
 
aragon said:
I don't understand how benchmarking an operating system in a state that it would never normally be used is of any relevance. Why do we care how operating systems perform in single user mode?
Click to expand...

When benchmarking two different systems, you have to eliminate as many variables as possible, leaving only the 1 or 2 variables that you are trying to test. Everything else should be a constant (identical between systems).

For example, in the Phoronix benchmark, the following are different:
  • CPU speed
  • amount of RAM
  • graphics hardware
  • harddrive
  • OS
  • filesystem
  • version of Xorg
  • version of video driver
  • desktop environment
  • number and type of daemons running in the background
  • probably other things as well

One system is slower than the other. Can anyone definitively say that the slowness is solely due to the OS? Nope. It could be from anything on that list.

Maybe single-user mode isn't the "best" way to benchmark two different OSes. But it eliminates a lot of the differences between the two.

What's annoying about Phoronix is that they have the hardware, they have the software, and they have the time to run crappy benchmarks like the above, so why not spend a little more time, and do the benchmarks properly?

And it wouldn't be that hard to do:
  • Use the exact same piece of hardware for all tests (or two identically configured systems, but that adds another variable to the equation since two "identical" systems rarely ever are)
  • Configure both OSes with the exact same versions of all software and drivers
  • Run the tests multiple times with different filesystems (ext3, ext4, btrfs, zfs on Linux; UFS w/softupdates, UFS w/SU-J, zfs on FreeBSD) to try and eliminate that as a primary variable

Failure to do the above is nothing more than link-/flame-bait without any real value.

Unfortunately, Phoronix has no inclination or desire to do anything beyond link-/flame-bait.
 
alie said:
they are comparing 2 different OS with different hardware...
Click to expand...
Different hardware? Where?

The textual differences in the hardware table are most likely artifacts of their benchmark collecting hardware information from completely different kernels.
 
alie said:
Another bad benchmarking... i am speechless hahaha... they are comparing 2 different OS with different hardware...

http://www.phoronix.com/scan.php?page=news_item&px=MTAyNzA
Click to expand...

The tests were performed on the same hardware. Here'e a clip from a posting to freebsd-stable today

On 12/15/2011 04:41 AM, Michael Ross wrote:
> Am 15.12.2011, 11:10 Uhr, schrieb Michael Larabel
> <michael.larabel@phoronix.com>:
>
>> On 12/15/2011 02:48 AM, Michael Ross wrote:
>
>>> Anyway these tests were performed on different hardware, FWIW.
>>> And with different filesystems, different compilers, different GUIs...
>>>
>>>
>>
>> No, the same hardware was used for each OS.
>>
>
> The picture under the heading "System Hardware / Software" does not
> reflect that.
>
> Motherboard description differs, Chipset description for FreeBSD is
> empty.
>

I was the on that carried out the testing and know that it was on the same system.

All of the testing, including the system tables, is fully automated.
Under FreeBSD sometimes the parsing of some component strings isn't as nice as Linux and other supported operating systems by the Phoronix Test Suite. For the BSD motherboard string parsing it's grabbing hw.vendor/hw.product from sysctl. Is there a better place to read the motherboard DMI information from?

-- Michael
Click to expand...
 
@Everyone

Since the tests were in fact on the same hardware, as the updated article indicates, can someone please validate these test.

I ask because FreeBSD is significantly slower and this should be addressed.

Thanks in advance developers for all that you do.
 
There has been a lot of discussion of this on the freebsd-current mailing list. One of the big questions was why ZFS was used, when UFS was much more comparable to ext4. The answer, as I saw it, was that FreeBSD had been installed on ZFS, so that's what they used. That's not so much benchmarking as "hey, let's see what this does." FreeBSD prerelease kernels have debug code that slows them down; did they build one without that? It's not mentioned. Useful benchmarks can be used to draw conclusions, but there are so many variables here that it's impossible to tell.

The author has been involved in those emails, so there's hope they will be more specific in future tests.
 
wblock@

Thanks for the comment.

Has there been any talk of someone reproducing the test results.

Also, you mentioned this topic has been talked about a lot on the mail list. Where can I read that conversation on the web?
 
frooyo said:
Has there been any talk of someone reproducing the test results.
Click to expand...

If so, I missed it. I think the benchmark is downloadable, but unless you have the same hardware... or are planning on duplicating both their Linux and FreeBSD tests. If you are planning on doing a useful benchmark, it would be good to ask for suggestions first. It helps to be thick-skinned, as benchmarks are easy to do wrong, hard to do right, and generally a boring, painstaking, thankless chore. Science, in other words.

Also, you mentioned this topic has been talked about a lot on the mail list. Where can I read that conversation on the web?
Click to expand...

Much of it is here: http://lists.freebsd.org/pipermail/freebsd-current/2011-December/thread.html. Search for "benchmark".
 
wblock@ said:
If so, I missed it. I think the benchmark is downloadable, but unless you have the same hardware... or are planning on duplicating both their Linux and FreeBSD tests. If you are planning on doing a useful benchmark, it would be good to ask for suggestions first. It helps to be thick-skinned, as benchmarks are easy to do wrong, hard to do right, and generally a boring, painstaking, thankless chore. Science, in other words.

Much of it is here: http://lists.freebsd.org/pipermail/freebsd-current/2011-December/thread.html. Search for "benchmark".
Click to expand...

Why do I need the us the same hardware they did?

Seems like I should just be able to fire up Virtualbox on my machine. Load Ubuntu in one instance, FreeBSD in the other instance.

Then run the tests and show which operating system is proportionally faster/slower.

(Obviously not run the images at the same time)
 
See, that's the tricky part. Different hardware might have vastly different reactions to the same code. CPU and disk cache, bus bandwidth, there are thousands of things that would be changing. Running in a VM is a big difference, too. If you can remove all the variables except one--or in reality, a few--then you can get meaningful comparisons.

If you run the same benchmarks in a VM and FreeBSD is vastly faster... what could you point to and say "My test shows the opposite of theirs, and here's why mine was valid."
 
wblock@ said:
See, that's the tricky part. Different hardware might have vastly different reactions to the same code. CPU and disk cache, bus bandwidth, there are thousands of things that would be changing. Running in a VM is a big difference, too. If you can remove all the variables except one--or in reality, a few--then you can get meaningful comparisons.

If you run the same benchmarks in a VM and FreeBSD is vastly faster... what could you point to and say "My test shows the opposite of theirs, and here's why mine was valid."
Click to expand...

With that mindset, you're imply that benchmark test should never be performed because they are never useful.

Is that correct?

(I hope not)
 
wblock

Since this topic has been obviously exhausted at length in the mail-list, I'll stop asking questions here.

Thanks again for everything you do for FreeBSD
 
Good benchmarks are very useful. The way they are done is to minimize variables so you can tell what is different between A and B. So if you want to compare typical speed for a user-oriented release, compare an Ubuntu release with 9.0-RELEASE, not a prerelease version. Or at least rebuild the kernel to disable the debugging code, as would be done for a real release. Give them comparable filesystems, ext4 versus UFS. Run the benchmarks on the exact same machine, multiple times, throwing away the first result to minimize the effect of filesystem buffering. That would be a reasonable start.
 
You must log in or register to reply here.
Back
Top