oliverh said:
>Silly synthetic benchmarks aren't of any help. Comparing heavily tuned operating systems like Ubuntu with stock operating systems like FreeBSD is nonsense, comparing alpha-crap-FS like brtfs with a mature FS like ZFS too. Phoronix is an eldorado for hardware junkies with lots of time at hand.
I think that is too harsh. Phoronix has interesting articles; and yes benchmarking is always controversial because it can always be debated what the numbers actually mean. In other words, often the conclusions humans attach to some numbers are wrong; not the numbers themselves. This is because the number does not represent their understanding of what is being tested. Thus, when benchmarking it is very hard to make definite conclusions unless you're doing a really thorough test setup with alot of variables involved.
However, it all depends on what you want to test, what kind of conclusions you would like to make. Comparing ZFS directly against Ext4 is pretty much impossible i think. But if you're a user that won't do any tuning and will just use an OS' filesystem as-is, then benchmarks done with heavy tuning won't matter to you; you're only interested in out-of-the-box performance.
When you test Linux and FreeBSD setups for filesystem performance, what you're actually comparing is how two different
Operating Systems perform with an out-of-the-box configuration under the circumstances you tested it with.
Likewise, doing tests which use I/O only partly like compression tests, can still be of interest to the user. It won't be an all-I/O benchmark so you're testing alot of variables here; but still could help answer the question whether different filesystems are actually noticeable when doing your casual tasks. If compressing files is one of them, such a test can still be worthwhile.
Also keep in mind Phoronix isn't commercial (or at least doesn't appear to be). If you think they could be doing the tests better, then help them with constructive criticism and turn it into advise rather than discrediting a nice hobby website. Also keep in mind that benchmarking properly is extremely hard. I've not seen a single review yet where i think it comes close to a perfect test setup.
My thoughts.
-sub