While interesting, those benchmarks are only useful if you implement something remotely similar to the actual benchmarks. Just glazing over them, they seem to be "unfairly" targeted at low-level languages.
That said, they are fun to look at. I just had a wtf-moment looking at this:
Note that the "alternative" Lisp SBCL and Java 7 programs both outperform Fortran.
Of course I agree with you on wtf-moments. WTF makes Ruby take an hour, and SBCL take 10 seconds? That's two orders of magnitude! But no matter, I imagine a more clever Ruby programmer could reduce that, or just call out to a native library.
> A program that simply switched according the command line arg and then printed…
Of course making programs do less can improve speed, and a great way of doing that is compile-time computation via macros! You can finish the program before it's even run.
> Is "a more clever Ruby programmer" some kind of equivalent to "a sufficiently smart compiler"?
No, since we assume human intelligence here. :P As the Graphics Programming Black Book puts it in the Chapter 1 title, "The Best Optimizer is between Your Ears".
> When is a Ruby program fast? When it's written in C ;-)
I've spent enough time asking for programs for the benchmarks game in Ruby forums, to start to doubt whether the "more clever Ruby programmer" will ever come forward ;-)
Maybe "a more clever Ruby programmer" always drops-down to C?
Maybe there's only a more clever Rails programmer :-)
> Don't make "unfair" accusations -- say why you think that
Because there seems to be a bias in selecting the problems solved in the benchmark games: to be "fair" they should be randomly selected from a pool of all possible problems solved with computer programs. Or, maybe, the frequency of these problems in the real world should be taken into account?
Of course, they're not actually unfair since they hide nothing. The error is in the interpretation, e.g. "My program will be faster if I write it in language X instead of Y."
The nice link you posted sums it up well:
"Programming languages are compared against each other as though their designers intended them to be used for the exact same purpose - that just isn't so."
1) You implied "unfair" by opining on what they would have to do to be "fair"; and you said something less vague than "unfair", you said "unfairly" targeted at low-level languages. If you don't mean to say they are "unfair" then your words are going to confuse ;-)
3) GP? Your opining about "selecting the problems" doesn't support the claim "targeted at low-level languages" anymore than it supports the claim targeted at high-level languages :-)
As you noted the important thing to remember is that timing measurements are not promises, and they aren't general answers to the question - Will my program be faster if I write it in language X?
That said, they are fun to look at. I just had a wtf-moment looking at this:
http://shootout.alioth.debian.org/u32/performance.php?test=n...
~20 seconds vs ~20 minutes??