Benchmarketing

They all do it, all the OEMs to one degree or another. Like statistics, you can cherry pick or narrow the focus and tell whatever story you want. My first realization of this was the Apple “snail” ads in the 1990s:

If you pay attention you’ll notice it’s purely an integer test (small disclaimer at the bottom), which is about the only thing the chip was better at. Moreover, ZDNet did an expose that showed how Apple used a compiler optimized for the PowerPC CPU and a generic x86 compile on the Pentium 2. The difference wouldn’t have been nearly as great with equal treatment. As a Mac owner at the time I was disgusted with Apple for the blatant deception.

But, as I mentioned, they pretty much all do it at one time or another. Like Samsung recently throttling everything except popular benchmarks on their phones. So this isn’t about piling on one company or one reviewer or giving vent to ecosystem partisanship.

2 Likes

Rene Ritchie just came out with a rant against the overuse and misuse of benchmarking from tech “journalists”:

Somewhat weakened due to his guilt in the past, doing the same. And this comes as he’s transitioning to a new career, some suspect a job at Apple. He’s already quit MacBreak Weakly likely because it would conflict with whatever he’s going into.

It also comes after the M2 MacBook Air throttling controversy … and the M2 MacBooks having slower SSD performance for the 256GB SKU.

So, with reservations, I think he still makes some good points.

Even better than such cherry-picked deception is the Microsoft Battery Benchmark, which I just can’t get over. I’ve ranted about this before, but here it is again: “Testing consisted of full battery discharge with a mixture of active use and modern standby.” The device is basically off for an unspecified amount of time, how is that in any way a helpful benchmark?? :exploding_head:

As for computing performance, these days there are so many different benchmarks for specialized tasks that I basically stopped trying to interpret them. I just buy something that is reasonably up to date and get on with life. Like the SLS, which is by no means the fastest thing out there, but it checks enough boxes for it to be a good choice for me.

Edit: the thread title is great, didn’t even notice it at first :joy:

1 Like

Yes and what’s sad, but should serve as reminder to Apple is that initially the Power PC was generally significantly faster than what Intel had at the time, but then AMD shocked everyone with the original Athlons and then Intel got serious with the Pentiums.

Proof that Apple shouldn’t let their guard down, and that technology innovation comes in waves, and that the top surfer of the current wave is often the laggard on the next.

PS: One of the legends of the time though I don’t know it for sure to be true, is that a major reason Intel got it in gear to respond to the initial PowerPC chips was the Bill Gates told Intel “they were weeks aways from shipping a PowerPC native version of Windows”

And of of course the punchline there was that in reality it was actually just Gates being Gates and that it didn’t actually exist (and in fact IBM’s existing agreements with MS expressly prohibited it)

2 Likes