FutureMark is solvent and wealthy because of its 3DMark series of benchmarking tools. And NVIDIA is wealthy because it has made the market believe, generally rightly, that its graphics cards were the best performers on a wide range of measures. Quite a few graphics card companies have gone down to dusty death competing with it.
Love of creating and making and selling a good product is what gets many companies started, but after a while the scorecard and purpose for a company and its officers and technicians becomes money and market power. FutureMark is in a very dicey conflict of interest, therefore. Its goal, maximizing income, is being pursued by a) charging companies for advance participation and access during test development; and b) selling a purportedly neutral and objective test suite to all comers. Any “Beta Program” graphics company will be in possession of information about exactly how the tests will measure performance, and will be hard pressed to resist the temptation to tweak drivers to accommodate the tests. These tests have immense market impact, both as sales tools (if you get good scores) and as the basis for expert opinion on review sites like this one.
How to Decide?
In the absence of hands-on direct experience with competing cards in the exact applications (usually games) they will be using, consumers must use some other measure. Brand loyalty, received opinion, the confident suggestions of sales clerks, and reviews, will all influence buying decisions. Given two cards of similar pricing and claimed performance, even the tiniest edge in benchmark scores can make the difference. Often such differences are insignificant for their purposes, or maybe for any purposes. But if the scores are fudged, the situation is much worse. The “error bar” around expected and actual performance in their intended uses increases dramatically. That is, the likelihood that they aren’t getting what they think they’re paying for goes way up.
The specific case in the news has come down to the claim by NVIDIA, more or less backed by FutureMark, that its driver tailoring of FPS scoring when it detects that an FPS benchmark is being run at the expense of image quality is actually optimization, not fudging or deception. This is egregiously transparent hooey. The only benefit of having that optimization present is to NVIDIA sales. The consumer will never see the optimization in action, and so will never get those extra FPS–and will never be subjected to the image degradation, either. That makes the scores a con job.
FutureMark’s bread, in the short and medium term, is buttered on the manufacturer side. The pre-release “Beta” membership fee can be perceived as a kind of blackmail by FutureMark. It certainly leaves wide open the suspicion that those who pay get special treatment and information, and those who don’t will have to take their chances. (Theoretically, it could even go so far as to involve deliberate crippling of performance on competitors’ products. Remember Dr. DOS?) So FutureMark gets Money and Power. On the other hand, manufacturers know that their fees make a substantial contribution to FutureMark’s profit margin, and can press for special consideration.
The only solution is to totally separate FutureMark’s income stream from the manufacturers. It should be prohibited from accepting dime one from any of them for any service whatsoever. If there were an umbrella organization that could be billed without making any contact with the manufacturers, that might fly, but it would still be a bit dicey. In any case, there should be no way for FutureMark to benefit from favoring one manufacturer’s product over others. We’re talking about the potential for serious collusion and fraud here; there is little or no room for compromise.
Right now consumers may only be able to rely on gaming benchmarks to find the real performance of graphics cards, but even this can be skewed to some degree. When we attended the Chaintech Reloaded event held in New Orleans a few weeks back, we witnessed nVidia’s mention of how closely it is working with game developers to build its games to perform better on its video cards. Of course this can be taken different ways, one of which is that this will make games perform better with technology used by nVidia. This often has to do with selective support for driver features and compression schemes, but there are many possible forms of “tuning”.
FutureMark’s best hope and greatest asset at the moment is that there isn’t a strong contender for a competitive comprehensive test suite. The door is sure wide open, though; once trust has been compromised or even lost, it’s a long road to recover it.
Author: Brian Hall