Presets, scoring, the different versions

Presets

Previously, to obtain an official 3DMark score we had to be happy with default settings ; however, now Vantage offers four presets, or in other words, four modes that can give you an official score. These are: Entry, Performance, High and Extreme. The letters E, P, H or X precede the score for clearer identification. The purpose of these presets is to enable measuring the performances of different graphic solutions in more relevant conditions. On the other hand, results lose a bit of clarity and we can safely assume that certain PC manufacturers may abuse this system by forgetting to specify the preset used when announcing figures.

Presets have an influence on graphic details but also on the resolution and level of anti-aliasing applied:


In addition to the different options, presets affect the way in which the overall score is calculated. The respective weight of CPU and GPU scores used in the calculation of the harmonic mean is thus modified depending on the preset in order to reduce the influence of the CPU when the graphic load increases.



Cx constants are used to bring scores down to similar levels in the different tests (in Fps for GPUs and in Ops for CPUs) in order that one test does not influence results more than another – or at least each has an impact on the overall score as determined by Futuremark.

The different versions

There are four versions of 3DMark Vantage. The first is free and you only have to download the program. It is limited to the Performance preset by default, there is no access to the more specific tests nor any graphic options. In addition, and this is new, you can only launch the test a single time! You will have to pay $6.95 (for the Basic version) in order to run tests several times while there are otherwise still the same limitations. Of course, this isn’t much to pay, but it’s only a benchmark and not too visually impressive at that.


To obtain a complete version (Advanced), the price is $19.95. All tests and options are then accessible. Here, only launching by command line isn’t possible and for this function you will have to move up to the Professional version which costs significantly more at $495.

Criticism

Criticism

Futuremark decided to pass on DirectX 10.1 for some unknown reason. This is rather strange for a benchmark that is supposed to look towards the future, and especially because the new version places anti-aliasing under the spotlight and DirectX 10.1 adds a number of innovations in this area. The choice was probably made to make things easier and avoid specific development for DirectX 10.1.

On the other hand, Futuremark did devote specific development to the PPU, Ageia’s physics accelerator, which however is now dead and buried. Why? In our opinion, there were two reasons. First and foremost, the development of this benchmark started before Nvidia’s acquisition of Ageia and therefore before the announcement of the death of the PPU (which either way wasn’t as successful as expected and was destined to die out). After that, there was the fact that Nvidia was going to enable its GeForce 8 and 9 GPUs (or only some of them) to accelerate the PhysX PPU API (different from the general PhysX API which doesn’t support any acceleration) and therefore behave like a PPU.

The PhysX PPU API is not used in the two GPU tests which process certain physics effects with the GPU via DirectX and is therefore reserved to only one of the two CPU tests. This choice appears equally strange because a GPU will be able to boost the CPU score in situations where the graphic rendering load is light. It seems obvious to us that in future games that may use the GPU to accelerate physics, the GPU will also be subject to a tough challenge in terms of graphic rendering.

This is all the more incoherent that Futuremark has decided to strongly reduce the impact of the CPU on the overall score, justifying this choice by the fact that the CPU is less important than the GPU for gamers. If the CPU is indeed less important, this is because it’s not entirely saturated and is therefore "free" to process physics, contrary to the GPU. So why did they opt for a scoring system where the GPU can increase the overall score by accelerating physics in CPU test results?

Also, why was the influence of the CPU reduced so much? Does this correspond to what will happen in the future? In our opinion, Futuremark didn’t make this choice based on technical thinking but rather it was an easy way to reduce the influence of multi-core processors which are currently used very little in games beyond 2 cores. At the same time, higher performance CPU cores, whether this is in terms of architecture or frequency, are also penalized.

Still on the same subject, why is the CPU score less important in the Extreme preset than in Entry or Performance? On the one hand, we have to admit that in high resolution and with anti-aliasing the GPU counts more than the CPU even if this is only half true because the CPU always has to provide enough images. However, it’s harder to accept that Futuremark gives more importance to the CPU in entry level machines (which are tested in Entry mode) than in future high end machines (which will be tested in Extreme). What are they implying? That a big CPU is more important for an entry level machine than for the ultra high end?

We should point out that with the formula used in Extreme mode, doubling the CPU score doesn’t even change the overall score by 1% !

A few results

A few results

We conducted a few tests with the new 3DMark and optimized drivers provided by AMD and Nvidia. The smaller tests were deliberately left aside, because their results can sometimes be as much as four times higher when comparing the latest official drivers and the optimized ones. While waiting for AMD and Nvidia to push optimizations to a maximum, these results are of no interest.

We ran tests with a GeForce 8800 GT and Radeon HD 3870, or the current reference cards, combined with an X38 platform equipped with 2 GB of DDR3 and a Core 2 Extreme 9770. First of all, this was done in the Performance preset ("medium"), the default mode which is available with the basic version of the benchmark :


In this mode, the GeForce 8800 GT has 10% higher performances overall when compared to the Radeon HD 3870. The gains essentially coming from the first GPU test.

Next we tested the eXtreme mode, or in 1920x1200 with FSAA 4x. This severely limited the graphic cards in terms of framerate:


The GeForce 8800 GT does better under this load, increasing its lead to 20% higher performances than the Radeon.

Given that the 3DMark Vantage supports the PhysX card (and after having dusted it off), we added it to the configuration equipped with the GeForce 8800 GT in order to observe gains. We purposefully modified only the test results that take into account this accelerator and manually calculated the score in order to avoid variations in the other tests.


The PhysX accelerator increases the CPU2 results by more than 50% which translates into a CPU score that is 15% higher. However, the impact on the overall score, although in the Performance preset, was only 1% given the formula used. With the Extreme setting, the impact is only 0.06% or less than the normal variation in scores between the different tests.

Finally, we tried to obtain the biggest scores possible with the hardware we had on hand. This was a Skulltrail system with two Core 2 Extreme 9775s and of course in Quad SLI with two GeForce 9800 GX2. Unfortunately, activation of Quad SLI resulted in crashing the benchmark. Moreover, because triple SLI wasn’t functional with the Skulltrail, we looked to Quad CrossFire X with two Radeon HD 3870 X2s. In this case, there was no crash but there were visual bugs. We still give you the results so that you have an idea of what such an ultra high end system is capable. Also, the PhysX card was a part of the system.

Conclusion

Conclusion

We were rather disappointed by Futuremark’s new benchmark and expected significantly more, as much in form as in content. It seems obvious that the 3DMark/GeForce FX affair had an impact on Futuremark’s freedom of development, meaning they had to accept that it’s not possible and can even be dangerous to cross much stronger opponents. Moreover, this can brought about the creation of an objective benchmark which gives results that are totally different compared to those in games whose purpose isn’t to be objective in their technical choices but rather to match common configurations.

Futuremark is pragmatic and now works more in a direction that will avoid confrontation with dominant trends in the industry and avoids making waves. This means their development is no longer done according to their own technical vision of future graphic engines, and they no longer simply show results obtained in their benchmark that could then be different than performances in current games. The risk was that they could anger certain manufacturers which sometimes spend large sums to enjoy an influence in their technical choices.

Futuremark is indeed aware that in the commercial success of its benchmark, the "political" aspect has taken on great importance compared to the technical aspect and the way certain components have an impact on scores. For this reason, Futuremark seems to have developed a few test scenes without worrying about the overall pertinence of results and then think that anyway they will be manipulated to provide a representative score. But we wonder representative of what?

We can definitively say that the pertinence of this benchmark no longer comes from the technical aspect or the objective result obtained in the different scenes, but rather the "political" one or the subjective manner in which the scenes are interpreted by Futuremark. This doesn’t mean that results are not of interest but more that they are subject to significant criticism that cannot be defended on a technical level. This was already the case before; however, it’s even more accentuated with 3DMark Vantage.

Is the absence of DirectX 10.1 support related to a technical choice (it’s of little use) or more a political one (crossing Nvidia which largely dominates the graphic market but does not offers its support)? Is the weight of the GPU and CPU in overall scores based on a technical choice (a big GPU is more important than a big CPU in gaming) or a political one (in line with current Nvidia lobbying in which AMD has followed suit)? Is the variable weighting of scores depending on the preset based on a technical choice (and if so which one?) or is it politically based to make everyone happy (favoring the CPU in machines poorly equipped in GPUs to please Intel, while doing the opposite on the high end to please GPU manufacturers) ?