Thursday, June 7, 2007

XP vs. Vista - A Tale of Framerates: Redux

We recently published an article comparing framerate performance in XP versus Vista on NVIDIA hardware. Our readership wanted to see what ATI could do with the same task so that we could see if Vista was really to blame for what we saw.

Introduction

We recently published an article that compared framerate performance between Vista and Windows XP in multiple current gaming titles. We wanted to do this because there were many claims that Vista was a poor gaming operating system, or at least that it didn’t perform as well as XP on the same hardware. Up to now, it’s been claimed that poor driver support has crippled Microsoft’s new operating system. We wanted to try to associate hard numbers to this issue and take an industry accounting of what users can expect from Vista. We even went so far as to test two different video cards and driver sets. Both were from NVIDIA, and included a 7600 GS and an 8800 GTS 320MB.

However, our scope was more limited than we realized. After the article, a consistent criticism was that we did not include ATI hardware in our evaluation, so any claims that we made were unqualified. Our readers were absolutely right. One of the reasons that our content is better than most publications is that because we have the best readership in the industry and we listen to them.

It wasn’t a conscious effort on our part to specifically exclude ATI hardware from the original article. It was merely the fact that we had NVIDIA hardware on hand at the time, so we used it. Given ATI’s track record lately, we simply did not have any ATI cards on hand in our former [H] Consumer offices in Austin. The original plan was to just use one video card, but later we thought that two would be better. Who knew that three was actually the right number? We also wanted to keep the experiment as simple as possible. We wanted the article to be about operating systems, not hardware. For this reason, including ATI wasn’t at the forefront of our minds.

Another criticism was that we used "old" drivers from NVIDIA. This was mostly very bad timing. The article was in its very last stages when the new drivers were released and had already been written and approved for publish (not a quick or easy process). That said, it would definitely have been best to include the latest drivers in the article. We simply didn’t know that new drivers would be coming out. As of late, NVIDIA has been of little help as they don’t know when their own drivers are pushing out. To address this issue in this ATI article, we contacted ATI to ask them when their next driver release would be. They did us one better – they gave us the beta drivers that were going to be released as WHQL on May 23rd, which are now known as Catalyst 7.5. In this instance, we’re dealing with the latest and greatest. And just for good measure, we took a few of the games that performed extra-poorly with our NVIDIA hardware in our last article, and gave them another run on the new 158.18/158.22 drivers to see if there was any effect.

Test Set-Up

The test will be conducted essentially exactly as before, except that we now have an ATI X1950XTX 512MB as our video card. The rest of the system is as before, provided by Puget Custom Computers.

Article Image

Again, we will be using the former [H] Consumer gaming line-up, which is what was used in the previous article. As before, we will be mirroring our gaming experiences between the two operating systems as closely as humanly possible. Some titles are easier because of saved games and consistent bot behavior. Others are more difficult because of the complexity of the AI in the game.

Battlefield 2142

We had no problems with getting this title to run on the new hardware. This game was a bit difficult to test because there is no save game option and the AI behaves differently every time. We did our best to stay in the same area of the map between sessions…and not die.

Article Image

Graphics Settings

Article Image

Image Quality – Vista on Top

Article Image Article Image

Article Image Article Image

Company of Heroes

We had no problems getting CoH to run on our system. Installation, patching, and gameplay went smoothly and caused no issues on the ATI video card and driver set.

Article Image

Graphics Settings

Article Image

Image Quality – Vista on Top

Article Image Article Image

Article Image Article Image

Elder Scrolls IV: Oblivion

We also had no issues with this title. In our 30 Days with Vista article, we experienced an anomaly where the floor of our opening sequence jail cell swallowed Uriel Septim and the game was unable to progress. Thankfully, we experienced nothing like that and the ATI hardware was able to play this title without any problems.

Article Image

Graphics Settings

Article Image Article Image Article Image

Image Quality – Vista on Top

Article Image Article Image Article Image

Article Image Article Image Article Image

Flight Simulator X

We experienced a strange event while launching this title. For a game that was designed with Vista in mind, it surprised us that the UI had to change from Aero to “Vista Basic” for the game to run. None of our other titles required this. Otherwise, the game installed and played perfectly fine.

Article Image

Graphics Settings

Article Image Article Image

Image Quality – Vista on Top

Article Image Article Image

Article Image Article Image

Need for Speed: Carbon

As in our previous article, we experienced some weirdness with this title in Vista. On the ATI driver set, we had crashes to desktop as soon as we double-clicked the icon and the game attempted to launch. We set the program to run as the administrator and run in XP SP2 compatibility mode, but still no dice. We went online to check for a solution and found out that you actually have to use Windows 98 compatibility mode to run this game. Once we had that selected, the game ran without any problems. As with the 8800 GTS in our previous article, we couldn’t get FRAPS to show up to do our framerate capture. It turned out that we had to set FRAPS to run as the administrator as well for it to work in Carbon.

Article Image

Graphics Settings

Article Image Article Image Article Image

Image Quality – Vista on Top

Article Image Article Image

Article Image Article Image

Prey

Again, Prey was a problem in Vista. We had to set the game to run as administrator and select XP SP2 compatibility mode for the game to run. We had to also set FRAPS to run as the administrator for it to function in the game. Otherwise, gameplay was smooth and we experienced no problems.

Article Image

Graphics Settings

Article Image Article Image Article Image

Image Quality – Vista on Top

Article Image Article Image

Article Image Article Image

The Sims 2

We had no problems running this title. Installation, patching, and gameplay went off without a hitch on either OS.

Article Image

Graphics Settings

Article Image

Image Quality – Vista on Top

Article Image Article Image

Article Image Article Image

World of Warcraft

We had some major problems getting this game to patch under Vista in our previous article, but we did not experience those issues in this case. After we successfully patched our game, the ATI blazed through the game without any issue.

Article Image

Graphics Settings

Article Image

Image Quality – Vista on Top

Article Image Article Image

Article Image Article Image

NVIDIA Hardware Revisited

As we stated in the introduction, we wanted to see if the new 158.xx driver revision from NVIDIA had addressed some of the significant performance gaps that we observed in our previous article. We retested on the 8800 GTS 320MB and only included the games that seemed most affected by either the operating system or poor support from NVIDIA.

Battlefield 2142

Per our previous experience, we had no problems with getting this game to run.

Article Image

Graphics Settings

Article Image

Image Quality – Vista on Top

Article Image Article Image

Article Image Article Image

Company of Heroes

This game also presented no problems.

Article Image

Graphics Settings

Article Image

Image Quality – Vista on Top

Article Image Article Image

Article Image Article Image

Elder Scrolls IV: Oblivion

We also had no issues with this title.

Article Image

Graphics Settings

Article Image Article Image Article Image

Image Quality – Vista on Top

Article Image Article Image Article Image

Article Image Article Image Article Image

Need for Speed: Carbon

As in our previous article, we had to set Carbon to run in Windows 98 compatibility mode and FRAPS also had to be set to run as administrator.

Article Image

Graphics Settings

Article Image Article Image Article Image

Image Quality – Vista on Top

Article Image Article Image

Article Image Article Image

Conclusion

We presented each game individually, but here they are all together.

Article Image Article Image

As a comparison, here are the graphs from our previous article with NVIDIA hardware.

Article Image Article Image

The results for ATI were largely very impressive. On ATI hardware, we saw that there was generally only a modest difference between XP and Vista in performance. In our previous article, we noted dramatic framerate discrepancies between the two OSes and perhaps drew a bit too much attention to Windows Vista as being the common denominator. As our readers noted, it would have been better to qualify those statements by stating that our experiences were with NVIDIA hardware alone, and it’s not possible to tell if the problem is intrinsic to the operating system, the NVIDIA hardware or driver revisions, or possibly all of the above.

We’ll make that distinction now – the implementation of the driver revision that we used for our NVIDIA hardware was woefully underperforming in a Vista environment. Again, NVIDIA recently released a new driver set, and we wanted to see if there was any effect in some of the games that were disappointing when run under Vista. From the graph above, we see that the new driver set has closed some of the gaps, while others remain wide open. This is interesting to see, as ATI seems to largely have its ducks in a row concerning Vista.

We were intrigued by the 158.xx revision of the NVIDIA driver set as some games showed marked improvement since our last go-around. However, some titles still exhibited large performance gaps - notably, Need for Speed: Carbon. It's important to note, however, that EA has not patched this title since before the retail release of Vista (about six months ago). Between the recent developer patches and perhaps the new driver revision, the performance gap for Company of Heroes closed to a much more reasonable interval.

It's still very interesting to see the overall trend, though. Even on ATI hardware, seven of our eight gaming titles showed a higher framerate in XP over Vista. In the one case where Vista outpaced XP, it was by an average of 0.9fps. Even on the new NVIDIA drivers, our panel showed the same trend.

Now that we've used new drivers and a more comprehensive panel of hardware, we're at a point where we have to look at this profound trend in the data as significant. Even though it's only by a few frames here and there, it's curious to see that Vista continues to underperform (albeit slightly in most cases) in 3D gaming applications compared to XP.

We hope that this follow-up has placated some of the curiosities about what ATI is doing to make sure that gamers are happy with their gaming experiences in Vista. It’s obvious now that ATI should have been included in our previous round-up as it would have been an interesting contrasting element and would have made the article more complete. In effect, NVIDIA got a pass while we put the blame on Microsoft. From our results, it looks like some blame still does go to Microsoft, but it's also clear that NVIDIA's previous driver revision was largely underperforming. We want to express our thanks to our readers for their thoughts and always holding us accountable for the content that we produce.

Source: enthusiast.hardocp.com

No comments: