Test Center benchmarks: Windows 7 unmasked

Measured by runtime specs and performance benchmarks, Windows 7 M3 looks like Vista, and it runs like Vista. Welcome to Windows Vista R2!

It's here! After months of speculation, Windows 7 was finally unveiled last month at Microsoft's Professional Developers Conference (PDC). Through a series of well-orchestrated keynote presentations and supporting breakout sessions, Microsoft walked conference attendees through the highlights of its new desktop OS: better performance, an improved user experience, and some nifty media-sharing features. Overall, Microsoft's pitch was quite compelling, and the PDC crowd was practically salivating at the chance to play with Microsoft's latest and greatest.

But after the stage props came down, and after the projectors finally went cold, attendees were left with a pre-beta copy of something that looked less like a new OS than the repackaging of an old one. At least that was my impression after I started exploring the Windows 7 M3 (Milestone 3) bits that came on my shiny new 160GB Western Digital USB hard disk (one of the better tchotchkes I've received at a conference). As I reported on my Enterprise Desktop blog, the more I dug into Windows 7, the more I saw an OS that looked and felt like a slightly tweaked version of Windows Vista.

Just what was so new about Microsoft's next Windows, apart from a rejuggled UI? Windows 7 appeared to suck memory like Vista, to consume CPU like Vista, and to have the same consumer focus. How would this product be received by enterprise customers, the vast majority of whom had soundly rejected its predecessor? After all, if Vista wasn't good enough for big business, then surely a Vista-derived encore would meet with a similarly chilly reception.

If any pre-beta software ever called for a close look and benchmark testing, Windows 7 M3 was it. With so many questions to answer, and the fate of Windows in the enterprise hanging in the balance, I rolled up my sleeves and dove in. I started by examining Windows 7's innards -- the kernel and other low-level structures -- then slowly worked my way out to subsystem behavior and application runtime characteristics. Because one of the focal points of Microsoft's keynote presentation was improved performance, I looked for signs that Windows 7 would be faster, more responsive, and less resource-intensive than the bloated Windows Vista.

Note: All the test tools I used for this article are freely available from the exo.performance.network Web site. You can also test your current PC for Windows 7 compatibility now, and then monitor Windows 7 performance on your own system when it enters public beta later this year, using InfoWorld's free Windows Sentinel tool.

The view from inside: a minor tweak to Vista
As I mentioned in the intro, I began my exploration of Windows 7 by poking around the OS's innards. Using a combination of the Windows Performance Monitor utility and some reference data I'd gathered from Windows Vista and XP, I began comparing the runtime structure and composition of various OS processes and services.

First up was the Windows 7 kernel -- aka the System process. When comparing Windows versions, it's always good to start with the kernel because this is where the most fundamental changes take place. For example, when Microsoft went from Windows 2000 to Windows XP, the System process gained 21 execution threads in its default configuration. Likewise, when Microsoft introduced Windows Vista, the kernel gained 39 execution threads.

In fact, the kernel in each major new version of the Windows OS has spawned a different, typically higher number of threads. So when I examined Windows 7 and found a nearly identical thread count (97 to 100) for the System process, I knew right away that I was dealing with a minor point-type of release, as opposed to a major update or rewrite. This was not "MinWin," the mythical, streamlined new Windows kernel that promised a clean break with the bloated Vista.

Next up: the memory footprint for the kernel. Like the System process, memory footprint tends to vary widely from version to version, with successive releases showing an ever-expanding kernel working set. Based on the thread count values I observed in my previous example, I anticipated that the working set would be similar to Vista's. And in keeping with my initial suspicions, Windows 7 M3's System process indeed consumed a similar amount of memory (3.5MB to 4.5MB). So far, the new OS was looking a lot like Vista.

In fact, as I worked my way through the process lists of the two operating systems, I was struck by the extent of the similarities. Key subsystems, including the Desktop Window Manager (dwm) and Client/Server Runtime (csrss), sported similar thread counts and working set sizes. When viewed side by side in Performance Monitor, Vista and Windows 7 were virtually indistinguishable.

Two of Microsoft's selling points for Windows 7 have been improved performance and lower resource requirements when compared to Vista. However, these sorts of gains traditionally have been difficult to realize without significantly modifying the OS at a kernel level, and the telltale signs were showing that the changes were minor. What would happen if I threw some test workloads at this OS?

Benchmarking Windows 7 Milestone 3
To test Windows 7, I employed the DMS Clarity Studio tools suite -- the same tools I used in March to show that Vista was 40 percent slower than Windows XP. Using a combination of the Clarity Studio's ADO (ActiveX Data Objects), MAPI (Messaging Application Programming Interface), and WMP (Windows Media Player) Stress workload objects, I was able to simulate a complex, multiprocess workload under Windows 7 consisting of client/server database, workflow, and streaming media playback tasks. I used the DMS Clarity Tracker agent to record system and process metrics during the test scenarios. All tests were conducted against a 2GB Core 2 Duo (T7200) laptop PC (the Dell XPS M1710) configured to dual-boot between Windows 7 Ultimate M3 and Windows Vista Ultimate SP1.

A nine-way test scenario, involving three concurrent instances of each workload object, turned in nearly identical average transaction times under Windows 7 M3 and Windows Vista. In fact, the scores were so close -- less than a 5 percent delta (in favor of Vista) on the database tasks, and a roughly 2 percent delta (in favor of Windows 7) on the workflow tasks -- that they fell within what I'd typically consider the margin of error for this sort of test.

An analysis of system and process metrics collected during testing showed both operating systems consuming a similar amount of RAM -- 637MB for Vista and 658MB for Windows 7 -- across all active processes, while the overall thread count showed a modest reduction under Windows 7 (712 versus 810 for Vista). Interestingly, Clarity Studio's process objects for the database workloads (ADO Stress) spent significantly less time (61 percent less) executing in kernel mode under Windows 7 than under Vista, perhaps indicating that some of the code paths related to Jet 4.0 database access have been moved into user mode. This could conceivably explain the modest, 5 percent slowdown of these same workloads under Windows 7. The additional Local Procedure Call overhead of moving portions of MDAC (Microsoft Data Access Components) out of the kernel would most certainly be felt by a time-sensitive, looping transactional workload like ADO Stress.

In a nutshell, Windows 7 M3 is a virtual twin of Vista when it comes to performance. The few minor variations I observed during comparative testing are easily explained away by slight tweaks to the kernel (such as the aforementioned MDAC changes); they certainly don't indicate a significant performance overhaul. More important, these variations pale in comparison with the 40 percent gap in performance I've observed between Vista and Windows XP SP3. From a raw throughput perspective, Windows 7 promises to perform as poorly as its predecessor. "Pre-beta" notwithstanding, the reality is that any hope for closing of the performance gap with Windows XP is unlikely to materialize in Windows 7.

Visual cues and other hints show Vista under the hood
Much has been made, by Microsoft and others, of the significant changes to the Explorer UI under Windows 7. The M3 build provided at PDC lacks certain key features -- such as the reengineered task bar --but clearly shows that Microsoft felt the need to do some housecleaning. Printers and other peripherals, including Bluetooth devices, are now managed through a single Control Panel applet, the Device Stage. Some Control Panel items have been eliminated entirely, while others have been shifted around or rolled into neighboring categories (for example, Security and Maintenance).

Overall, the changes are mostly superficial. Even the new Task Bar is simply a twist on the existing Explorer UI model, not to mention a blatant rip-off of the Mac OS X dock. Moreover, none of Windows 7's UI goodness is the result of any architectural changes to the OS. The underpinnings are still clearly based on Vista, which explains why most Vista device drivers and services install without a hitch under Windows 7 M3. Not all Vista drivers behave, however; see the next section for some potential trouble spots.

Otherwise, Windows 7 operates much like Vista. There are subtle visual tweaks here and there, but nothing on the level of the dramatic XP-to-Vista transition. Ironically, Vista users may be more annoyed by the UI changes than users coming from XP. Because the Windows 7 and Vista Aero experiences are so similar, seasoned users of Vista will be more likely to look in the wrong places for common functions. By contrast, XP users won't be burdened with now-outdated Aero navigation skills.

Potential pitfalls: compatibility woes continues
One of the more surprising results of my Windows 7 M3 testing was the number of unexpected compatibility issues that plagued each stage of the process. For example, Daemon Tools, an ISO image-mounting utility that works great under Windows XP and Vista, refused to install under Windows 7. Worse still, when I tried to work around the problem -- by using the Compatibility tab in the MSI file's properties dialog -- I found myself stuck in an endless loop of failed installations and mandatory reboots.

The apparent source of the problem -- an incompatibility between the low-level CD/DVD-ROM simulator driver (SPD version 1.56) and Windows 7 -- was difficult to fathom, considering the kernel composition seemed so similar to Vista's. In fact, when I relayed my experience to some of the Microsoft folks on the show floor of the PDC Partner Pavilion, they were equally puzzled. They also seemed alarmed that Windows 7 was incompatible with anything that ran properly under Vista, a reaction I interpreted as tacit endorsement of my understanding of the Windows 7 kernel.

Unfortunately, that wasn't the end of my compatibility headaches. Running under Windows 7 M3 on a Dell Precision M6400 mobile workstation (review to come), Skype 3.8 would randomly crash without warning. Skype showed no debug dialogs -- it just disappeared. But the worst compatibility issue, and also the most alarming from an architectural standpoint, was with VMware Workstation 6.5.

After installing VMware Workstation on the Dell M6400, I was unable to launch any virtual machines. The VMware UI shell couldn't communicate with the VMware Authorization Service, a problem I eventually worked around by running the shell in Administrator Mode. But this "fix" came at a price: the loss of drag and drop between Windows 7 and the VMware guest OS.

Nor was this the only issue I encountered with VMware Workstation. It seems that the VMware Bridge Protocol -- another of those Vista-compatible device drivers you would expect to work unmodified under Windows 7 -- was nonfunctional, rendering VMware's bridged networking mode useless. (In VMware Workstation, bridged networking is the virtual network configuration that allows the guest OS to communicate with the host OS.)

As of this writing, I have yet to confirm the exact nature of the VMware issues. I suspect the VMware Authorization Service bug had something to do with Windows 7's revamped UAC (User Account Control) system. An attempt to bolster system security by requiring the user to confirm certain common functions via a dialog prompt, the nettlesome UAC has been one of the more unpopular features of Vista. In Windows 7, Microsoft has responded to the criticism by suppressing most UAC prompts by default. Because VMware Workstation 6.5 was designed to work seamlessly with the older, more intrusive Vista model, my guess is that Windows 7's attempts to suppress UAC breaks the VMware shell's UAC interaction logic; it's no longer able to traverse the various process privilege levels and speak to its Authorization Service component.

Just how many Vista-compatible applications will break in this manner is anybody's guess. But as a person charged with supporting a UAC-aware software product on Windows, I'm genuinely concerned. Even more disturbing are the unexplained driver compatibility issues. If Windows 7 really is Vista at its core -- as the close similarity of their System process, memory, and performance profiles suggests -- then the fact that Microsoft has still managed to break applications as popular as Daemon Tools and Skype (both have tens of millions of users) is disconcerting and perhaps even alarming. At the very least, it doesn't bode well for Microsoft's promises to make the Vista-to-Windows 7 transition truly seamless.

1 2 Page 1
Page 1 of 2