Our tests show that Windows Vista and Office 2007 not only smash Redmond's previous records for weight gain, but given the same hardware diet, run at less than half the speed of generation XP
What Intel giveth, Microsoft taketh away. Such has been the conventional wisdom surrounding the Windows/Intel (aka Wintel) duopoly since the early days of Windows 95. In practical terms, it means that performance advancements on the hardware side are quickly consumed by the ever-increasing complexity of the Windows/Office code base. Case in point: Microsoft Office 2007, which, when deployed on Windows Vista, consumes more than 12 times as much memory and nearly three times as much processing power as the version that graced PCs just seven short years ago, Office 2000.
Despite years of real-world experience with both sides of the duopoly, few organizations have taken the time to directly quantify what my colleagues and I at Intel used to call The Great Moore's Law Compensator (TGMLC). In fact, the hard numbers above represent what is perhaps the first-ever attempt to accurately measure the evolution of the Windows/Office platform in terms of real-world hardware system requirements and resource consumption. In this article I hope to further quantify the impact of TGMLC and to track its effects across four distinct generations of Microsoft's desktop computing software stack.
To accomplish my goal, I'll be employing a cross-version test script – OfficeBench – and executing it against different combinations of Windows and Office: Windows 2000 and Office 2000; Windows XP (SP1) and Office XP; Windows XP (SP2) and Office 2003; and Windows Vista and Office 2007. Tests were first conducted in a controlled virtual machine environment under VMware and then repeated on different generations of Intel desktop and mobile hardware to assess each stack's impact on hardware from the corresponding era.
What does this all mean for Windows IT shops? Should they upgrade to Vista and Office 2007? Or should they stick with Windows XP and Office XP or Office 2003? As I’ve argued in a previous article, “Death Match: Windows Vista vs. XP,” most IT organizations will find they can safely skip a generation and avoid Vista altogether. In addition to sending a message to Microsoft that IT won’t tolerate bloat-ware, it also buys you time to allow the hardware cycle to catch-up with what will hopefully remain a static software target, or at least a slower-moving one (through Windows 7) – a way of putting TGMLC to work for you.
[ Monitor your own Windows and Office system performance with the new Windows Sentinel toolsfrom InfoWorld, which include the DMS Clarity Tracker Agent used in these tests. Share your questions and experiences in the companion blog.]
About OfficeBench: The OfficeBench test script is a version-independent benchmark tool that uses OLE automation to drive Microsoft Word, Excel, PowerPoint, and Internet Explorer through a series of common business productivity tasks. These include assembling and formatting a compound document and support workbooks and presentation materials, as well as gathering data through simulated browsing of a Web-based research database. (For more detail, see "Cross-generational Windows/Office performance: About OfficeBench.") OfficeBench is available for free download from the exo.performance.network Web site as part of the DMS Clarity Studio testing framework.
The Stone Age: Windows 2000/Office 2000
Back in 1999, when I was working as an advisor to Intel's Desktop Architecture Labs (DAL), I remember how thrilled we all were to get our hands on Windows 2000 and Office 2000 – finally, a version of the Windows/Office stack that could leverage all of the desktop horsepower we were building into the next-generation Pentium 4 platform. I remember it was also the first time I had a fully scriptable version of the Office suite to work with (previous versions had supported OLE automation only in Word and Excel). Shortly thereafter, the first version of OfficeBench was born, and I began my odyssey of chronicling TGMLC through the years.
First off, let me characterize the state-of-the-art at the time. The Pentium 4 CPU was about to be unveiled and the standard configuration in our test labs was a single-CPU system with 128MB of RDRAM and an IDE hard disk. A joke by today's standards, this was considered a true power-user configuration suitable for heavy number-crunching or even lightweight engineering workstation applications. It was also only marginally faster than the previous-generation Pentium III, a fact that Intel tried hard to hide by cranking up the CPU clock to 1.5GHz and turning its competition with rival AMD into a drag race.
Sadly, I didn't have access to an original Pentium 4 system for this article. My engineering test bed was long ago scrapped for parts, and I doubt that many of these old i840 chip-set-based boxes are still in use outside of the third world. However, I could at least evaluate the software stack itself. Through the magic of virtualization, we can see that, even with only 128MB of RAM, a Windows 2000-based configuration had plenty of room to perform. During OfficeBench testing, the entire suite consumed only 9MB of RAM, while the overall OS footprint never exceeded 132MB of RAM, roughly half of the available memory. Clearly this was a lean, mean version of Windows/Office. It chewed through the test script a full 17 percent faster than its nearest competitor, Windows XP (SP1) and Office XP. View the overall test results. View more detailed test results at xpnet.com.
The Bronze Age: Windows XP/Office XP
The introduction of Windows XP in 2001 marked the first mainstream (not just for business users) version of Windows to incorporate the Windows NT kernel. In addition to better plug-and-play support and other improvements, XP sported a revamped user interface with true-color icons and lots of shiny, beveled effects. Not wanting to look out of style, and smelling another sell-up opportunity, the Office group rushed out Microsoft Office XP (aka Office 10), which was nothing more than a slightly tweaked version of Office 2000 with some UI updates.
Hardware had evolved a bit in the two years since the Windows 2000 launch. For starters, Intel had all but abandoned its ill-fated partnership with Rambus. New Intel designs featured the more widely supported DDR-SDRAM, while CPU frequencies were edging above 2GHz. Intel also upped the L2 cache size of the Pentium 4 core from 256KB to 512KB (the Northwood redesign) in an attempt to fill the chip's stall-prone 20-stage integer pipeline. Default RAM configurations were now routinely in the 256MB range, while disk drives sported ATA-100 interfaces.
Windows XP, especially in the pre-SP2 timeframe, wasn't all that more resource intensive than Windows 2000. It wasn't until later, as Microsoft piled on the security fixes and users started running anti-virus and anti-spyware tools by default, that XP began to put on significant weight. Also, the relatively modest nature of the changes from Office 2000 to Office XP translated into only a minimal increase in system requirements. For example, overall working set size for the entire Office XP suite during OfficeBench testing under VMware was only 10MB, just 1MB higher than Office 2000, while CPU utilization actually fell 1 percent across the three applications (Word, Excel, and PowerPoint). This did not, however, translate into equivalent performance. As I noted before, Office XP on Windows XP took 17 percent longer than Office 2000 on Windows 2000 to complete the same OfficeBench test script. View the overall test results. View more detailed test results at xpnet.com.
I was fortunate enough to be able to dig up a representative system of that era: A 2GHz Pentium 4 system with 256MB of RAM and integrated Intel Extreme graphics. Running the combination of Windows XP (SP1) and Office XP on bare iron allowed me to evaluate additional metrics, including the overall stress level being placed on the CPU.
By sampling the Processor Queue Length (I ran the DMS Clarity Tracker Agent in parallel with Clarity Studio and OfficeBench), I was able to determine that this legacy box was only moderately stressed by the workload. With an average Queue Length of three ready threads, the CPU was busy but still not buried under the computing load. In other words, given the workload at hand, the hardware seemed capable of executing it while remaining responsive to the end-user (a trend I saw more of as testing progressed).
The Industrial Revolution: Windows XP/Office 2003
Office 2003 arrived during a time of real upheaval at Microsoft. The company's next major Windows release, code-named Longhorn, was behind schedule and the development team was sidetracked by a string of security breaches in the Windows XP code base. The resulting fix, Windows XP Service Pack 2, was more of a relaunch than a mere update. Whole sections of the OS core were either replaced or rewritten, and new technologies – such as Windows Defender and a revamped firewall – added layers of code to a rapidly bloating platform.
Into this mess walked Office 2003, which, among other things, tried to bridge the gap between Windows and the Web through support for XML and the ability to store documents as HTML files. Unlike Office XP, Office 2003 was not a minor upgrade but a major overhaul of the suite. And the result was, not surprisingly, more bloating of the Windows/Office footprint. Office suite memory consumption went up modestly to 13MB during OfficeBench testing, while CPU utilization remained constant versus previous builds, despite the fact that the suite was spinning an extra four execution threads (the overall thread count was up by 15).
Where the bloat took its toll, however, was in raw application throughput. Completion times under VMware increased another 8 percent vs. Office XP, putting the Windows XP (SP2) and Office 2003 combination a full 25 percent off the pace of the original Windows 2000/Office 2000 numbers from three years earlier. In other words, with all else being equal – hardware, environment, configuration – Microsoft's desktop computing stack was losing in excess of 8 percent throughput per year due to increased code path complexity and other delays. View the overall test results. View more detailed test results at xpnet.com.
Of course, all else was not equal. Windows XP (SP2) and Office 2003 were born into a world of 3GHz CPUs, 1GB of RAM, SATA disks, and symmetrical multithreading (that is, Intel Hyper-Threading). This added hardware muscle served to offset the growing complexity of Windows and Office, allowing a newer system to achieve OfficeBench times slightly better (about 5 percent) than a legacy Pentium 4 system, despite the latter having a less demanding code path (TGMLC in action once again).
Welcome to the 21st century: Windows Vista/Office 2007
Given the extended delay of Windows Vista and its accompanying Office release, Microsoft Office 2007, I was understandably concerned about the level of bloat that might have slipped into the code base. After all, Microsoft was promising the world with Vista, and early betas of Office showed a radically updated interface (the Office "ribbon") as well as a new, open file format and other nods to the anti-establishment types. Little did I know that Microsoft would eventually trump even my worst predictions. Not only is Vista and Office 2007 the most bloated desktop software stack ever to emerge from Redmond, its system requirements are so out of proportion with recent hardware trends that only the latest and greatest from Intel or AMD can support its epically porcine girth.
Let's start with the memory footprint. The average combined working set for Word, Excel, and PowerPoint 2007 when running the OfficeBench test script is 109MB. By contrast, Office 2000 consumed a paltry 9MB, which translates into a 12-fold increase in memory consumption (170 percent per year since 2000!). To be fair, previous builds of Office benefited from a peculiar behavior common to all pre-Office 12 versions: When minimized to the task bar, each Office application would release much of its noncritical working set memory. This resulted in a much smaller memory footprint, as measured by the Windows performance counters (which are employed by the DMS Clarity Tracker Agent used in my tests).
Microsoft has discontinued this practice with Office 2007, resulting in much higher average working set results. However, even factoring in this behavioral change, the working set for Office 2007 is truly massive. Combined with an average boot-time image of more than 500MB for even the minimal Windows Vista code base, it seems clear that any system configuration that specifies less than 1GB of RAM is a nonstarter with this version. And none of the above explains the significantly higher CPU demands of Office 2007, which are nearly double that of Office 2003 (peak utilization of 73 percent versus 39 percent). Likewise, the number of execution threads spawned by Office 2007 (32) is up, as is the total thread count for the entire software stack, which is nearly double the previous version (615 versus 370). View the overall test results. View more detailed test results at xpnet.com. Compare memory consumption.
Clearly, this latest generation of the Windows/Office desktop stack was designed with the next generation of hardware in mind. And in keeping with the TGMLC pattern, today's latest and greatest hardware is indeed up to the challenge. Dual cores, combined with 4MB or more of L2 cache, have helped to sop up the nearly doubled thread count, while 2GB standard RAM configurations are mitigating the nearly 1GB memory footprint of Vista and Office 2007.
The net result is that, surprise, Vista and Office 2007 on today’s state-of-the-art hardware delivers throughput that's still only 22 percent slower than Windows XP and Office 2003 on the previous generation of state-of-the-art hardware. In other words, the hardware gets faster, the code base gets fatter, and the user experience, as measured in terms of application response times and overall execution throughput, remains relatively intact. The Great Moore's Law Compensator is vindicated.
Give and take
The conventional wisdom regarding PC evolution, that Microsoft devours every Intel advance, continues to hold true right up through the current generation of Windows Vista and Office 2007. What's shocking, however, is the way that the IT community as a whole has grown to accept the status quo. There is a sense of inevitability attached to the concept of the Wintel duopoly, a feeling that the upgrade treadmill has become a part of the industry's DNA. Forces that challenge the status quo – such as Linux, Google, and Apple – are seen as working against the very fabric of the computing landscape.
But as Microsoft is learning, you can only push your customers so far before they push back. In the case of Windows Vista, the combination of heavy hardware requirements and few tangible benefits to IT has resulted in a mass rejection of Microsoft’s latest and greatest. Companies are finally saying enough is enough and stepping off the treadmill, at least for a while. Microsoft’s challenge will be to woo these customers back, and they can start by taking a hard look at their OS and application development practices. Instead of targeting the next generation of hardware, Microsoft engineers should try making sure that their new features and functions work well on the hardware of today, thus guaranteeing that they won’t overshoot their target and disrupt the fragile TGMLC balance. Maybe then customers will start looking forward to upgrading for a change.
Windows 7 is suddenly telling users it isn't genuine -- and it has nothing to do with Windows being...
Windows users are reporting significant problems with four more October Black Tuesday patches
The larger design is very welcome, but there's much more to the iPhone 6 than a bigger screen
Sponsored by Rackspace
Sponsored by Nuage Networks
Sponsored by Fibre Channel Industry Association
The end of typing as we know it may be a gesture, a blink, or a thought away
Surface Pro 3 is relatively glitch-free after its 10th firmware update, released last week, resolved...
As these examples of well-known applications show, the source code behind proprietary software doesn’t...
Pretenders to the messaging throne have been many, but email still reigns. Here’s how that might change...