A classic NT/Windows 7 gem from the blogosphere

One of the best parts of blogging is the opportunity to interact with readers. Through the comments mechanism attached to virtually every blogging platform, both fans of a blog and its detractors have an opportunity to voice their opinions. And sometimes, the contents of a blog site’s comments section surpasses – in terms of intrinsic value – the original posting. Case in point: The recent rant by Mahmoud Al-Qud

One of the best parts of blogging is the opportunity to interact with readers. Through the comments mechanism attached to virtually every blogging platform, both fans of a blog and its detractors have an opportunity to voice their opinions. And sometimes, the contents of a blog site’s comments section surpasses – in terms of intrinsic value – the original posting.

Case in point: The recent rant by Mahmoud Al-Qudsi over at NeoSmart.net. In his posting, Mr. Al-Qudsi is attempting to discredit the anonymous author of the mysterious “Shipping 7” blog, a person who claims to be a Microsoft insider working on the next version of Windows. After blathering on for several paragraphs about what he believed were amateur “mistakes” in one of the “Shipping 7” author’s web postings – and how this should discredit the blog as a source of insider info – one of Mr. Al-Qudsi’s own readers (username “AnthonyTP”) takes him out behind the proverbial “woodshed” for a quick lesson on Windows history and architecture. What follows is one of the most concise, accurate and informative defenses of the Windows NT architecture I’ve ever read:

NT is NOT like *nix kernels or even other MACH variants like OS X. There is a large element of simplicity to its complexity when it is broken down.

The basic NT kernel and design is also very modular, so by comparing it to the Linux kernel is going to get you into trouble.

NT is a hybrid kernel that took a fairly elegant approach to kernel design, especially for the early 90s, as it was a new conceptual kernel of the time, taking some existing kernel architectu[r]al models and some of the best kernel theories that had not been implemented before.

For example NT has some MACH conceptual ideas and lightweight kernel API set for performance, but then hands this off to increasing levels of APIs co[m]plexity. This gave NT the ability to be very lightweight at the core, but have extended functionality that didn't weigh the core kernel down.

NT's HAL for example was under 64KB, and even on Vista is still 256KB(slightly larger in Vista x64), which for modern hardware is still extremely small. Going from the HAL to the lower kernel and API layers is still very small, especially compared to other OS kernel models in use today. This is how and why the MS Embedded OSes (XP/Vista) can and do work so well, as NT was simply broken apart as needed for the Embedded versions to make them very light (As in used in a router light).

The essential design of the NT kernel is both object based and a client/server model. This is not normal, or something that you will find in any other consumer level OS out there.

What the client/server model of the kernel of NT allows is the light core kernel with limited APIs secondary NT APIs and then subsystems for the actual OS clients that run on top of it. This is why even Win32/Win64 is just a subsystem that sits on NT, and can be ripped off of NT at anytime, as MS has demo[n]strated for YEARS.

The client/server kernel design of NT is also how MS can and does include a BSD or V5 UNIX subsystem for Vista, as the UNIX subsystem is just another OS sitting on the NT kernel client/server model. The UNIX sub[s]ystem in Vista is equal to Win32/64, as they are both sitting on the NT kernel in the same way. Win32/64 only has preference as it is the the default OS subsystem. By both Win32/64 sub[s]ystem and the UNIX subsystem running together on top of NT, they both get the drivers and benefits of the NT layered kernel, but also can cross communicate via the NT kernel, and all of this happens with no emulation layers.

For people truly interested in the NT design, source code of the NT kernel can be obtained from Microsoft for academic purposes, just search www.microsoft.com and be a teacher of some sort.

This gives academic minded people access to and a greater understanding the NT architecture from the HAL to the API layers and subsystems that run on it.

Based on the NT design, I would have to correct you in asserting that ripping out EXT3 from Linux would be as hard as adding in Reiser as a comparison. NT and Linux kernels are VERY different in this regard. Take the recent Linux arguments about the HardLocks code that is giving Linux trouble with multi-processor granularity.

Changes to the older hardlock mechanism in Linux requires a substa[n[tial amount of work and tricks to bypass, and is why things are not going so well. With an NT design, something like this is API wrapped at a low level, and changing the mechanism would be rather simple, not require someone to sift through 9000 lines of code to change the setting.

Another example would be Vista, and Microsoft adding on a Video driver model. (Not just new drivers, but a new model of how they are handled from the kernel as well.) This was easier on NT than it would be on a Linux or even OS X kernel design because of the NT design. This is also how Microsoft was able to add in the new WDDM while still allowing the XP drivers(XPDM) to operatate on Vista if needed. That alone would be a nightmare situation for both driver and kernel level feature support if this was tried on Linux or OS X.

The WDDM not only splits the video driver to a shared kernel/user level model, but it works with the NT kernel of Vista to things no other consumer currently can do. Features like transparent GPU RAM vir[t]ualization, an OS level scheduler for the GPU, creating a pre-emptive state of GPU multitasking controlled by the OS (WDDM), and even multi-GPU processor support that works with the GPU multitasker. The WDDM in Vista allows several 3D applications on screen at the same time, and gives them all extra VRAM than what is available on the GPU hardware, and because the OS handles the multi-tasking, no applicati[o]n can steal the GPU to halt responsiveness, but yet even with demanding games running at the same time, they only lose a few FPS running side by side compared to them running exclusively full screen.

So you can see that dropping in the WDDM to Vista was a good achievement in that it works as transparently as it does, but this would not have been possible if it weren't for the NT achitecture that allows for massive kernel level conceptual changes by adding in new API kernel level constructs that layer on the lightweight lower level NT kernel APIs.

After taking a bit of this in, you can maybe get a glimpse of the simplicity and the complexity of the NT architecture and why the MinWin used at the demo[n]stration that started a lot of this, is no different than the internal MinWin that Microsoft has used and showcased internally for years. It is nothing more than the NT kernel at the basic level with select API layers used, which is something that can be fairly easily done with NT. And yes technically the NT kernel can be fairly small when you remove the upper level kernel level API interfaces and especially the subsystems like Win32/64 that sit on top of them.

This is why when people call for Microsoft to write NT from scratch, many OS engineers/theorists like myself scratch our heads at the level of understanding. Rewrite Win32, sure why not, rewrite NT, a sadly bad idea. NT is more advanced and useful than what people give MS credit for, especially if lightweight, extensible, fast, and modular are your design goals for a kernel.

Also a thing of note, you call BSD a kernel, it technically is a set of APIs, an 'interface' to a kernel not an actual kernel, and on OS X would be the API wrappers around the modified MACH system calls, not the OS X kernel.

Like I said, powerful stuff. Personally, I knew (or at least suspected) much of this already – after working with, and developing for, a platform on a daily basis for over 15 years, you tend to pick-up things. Still, it was nice to see the case for NT’s internal sophistication/purity so eloquently presented by someone with the technical chops to pull it all together (sort of like a poor man’s Mark Russinovich, only less confrontational towards Linux).

It also reinforces my belief that Microsoft blew it with Vista by burying what is still a technically sophisticated, state-of-the-art OS under a mountain of consumer-focused crap. As an original NT purist (I literally wrote the book on “Migrating to Windows NT” back in 1993 – look it up), I’ve bemoaned each new concession to the Windows 9x/Me team holdovers, from win32k.sys in Windows NT 4.0  to the god-awful “Luna” UI theme in XP. For people like me, Vista is an abomination – what happens when big decisions are left in the hands of the “crayons and construction paper” crowd.

Oh well, at least we still have Server 2008...

Related:

Copyright © 2008 IDG Communications, Inc.