What's holding software back?

I recently downloaded and ran a 1981-era copy of VisiCalc. The VC.COM file weighs in at less than 28KB, which, as co-creator Dan Bricklin says, is smaller than many GIF and JPG files on today's Web.

Free IT resource Hear how top CIOs turn change into a competitive advantage. Sponsored by HP Free IT resource Attend the SOA Executive Forum: Breaking SOA Bottlenecks SOAExecForum.com/may2007 Sponsored by InfoWorld

Recounting the birth of the spreadsheet, Bricklin makes several other cogent observations. The essential nature of the spreadsheet, he rightly notes, has changed very little in nearly a quarter century, from VisiCalc's introduction in 1979 to Microsoft Excel's 2003 upgrade. Perhaps equally remarkable is that VisiCalc can still run on today's computers — in a DOS box on Windows Server 2003, for example.

Curmudgeons love to pounce on these facts as evidence that we've squandered the riches bequeathed to us by Moore's law. Compared with a 1981 PC, a modern one is 500 times faster, has 2,000 times as many transistors, and can store 8,000 times as much data. It would be hard to argue convincingly that Excel 2003 surpasses VisiCalc by these orders of magnitude. Even so, few would choose to go back. Software advances at a different rate, but it does move forward.

It's not the rate of advance that worries me so much as the method of propulsion. The arc of software progress is defined not by increasing speed or capacity, but by the growing complexity of the data, events, messages, and relationships flowing through software systems. We deal with that complexity by layering abstractions on top of abstractions. A file system abstracts block device drivers; the Java and .Net virtual machines abstract CPUs. It's a brilliant strategy. We get to focus on the higher-order abstractions and pretend that the primitive ones don't exist. But in reality they still do. Nothing that's been hidden ever really goes away. And when abstractions leak, as they always do, things can get messy.

Recently, while working with the latest versions of Windows Server, Exchange, Office, SharePoint Services, and Live Communication Server, I ran into a classic leaky-abstraction problem. As has often been the case throughout the years, it had to do with IIS and security permissions. Back in the '90s, there was already confusion between lower-level permissions applied to the NTFS (NT File System) and higher-level permissions that were defined in the IIS metabase and that governed virtual Web sites. The new century brings another level of abstraction: .Net. After checking all the usual suspects, I remembered that the service giving me trouble — SharePoint — is based on ASP.Net. In that realm, there's a new layer of security permissions that segregate sets of .Net assemblies into trust zones. These permissions are declared in a file called machine.config, and it was there that I found and fixed the problem.

This isn't just a Windows phenomenon, of course. Every major software system has, at its core, what Dave Winer likes to call a lizard brain. The roots of Linux reach down through many layers to its lizard brain. Mac OS X archeologists can explore not only that same deep history but also a parallel 15-year NextStep legacy. Every software architect longs for a chance to reorganize — or as they like to say, "refactor" — to a simpler and stronger foundation for new layered abstractions. Few organizations have the resources to maintain and evolve a working system while mercilessly refactoring to produce its successor. Microsoft is among the lucky few. We'll see, in a couple of years, how well Longhorn has exploited that rare opportunity.