Exploring the deep structure of code

It’s been a while since I’ve written code in Java. This week, however, I updated my installation of Eclipse, dusted off an old Java program, and saw that ancient artifact in a whole new way.

Free IT resource Open Source Business Conference (OSBC) May 22-23, 2007 Sponsored by OSBC Free IT resource Virtualization Insights from Top Experts - Learn how virtualization gets real! Sponsored by Dell

Now, I’ve never been an IDE kind of guy. It’s always been Emacs for me, and in InfoWorld circles, I’m not alone in that preference. Both Chad Dickerson and Maggie Biggs have also made the case for the no-frills approach. At the end of the day, programming still comes down to pushing lines of code around in text files, and until we change that paradigm, code editors built to push lines of code around in text files are going to be hard to beat.

Eclipse 3.0 doesn’t complete the paradigm shift, but its powerful refactoring tools hint at things to come. For starters, the ability to rename a method globally, across all of a project’s Java files as well as its unit tests, is convenient. Yet, that’s really just a fancy search-and-replace. More impressive is the ability to restructure code. Eclipse can, for example, convert a code fragment into a method and a call to that method, or it can promote and demote class members within a class hierarchy.

This transformational magic requires some actual analysis of the code. Under the covers, Eclipse maintains an AST (abstract syntax tree) — the deep structure (as linguist Noam Chomsky would say) that underlies the surface structure that we write, read, and edit as lines of Java code. With Eclipse, you begin to get the feeling that you’re working directly with the AST and that the Java text is only a convenient way to visualize it.

It’s still only an illusion, though. When you check your project back into the repository, you’re checking in lines of code, not an AST. The refactoring that you did will be visible to the rest of the project team as line-oriented diffs, not as tree transformations.

At one time or another, every programmer has imagined what it would be like to work directly with the deep structure of code. Some of the best minds in the business are working to make that happen. The legendary Charles Simonyi, who left Microsoft a couple of years ago to pursue his vision of “intentional programming,” says deep structure is at the core of the toolset that his new company, Intentional Software, is building. Sergey Dmitriev shares Simonyi’s vision, and his company — JetBrains, creator of the IntelliJ IDEA Java IDE — wants to do something similar with its next-generation toolset.

These projects are still under wraps, but another champion of deep structure is working out in the open. Jonathan Edwards, currently a visiting engineer with the Massachusetts Institute of Technology’s Software Design Group, has built a prototype system that he is demonstrating in a screencast.

There are big ideas at work here. In Edwards’ prototype, programming, testing, and debugging are just different ways of interacting with a program’s tree structure. Edwards’ 2004 OOPSLA paper, “Example Centric Programming ”, explores one of the benefits of this arrangement: The examples — or “use cases” — that drive program design are worked out in the context of the living and evolving program.

We’ve all heard stuff like this before. I may yet go to my grave without Emacs ever having been pried from my cold dead fingers. It’s worth pondering, now and then, though, what we could do with tools that didn’t think of programs as strings of text.