The rise of open infrastructure

When entrepreneurs pitch their software-as-a-service ideas to me, I always ask how they plan to compete with what I call the galactic clusters -- Amazon, Google, Microsoft, and Yahoo. These giants have set a high bar for Internet-scale operations, and they’re relentlessly pushing it higher.

The answer usually comes back: “We’re confident we can scale out as needed.” Maybe yes, maybe no. A lot depends on architectural choices and operational competence. But either way, if you are merely a planet, you don’t want to butt heads with a galaxy.

If you can’t beat ’em, join ’em. In the days of the old giants, that meant moving into a platform ecosystem, such as Microsoft’s or Apple’s, and then nimbly occupying the available niches until the landlord decided to kick you out.

Will the era of the new giants be any different? Tim O’Reilly raised this key question on his blog. “Being a developer ‘on someone’s platform’ may ultimately mean running your app in their datacenter,” he wrote, adding, “Microsoft has a key advantage over open source, because the Windows Live team and the Windows Server and tools team work far more closely together than open source projects work with companies such as Yahoo!, Amazon, or Google.”

It’s both a stunning observation and a stirring call to action. Neil McAllister, who edits this column and writes the Open Enterprise column, said last week that Microsoft can no longer laugh off the long-heralded “year of the Linux desktop.” I agree, and I saluted Neil the other night as I installed the latest Ubuntu release on my aging ThinkPad. But the desktop isn’t the battleground it once was. I float like a butterfly from Windows to OS X to Linux. My home is in the cloud, and that’s the next frontier for the champions of free and open commodity infrastructure.

At the moment, it seems very unlikely that a motley crew of volunteers distributed around the globe will be able to match the economies of scale and the military discipline that make today’s giant clusters the awesome powers that they are. But shouldn’t we have learned, by now, to expect the unexpected?

We’ve already seen how open source software projects harness collective effort to produce quality results. We’re now seeing how open content projects such as Wikipedia do the same. Can open infrastructure be far behind?

Arguably it’s already here. Yochai Benkler, author of The Wealth of Networks, notes that if we regard the P2P file-sharing networks from a technical rather than a political/legal perspective, we observe the evolution of robust decentralized storage systems. These systems could well represent a bigger threat to Amazon’s metered disk in the cloud, S3, than any of Amazon’s galactic peers will.

Operating on a smaller scale but at a higher level in the stack, open content delivery networks such as CoralCDN, which I mentioned a year ago, will challenge proprietary CDNs (content delivery networks) such as Akamai. If you haven’t tried CoralCDN, just append .nyud.net:8080 to the domain name of any Web URL and watch this amazing experimental cache serve up the page.

Beyond CDNs lie service delivery networks, a new opportunity that even the galactic incumbents have yet to seize. If I were the next Linus Torvalds, itching to create the Linux of open infrastructure, this is where I’d scratch. Innovation in open source was about process more than technology. Innovation in open infrastructure will require both.