Building connection engines with metadata

In “Scan This Book!” -- a May 14 manifesto published in The New York Times Magazine -- Wired’s Kevin Kelly explores the copyright battle provoked by Google’s ambition to digitize millions of library books. It’s ultimately a clash of business models, he concludes. In a networked world, where copying is implicit in every transfer of information, copies lose their direct economic value but gain indirect value as “discovery tools” that attract attention, sponsorship, and subscription.

Free IT resource Hear how top CIOs turn change into a competitive advantage. Sponsored by HP Free IT resource Attend the SOA Executive Forum: Breaking SOA Bottlenecks SOAExecForum.com/may2007 Sponsored by InfoWorld

Search is the game-changer. “Things can be found by search,” Kelly adds, “only if they radiate potential connections.” Yes, but that leads to a more nuanced view of search than the one Google and its competitors have popularized. For the past few months I’ve been revamping search on InfoWorld.com. Full-text search works more effectively now and is augmented by streams of metadata and by RSS syndication. It’s all about making the site a better connection engine.

On my blog, I’ve chronicled the development of a pair of applications called InfoWorld Power Search and InfoWorld Metadata Explorer. Both exploit three kinds of metadata to turbocharge the discovery of InfoWorld articles: first, structured document titles that include key attributes, such as date, type, and author; second, tags assigned by way of del.icio.us; third, subtitles and lead paragraphs.

In InfoWorld Power Search, aka iws, I use these metadata streams to add value to the output of our Ultraseek search engine. The raw Ultraseek results are ordered by relevance, but that’s begging the question: Relevant to whom? For what purpose? Using iws, you can order results by date, type, and author, and you can evaluate the results in the context of their tags, subtitles, and lead paragraphs.

In InfoWorld Metadata Explorer, aka iwx, the same metadata streams add value to del.icio.us. Compare the results for the tag “vista,” for example, in del.icio.us and in iwx. It’s the same set of URLs, but in iwx they’re decorated with extra metadata. Those metadata elements aren’t just passively displayed; they’re active filters, too. Clicking a tag filters the view to include just items with that tag. Clicking an author’s name adds another filter for items by that author.

These applications blend search and navigation in interesting and powerful ways. Because every view is fully specified by a URL, they radiate a lot of connections for people to use. Under the covers, they also use RSS feeds to radiate connections that people and machines alike can use.

Like many sites, InfoWorld.com offers a set of topical RSS feeds. Now that every iwx view can be seen through an RSS lens, that set is vastly enlarged. Suddenly we have feeds for Vista, Cisco, and many other topics. Cool! But not in the obvious way. Before iwx offered an RSS feed of InfoWorld’s Vista articles, del.icio.us did. Frankly, neither version is very interesting in and of itself. If you want to syndicate articles about Vista, ours are only some of the ones you’ll want to see in that feed.

Aggregated views are the ticket. And the key point is that the iwx version of the feed encourages smarter aggregation. Metadata is what makes the interactive experience in iwx more compelling than its del.icio.us counterpart. By syndicating that metadata, I’m inviting others to more richly contextualize their aggregations of our stuff.

If other publishers will return the favor, I’ll gladly make better use of theirs. Any takers?