Degrees of freedom

Radio UserLand doesn't produce WSDL wrappers for SOAP services, so I used Virtuoso's low-level SOAP client to retrieve the data as a PL vector — essentially, a list of lists. Once I had a PL routine that completed this task, the data could in turn be wrapped up as a SOAP service. That enabled me to use Virtuoso's WSDL auto-generator to make the raw Radio service WSDL-consumable — a neat bootstrapping trick! To pull this off, I wrote an XML Schema fragment that defines an ArrayOfString and fed it to Virtuoso's soap_dt_define function to define the new type. Finally, I used Virtuoso's auto-generated test harness to invoke the service through the WSDL wrapper.

Free IT resource TechNet: More ways to know it, share it, and keep it running. Sponsored by Microsoft Free IT resource Attend the SOA Executive Forum: Breaking SOA Bottlenecks SOAExecForum.com/may2007 Sponsored by InfoWorld Virtuoso Universal Server 3.0 Beta OpenLink Software, openlinksw.com Beta   Business Case: Enterprise middleware that unifies SQL, text, XML, and object data will play an increasingly vital role in delivering business data to users. Technology Case: Virtuoso 3.0's capability of hosting .Net languages, on both Windows and Linux, creates exciting new options for integrators. Pros: + Aggressive support for publishing Web services from local or foreign databases and for consuming them from anywhere + Equipped to index and search XML data + All-in-one combination of universal database, application server, and Web-services-savvy integration toolkit Cons: - Activation threshold for learning to use this complex, multifunctional product is high Cost: Starts at $599 for five concurrent users Platforms: Windows, Linux, Solaris, AIX, HP-UX, Mac OS X About our Reviews and Scoring Methodology

The next step was to define the database schema for the news feeds. Because they are by definition valid XML files, I chose a simple two-column structure:

CREATE TABLE feeds (feedname VARCHAR, feeddata LONG VARCHAR)

For the BLOBs in the feeddata column, I created an XML-aware text index, like so:

CREATE TEXT INDEX XML on feeds (feeddata)

One way to populate this table would be to fetch the news feeds using Virtuoso's native Web client. Virtuoso PL is a capable scripting language, but it's nice to know that other, more familiar languages are an option. This was a great opportunity to try out the new C# support. On a Windows XP machine equipped with Version 1.0 of the .Net Framework, I wrote a C# method to fetch an URL and to ensure that its XML is well-formed. Then I compiled the method into the server's directory, used a PL procedure to import it, and used another PL procedure to invoke it once per feed.

Experiments in XML

On my first try I ran into a glitch in the beta software. Some XML files validated by the C# code were nevertheless rejected by Virtuoso's parser and could not be added to the database. OpenLink has since verified and fixed the problem, but it did serve to illustrate an interesting point. A C# (or Java) method called from a Virtuoso stored procedure is transacted, so the failures caused automatic rollbacks.

While OpenLink was analyzing the bug I found, I switched to the simpler Web-client approach to populate the database and try some XML queries. I've sometimes wondered about the distribution of RSS versions across the 100 news feeds I subscribe to; what's the breakdown among the RSS 0.9x, RSS 1.0, and RSS 2.0 versions? With the XML text index in place, I was able to select the union of two XPath-enabled queries — one targeting the version attribute of RSS 0.9x and 2.0 feeds, the other isolating the RDF element of RSS 1.0 feeds — and to group the results by frequency of version.

I could have piped these results through Virtuoso's XSLT processor and displayed them using its built-in Web server. But I decided to chart the data. To do that, I used Virtuoso's SQL-to-XML converter — essentially the same as SQL Server's FOR XML AUTO data-shaping method — to render the results as XML. I routed the results of the FOR XML query to Virtuoso's WebDAV repository, which can cache such results or regenerate them on demand. Then I pointed Excel 11 at Virtuoso's WebDAV server, imported the "file" into an XML data range, and charted it. Once it was all working, I repeated the same procedure on Red Hat Linux 8.0.

Think about the pathways involved here. Data moves from a SOAP service in Radio UserLand, through an auto-generated WSDL wrapper, into a database stored procedure, which calls out to the Web through a C# extension and stores results in an indexed XML database. Then an XPath-enabled SQL query gathers results, converts them to XML, and virtualizes them as a WebDAV resource, which Excel finally reads and analyzes. Too many degrees of freedom? Not to my way of thinking. "It's just data," says Apache officer and IBM senior developer Sam Ruby. Of course data takes many forms and flows in many ways. I can't think of another product that supports as many data pathways — or does so as intelligently — as Virtuoso.