Truviso: New tricks with old SQL

Event stream processing startup brings tried-and-true database technology to a new world of real-time data management

If Werner Heisenberg had pursued business analytics in the early 21st century instead of quantum physics in the early 20th, he might have struck on a different uncertainty principle: If your analysis and decision-making capabilities don't keep pace with the explosion of critical event data across enterprise networks today, soon you won't know the state of your business, the direction it's heading, or how fast it's getting there. 

A host of technology companies are working on the science behind taking these measurements. Perhaps the newest of these event stream processing or complex event processing vendors is Truviso (formerly Amalgamated Insight), a Silicon Valley startup founded on work led by co-founder and CTO Michael Franklin at U.C. Berkeley, where Franklin is a professor of computer science.

The seed of Truviso was planted in Franklin's frustration at what he considered wrong-headed approaches to the analysis problem. Many seemed to think that analyzing event streams -- be they RFID events from sensor networks, service events from SOA networks, system events from traditional networks, events from financial markets, or other kinds of electronic transactions -- was merely a matter of plumbing. If you could just get this device talking with that device, you would somehow enable pervasive computing.

"And my view," Franklin recalls, "was that once you solved the plumbing problems, what you've got is a data management problem. Because you've got huge volumes of data being created all over the place. Lots of applications need that data, and lots of users need insight into what's going on in these dispersed environments."

Beyond the plumbing, others were trying to reinvent the wheel. Attending an RFID standardization meeting as a guest, Franklin listened as group after group, each working at a different level of the system, described the data at their level and how to access it. And he realized, each one was defining a data model and a query language, and each one was different. That's when the seed sprouted.

"Sitting there as a database guy, I was just getting irritated," he says. "Jeez, why wouldn't you just use SQL for all of this stuff across all the levels? And on the way home from that meeting, I realized, well, you know, why don't I just do that."

Truviso's three-part solution combines a stream processing engine, a Java-based integration framework, and a visualization front end based on Adobe Flex. Naturally, the processing engine, which is embedded into the open-source PostgreSQL database, uses SQL to continuously query events as they flow past.

"It lets you write SQL queries that are basically always running," Franklin explains. "They're always producing the answer so far based on the data they've seen so far."

To enable SQL to query infinite data streams, Truviso extended it in two ways, adding a windowing capability to restrict the amount of data it would be exposed to in any instant and adding some "syntactic sugar" to define event patterns you might want to inspect. The "continuous analysis" engine is adaptive, Franklin says, meaning you can add or remove queries on the fly.

Truviso sees the decision to embed the core processing engine in Postgres not only as key to advancing the development effort, but also as a key differentiator in the marketplace. Franklin notes that it gave the solution complete SQL, JDBC, and ODBC interfaces, internationalization, and all of the extensibility of Postgres right out of the box.

As Boyd Pearce, the company's president and CEO, points out, Postgres lets users of Truviso's engine to write SQL that joins tables and streams, provides subqueries and views, and does all the magic that SQL can do. Programmers already know how to code in it.

"We got all that based on having Postgres," Pearce notes. "But the other thing we've found in dealing with customers in our application space is that if you don't have a relational database that speaks full SQL and lets people program, you're hugely disadvantaged in terms of doing what the customer needs."

Noting that Postgres allowed Truviso "to jump in front of a 10-year database systems development curve," Franklin sees a larger lesson in building on the open-source database.

"We really hit upon what I think is the future of enterprise software development," he concludes. "If you start from scratch today, if you're building anything complex, you're crazy. That's just not going to be a viable way to build enterprise software in the future. You're going to have to leverage open source and other available tools that are out there."