Event stream processing advances Web services and RFID

Mark Palmer

In the late 1980s mark palmer helped build a trading system for Bankers Trust in London. Back then, the finance industry was driving event-stream processing, the technology that absorbs high-volume and rapid-fire flows of event data and then performs both historical and real-time analysis of those events. Nowadays, of course, with the advent of Web services on the one hand and RFID tags on the other, our networks are flooded with events created by both virtual and physical transactions. As vice president of event-stream processing for the real-time division at Progress Software, Mark Palmer is one of the innovators working to define this emerging discipline and to adapt databases and query languages to the new challenges it presents.

In the mid-1990s Palmer worked at Object Design. Although its ObjectStore database was a good foundation for event processing, it had to be adapted for time-series data. Another key enhancement was a pipelined architecture, which made it possible to consume tens of thousands of events per second and then efficiently answer queries such as: “Replay the events associated with dock door No. 20 between 3:30 and 4:15 yesterday afternoon.” These innovations are the basis of what’s now known as the Progress Event Engine.

The ObjectStore database morphed into an XML-oriented product, eXcelon, which was acquired by Progress Software in 2002. Palmer, meanwhile, was pursuing other interests, but reunited with the product two years ago when Progress Software brought him on board to lead its event processing initiative.

A key goal was to beef up the event engine’s real-time analytical capability. After an 18-month investigation Palmer settled on Apama, a company Progress acquired in April. “Historical access with real-time extensions is what we were building, while Apama’s strength is real-time analysis,” he says. His own team had created a basic query language. Now the event database is being outfitted with Apama’s more sophisticated query technology, and Palmer is responsible for weaving these separate innovations into a common platform.

“As event-stream processing matures, we’ll need standard ways to represent and query event data. Such standards are now in the scoping phase, under the aegis of the EPCglobal consortium,” Palmer says.

“People are mostly excited about RFID tags right now,” Palmer adds. “But when we look back in a few years, we’ll see that this really was the beginning of a whole new style of computing.”