Apama and StreamBase unearth meaning in disjointed data streams

StreamBase’s command-line debug mode provides basic break points, operation stepping, and log review, but I’d rather see it brought up to par with the rest of the visual tools. In contrast, Apama’s additional Research Studio showed more detailed insight and control during testing.

Free IT resource Hear how top CIOs turn change into a competitive advantage. Sponsored by HP Free IT resource Attend the SOA Executive Forum: Breaking SOA Bottlenecks SOAExecForum.com/may2007 Sponsored by InfoWorld

Apama’s process for deployment and simulation testing, however, is somewhat less integrated. Within the modeler, I had to package up scenario and dashboard scripts, fire up the localized event-monitor engine, start a separate Deployment Tool to inject requisite support files and simulation scripts, bring up a Web browser to deploy my app, and then, finally, begin testing.

In fairness, most of the steps outlined above are mere one-click operations, and this release now auto-loads application-dependent service monitors. Although separate tools can benefit distributed dev environments, the entire process should also be streamlined directly into the Windows IDE.

Apama’s native debugging capabilities were mediocre, but a companion app — EventStore 2.4 with its Research Studio — adds some enterprise-rich features.

EventStore maintains running collections of live event data, without impact to throughput, and syncs a design-time replica for use in testing. Research Studio is able to isolate specific event groups and time frames, simulate event-only data, and run complete historic playbacks with timing information and external stream feeds. This can go a long way toward failure analysis and back testing.

Wading in data

Both vendors provide some semblance of load balancing and availability assurance. Apama’s low-latency options can accelerate queue management processing, and its channel routers can push a stream across a predefined channel for processing on an alternate engine.

StreamBase supports clustering and load balancing as well. At dev time, however, you simply need to select concurrency for a module. Work is automatically distributed among available processors, or configured cluster farm.

Both offer fail-over features. Apama checkpoints are only available through the Enterprise Management and Monitoring console, which must remain running. But it can restart engines using the most recent checkpoint and can be used to manage interface adapters and routers as well.

StreamBase runs checkpoints for recovery, too, but high availability cannot be run concurrently with its cluster mode.

For truly mission-critical applications, the gap between checkpoints may not be sufficient for transactional reliability.  You’ll still need to incorporate hand-coded synchronization routines within your client apps.

Downstream

Although stream processing isn’t new, it hasn’t fully evolved. In the future, ESP systems must develop facilities that more innately finesse complex correlations out of event clouds. Although StreamBase at least exposes time as a native data type, for example, future iterations must expand to handle causality among events and offer easy relationship algorithms beyond mere temporal reference in order to simplify truly complex event processing.

But we’ve come a long way in the quest to monitor and streamline operations. For quick development — without sacrifice to long-term flexibility — both Apama and StreamBase lay solid groundwork that will help companies manually glean meaning from complex streams and begin capitalizing on automated, real-time awareness and response.