Bring business analysis to streaming events

AptSoft's and iSpheres' complex event processing solutions provide effective ways to sift business meaning from system events in real time

It’s no secret to IT people, or any business executive worth his Beemer, that an amazing wealth of actionable business intelligence is coursing through enterprise applications, databases, and even system logs nearly every moment of every day. The problem has not only been plucking the meaningful events from the unimportant ones but also finding the often seemingly unrelated patterns between them, and doing so before it’s too late to make a difference -- before the supplier raises the price, the shopper leaves the Web site, or the scammer transfers the funds. 

CEP (complex event processing) promises to help you capitalize on the rich information hidden in your data streams -- information typically tossed away in log files and data warehouses. CEP helps users correlate complex events (often seemingly unconnected ones), spot trends, trap anomalies, and trigger alerts that can be used to seize opportunities or mitigate risks.

Tapping into distributed message buses to unearth important combinations and patterns of events that span multiple, disparate systems and lengthy, dynamic business processes, CEP meets challenges that traditional analysis and automation tools are inadequately suited to address. As a result, CEP is catching on in industries such as banking and insurance, where early adopters are using it to detect fraud and compliance violations. Other promising applications of CEP are RFID and CRM, where the objects, actions, and rules for handling them are constantly changing.

The science behind CEP is the ability to create high-level “business” events from numerous low-level system events. CEP events are created by filtering real-time data (from middleware, applications, databases, and Web services, for example) and infusing it with defining detail such as dependencies or causal relationships discovered by correlating other events. 

For example, you could construct a CRM event from an “abandoned shopping cart” message (parsing out the transaction, customer ID, and time), using other filters to extract the value of goods in the cart and tapping the correlation capabilities of the CEP system to add causal indicators such as whether the commerce site was suffering performance problems. Your CRM event might also include customer value or rank from the customer database.

CEP solutions don’t discover meaningful events and patterns on their own. You need to define what you’re looking for. One of the biggest challenges for CEP vendors has been to develop an EPL (event processing language) that is lightweight enough to run in real time, powerful enough to correlate complex data and manage anomalies, and yet easy enough to be used by business analysts.

Regardless of whether the EPL is accessible to humans, the ongoing participation of business analysts is key to the success of any CEP project. Click for larger view. Configuring meaningful CEP rules and filters -- and adapting them to evolving business conditions -- will require a deep understanding of the underlying business process.

Regardless of whether the EPL is accessible to humans, the ongoing participation of business analysts is key to the success of any CEP project. Click for larger view. Configuring meaningful CEP rules and filters -- and adapting them to evolving business conditions -- will require a deep understanding of the underlying business process.

If you get it right, CEP can be far more flexible, responsive, and cost-effective than using traditional BAM (business activity monitoring) and BPM solutions. CEP doesn’t require ripping out or recoding existing systems, and it is actually best applied in modest increments -- allowing for frequent and easy comparison of ROI metrics to rule sets.

Following is a look at products from two pioneers offering business-grade CEP applications. AptSoft Director for CEP 4.0 provides an infrastructure for CEP well-suited to apps such as fraud detection and CRM, whereas iSpheres Event Server 5.2 focuses on financial apps, RFID, and CRM. Both are doing great work applying CEP to their respective areas of expertise.

AptSoft Director for CEP 4.0

AptSoft took the wraps off its shiny new Director for CEP 4.0 release in February. The solution layers a Java-based event-processing engine atop metadata and policy modules for collecting and analyzing event stream data. Perhaps most importantly, Director provides easy-to-use, plain-English tools that business analysts can use to construct the complex rule sets and filters required to draw business intelligence from low-level events.

Factor in provisions for dynamic data retrieval, active insight and monitoring, and support for nonlinear, stateful workflow, and Director shapes up to be one of the more advanced CEP suites on the market.

Capable of processing roughly 1,600 events per second, Director is well-suited to small and midsize projects in need of flexible rule creation and maintenance. Included templates -- such as those for bridging communication with CRM applications and Salesnet -- help put development over the top.

Setting up Director was a rather tedious process, requiring a good deal of manual tweaking to the underlying test bed and post-installation modifications. AptSoft would do well to beef up the wizard-driven configuration tools. Nevertheless, all went off without a hitch.

Event processing in Director involves capturing the message streams from your business systems -- or Touchpoints -- such as off a JMS (Java Message Service) message queue to create intermediate event objects for analysis. Successful matches to comparative rules, which can include relationship and temporal assessment, can invoke actions such as an alert or an e-mail to a customer, or they can spawn an entirely new event.

Touchpoints can come from any JMS-ready source, Web service, or via a custom adapter, allowing you to incorporate transactional data from your sales-force automation application, call center systems, EAI applications, or Web site. Although AptSoft doesn’t offer any custom integration adapters itself, partner iWay Software can assist, or you can do it yourself using the modest SDK.

Design is definitely a strong suit for AptSoft. The plain-English syntax and graphical tools used to create pattern matching rules and temporal filters are inspired. Compared with iSpheres EPL, creating rule sets and event flows was a snap. Nontechnical business consultants will be able to sit down and immediately become productive -- really. Deployment to the servers is a one-click process.

Director’s intermediate objects can also dynamically draw in static data sources without custom coding, so rules can easily combine RDBMS queries -- for a customer’s VIP status, for example -- with real-time inputs to produce finely tuned actions such as placing a customer service ticket on the call-back queue.

The functions for time-based correlation and sequencing are superb. For capturing events in nonlinear flows, the optional workflow designer module’s graphical tools proved quite nimble. The Interact module, also available at additional cost, allows you to build human interaction into the system: An exception alert -- anything that requires human intervention or approval -- fires off an e-mail that solicits responses for continued processing.

A number of other features make Director stand out. In-memory queuing of the event dispatcher improves nonpersistent rules processing and transaction speed. Hot fail-over boosts reliability, and newly improved clustering capability helps ensure availability of mission-critical processes.

An onboard rules repository provides basic check-in/check-out, and template adapters for parsing IBM Lotus Notes and Novell GroupWise e-mail help Director slip into enterprise communications infrastructures.

Nevertheless, Director has room to grow. Event flow simulation would be a useful addition, and I would like to see the opportunity to import processes from external modeling tools. AptSoft indicated it would begin adding BPEL (Business Process Execution Language) support in the near future.

Furthermore, although the browser-based dashboard provides decent real-time monitoring of run-time activities, reporting facilities proved  lackluster. In most cases, reports amounted to little more than static data dumps. Support for user-customizable reports and tools for onboard data analysis, as well as deep drill-down capabilities with correlation and grouping, would go a long way toward unearthing the wealth of event data buried in noisy messaging environments.

Using AptSoft Director for CEP 4.0, businesses can begin to comprehend the real patterns and trends developing within their event-based ecosystems. Requiring no modification to existing systems, and priced at a fraction of the cost of less-flexible BPM solutions, Director makes it possible to draw meaningful correlations among the blizzard of low-level events otherwise obscured from decision-making processes.

iSpheresEvent Server 5.2

The iSpheres Event Server 5.2 and its companion IDE, EPL Studio, deliver a lightweight but sturdy platform for CEP applications focusing on activities within relatively narrow time windows.

Clocking in at more than 470,000 events per second, this Java-based package -- adapted from research out of a Caltech/U.S. Air Force project in the late 1990s -- can burn through streams of real-time data with exhaustive analysis. On the downside, Event Server lacks the comprehensive persistence, clustering, and load balancing capabilities of AptSoft Director. Limited means to uniformly administer distributed CEP farms and only bare-bones reporting also make Event Server a rougher ride than Director, but still well worth a look.

Installation and setup of the Event Server was a swift and truly painless process, due in part to the lightweight infrastructure on which it runs. Event Server consists of an event-monitoring engine -- capable of capturing and processing JMS and SOAP data streams -- and several console and browser-based management interfaces for basic local and remote server administration.

As opposed to AptSoft Director, Event Server neither requires nor supports an RDBMS for persisting  events. Instead of pushing data out to a relational database, Event Server uses a flat file on disk. Nor is there a need for a Web or application server, as iSpheres bundles the Jetty HTTP Server and Servlet container.

This low-overhead assembly is an efficient processing mechanism, capable of standing up to heavy volume, but it comes at the expense of enterprise-grade persistence over long-running, concurrent processes. Event Server would be well-suited to detecting an arbitrage opportunity or for monitoring brokers for market timing, where the tell-tale events happen quickly and in no particular order. It would be less suited to monitoring longer running processes that involve parallel, concurrent transactions, such as those found in commercial lending and other financing scenarios.

At this stage of the game, all EPLs are proprietary; no open standards yet exist. Whereas AptSoft’s rule language is friendly to businesspeople, iSpheres’ is quite friendly to developers. iSpheres EPL is fully extensible, allowing users to add new functions and extend the functionality of EPL’s keywords and components. Enterprise developers will appreciate the ability to tinker under the hood to meet custom specifications.

I found the EPL syntax clean and unambiguous. Using its On-When-Then framework (with Catch for exception handling) I could set up triggers, conditions, and actions in response to stream events. The EPL Studio -- used for programming in EPL -- offers a menu-driven UI for defining Event Server applications, enabling me to specify stream and data providers, build listeners, and construct templates and event definitions -- EPL rules -- over time-based constraints. With features such as one-click EPL code verification and dependency checking, EPL Studio proved helpful.

On the downside, the coding editor offered only stark basics, namely color-coded keywords. Niceties found in commercial development environments -- syntax checking, type ahead, auto-indent -- are nonexistent. Worse, EPL provides none of the plain-English abstraction found in AptSoft.

Event Server also lacks a code repository, simulation tools, graphical event flow modeling, and an integrated test facility. Debugging can be enabled at the server level, but the IDE offers no interface for it. You must leave the console to run the newly created server. On the plus side, I found the multi-event macros and named views beneficial in defining complex event situations.

Event Server’s reporting facilities amounted to little more than text screens, with no ability to sort or correlate among events. You will need to build your own or bridge the reporting gap with external BI tools for analysis. Officials at iSpheres indicated the next release would offer a report module.

Administrative tools for server and process management provided granular views of operating parameters. I could not only see current streams but even change and add event definitions -- that is, EPL rules -- to my server on the fly. Although change controls would be a nice addition, this does show how quickly you could update processing rules in fluid business circumstances.

1 2 Page 1
Page 1 of 2