Tuesday, October 02, 2007

herenow Complex Event Processing I


We haven’t fully launched into the Fall 2007 series of SOA analyst podcasts yet. Dana says it will start up again some time this month.

One topic I’d like to discuss is how complex event processing (CEP)—sometimes known as event stream processing (ESP), event processing (EP), or event-driven architecture (EDA)--relates to SOA. CEP is the term popularized by Dr. David Luckham, an IT visionary who created a new architectural school of distributed computing around the notion. EDA is a popularized by Roy Schulte of Gartner, and is half technical, half business in thrust, often used in the same conceptual breath as “just-in-time business.” ESP is associated with real-time pub-sub middleware approaches such as traditional message brokering or queuing systems. EP is the shortened term that is increasingly used to stand for them all, reflecting the fact that the similarities among these notions are more important than the distinctions.

For the sake of this discussion (herenow in this post, and eventually in the anticipated podcast), I’ll alternate between them all. I’ll use CEP to key in on the growing complexity of this approach, of the events being processed, and of the conceptual, interactivity, and visualization approaches necessary for extracting actionable intelligence from them all. I’ll use ESP to focus on the streaming, real-time, continuous flow of events. I’ll use EDA to focus on the need for a ubiquitous event bus, across which all nodes are event-enabled, so that services, applications, business processes, and other endpoints can engage in event-driven notifications, publish-subscribe, and so forth, at all architectural layers from the physical on up. And I’ll use EP when I’m too tired to split hairs anymore and just need a pithy catch-all.

I’ve heard it said on many occasions that SOA and EDA are polar opposites—that SOA presumes request-response two-way interaction patterns among endpoints, whereas EDA supports both two-way notification/acknowledgment and one-way publish-subscribe interactions. I disagree with that limiting characterization of SOA, because I see SOA, as a paradigm, being independent of the interaction patterns on the underlying “enterprise service bus” (ESB) or what have you. SOA simply refers to a development and integration paradigm that focuses on maximizing the reuse, sharing, and interoperability among distributed resources by means of a standard, ubiquitous interoperability backplane, fabric, environment, mesh, etc (call it ESB for short). The resources being reused via SOA can conceivably be event sources, and the ESB can conceivably support the requisite pub-sub, notifications, etc to enable this reuse/sharing.

So in that sense I see EDA/CEP/ESP/EP as a subset of SOA—i.e., referring to just another category of interaction patterns traversing the ESB to further the goals of SOA.

But on another level, paradoxically, I see EDA as in some way a master umbrella concept that subsumes SOA—i.e., as a superset of SOA. To explain what I mean by that, let’s focus on SOA’s core concept: the service. When you look at services in the context of SOA governance, you realize that a service is fundamentally just a string of events associated with the life-cycle of some network-accessible resource:

  • Event: service proposed
  • Event: service authorized
  • Event: service created
  • Event: service tested
  • Event: service provisioned
  • Event: service called
  • Event: service responded
  • Event: service monitored
  • Event: service managed
  • Event: service modified
  • Event: service deprovisioned

Essentially, then, SOA governance is the sum total of practices and policies that manage the complexity (CEP: developing, orchestrating, monitoring, managing increasingly complex composite services) and the streams (ESP: myriad, concurrent, overlapping, real-time) of events throughout ubiquitous (EDA: IT and business) distributed architectures.