This document provides an overview of complex event processing (CEP). It begins by defining what an event and complex event are. It then defines CEP as the processing of multiple events to identify meaningful patterns within an event cloud. Some key CEP techniques discussed include event detection, correlation, hierarchies, and causal relationships. An example is given of how CEP could be used for applications like emergency response systems and credit card fraud detection. The document also discusses related concepts like event-driven architecture and how CEP compares and relates to event stream processing.

1. Mark
Proctor
Project Lead
The SkyNet funding bill is passed.
The system goes online on August 4th, 1997.
Human decisions are removed from strategic defense.
SkyNet begins to learn at a geometric rate.
It becomes self-aware at 2:14am Eastern time, August 29th
In a panic, they try to pull the plug.
And, Skynet fights back

2. 2
What is C omplex E vent Proces s ing?

3. 3
What is C omplex E vent Proces s ing?
My answer: the ability to make more money than your competitors. ;)

4. 4
What is C omplex E vent Proces s ing
1. Detect 2. Correlate
3. React

5. 5
Time is Money
Business Event
Value Loss
Business Value
Reaction
Time
Time Loss
Adapted from a presentation by James Taylor, Sep/2011

6. 6
Terminology: Event
F or th e s cop e of th is p re s e ntation:
“An e ve nt is a
s ignificant
change of s tate
at a p articu lar point in time”

7. 7
Terminology: Complex Event
“C omplex E vent, is an ab s traction of oth e r e ve nts
calle d its m e m b e rs .”
E xam p le s :
The 1929 s toc k market cras h – an ab s traction d e noting m any th ou s and s
of m e m b e r e ve nts , inclu d ing ind ivid u al s tock trad e s )
The 2004 Indones ian Ts unami – an ab s traction of m any natu ral e ve nts
A c ompleted s tock purc has e -an ab s traction of th e e ve nts in a
trans action to p u rch as e th e s tock
A s uc c es s ful on-line s hopping c art c hec kout – an ab s traction of
s h op p ing cart e ve nts on an on-line we b s ite
S ource : http :/com p l xe ve nts.com
/ e

8. 8
Terminology: CEP
“C omplex E vent Proces s ing, or C E P , is p rim arily
an e ve nt p roce s s ing conce p t th at d e als with th e
tas k of p roce s s ing m u ltip le e ve nts with th e goal
of identifying the meaningful events with in th e
e ve nt clou d .
C E P e m p loys te ch niqu e s s u ch as detection of
com p le x p atte rns of m any e ve nts , e ve nt
correlation and abs traction, e ve nt h ie rarch ie s ,
and re lations h ip s b e twe e n e ve nts s u ch as
cau s ality, m e m b e rs h ip , and tim ing, and e ve nt-
d rive n p roce s s e s .”
-- wikip e d ia

9. 9
Terminology: CEP
E xam p le s :
E m e rge ncy R e s p ons e S ys te m s
C re d it C ard F rau d D e te ction
Logis tics R e al-Tim e Aware ne s s s olu tion
N e onatal IC U : infant vital s igns m onitoring

10. 10
Terminology: CEP vs ESP
C omplex E vent Proces s ing, or C E P , and E vent
S tream Proces s ing, or E S P , are two
te ch nologie s th at we re b orn s e p arate , b u t
conve rge d .
An ove rsim p l
ification: In th e ir origins ...
E ve nt S tre am P roce s s ing focu s e d on th e ab ility to p roce s s h igh
volu m e s treams of e ve nts .
C om p le x E ve nt P roce s s ing focu s e d on d e fining, d e te cting and
p roce s s ing th e relations hips am ong e ve nts .

11. 11
Terminology: CEP and ESP
F or th e s cop e of th is p re s e ntation:
“C E P is u s e d as a com m on te rm
m e aning b oth C E P and E S P.”

12. 12
Terminology: EDA
“E vent Driven A rc hitecture (E DA ) is a s oftware arch ite ctu re
p atte rn p rom oting th e p rod u ction, d e te ction, cons u m p tion of,
and re action to e ve nts . An e ve nt can b e d e fine d as "a
s ignificant ch ange in s tate "[1 ]. F or e xam p le , wh e n a
cons u m e r p u rch as e s a car, th e car's s tate ch ange s from "for
s ale " to "s old ". A car d e ale r's s ys te m arch ite ctu re m ay tre at
th is s tate ch ange as an e ve nt to b e p rod u ce d , p u b lis h e d ,
d e te cte d and cons u m e d b y variou s ap p lications with in th e
arch ite ctu re .”
h ttp :/ e n.wikip e d ia.org/
/ wiki/ ve nt_ D rive n_ Arch ite ctu re
E

13. 13
EDA vs CEP
CEP is a component of the EDA
Source: http://elementallinks.typepad.com/.shared/image.html?/photos/uncategorized/simple_event_flow.gif

14. 14
E DA vs S OA
E D A is **not** S O A 2.0
C om p le m e ntary arch ite ctu re s
M e tap h or
In ou r b od y:
S O A is u s e d to b u ild ou r m u s cle s and organs
E D A is u s e d to b u ild ou r s e ns ory s ys te m

15. 15
E vent Driven A rchitectures
edBPM + E DM

16. 16
E DA vs S OA
Source: http://soa-eda.blogspot.com/2006/11/how-eda-extends-soa-and-why-it-is.html

17. 17
C omplex E vent Proces s ing
A few characteris tics of common C E P s cenarios :
H u ge volu m e of e ve nts , b u t only a fe w of re al inte re s t
U s u ally e ve nts are im m u tab le
U s u ally qu e rie s / le s h ave to ru n in re active m od e
ru
S trong te m p oral re lations h ip s b e twe e n e ve nts
Ind ivid u al e ve nts are u s u ally not im p ortant
Th e com p os ition and aggre gation of e ve nts is im p ortant

18. 18
JBOSS Enterprise BRMS v5.2

19. 19
S mart S olutions
CEP
Business Rules Business Events Processing
Seamlessly Integrated (Re)Active component
Platform
Business Processes Business Ontologies

20. 20

21. 21
Drools Fus ion: E nables …
E vent Detection:
F rom an e ve nt clou d or s e t of s tre am s , s e le ct all th e m e aningfu l
e ve nts , and only th e m .
[Temporal] E vent C orrelation:
Ab ility to corre late e ve nts and facts d e claring b oth te m p oral and
non-te m p oral cons traints b e twe e n th e m .
Ab ility to re as on ove r e ve nt aggre gation
E vent A bs traction:
Ab ility to com p os e com p le x e ve nts from atom ic e ve nts AN D re as on
ove r th e m

22. 22
Drools Fus ion
Features :
E ve nt S e m antics as F irs t C las s C itize ns
Allow D e te ction, C orre lation and C om p os ition
Te m p oral C ons traints
S e s s ion C lock
S tre am P roce s s ing
S lid ing Wind ows
C E P volu m e s (s calab ility)
(R e )Active R u le s
D ata Load e rs for Inp u t

23. 23
Demo
Twitter S tream C E P Demo:
Lis te n to th e Twitte r S tre am AP I
Twitte r4J AP I
Lis te ns to a rand om s am p le of twe e ts
D e te cts p atte rns and re acts
D rools F u s ion
S im p le one p roce s s (m u lti-th re ad ) d e m o
F ocu s on s p e cific fe atu re s

24. 24
E vent Declaration and S emantics
// declaring existing class
import some.package.VoiceCall E ve nt s e m antics :
declare VoiceCall
@role( event )
P oint-in-tim e and Inte rval
@timestamp( calltime )
@duration( duration )
end An e ve nt is a fact with a fe w s p e cial
// generating an event class
ch aracte ris tics :
declare StockTick
U s u ally im m u tab le , b u t not e nforce d
@role( event )
S trong te m p oral re lations h ip s
symbol : String
price : double Life cycle m ay b e m anage d
end
Allow u s e of s lid ing wind ows
“All e ve nts are facts , b u t not all facts
are e ve nts .”

25. 25
Temporal Reas oning
S e m antics for:
time: d is cre te
events : p oint-in-tim e and inte rval
Ab ility to e xp re s s te m p oral re lations h ip s :
Alle n’s 1 3 te m p oral op e rators
J ames F. A llen d e fine d th e 1 3 p os s ib le te m p oral re lations
b e twe e n two e ve nts .
E iko Yoneki and J ean B acon d e fine d a u nifie d s e m antics for
e ve nt corre lation ove r tim e and s p ace .

26. 26
Temporal Relations hips
rule “Shipment not picked up in time”
when
Shipment( $pickupTime : scheduledPickupTime )
not ShipmentPickup( this before $pickupTime )
then
// shipment not picked up... action required.
end

27. 27
Temporal Relations hips
rule “Shipment not picked up in time”
when
Shipment( $pickupTime : scheduledPickupTime )
not ShipmentPickup( this before $pickupTime )
then
// shipment not picked up... Action required.
end
Temporal
Relationship

28. 28
A llen’s 13 Temporal Operators
Point-Point Point-Interval Interval-Interval
A
A before B B
A
A meets B B
A
A overlaps B
B
A
A finishes B B
A
A includes B
B
A
A starts B B
A
A coincides B
B

29. 29
A llen’s 13 Temporal Operators
Point-Point Point-Interval Interval-Interval
A
A after B B
A
A metBy B B
A
A overlapedBy B
B
A
A finishedBy B B
A
A during B
B
A
A finishes B B

30. 30
S treams : S imple E xample S cenario

31. 31
S tream S upport (entry-points )
A s coping abs traction for s tream s upport
R u le com p ile r gath e r all e ntry-p oint d e clarations and e xp os e th e m
th rou gh th e s e s s ion AP I
E ngine m anage s all th e s cop ing and s ynch ronization b e h ind th e
s ce ne s .
rule “Stock Trade Correlation”
when
$c : Customer( type == “VIP” )
BuyOrderEvent( customer == $c, $id : id ) from entry-point “Home Broker Stream”
BuyAckEvent( sourceEvent == $id ) from entry-point “Stock Trader Stream”
then
// take some action
end

32. 32
Cloud Mode, Stream Mode, Session Clock
CLOUD STREAM
N o notion of “flow of tim e ”: th e N otion of “flow of tim e ”:
e ngine s e e s all facts with ou t conce p t of “now”
re gard to tim e S e s s ion C lock h as an active
N o attach e d S e s s ion C lock role s ynch ronizing th e
N o re qu ire m e nts on e ve nt re as oning
ord e ring E ve nt S tre am s m u s t b e
N o au tom atic e ve nt life cycle ord e re d
m anage m e nt Au tom atic e ve nt life cycle
N o s lid ing wind ow s u p p ort m anage m e nt
S lid ing wind ow s u p p ort
Au tom atic ru le d e laying on
ab s e nce of facts

33. 33
Reference C lock
R e fe re nce clock d e fine s th e flow of tim e
N am e d S es s ion C lock
is as s igne d to e ach s e s s ion cre ate d
S ynch ronize s tim e s e ns itive op e rations
d u ration ru le s
e ve nt s tre am s
p roce s s tim e rs
s lid ing wind ows

34. 34
S es s ion C lock
Uses the strategy pattern and multiple implementations:
Real-time operation
Tests
Simulations
etc

35. 35
S es s ion C lock
Selecting the session clock:
API:
KnowledgeSessionConfiguration conf = ...
conf.setOption( ClockTypeOption.get( “realtime” ) );
System Property or Configuration File:
drools.clockType = pseudo

36. 36
S liding Window S upport
Allows re as oning ove r a m oving wind ow of “inte re s t”
Tim e
Le ngth
Sliding window 1
Sliding window 2

37. 37
S liding Window S upport
Allows re as oning ove r a m oving wind ow of “inte re s t”
Tim e
Le ngth
Sliding window 1
Sliding window 2
Joined window

38. 38
S liding Window S upport
Allows re as oning ove r a m oving wind ow of “inte re s t”
Tim e
Le ngth
rule “Average Order Value over 12 hours”
when
$c : Customer()
$a : Number() from accumulate (
BuyOrder( customer == $c, $p : price )
over window:time( 12h ),
average( $p ) )
then
// do something
end

39. 39
Delaying Rules
Negative patterns may require rule firings to be delayed.
rule “Order timeout”
when
$bse : BuyShares ( $id : id )
not BuySharesAck( id == $id, this after[0s,30s] $bse )
then
// Buy order was not acknowledged. Cancel operation
// by timeout.
end

40. 40
Delaying Rules
Negative patterns may require rule firings to be delayed.
rule “Order timeout”
when
$bse : BuyShares ( $id : id )
not BuySharesAck( id == $id, this after[0s,30s] $bse )
then
// Buy order was not acknowledged. Cancel operation
// by timeout.
end
Forces the rule to wait for 30 seconds before firing, because the
acknowledgement may arrive at any time!

41. 41
Temporal Dimens ion
R e qu ire s th e s u p p ort to th e te m p oral d im e ns ion
A ru le / e ry m igh t m atch in a give n p oint in tim e , and not m atch in
qu
th e s u b s e qu e nt p oint in tim e
Th at is th e s ingle m os t d ifficu lt re qu ire m e nt to s u p p ort in a way
th at th e e ngine :
s tays d e te rm inis tic
s tays a h igh -p e rform ance e ngine
Ach ie ve d m os tly b y com p ile tim e op tim izations th at e nab le :
cons traint tigh te ning
m atch s p ace narrowing
m e m ory m anage m e nt

42. 42
Temporal Dimens ion S upport
C E P s ce narios are s tate fu l b y natu re .
E ve nts u s u ally are only inte re s ting d u ring a s h ort p e riod of
tim e .
H ard for ap p lications to know wh e n e ve nts are not ne ce s s ary
anym ore
Te m p oral cons traints and s lid ing wind ows d e s crib e s u ch “wind ow of
inte re s t”

43. 43
S imple E xample Rule
rule “Bag was lost”
when
$b : BagScannedEvent() from entry-point “check-in”
not BagScannedEvent( id == $b.id, this after[0s,5m] $b )
from entry-point “pre-load”
then
// Bag was lost, do something
end
Easy to “see” that the only temporal relationship between the
events defines a 5 minutes interest window.

44. 44
C alendars
rule "weekdays are high priority"
calendars "weekday"
timer (int:0 1h) Execute now and after
1 hour duration
when
Alarm()
then
send( "priority high - we have an alarm” );
end
rule "weekend are low priority"
calendars "weekend"
timer (int:0 4h) Execute now and after
4 hour duration
when
Alarm()
then
send( "priority low - we have an alarm” );
end

45. 45
Timers
Field Name Mandatory? Allowed Values Allowed Special Characters
Seconds YES 0-59 ,-*/
Minutes YES 0-59 ,-*/
Hours YES 0-23 ,-*/
Day of month YES 1-31 ,-*?/LW
Month YES 1-12 or JAN-DEC ,-*/
Day of week YES 1-7 or SUN-SAT ,-*?/L#
Year NO empty, 1970-2099 ,-*/
Send alert every quarter of an
hour
rule “name”
timer ( cron: 0 0/15 * * * * )
when
Alarm( )
then
sendEmail( ”Alert Alert Alert!!!” )

46. 46
A bs tract E xample Rule
rule “Abstract event relationship example”
when
$a : A()
$b : B( this after[-2, 2] $a )
$c : C( this after[-3, 4] $a )
$d : D( this after[ 1, 2] $b, this after[2,3] $c)
not E( this after[ 1,10] $d )
then
// Bag was lost, do something
end
How about now? What is the temporal relationship between
A and E?

47. 47
Temporal Dependency Matrix
[-2,2] B [1,2]
[1,10]
A D E
[-3,4] C [2,3]
Constraint tightening
A B C D E
A [ 0, 0 ] [ -2, 2 ] [ -3, 2 ] [ -1, 4 ] [ 0, 14 ]
B [ -2, 2 ] [ 0, 0 ] [ -2, 0 ] [ 1, 2 ] [ 2, 12 ]
C [ -2, 3 ] [ 0, 2 ] [ 0, 0 ] [ 2, 3 ] [ 3, 13 ]
D [ -4, 1 ] [ -2, -1 ] [ -3, -2 ] [ 0, 0 ] [ 1, 10 ]
E [ -14, 0 ] [ -12, -2 ] [ -13, -3 ] [-10,-1 ] [ 0, 0 ]

48. 48
C E P A pplied at FedE x C us tom C ritical
* Presented by Adam Mollemkopf at ORF 2009

49. 49
C E P A pplied at FedE x C us tom C ritical
* Presented by Adam Mollemkopf at ORF 2009

50. 50
C E P A pplied at FedE x C us tom C ritical
* Presented by Adam Mollemkopf at ORF 2009
At least 50% of Alerts can be reasoned automatically, promoting
staff savings and improved Customer and Driver experiences.
Risk Avoidance via pro-active monitoring
Reduction in insurance claims and shipment service failures
Minimum 30% efficiency gains in shipment monitoring , saving at
least 15% of Operations staff cost.

51. 51
C E P A pplied at FedE x C us tom C ritical
* Presented by Adam Mollemkopf at ORF 2009
Some numbers (from early 2010):
24 x 7 sessions, no downtime
Average of 500k+ facts/events concurrently in memory
Business hours: 1M+ facts/events concurrently
Response time for reasoning cycles:
Average: 150 ms
Peak: 1.2 sec
Several hundred rules

52. 52
Differential Update
Differential Update (a.k.a. “true modify”)
Implements a real “modify/update” operation, instead of
retract+assert.
Reuses tuples, reduces GC stress, improves performance

53. Q&A
Drools project s ite:
h ttp :/ www.d rools .org ( h ttp :/ www.j os s .org/ rools / )
/ / b d
Doc umentation:
h ttp :/ www.j os s .org/ rools / ocu m e ntation.h tm l
/ b d d
Edson Tirelli – etirelli@redhat.com