EPTS DEBS2011 Event Processing Reference Architecture and Patterns Tutorial v1 2

epts
event processing technical society

Tutorial v1.2

epts
Content by: members of the
EPTS Reference Architecture Working Group

Event Processing Adrian Paschke
(Freie Universitaet Berlin)
Paul Vincent
Reference Architecture (TIBCO Software)
Catherine Moxey
and Design Patterns (IBM)
Alex Alves
(Oracle)

epts

Tutorial

• Event Processing is an increasingly “mainstream”
view in IT, if not widely in production
• Event Processing Functional Reference Architecture
(FRA) describes the functions involved in event
processing
• Functions in the FRA can themselves be described as
event processing patterns
• Most technology providers can support the FRA and
its associated design patterns

2

epts

Agenda

• Introduction to the EPTS Functional Reference
Architecture, and what it contains
• Mapping Functions to Patterns: some sample
Event Processing Patterns
• EPTS member pattern implementation examples
• Summary and future work on EP Patterns

3

epts

Introduction to Event Processing

• Event-centric view of IT
• Events are
– Sent and Received
– Aggregated, Transformed into Data, Deleted
– Processed in queries, rules etc
– Cause actions like processes, service invocations, etc

4

epts

Introduction to Event Processing

Event Processing,
Complex Event Processing,
Event Stream Processing
Event Processing Rules, etc.
Event Type Definitions,

Event Abstraction, Event Pattern
Detection, Event Composition

Event Management
etc. etc.

Events Derived
Events

Event Producer Event Consumer
(Event Source, (Event sink, event
Event Emitter) handler, event
listener)

Design time Run time Administration

epts

Motivation and Benefits

• Motivation
– Event Processing is evolving and exploiting many technologies
– Potential adopters (stakeholders) need a reference to
understand suppliers‟ architectures and solutions.
• Benefits
– a Reference Architecture predefines
customizable abstract frames of reference
for specific stakeholder concerns and application domains.
• Aids reuse of successful EP architectures
for frequently occurring EP design problems
– Underlying Architecture is a Reference Model that defines the
terminology and components in Event Processing architectures

epts

Reference Architecture Viewpoints
Viewpoint Viewpoint
Element Engineering EP Managing EP Business with EP
Architecture Architecture Architecture
Concepts How to How to apply? How to utilize / sell /
implement? own?
Stakeholders Architects / Project Manager Decision Maker,
Engineers Customer, Provider
Concerns Effective Operational Strategic and
construction and Management tactical
deployment management
Techniques / Modeling, IT (service/appl) Monitoring,
Languages Engineering management, Enterprise Decision
project Management,
management Governance
10

epts

Stakeholders and Viewpoints covered

Viewpoints (for CEP / stateful EP)
Stakeholder Component Functional
Decision Maker [Not applicable] Inputs, Outputs and
/ End User Processing Requirements
View
Architect / Solution Components Functions carried out in
Designer View CEP View

TBD (2012?) Focus so far

11

epts

Functional Reference Architecture

• Architect and Developer perspective
– includes the 3 main functions
(development, run-time and administration),
– targets primarily the automated event processing
operations
• Run-time functions in 2 main groups:
– the event infrastructure (sources and consumers)
external to the event processor under consideration,
– the event processor.

epts

Functional View source: definitions of EP
Event Processing, Complex
Event Processing, Event
Stream Processing
Event Processing Rules, etc.
Event Type Definitions,

Event Abstraction, Event Pattern
Detection, Event Composition

Event Management
etc. etc.

Events Derived
Events

Event Producer Event Consumer
(Event Source, (Event sink, event
Event Emitter) handler, event
listener)


epts

Reference Architecture: Functional View
Definition, Modeling, (continuous) Improvement

Event Reaction
Assessment, Routing, Prediction,
Discovery, Learning
(Pattern, Control, Rule, Query, RegEx.etc)

Event Process Monitoring, Control
0..*

State Management
Complex Event Detection
Event and Complex Event

Consolidation, Composition,
Aggregation
0..*
Event Analysis
Analytics, Transforms, Tracking,
Scoring, Rating, Classification
0..*
Event Preparation
Identification, Selection, Filtering,
Monitoring, Enrichment
0..*

Event Production Event Consumption
Publication, Dashboard, Apps,
Retrieval External Reaction


epts

Reference Architecture: Functional View / Runtime

Event Reaction
Discovery, Learning
0..*

State Management
Aggregation
0..*
Event Analysis
0..*
Event Preparation
0..*


Run time

epts


Event Production: the source of events for event processing.
• Event Publication: As a part of event production, events may be published
onto a communication mechanism (eg event bus) for use by event
consumers (including participants in event processing). This is analogous to
a "push" system for obtaining events.
• Event Retrieval: As a part of event production, events may be explicitly
retrieved from some detection system. This is analogous to a "pull" system
for obtaining events.


epts


Event Consumption: the process of using events from event publication and
processing. Event processing itself can be an event consumer, although for
the purposes of the reference architecture, event consumers are meant to
indicate downstream consumers of events generated in event processing.
• Dashboard: a type of event consumer that displays events as they occur to
some user community.
• Applications: a type of event consumer if it consumes events for its own
processes.
• External Reaction: caused through some event consumption, as the result
of some hardware or software process.


epts

Event Preparation: the process of preparing the event and associated payload and
metadata for further stages of event processing.
• Entity Identification: incoming events will need to be identified relative to prior events, such
as associating events with particular sources or sensors.
• Event Selection: particular events may be selected for further analysis. Different parts of
event processing may require different selections of events. See also event filtering.
• Event Filtering: a stream or list of events may be filtered on some payload or metadata
information such that some subset is selected for further processing.
• Event Monitoring: particular types of events may be monitored for selection for further
processing. This may utilise specific mechanisms external to the event processing such as
exploiting event production features.
• Event Enrichment: events may be "enriched" through knowledge gained through previous
events or data.

Event Preparation

epts

Event Analysis: the process of analysing suitably prepared events and their
payloads and metadata for useful information.
• Event Analytics: the use of statistical methods to derive additional information about an
event or set of events.
• Event Transforms: processes carried out on event payloads or data, either related to
event preparation, analysis or processing.
• Event Tracking: where events related to some entity are used to identify state changes in
that entity.
• Event Scoring: the process by which events are ranked using a score, usually as a part of
a statistical analysis of a set of events. See also Event Analytics
• Event Rating: where events are compared to others to associate some importance or
other, possibly relative, measurement to the event.
• Event Classification: where events are associated with some classification scheme for
use in downstream processing.

Event Analysis

epts


Complex Event Detection: the process by which event analysis results in the
creation of new event information, or the update of existing complex events.
• Event Consolidation: combining disparate events together into a "main" or "primary"
event. See also event aggregation.
• Event Composition: composing new, complex events from existing, possibly source,
events.
• Event Aggregation: combining events to provide new or useful information, such as trend
information and event statistics. Similar to event consolidation.

Aggregation

epts

Event Reaction: the process subsequent to event analysis and complex event
detection to handle the results of analysis and detection.
• Event Assessment: the process by which an event is assessed for inclusion in some
process, incorporation in some other event, etc.
• Event Routing: the process by which an event is redirected to some process, computation
element, or other event sink.
• Event Prediction: where the reaction to some event processing is that some new event is
predicted to occur.
• Event Discovery: where the reaction to some event processing is the disclosure of a new,
typically complex, event type.
• Note that event prediction is predicting some future event, usually of a known type, whereas event
discovery is the uncovering of a new event type. See also event-based learning.

• Event-based Learning: the reaction to some event processing that uses new event
information to add to some, typically statistical-based, understanding of events.
• Note that event-based learning is a specialisation of
general machine learning and predictive analytics.
Event Reaction
Discovery, Learning

epts

Reference Architecture: Functional View / Design time
Definition, Modeling, (continuous) Improvement

Covers the definition, modeling, improvement /
(Pattern, Control, Rule, Query, RegEx.etc)

maintenance of the artifacts
used in event processing:
Event and Complex Event

• event definitions, including event metadata and payloads,
• event and event object organisations and structures,
• event processing transformations / queries / rules /
procedures / flows / states / decisions /
expressions (although these can sometimes be
considered as administrative updates in some situations)

Design time

epts

Reference Architecture: Functional View / Administration

Administrative concepts of

Event Process Monitoring, Control
monitoring and control. This may involve
• starting and stopping the application and event
processing elements, including application monitors
• providing and updating security levels to event inputs and
outputs (also can design-time)
• management of high availability and reliability resources,
such as hot standby processes
• resource utilisation monitoring of the event processing
components
• process updates, such as how-swapping of event
processing definitions to newer versions.

Administration

epts

Agenda


24

epts


• Functions from Reference Architecture are a
guide to possible low-level patterns
Event Reaction
Discovery, Learning

Aggregation

Event Analysis

Event Preparation

25

epts

Which Classification of Patterns to Use?

• Common EP Functions?
As described in the Reference Architecture
• Other sources?
Recent literature

26

epts

Comparison of EPTS RA Fns and EPIA EP Agents
Event Reaction
Discovery, Learning

Aggregation

Event Analysis
Analytics, Transforms, Tracking, EPA

Event Preparation Filter Transformation Pattern detect

Translate Aggregate Split Compose

Enrich Project

27

epts

Event Preparation

28

epts

Event Analysis

29

epts


30

epts

Event Reaction

31

epts

What do we want to cover in an EP Pattern?

Type / Operands Operation /
synonym Operators

Event Filtering: a stream or list of events may be filtered
on some payload or metadata information
such that some subset is selected for further processing.

Rationale /
Mapping to business rule
EPL template

32

epts

EP MetaPattern version 1

• Name & Alternative Names
• Classification versus other references
– EPTS RA and other references e.g. “EP in Action” book
• Description
– Role, and Business Rule type specification / requirement
• Structure
– EPTS Glossary terms
• Implementations

33

epts

EP MetaPattern future versions?

• Name & Alternative Names
• Classification versus other references
– EPTS RA and other references e.g. “EP in Action” book
• Description may use revised EP Fns
classification / ontology
– Role, and Business Rule type specification / requirement
• Structure
– EPTS Glossary terms
• Implementations may use EPL classification
per EPTS Languages WG
34

epts

EP Pattern Implementations
• Differ in Implementation type (Event Processing
Language, Execution semantics, etc)
• Sampled systems we cover:
1.TIBCO BusinessEvents Rete-production-rule-engine
(with continuous query and event-expression options)
2.Oracle CEP Stream-processing engine
3.Prova Logic Programming Semantic CEP Rule Engine
4.IBM WebSphere Decision Server event-driven
production-rule-engine
& IBM Infosphere Streams stream processing engine
35

epts

Agenda


36

epts

Some EP Patterns to review in more detail
1. Filter: filter out all events that have some property in their
payload / data
– E.g. customer purchases: filter out those with values < $100
2. Enrichment: add some information based on prior events
– E.g. customer purchase: add the customer history to the
purchase event
3. Aggregation: add prior events together for new information
– E.g. customer purchase history: track the sum total of order
values for a customer
4. Routing: based on the purchase type pass the event on to the
appropriate service
– E.g. customer purchase event: pass on to a provisioning service
37
based on the type of product / product classification

epts

payload / data
purchase event
3. Aggregation: add some information based on prior events
4. Routing: based on the purchase type pass the event on to
the appropriate service
– E.g. customer purchase event: pass on to a provisioning
38
service based on the type of product / product classification

epts

Filter Pattern: Classification (1)

• EPTS Reference Architecture:
– "During event preparation, a stream or list of events may
be filtered on some payload or metadata information
such that some subset is selected for further processing."
• EPIA:
– Filter (EPA or Event Processing Agent) performs filtering
only and has no matching or derivation steps, so it does
not transform the input event.

39

epts

Filter Pattern: Description (2)

• Role / Explanation
– Comparing some property of the event (an attribute or
metadata) with some other value (from some other event, or
data)
• Associated Business Rule Specification
– As a constraint: a selection (to be enforced during the
processing of events):
All <event entities>
that have Filter
<filter expression>
event event
f(event)
must have
<some status>. discarded
event
40

epts

Filter Pattern: Structure (3)

• EPTS Glossary comparison
– A filter pattern can relate to several terms in the EPTS
Glossary
• can be specified in an event pattern (A template containing
event templates, relational operators and variables...)
• .. by an event pattern constraint (A Boolean condition that
must be satisfied by the events observed in a system...)
• .. as part of an event processing rule (A prescribed method
for processing events.)
• .. or as part of event stream processing (Computing on inputs
that are event streams.)
–
41

epts

Filter Pattern: Implementations (4)

• TIBCO, Oracle, Prova and IBM implementations for
this pattern

Filter
event event
f(event)

discarded
event

42

epts

Filter Pattern: Implementation 1a
TIBCO BusinessEvents: rules
• General pattern:
<declare> event event

<if> event-boolean-expression Filter
f(event)

<then> update-event-with-fact event
OR
action-for-filtered-event
Operation on
• Effect: ignore filtered events event
on operations on non-filtered events

43

epts

Filter Pattern: Implementation 1b
TIBCO BusinessEvents: event removal rules
• Implementation type: Rete-based rule-driven system
• Alternate pattern:
<declare> event
event
<if>
NOT Filter
NOT event-boolean-expression
!f(event)
<then>
consume-event event

• Effect: removed filtered events from
system (and ALL subsequent operations!)

44

epts

Filter Pattern: Implementation 1c
TIBCO BusinessEvents: State transitions
<state models for concept C> event
<route from State n to n+1>
<on> event Filter
<where> event-filter query(event)

• Effect: change state for some concept State
transition
• Notes: this is a specialisation of the
general rule pattern!

45

epts

Filter Pattern: Implementation 1d
TIBCO BusinessEvents: Queries
<select> output event
<from> event
<where> event-filter Filter
query(event)

<policy> notification Policy Event
event subset
• Effect: collect filtered events for
event operations per some policy
Operations on
event subset
• Note: the pattern matching framework on policy event
has a similar notion
46 but a different surface syntax

epts

Oracle CEP: queries

• General pattern: event
SELECT <property>
FROM <STREAM> predicate
WHERE <predicate-condition>

event

• Effect: discard events whose predicate Operation on
evaluate to false event

47

epts

Oracle CEP: model

• Scenario:
– Select only stocks from the stock tick stream whose
symbols are „AAA‟.
• Application Model

48

epts

Oracle CEP: query

Specify STREAM source

SELECT *
FROM StockTickStream
Define predicate for
WHERE symbol = „AAA‟ filtering

49

epts

Prova: rules
• Implementation type: backward reasoning logical filter
• General pattern template:
rcvMsg / rcvMult <Event Msg Pattern(s)> :- event
<logical filter condition(s)>,
… event
... .
Event pattern +
<logical derivation rule 1> :- logical rule filter
<filter conditions>.
Backward
<logical derivation rule 2> :- Resoning
Rule
event
<filter conditions>. chaning
...

• Effect: Unify received events with Operation on
event pattern definition and apply logical event
50 filter rules (derivation rules with rule chaining)

epts

Prova: Messaging Reaction Rules in Prova
• Send a message
sendMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
• Receive a message
rcvMsg(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)
• Receive multiple messages
rcvMult(XID,Protocol,Agent,Performative,[Predicate|Args]|Context)

Description:
– XID is the conversation identifier
– Protocol: protocol e.g. self, jms, esb etc.
– Agent: denotes the target or sender of the message
– Performative: pragmatic context, e.g. FIPA Agent Communication
– [Predicate|Args] or Predicate(Arg1,..,Argn): Message payload

epts

Prova: Example
% Filter for stocks starting with „A“ and price > 100
rcvMult(SID,stream,“S&P500“, inform, tick(S,P,T)) :-
S = "A.*",
P > 100,
sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).

Example with rule chaining
% Filter for stocks starting with „A“ and price > 100
filter(S,P),
sendMsg(SID2,esb,"epa1", inform, happens(tick(S,P),T).
filter(Symbol,Price) :-
Price > 100,
Symbol = "A.*".

epts

Filter Pattern: Implementation 3d
Prova: Semantic Event Processing Example
Semantic Query Filer:
Stocks of companies, which have production facilities in Europe and
produce products out of metal and
Have more than 10,000 employees.
Event Stream
{(Name, “OPEL”)(Price, 45)(Volume, 2000)(Time, 1) }
{(Name, “SAP”)(Price, 65)(Volume, 1000) (Time, 2)}

{(OPEL, is_a, car_manufacturer),
(car_manufacturer, build, Cars),
(Cars, are_build_from, Metall),
Semantic (OPEL, hat_production_facilities_in, Germany),
Knowledge Base (Germany, is_in, Europe)
(OPEL, is_a, Major_corporation),
(Major_corporation, have, over_10,000_employees)}

rcvMult(SID,stream,“S&P500“, inform,
tick(Name^^car:Major_corporation,P^^currency:Dollar,
T^^time:Timepoint)) :- … <semantic filter inference> .

epts

IBM WebSphere Decision Server: Business Events
component
event event
• Example:
Define a filter to only process events with a
Filter
purchase value greater than £100

event

Operation on
• Effect: Only process events
event
which match the filter

55

epts

IBM InfoSphere Streams: Streams Processing Language
• Example:
composite Main {
graph
stream<rstring name, uint32 age> Beat = Beacon() {}
stream<rstring name, uint32 age> Youngs = Filter(Beat)
{
param filter : age < 30u;
}
(stream<Beat> Younger; stream<Beat> Older) = Filter(Beat)
{
param filter : age < 30u;
}
}

• Effect: filter out into streams of older and younger ages
56

epts

payload / data
purchase event
57

epts

Enrichment Pattern: Classification (1)

• Alternative Names:
– Event Annotation, Semantic Event
– "During event preparation, events may be "enriched"
through knowledge gained through previous events or
data. "
• EPIA:
– Enrich (EPA): a translate EPA that takes a single input
event, uses it to query data from a global state element,
and creates a derived event which includes the attributes
from the original event, possibly with modified values...
58

epts

Enrichment Pattern: Description (2)

– Updating some attribute or metadata of the event with some
other values, possibly computed, from some other event or
data
– a selection (to be enforced during the processing of events):
All <event entities> that are also <enriched data
expression>
Other data
must have <some status>. or event

Enrichment
event event
f(event)
59

epts

Enrichment Pattern: Structure (3)

– Enrichment can be specified in an event pattern (pattern
definition containing additional meta data, semantic
knowledge)
– .. as part of an event processing rule (A method for
enriching events during processing, e.g., by analysing the
meta data and querying external data sources or
inferencing semantic knowledge bases)
– .. or as part of event stream processing (additional
additional information to event stream data, e.g. by
querying external data sources)

60

epts

Enrichment Pattern: Implementations (4)

this pattern

Other data
or event

Enrichment
event event
f(event)

61

epts

Enrichment Pattern: Implementation 1
TIBCO BusinessEvents: Rules
• General pattern: Enrich
<declare> event, data data
event

<if> event-data-match Identify
enrichment data

<then> update-event-with-data Enrich
Operation
• Effect: find appropriate data for on event
events and apply enrichment
operations on them
Event

62

epts

Enrichment Pattern: Implementation 2a
Oracle CEP: queries
event
SELECT <property>
FROM <STREAM>[WIN],<RELATION>
join-condition
WHERE <join-condition>

event
• Effect: enrich events from stream
with contextual data from relation
where condition applies. Operation on
event

63

epts

Enrichment Pattern: Implementation 2b
Oracle CEP: model

• Scenario:
– For every stock event from stock tick stream whose bid is
greater than 5.0, find its company‟s location.

64

epts

Enrichment Pattern: Implementation 2c
Oracle CEP: queries

No need for window
SELECT event.symbol, location operator on a „relation‟
FROM
StockTickStream [NOW] AS event,
StockTable AS data
WHERE event.symbol = data.symbol AND
event.lastBid > 5.0

65

epts

Enrichment Pattern: Implementation 2d
Oracle CEP: table metaphor
StockTable
Symbol Full-name Location
AAA The AAA company San Francisco

BBB The BBB company San Jose

Time Input (StockTickStream) Output (QUERY)
1 {“AAA”, 10.0, 10.5} {“AAA”, “San…”}

2 {“BBB”, 11.0, 12.5} {“BBB”, “San Jose”}

3 {“AAA”, 4.0, 4.5}

66

epts

Enrichment Pattern: Implementation 3a
Prova: rules
rcvMsg / rcvMult <Event Msg Pattern(s)> :- event
<query built-in(s)>,
<enrichment condition(s)> External
... . Query data
<logical enrichment rule 1> :-
<enrichment operations>. Enrichment
rules

• Effect: query data from external source event*
and enrich the event
Operation on
enriched event
67

epts

Enrichment Pattern: Implementation 3b
Prova: External Data and Object Integration
• Java
..., L=java.util.ArrayList(),, ..., java.lang.String(“Hello”)

• File Input / Output
..., fopen(File,Reader), ...

• XML (DOM)
document(DomTree,DocumentReader) :-
XML(DocumenReader),
• SQL
…,sql_select(DB,cla,[pdb_id,“1alx“],[px,Domain]).
• RDF
...,rdf(http://...,"rdfs",Subject,"rdf_type","gene1_Gene"),
• XQuery
..., XQuery = ' for $name in
StatisticsURL//Author[0]/@name/text() return $name',
xquery_select(XQuery,name(ExpertName)),
• SPARQL
...,sparql_select(SparqlQuery,name(Name),class(Class),
definition(Def)),

epts

Enrichment Pattern: Implementation 3c
Prova: Example with SPARQL Query
% Filter for car manufacturer stocks and enrich the stock tick
event with data from Wikipedia (DBPedia) about the manufacturer
and the luxury cars
carManufacturer(S,Man), % filter car manufacturers
findall([luxuryCar|Data],luxuryCar(Man,Name,Car),Data), % query
EnrichedData = [S,Data], % enrich with additional data
sendMsg(SID2,esb,"epa1", inform, happens(tick(EnrichedData,P),T).

% rule implementing the query on DBPedia using SPARQL query
luxuryCar(Manufacturer,Name,Car) :-
Query="SELECT ?manufacturer ?name ?car % SPARQL RDF Query
WHERE {?car <http://purl.org/dc/terms/subject>
<http://dbpedia.org/resource/Category:Luxury_vehicles> .
?car foaf:name ?name .
?car dbo:manufacturer ?man .
?man foaf:name ?manufacturer.
} ORDER by ?manufacturer ?name“,
sparql_select(Query,manufacturer(Manufacturer),name(Name),car(Car)).

epts

Enrichment Pattern: Implementation 4
component
• Example: Define a Mapped Key expression, where a
field in the event is used to look up objects in a
database table
event

Mapped Key
look up

Enriched
event

• Effect: required enrichment data is obtained
70
from the data base and added to event

epts

payload / data
purchase event
71

epts

Aggregation Pattern: Classification (1)

– Event Summarization, Composite event, Complex event
– "During complex event detection, combining events to
provide new or useful information, such as trend
information and event statistics. Similar to event
consolidation "
• EPIA:
– Aggregate (EPA): a transformation EPA that takes as
input a collection of events and creates a single derived
event by applying a function over the input events.
72

epts

Aggregation Pattern: Description (2)
– Aggregation is used to describe several events as a single
composite event, generally summarizing a metric of a set of
component events.
– For example: generate a (composite) event that contains the
medium price of a stock over a 10 minute stream of events.
– a summarisation (to be enforced during event processing):
The <aggregation fn> of <event entities> that
are <selection constraint>
must have
Aggregation
<some constraint>. event event
event f(events)
73
event

epts

Aggregation Pattern: Structure (3)

– The term aggregate event is sometimes used for some
forms of composite or derived event.
– Composite event: a derived, complex event
• is created by combining base events using a specific set of
event constructors such as disjunction, conjunction,
sequence, etc.
• always includes the base (member) events from which it is
derived.
– Derived event (/synthesized event): an event that is
generated as a result of applying a method or process to
one or more other events.
74

epts

Aggregation Pattern: Implementations (4)

this pattern

Aggregation
event event
event f(events)
event

75

epts

Aggregation Pattern: Implementation 1
• General pattern: Aggregation
<declare> event, object
event
aggregation-object
Identify
<if> event for
event-member-of-aggregation aggregation
<then>
update-aggregation-object Aggregate
with event-data Operation
on event

• Effect: construct aggregation Updated
(event) object for each incoming event Aggregation
76 Object

epts

Aggregation Pattern: Implementation 2a
Oracle CEP: queries
• General pattern template: Aggregation
object
event
SELECT <summary-property>
FROM <STREAM>[WIN]
Identify
WHERE <predicate>
event for
GROUP BY <aggregation-identity> aggregation

• Effect: summarizes the properties of Aggregate
several simple event into a new Operation
complex event on event

Updated
Aggregation
77 Object

epts

Aggregation Pattern: Implementation 2b
Oracle CEP: model

• Scenario:
– Output the average bid and ask price of a stock in the
last 10 seconds.

78

epts

Aggregation Pattern: Implementation 2c
Oracle CEP: queries

SELECT symbol, AVG(bid), AVG(ask)

FROM
StockTickStream [RANGE 10 SECONDS]

GROUP BY
symbol

79

epts

Aggregation Pattern: Implementation 2d
Oracle CEP: table metaphor

Time Input Output

1s {“AAA”, 10.0, 12.0} {“AAA”, 10.0, 12.0}

4s {“AAA”, 12.0, 14.0} {“AAA”, 11.0, 13.0}

{“AAA”, 11.0, 13},
9s {“BBB”, 4.0, 5.0}
{“BBB”, 4.0, 5.0}

15s {“BBB”, 8.0, 10.0} {“BBB”, 6.0, 7.5}
> 10s
20s {“BBB”, 8.0, 10.0}

80

epts

Prova: rules Aggregation
object
event
<rule_head> :- Check & Update
<Create Aggregator>, Timer / Size Count
@group(<reaction group>)
@timer|size(Start,Interval,Aggregator) Identify event
rcvMsg <EventPattern> [<Aggregate Operation>]. aggregation
for
or
<rule_head> :- Aggregate
@or(<reaction group>) Operation
rcvMsg <Aggregator>, on event
... <consume Aggregator>.

• Effect: Repeated incremental Updated
Aggregation
Consume
Aggregation
aggregations over new events Object Object

81

epts

Prova: Example with Time Counter
% This reaction operates indefinitely. When the timer
elapses (after 25 ms), the groupby map Counter is sent
as part of the aggregation event and consumed in or
group, and the timer is reset back to the second
argument of @timer.

groupby_rate() :-
Counter = ws.prova.eventing.MapCounter(), % Aggr. Obj.
@group(g1) @timer(25,25,Counter) % timer every 25 ms
rcvMsg(XID,stream,From,inform,tick(S,P,T)) % event
[IM=T,Counter.incrementAt(IM)]. % aggr. operation

groupby_rate() :-
% receive the aggregation counter in the or reaction
@or(g1) rcvMsg(XID,self,From,or,[Counter]),
... <consume the Counter aggreation object>.

epts

component
• General Pattern:
event event
Define how the
events should be
accumulated
Add to array
... then when
processing the
events, use the Intermediate
relevant Object Array
aggregator

• Effect of the example: sum up the value of a Operation
customer‟s purchases over a week on aggregate
83

epts

IBM InfoSphere Streams: Streams Processing Language
• Example:
// sliding window
stream<Beat, tuple<int32 maxSalary, uint32 ageOfMaxSalary> >
Agg4 = Aggregate(Beat)
{
window
Beat : sliding, time(10.5), count(10);
output
Agg4 : maxSalary = Max(salary), ageOfMaxSalary = ArgMax(salary, age);
}

• Effect of example: Aggregate the salary values in the
stream within a time-based window.
84

epts

payload / data
purchase event
85

epts

Routing Pattern: Classification (1)

– Event Summarization, Composite event, Complex event
– "During event reaction, event routing is the process by
which an event is redirected to some process,
computation element, or other event sink. "
• EPIA: no direct analogy, possibly maps to split /
compose / project
– Project (EPA): a translate EPA that takes an input event
and creates a single derived event containing a subset
of the attributes of the input event.
86

epts

Routing Pattern: Description (2)
– Routing is used to describe the process of adding one or
more new destinations to an event.
– For example: route an input event to the appropriate
specialist agent for that event type.
– a constraint (to be enforced during event processing):
The <event> that
Other data
satisfies <selection constraint> or event
must be assigned to
<destination>.
Routing
event event
dest = f(event+data)
87

epts

Routing Pattern: Structure (3)

– Routing (a process on events) is not defined, but is
related to associating an event to an event sink.
– Event sink (event consumer) is an entity that receives
events.
• Examples:
• ƒ
Software module
• ƒ
Database
• ƒ
Dashboard
• ƒ
Person

88

epts

Routing Pattern: Implementations (4)

this pattern

Other data
or event

Routing
event event
dest = f(event+data)

89

epts

Routing Pattern: Implementation 1
• General pattern: Routing
<declare> event, data
event
routing-data
Identify
<if> event for
event-matches-routing-data routing
<then>
Create routed
create-new-routed-event and
event
send-new-routed-event
event
• Effect: match events to routing rule
and construct / send new event
90

epts

Routing Pattern: Implementation 2a
Oracle CEP: query pattern
• General pattern template: Routing
data
event
Route 1: <filtering pattern> query
Route 2: <filtering pattern> query
Identify
Route n: <filtering pattern> query
event for
• Effect: Different predicates routing
associated to different queries
evaluate and select appropriate
Select destination
destinations

event

91

epts

Routing Pattern: Implementation 2b
Oracle CEP: model

• Scenario:
– Output stocks whose symbols start with „a-m‟ to
destination 1, and whose symbols start with „n-z‟ to
destination 2.

92

epts

Routing Pattern: Implementation 2c
Oracle CEP: queries

Destination-1:
SELECT *
WHERE symbol.matches(“^[a-m]”)

Destination-2:
SELECT *
WHERE symbol.matches(“^[n-z]”)

93

epts

Routing Pattern: Implementation 3 event
Prova: rules
• General pattern template: Identify
event for
rcvMsg <Event Msg Pattern> :- routing
<routing decisions>,
sendMsg <route event >. Routing Routing
Decisions data
<routing decision rule 1> :-
<decision conditions>.
Create routed
<routing decision rule 2> :- event
<decision conditions>.
...
Route
event
event
• Effect: Events are routed
according to the routing decision
94
rules

epts

Routing Pattern: Implementation 3b
Prova: Example with Agent (Sub-) Conversations
rcvMsg(XID,esb,From,query-ref,buy(Product) :-
routeTo(Agent,Product), % derive processing agent
% send order to Agent in new subconversation SID2
sendMsg(SID2,esb,Agent,query-ref,order(From,
Product)),
% receive confirmation from Agent for Product order
rcvMsg(SID2,esb,Agent,inform-ref,oder(From, Product)).

% route to event processing agent 1 if Product is
luxury
routeTo(epa1,Product) :- luxury(Product).
% route to epa 2 if Product is regular
routeTo(epa2,Product) :- regular(Product).

% a Product is luxury if the Product has a value over …
luxury(Product) :- price(Product,Value), Value >=
10000.
% a Product is regular if the Product ha a value below

epts

Routing Pattern: Implementation 4
IBM: Business Events component of WebSphere
Decision Server
Routing
event
• Example: Define an interaction set, information
which will send the request to supplier A,
when the purchase indicates supply
from Supplier A Identify
• event
routing

event

• Effect of example:
Events are routed as required
Consume
96 event

epts

Agenda


97

epts

Another Pattern Type

Other patterns may be required for
implementation, such as handling
produced-consumer interactions

epts

Application-Time Pattern

• Problem:
– Need to use application‟s view of time, instead of CPU
wall-clock.
– This is particularly useful when events originate from
different machines and need some way of synchronizing.
• Scenario:
– Again, consider bid and ask stream…
– However, bid and ask requests are time-stamped on
trader at the time that the order is placed.

99

epts


• Scenario:
– Seller places two ask requests respectively at time
8:00:00 and 8:00:12
– Buyer places one bid request at time 8:00:11

100

epts


• Scenario:
– Remember that we want to correlate using a 10 seconds
window, as anything older is stale…
– Hence first event from seller should not be considered,
and we should correlate the ask price of 11.0 with the bid
price of 9.5

101

epts


• Scenario:
– However, the reality is that the first two events could
arrive together in a burst in the exchange, and the third
could be delayed in the cloud…

102

epts


• Scenario:
– In this case, bid price of 9.5 would correlate to ask price of 10.0
and the exchange would loose money as the spread is lower…

103

epts

Application Time-stamped Pattern: Implementation 1
Oracle CEP

• Solution:
– The query does not change…
– However STREAMS must be configured to use
application time-stamps based upon some event
property, instead of having events being system-
timestamped…

104

epts

IBM Business Monitor
• Example: Define a correlation expression

„Order Placed‟ „Order Complete‟

event event
• Effect of example:
The duration of a customer order can be obtained from
the timestamps on start and end events
Correlate

Calculate
Duration
105

epts

Application Time Pattern: Implementation 3
Prova: rules
Event

rcvMsg <Event Msg Pattern(s)> :-
<add application timestamp to event>,
sendMsg <Adjusted Event>.
adjust event

• Effect: Any timestamp can be adjusted
(for time computations)
Updated
Event

106

epts

TIBCO BusinessEvents: PreProcessor Pattern
• General Pattern: adjust Event metadata (e.g.
Timestamp) in preprocessor for that event
Event
– events are generally non-monotonic!
– PreProcessor applies regardless of engine
used! Preprocess:
• Effect: adjust event
Any custom timestamp can be adjusted (for time
computations)

Note: If the problem is “out of order” events Updated
Event
(rather than synchronizing clocks) a
different pattern may be better!

107

epts

Use Cases

Illustrating using some of the patterns

epts

IBM Use Case: Utilizing Smart Meters
Large energy company improved customer responsiveness
using event processing

Use Case
• Utilized events (and lack of events) from
smart meters to detect service outages and
disruptions
• Empowered operations team to define
outage detection and response patterns
without coding or scripting

Smarter Business Outcomes
• Automated detection and response to service
Challenge: Reliance on customer calls outages
to detect and respond to outages and
reduce time to recovery while providing • Improved customer satisfaction by delivery of
more information to the customer estimated time to recovery alerts based on
customer preferences

epts

Energy Company Leveraging Smart Meters to Improve
Customer Responsiveness
Customer
Business Data
Events
2

1
3
Data Collection
Engine (DCE)

WebSphere
Business Events
1
4

Tivoli OMNIbus,
Network Manager,
and Impact
Power Outage
Disruption Management Customer Care
System (OMS) Manager(s) with IBM
Business Monitor
Location, Estimated Duration
of Disruption
Alert and Assist Impacted
Customers

110

epts

Oracle Use-case: Capital Markets

• Low-Latency Trading
– Algorithm Trading
• Trade Matching
– High throughput
– In-flight fraud
• Position Capture and Aggregation
• Risk and Analytics
– Pre/post trade analytics
– On-demand risk management
111

epts

TIBCO Use Case: Application Service Gateway
Telco high performance service policies using
event processing

Excellent service quality
Mobile Services
needed to avoid
difficult to monitor
customer churn
end to end

Auto detect and
fixing of issues Service quality affects
for better take-up of new services
112
user service

epts

Service Gateway architecture exploits event processing

• Standardized channel for
monitoring subscribers
• Standardized access to services
(e.g. Facebook, Google, etc)
• Consolidation of existing multiple
gateways
• Allows analysis of service usage,
subscriber behavior, service
performance, marketing campaign
success, etc
• Creates events for revenue
generation and reconciliation
• Handles Common Telco
Notifications - change-of-device,
contract, service- allowing
UpSell/Cross sell etc

epts

Service Gateway architecture exploits event processing

• Standardized channel for
monitoring subscribers
• Standardized access to services
(e.g. Facebook, Google, etc)
Security Pattern
• Consolidation of existing multiple
gateways
Decision / Policy
• Allows
Pattern analysis of service usage,
subscriber behavior, service
performance, marketing campaign
success, etc
Routing Pattern
• Creates events for revenue
generation and reconciliation
Transformation
• Handles Common Telco
Patterns
Notifications - change-of-device,
contract, service- allowing
UpSell/Cross sell etc

epts

Agenda


115

epts

Summary: Realization Patterns / Idioms

Event Reaction
Discovery, Learning

State Management
0..*
Aggregation
0..*
Event Analysis
0..*
Event Preparation
0..*

Application Time Dashboard, Apps,
Publication, Retrieval External Reaction

epts

Categorization of Event Processing Patterns

Source:Paschke, A.: Design Patterns for Complex Event Processing, DEBS'08, Rome, Italy, 2008
http://arxiv.org/abs/0806.1100v1

epts
event processing technical society Categorization of Patterns
• Categorization according to Good and Bad Solutions
– CEP Patterns
– CEP Anti-Patterns
• Categorization according to the Abstraction Level
– Guidelines and Best Practices
– Management patterns
– Architecture patterns
– Design patterns
– Mapping patterns
– Idioms / Realization patterns
– Smells / Refactoring patterns
• Categorization according to the Intended Goal
– Adoption patterns
– Business patterns
– Integration patterns
– Composite patterns:
– …
• Categorization according to the Management Level
– Strategic patterns
– Tactical patterns
Source:Paschke, A.: A Semantic Design Pattern Language for Complex Event Processing,
– Operational patterns Intelligent Event Processing, Papers from the 2009 AAAI Spring Symposium (2009) , p. 54-60.
http://www.aaai.org/Papers/Symposia/Spring/2009/SS-09-05/SS09-05-010.pdf

epts

Next Steps
• Continue evolution of EP Reference Architecture
– Eg Logical Architecture vs Systems Description
• Evolve the EP Pattern <<metapattern>> to be more useful for
developers and tools
• Expand coverage to all Functions as Patterns, + extend Functions as
necessary
• Planned Outputs:
– EPTS-RA Description white paper
– Wiley Book "Pattern Oriented Software Architecture: Architectures,
Models and Patterns for Event Processing" (already in preparation)
– Input for Event Processing Metamodels and associated standards

epts

Summary

• Reference Architecture provides a common set of
EP Functions that may be mapped
to other EP ontologies / taxonomies
• Provides the basis for discussing EP Patterns
• EP Patterns can be semantic, design or runtime
focused
• Further evolution of EP Pattern and examples is
likely!

120

epts

Thank you !

Acknowledgment to the Event Processing Technical Society
Reference Architecture working group members

121

epts

References
• EPTS Glossary
http://www.ep-ts.com/component/option,com_docman/task,doc_download/gid,66/Itemid,84/

EPTS DEBS2011 Event Processing Reference Architecture and Patterns Tutorial v1 2

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (15)

Similar a EPTS DEBS2011 Event Processing Reference Architecture and Patterns Tutorial v1 2

Similar a EPTS DEBS2011 Event Processing Reference Architecture and Patterns Tutorial v1 2 (20)

Último

Último (20)

EPTS DEBS2011 Event Processing Reference Architecture and Patterns Tutorial v1 2