"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
Esper - CEP Engine
1. Esper
The Power to Predict
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
2. Agenda
• Predictive Business
• Complex Event Processing
• Esper
• Demo
• Questions
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
3. Predictive Business
"In the future, every company must be able to
recognize what is coming early enough to take
evasive action or steer towards an emerging
opportunity."[Frederick W. Smith-CEO
FedEx]
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
4. Business Demands
High throughput
Complexity High Availability
Low latency Insight
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
5. Conventional Models
• RDBMS
• Adds delays into data storage
• Uses Polling Model
• Wasteful Use of Resources
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
6. Conventional Models
• Messaging Systems
• Process Unitary Events
• No Correlation support
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
7. Agenda
• Predictive Business
• Complex Event Processing
• Esper
• Demo
• Questions
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
9. Concepts - Event
An event is a piece of data that represents
that something happened in the real world,
or in software system.
e.g. Stock Tick, Weather, UserClick
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
10. Concepts - Complex Event
An event that could only happen when lots
of other events happen.
e.g. Housing Price Drop
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
11. Concepts - Event Stream
A linearly ordered (often by time)
sequence of events forms Event Stream.
e.g. Click Stream
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
12. Concepts - Event Cloud
A partially ordered set of events form
Event Cloud
e.g.
Set of all stock trades for GOOG within a 5
minute time window is an Event Stream. While all
Stocks sold in a business day is an Event Cloud.
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
13. Concepts - Event Cloud
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
14. ESP and CEP
Event Stream Processing is focused more
on high-speed querying of data in streams
of events and applying mathematical
algorithms to the event data.
Complex Event Processing includes event
data analysis, but places emphasis on
patterns of events, and abstracting and
simplifying information in the patterns.
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
15. CEP Engine
An event processor is an application that
performs operations on event objects,
including creating, reading, transforming,
aggregating, correlating or removing them.
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
16. CEP Engine
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
17. CEP Engine Characteristics
1. Keep the data moving
2. Query on streams
3. Handle Stream Imperfections
4. Integrate Stored and Streaming Data
5. Guarantee Data Safety and Availability
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
18. CEP Engine Characteristics
6. Partition and scale applications automatically
7. Process and Respond instantaneously
8. Interoperability
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
19. CEP in Application Architecture
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
20. Commercial CEP Products
• Weblogic Event Server
• Coral8
• TIBCO
• IBM
• StreamBase
• Progress
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
21. CEP/EDA and SOA [1/3]
EDA is SOA 2.0
EDA is Proactive SOA
EDA and SOA
Complementary
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
22. CEP/EDA and SOA [2/3]
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
23. CEP/EDA and SOA [3/3]
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
24. Real World Applications
Capital Markets
BAM - Business Activity Monitoring
Supply Chain Management
Flight Operations Monitoring
Fraud Detection
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
25. Agenda
• Predictive Business
• Complex Event Processing
• Esper
• Demo
• Questions
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
26. Esper
• Dual Licensing model
• Open Source GPL
• Esper Tech
• Q2/2007 BEA Partner
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
27. Good Documentation
• Nice reference document
• Patterns catalogue available online
• Lots of examples
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
28. Lightweight
• ANTLR
• CGLIB
• LOG4J
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
30. Pluggability
• Works with any Java Process
• OSGI Enabled
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
31. Architecture
ESPER Engine
EPServiceProvider
Event EPL
JDB
Objects Statements Confi
C
gurat
Adap
ion
Listeners ter
Subscribers
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
32. API Overview
• EPServiceProvider
• Acts as Engine
• EPStatement
• Queries written in EQL
• UpdateListener
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
33. Event
• POJOs
• Key value pairs (java.util.Map)
• XML Documents
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
34. Event
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
35. Event Properties
Type Syntax Example
Simple name sensor
Indexed name[index] sensor[0]
Mapped name(key) sensor(‘light’)
Nested name.nested sensor.value
Name
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
36. UpdateListener
Receives updated data as soon as
it is processed for the statement.
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
37. Subscriber
• Must provide a method by name update
to receive insert stream events row-by-
row
• The number and types of parameters
declared by the update method must
match the number and types of columns
as specified in the select clause, in the
same order as in the select clause
• JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
38. Subscriber
select orderId, price, count(*) from
OrderEvent
public class MySubscriber { ...
public void update(String orderId, double
price, long count) {...}
}
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
39. Esper Engine Integration
• Obtain an Esper engine instance
• Create a statement (using the Esper query
language)
• Register the statement with the engine
• Create a listener (by implementing a Java
interface that will be triggered when the
statement evaluates to true) and attach it to the
statement
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
40. Esper Engine Integration
//Register events
Configuration configuration = new Configuration();
configuration.addEventType("TempEvent",
BodyTemperatureEvent.class.getName());
// Get Esper CEP Engine
EPServiceProvider cepEngine = EPServiceProviderManager.getProvider("esper-sports-
poc", configuration);
// Create EPStatement
PlayerBMIStatement playerBMIStatement = new
PlayerBMIStatement(cepEngine.getEPAdministrator());
playerBMIStatement.addListener(new PlayerBMIListener());
// Dispatch Event
cepEngine.getEPRuntime().sendEvent(eventInstance);
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
41. EPL
Similar to SQL
Stream
Name BP
Event
Ross 110
Rachel 120
Joe 100
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
43. Streams
Input stream denotes new events
arriving, the term remove stream
denotes events leaving a data
window.
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
44. Windows
• win:length(4)
• win:time(4 sec)
• win:time_batch(4 sec)
• win:length_batch(4)
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
45. Batch Time Window
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
46. Named Windows
• Analogous to database views
• Created using SELECT queries
• Define window (view) first and
insert data
create window OrdersTimeWindow.win:time(30
sec) as select symbol, volume, price from
OrderEvent
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
47. Filters
select * from
Withdrawal(amount>=200).win:le
ngth(5)
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
48. Aggregations
• sum
• avg
• count
• min/max
• stddev
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
49. Output Rate Limiting
Engine dispatches to Listener
when output condition occurs.
e.g.
select irstream symbol, volume, price from
MarketData.win:time(5.5 sec) output every 1 seconds
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
50. Event Patterns
Pattern Expression = Pattern
atoms + Pattern Operators
every (spike=ServiceMeasurement(latency>20000)
or error=ServiceMeasurement(success=false))
select s from pattern [every s=SubmitOrderEvent ->
(timer:interval(30 sec) and not
OrderAcceptedEvent(theOrder.id=s.theOrder.id))]
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
51. RDBMS Integration
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
52. RDBMS Integration
select name, healthy_temp from
TempEvent.win:time(60 sec) temp, sql:mydb3 ['
select name,healthy_temp from Player where
name = ${temp.playerName} and $
{temp.temperature} > healthy_temp ']
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
53. Non Event Based Data Integration
public class InstrumentRepository {
public static Exchange[]
getExchangesForSymbol(String symbol) {
...
}
select ord, eligibleExchanges from TheOrder as ord,
method:com.ociweb.jnb.esper.InstrumentRepository.ge
tExchangesForSymbol(ord.symbol) as eligibleExchanges
}
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
54. CEP Design Patterns (1/9)
Filtering:
e.g. Click-stream analysis: a filter may be used to capture the trades that
originate from a certain set of IP addresses.
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
55. CEP Design Patterns (2/9)
In Memory Caching:
e.g. Click-stream analysis: a typical application may hold the recent clicks and
searches performed by the users, coupled with the relevant historical and reference
information.
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
56. CEP Design Patterns (3/9)
Aggregation over Windows:
e.g. Click-stream analysis: it is often useful to compute the number of visitors
who click on a particular link within a specified time interval.
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
57. CEP Design Patterns (4/9)
Database Lookups:
e.g. Trading: a trading application may look up historical price for a stock, or
certain information about an order, or certain rules and regulations stored in a
database.
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
58. CEP Design Patterns (5/9)
Database Writes:
e.g. Click-stream analysis: storing the raw click-stream history, together with
derived data, in the database
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
59. CEP Design Patterns (6/9)
Correlation (Joins):
e.g. Trading: correlating information from multiple exchanges to find arbitrage
opportunities.
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
60. CEP Design Patterns (7/9)
Event Pattern Matching:
e.g. Fraud detection: fraud patterns are often described as a sequence of
events, in one or more streams.
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
61. CEP Design Patterns (8/9)
Hierarchical Events:
e.g. Order Processing: Purchase Order can have multiple items which needs to
be processed differently.
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
62. CEP Design Patterns (9/9)
Dynamic Queries:
e.g. Trading environments: every trader can enter their subscriptions like
X=MSFT, Y=.1%, Z=10 am.
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
63. EsperIO
Provides adapters support:
• CSVAdapter
• JMSAdapter
• OpenTick Adapter
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
64. Demo
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
65. Performance Tips
// Better performance
select * from RFIDEvent
// Less good performance
select assetId, zone, xlocation, ylocation
from RFIDEvent
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
66. Conclusion
Do I really need it?
Do I have the right
infrastructure?
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
67. Conclusion
• Learning Curve of EPL
• No Standardization of EPL
• EPL is still evolving
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
68. Questions?
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
69. CEP in Dutch Market
• KLM - TIBCO
• Boekhandels Groep Nederlands
(BGN)
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009
70. Resources
Esper http://esper.codehaus.org/
http://complexevents.com/
CEP http://www.cepbusinesscast.com/
Open Source SOA
JTeam
Innovating J2EE Development
Wednesday, April 1, 2009