2. An example to kick-off this tutorial: Act: Passenger has been re-routed to another destination – send the luggage Act: Bag has reached to the wrong aircraft Notify: Bag has been checked but did not reach the ULD within 20 minutes Notify: Bag has been checked but did not reach the connecting flight THE LUGGAGE PERSPECTIVE: Across the 24 largest airlines more than 5.6 million bags went missing in 2006, this is an average of 15.7 bags per 1,000 travelers. 15% of the bags are never found. BBC News, April 4, 2007 Event processing can help here..
3. Outline of this tutorial What is behind event processing ? How is it related to other computing terms? Where are its roots? Event processing – architecture, building blocks The present: State of the practice in event processing – languages, implementation issues, challenges in implementing event processing applications The future: Trends and research challenges
4. What is behind event processing ? How is it related to other computing terms? Where are its roots?
5. What is “event processing” anyway? or Event processing is a form of computing that performs operations on events
6. Event processing and Data stream management? Aliases? One of them subset of the other? Totally unrelated concepts?
7. In computing we processed events since early days Network and System Management
9. What’s new? The analog: moving from files to DBMS In recent years – architectures, abstractions, and dedicated commercial products emerge to support functionality that was traditionally carried out within regular programming. For some applications it is an improvement in TCO; for others is breaking the cost-effectiveness barrier.
10. What is an event There are various definitions of events Event is an occurrence within a particular system or domain; the word event has double meaning: the real-world Occurrence as well as its computerized representation
12. Sometime we even react to the occurrence of multiple events I closed the deal with the Canadians We closed two huge deals in a single day, It is a good opportunity to send all the team to Some fun time in Singapore I closed the deal with the Australians
14. What we actually want to react to are – situations TOLL VILOATOR FRUSTRATED CUSTOMER Sometimes the situation is determined by detecting that some pattern occurred in the Flowing events. Toll violation Frustrated customer Sometimes the events can approximate or indicate with some certainty that the situation has occurred
19. Ancestor: Temporal databases There is a substantial temporal nature to event processing. Recently – also spatial and spatio-temporal functions are being added
24. Fast Flower Delivery Flower Store Van Driver Ranking and Reporting System Bid Request Delivery Bid Assignments, Bid alerts, Assign Alerts Control System GPS Location Location Service Location Driver’s Guild Ranking and reports Delivery confirmation Pick Up confirmation Ranked drivers / automatic assignment Bid System Store Preferences Delivery Request Assignment System Manual Assignment Assignment Assignments, Pick Up Alert Delivery Alert http://www.ep-ts.com/EventProcessingInAction
25. Event Processing Agent Context Event Channel Event Consumer Event Type Event Producer Global State The seven Building blocks
40. Filter EPA A filter EPA is an EPA that performs filtering only, and has no matching or derivation steps, so it does not transform the input event .
43. Pattern detection example Pattern name: Manual Assignment Preparation Pattern Type: relative N highest Context: Bid Interval Relevant event types: Delivery Bid Pattern parameter: N = 5; value = Ranking Cardinality: Single deferred Find the five highest bids within the bid interval Taken from the Fast Flower Delivery use case
44.
45. Pattern policies Evaluation policy —This determines when the matching sets are produced Deferred Cardinality policy —This determines how many matching sets are produced within a single context partition Repeated type policy —This determines what happens if the matching step encounters multiple events of the same type Every - Override – First – Last – with max/min value of…
46. Pattern policies – cont. Consumption policy —This specifies what happens to a participant event after it has been included in a matching set Consume Reuse Bounded reuse Order policy —This specifies how temporal order is defined By occurrence time – by detection time – by stream position – by attribute
47.
48. Context has three distinct roles (which may be combined) Partition the incoming events The events that relate to each customer are processed separately Grouping events together Different processing for Different context partitions Determining the processing Grouping together events that happened in the same hour at the same location
49. Context Definition A context is a named specification of conditions that groups event instances so that they can be processed in a related way. It assigns each event instance to one or more context partitions . A context may have one or more context dimensions. Temporal Spatial State Oriented Segmentation Oriented
50. Context Types Examples Spatial State Oriented Temporal Context “ Every day between 08:00 and 10:00 AM” “ A week after borrowing a disk” “ A time window bounded by TradingDayStart and TradingDayEnd events” “ 3 miles from the traffic accident location” “ Within an authorized zone in a manufactory” “ All Children 2-5 years old” “ All platinum customers” “ Airport security level is red” “ Weather is stormy” Segmentation Oriented
52. The present: State of the practice in event processing – languages, implementation issues, challenges in implementing event processing applications
53.
54. Existing Styles for EP languages (samples) Inference Rules State oriented Agent Oriented Imperative/ Script Based RuleCore Oracle Aleri Streambase Esper EventZero TIBCO WBE Apama Spade AMiT Netcool Impact * - if we add simple and mediated event processing the picture is even more diversified ECA Rules SQL extension Coral8 Agent Logic Starview XChange EQ Prova
59. Microsoft Streaminsights var topfive = (from window in inputStream.Snapshot() from e in window orderby e.f ascending, e.i descending select e).Take(5); var avgCount = from v in inputStream group v by v.i % 4 into eachGroup from window in eachGroup.Snapshot() select new { avgNumber = window.Avg(e => e.number) };
60. Esper EPL – FFD Example /* * Not delivered up after 10 mins (600 secs) of the request target delivery time */ insert into AlertW(requestId, message, driver, timestamp) select a.requestId, "not delivered", a.driver, current_timestamp() from pattern[ every a=Assignment (timer:interval(600 + (a.deliveryTime-current_timestamp)/1000) and not DeliveryConfirmation(requestId = a.requestId) and not NoOneToReceiveMSG(requestId = a.requestId)) ];
70. Optimizations Optimizations related to EPA assignment: partition, parallelism, Distribution, load balancing. Optimization related to the execution process: scheduling, routing optimizations and load shedding Optimizations related to the coding of specific EPAs: code optimization, state management.
78. Trend II: Going from monolithic to diversified Variety of functions Variety of Quality of Service requirements: Variety of platforms “ One size fits all” will not work – Instead a collection of building blocks that can fit together
79.
80. Trend IV: Going from programmer centered to semi-technical person centered Source: ebizQ Event processing market pulse
83. Emerging directions: Four directions to observe Multiple platforms – same look and feel The engineering of constructing EP applications Adding intelligence to Event processing Tailor-made optimizations
86. Emerging direction III: Event processing software engineering Best practices Methodologies Modeling & meta-modeling Design Patterns
87. Emerging direction IV: Intelligent event processing Offline and continuous mining of meaningful patterns in event histories Inexact event processing – handling inexact events and also false positives and false negatives Causality – a key for proactive, but also vital for provenance
88. Summary Event processing has emerged from multiple academic disciplines The state of the Practice is the 1 st generation of products – mainly engineering based Going to the next phase – many challenges that require collaboration of research and industry Already attracted coverage of analysts and all major software vendors The Dagstuhl grand challenge