At the 5th Elephant BigData conference in bangalore, india , 27-july-2012.
https://fifthelephant.talkfunnel.com/2012/384-build-your-own-real-time-analytics-and-visualization-enable-complex-event-processing-event-patterns-and-aggregates
Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
1. Build your own Real Time Analytics and
Visualization, Enable Complex Event
Processing, Event Patterns and Aggregates
Ramesh / Vishnu
Supply Chain - Platform Team
10. Complex Event Processing
● ElasticSearch as a Storage or Alternate DB
○ Faster on Lookup Queries than RDBMS
○ Can do simple predicate queries
○ Does not need multiple indexes (full text indexing)
○ Create fields out of interesting values
● Statsd layer is a sliding window counter
○ Within a sliding window we can do regex patterns
○ Aggregates
○ Deviations
○ This is a Key aspect of the SOA Monitoring System (Complex
patterns which need action)
Push the complex pattern back to ES or as a trigger for action
11. Use cases
● Every PO has a matching SO?
● Has a shelf in the warehouse just gone
empty?
● Where is the current pile up happening?
● Is the SLA being breached?
22. Are logs the only source of events?
● No - The database can be used as well.
● Events can be generated by capturing the
Updates/Inserts/Deletes being made to the
tables.
● These events can be published to an MQ to
speed up replication (batch processing) or sent
to the CEP engine.
23. Search CEP
Elastic
StatsD
Search
graphite
Graylog2
Change Data
Capture
Logstash
Database Application Server
24. Distribute
Replication Search CEP
General
Events
Query Log
Elastic
MQ StatsD
Search
graphite
Graylog2
Change Data
Capture
Logstash
log.cc
Database Application Server
27. Time to Sing
Mood of Mysql
Note:image is from http://www.technocation.org
28. Mood of Mysql
● Music is the best way to express how one feels.
● Well, Mysql has a soul too, it has a mood :)
● Mysql can sing through each query(good/bad) it gets.
● Every query, Mysql gets, is intercepted in log.cc and
sent acrross to an MQ Server. Subscribers to the
queue ,on receiving a message play a musical note
depending on the query they get.
29. Use case: Divide & Conquer General
query log
● Alternative to enabling general query log, which grows very
fast in size and disk space becomes a concern on the master
database.
● The queries are sent out to a queue on an MQ Server and an
army of subscribers who listen to the queue , log the query
on receiving a message.
● The general query log can now be distributed (among the
subscribers).
● More number of subscribers => smaller the log & easy to
rotate.