6. Metric Definition
• Not just count of events; but
• func of
• fields from one or more related events/entities
• on each event or a batch of events (for statistical
analysis)
• for a set of dimensions
10. Storage
• MultiDimensional support
• Optimized for Time series query
• Low query response times
• High write throughput
• Scalable
TSD*
* OpenTSDB does not support kerberose
11. Metric Definition
• Not scalable to write storm topologies for
each metrics
• Require DSL for non-tech folks
Introducing... Esper
12. Storm Topology - 1
Dim
Lookup
Dim
Lookup
Kafka
Spouts
Enricher
Bolts
Kafka
Bolts
{
id: a123-234,
time: 1234,
entityId: OD12
…
}
Event
{
id: a123-234,
time: 1234,
entityId: OD12
…
}
Enriched
Event
Dim
Store
17. HBase Time Batch Schema
Table 1 - Event Queue
• Key
<event_ns>_slot_<batchId>
batchId is constructed from event timestamp
• Value
(each column - Event JSON)
18. HBase Time Batch Schema
Table 2 - Event Queue Update Log
• Key
<event_ns>_log_<batchId>_<version>
batchId is constructed from event timestamp
version is timestamp at which batch was updated
• Value
Version