Patterns for Building Streaming Apps

Technical Lead
Patterns for Building Streaming Apps
Mohanadarshan Vivekanandalingam

Goal
● Streaming analytics? Streaming apps?
● The architecture of a streaming engine
● Understanding streaming constructs
● Applying patterns when building streaming apps
● Managing streaming patterns
● Deployment patterns

Why Streaming ?
Real-time
Near
Real-time
Offline
Constant low
milliseconds &
under
Low milliseconds
to
seconds
10s seconds
to
minutes
● A stream is series of events
● Almost all new data is streaming
● Detects conditions quickly
Image Source : https://www.ﬂickr.com/photos/plusbeautumeurs/33307049175

Why Streaming
Apps?
● Identify perishable insights
● Continuous integration
● Orchestration of business
processes
● Embedded execution of
code
● Sense, think, and act in real
time
- Forrester

How to Build a
Streaming App
Use a Streaming Processor
● Stream processor handles
data ﬂow, scalability, and
failure - you have to handle
the rest
● Publish data to a topic
● Write an end-to-end ﬂow to
consume events and processes
Code it Yourself
Use a Streaming SQL-based
Streaming Processor
● Write the queries using
streaming SQL

● To understand what stream
processing can do!
● Easy to solve common
problems in stream processing
● Where to use what?
● Learn best practices
Why Patterns for Streaming ?
Image Source : https://www.ﬂickr.com/photos/laurawoodillustration/6986871419

1. Streaming data preprocessing
2. Data store integration
3. Streaming data summarization
4. Interactive data search
5. KPI analysis and alerts
6. Event correlation and trend analysis
7. Real-time predictions
Streaming App Patterns

Stream Processor
Core
Streaming Engine
WSO2 Stream
Processor
Events
JMS, Thrift, SMTP, HTTP, MQTT, Kafka
Analytics Fabric
Complex Event
Processing
Incremental Time
Series Aggregation
Machine
Learning
Extension Store
Financialand
BankingAnalytics
RetailAnalytics
LocationAnalytics
OperationalAnalytics
SmartEnergy
Analytics
Custom Analytics
Solutions
...
Solutions
StatusMonitoring
Rule
Mgmt.
Dashboard

● Lightweight, lean, and cloud native
● Easy to learn streaming SQL (Siddhi SQL)
● High performance analytics with just 2 nodes (HA)
● Native support for streaming machine learning
● Long term aggregations without batch analytics
● Highly scalable deployment with exactly-once processing
● Tools for development and monitoring
● Tools for business users to write their own rules
Overview of WSO2 Stream Processor

Stream Processing
With WSO2 Stream Processor
Siddhi Streaming App
- Process events in a streaming manner
- Isolated unit with a set of queries, input and
output streams
- SQL Like Query Language
from Sales#window.time(1 hour)
select region, brand, avg(quantity) as AvgQuantity
group by region, brand
insert into LastHourSales ;
Stream
Processor
Siddhi App
{ Siddhi }
Input Streams Output Streams
Filter Aggregate
JoinTransform
Pattern
Siddhi Extensions

Use Case
Online Shopping Application
Place Order
Process
Order
Out for Delivery
Process
Payment
BANK

Use Case
• Monitor sales, supply, and delivery
• Optimize sales, supply, and delivery
• Predict demand/sales
• Recommend products when shopping
• Manage processing rules online
• Visualize real-time sales and delivery
Online Shopping Application

1. Streaming Data Pre-processing
• Consume events from multiple sources
• Convert them to streams
• Filter events
• Add defaults to missing ﬁelds
• Change event stream structure
Filtering, Add Defaults, and Projection
Filter
Transform
Process
Add
Defaults

Deﬁne Stream
@app:name(‘Online-Shopping-Analytics’)
define stream ProductPurchaseStream(userId
string, sessionId string, productId string,
qty double, price double);

Consume Events:
MQTT, HTTP, TCP, Kafka, JMS,
RabitMQ, etc.
Map Events to
Streams:
JSON, XML, Text, Binary, WSO2Event,
KeyValue, etc.
@source(type = http, …, @map(type = json, …))

Writing the Query
define stream ProductPurchaseStream
(userId string, sessionId string, productId
string, qty double, price double);
from ProductPurchaseStream
select *
insert into PossibleDiscountProductStream ;

Filter
from ProductPurchaseStream [qty > 5 and
productId == ‘XYZ’]
select *

Transformation &
Defaults
productId == ‘XYZ]
select userId, sessionId, productId, qty,
price * 1.17 as usdPrice,
‘USD’ as currency

Functions:
Inbuilt, Custom UDF or
Siddhi Extension
productId == ‘XYZ]
select userId, sessionId, productId, qty,
convertToUSD(price) as usdPrice, ‘USD’ as
currency

2. Data Store Integration
● Allows performing operations with the data store while
processing events on the ﬂy
Store, Retrieve, Remove, and Modify
● Provides a REST endpoint to query Data Store
● Query optimizations using Primary and Indexing keys
● Search ● Insert ● Delete ● Update ● Insert/Update

Deﬁne Table
(In Memory)
define stream ShoppingCheckoutStream(userId
string, sessionId string, amount double,
currency string );
define table UserPurchases(userId string,
sessionId string, amount double);

Primary Key &
Indexing
currency string );
@primaryKey(‘userId’)
@Index(‘sessionId’)
define table UserPurchases(userId string,

Table Backed by:
RDBMS, MongoDB, HBase, Cassandra, Solr,
Hazelcast, etc.
currency string );
@store(type=‘rdbms’, … )
define table UserTable(userId string,

Insert into Table
currency string );
@store(type=‘rdbms’, … )
define table UserTable(userId string,
from ShoppingCheckoutStream
select userId, sessionId, amount
insert into UserTable;

3. Streaming Data Summarization
● Can perform aggregations over short and long time periods
● Support for aggregations such as:
○ Sum
○ Count
○ Min/Max
○ Avg
○ etc.
Aggregations Over Time Periods

Aggregations Over a
Short Time
from ProductPurchaseStream#window.time(1 min)
select productId, sum(qty) totalQty,
currentTimeMillis() as timestamp
group by productId
insert into LastMinPurchaseStream;
Windows Sliding and Batch for Time,
Length, etc.

3. Streaming Data Summarization
Aggregations Over Long Time Periods
• Incremental aggregation for every
– second, minute, hour, day, year
• Support for out-of-order event arrival
• Fast data retrieval from memory and disk
for real time updates
Current Min
Current Hour
Sec
Min
Hour
0 - 1 - 5 ...
- 1
- 2 - 3 - 4 - 64 - 65 ...
- 2
- 124

Aggregations Over a
Long Time
define aggregation PurchaseAggregation
from ProductPurchaseStream
select productId, sum(price * qty) as
totalAmount, sum(qty) as noOfItems
group by productId
aggregate every seconds ... years ;
Deﬁne Aggregation

4. Interactive Data Search
Search Data Promptly
• Can perform data search on Data
Stores or pre-deﬁned aggregations.
• Supports both REST and Java APIs

Data Retrieval
Query
from PurchaseAggregation
on productId == ‘XYZ’
within 2018-05-01 2018-06-01
per ‘day’
select productId, totalAmount, noOfItems ;

Dashboard for
Business Users
• Generate dashboard and
widgets
• Fine grained permissions
– Dashboard level
– Widget level
– Data level
• Localization support
• Inter widget communication
• Shareable dashboards with
widget state persistence

5. KPI Analysis and Alerts
Generate Alerts Based on KPIs
• Identify KPIs using
– Filter, ifThenElse, having, etc.
• Send alerts using Sinks

Notify with
Event Sinks
define stream ShoppingPaymentStream(userId
string, name string, email string, sessionId
string, totalAmount double, address string,
isSuccess boolean);
@sink(type=‘email’, to=‘{{email}}’
@map(type=‘text’,
@payload(‘‘‘
Hi, {{name}}
Order placed successfully ...’’’))
define stream SuccessfulPaymentStream (userId
string,name string, email string, ...);
from ShoppingPaymentStream [isSuccess == true]
select *
insert into SuccessfulPaymentStream;

6. Event Correlation & Trend Analysis
CEP for Patterns and Sequences
• Identify complex patterns
– Followed by, non-occurrence, etc.
• Identify trends
– Peek, triple bottom, etc.

Pattern
Detect non-occurrence
currency string );
define stream ShoppingPaymentStream(userId
string, name string, email string, sessionId
string, totalAmount double, address string,
isSuccess boolean);
from every (e1 = ShoppingCheckoutStream)
-> not ShoppingPaymentStream
[sessionId == e1.sessionId]
for 15 min
select e1.sessionId, e1.totalAmount
insert into PaymentDelayedStream ;

Sequences
Identify Decreasing Trend
define stream LastMinPurchaseStream(productId
string, totalQty double, timestamp long);
partition with
(productId of LastMinPurchaseStream)
Begin
from every e1=LastMinPurchaseStream,
e2=LastMinPurchaseStream
[timestamp - e1.timestamp < 10 and
e1.totalQty > totalQty]*,
e3=LastMinPurchaseStream
[timestamp - e1.timestamp > 10 and
e2[last].totalQty > totalQty]
select e1.productId, e1.totalQty as
initialQty, e3.totalQty as finalQty
insert into ContinousSalesReductionStream ;
end;

7. Real-time Predictions
Using Machine Learning
• Use pre-created machine learning
models and perform predictions.
– PMML, TensorFlow, etc.
• Streaming Machine Learning
– Clustering, Classiﬁcation,
Regression
– Markov Models, Anomaly
detection, etc.
Image Source : https://www.ﬂickr.com/photos/149823084@N08/27871878168/

Machine Learning
Models for
Prediction
currency string );
from ShoppingCheckoutStream#pmml:predict
(“/home/user/ml.model”, userId)
select *
Insert into ShoppingPredictionStream ;

Continuous Learning
& Prediction
currency string );
define stream ShoppingPredictionStream(userId
string, predictedAmount double);
#streamingml:hoeffdingTreeTrain
(‘Model’, userId, amount);
...
#streamingml:hoeffdingTreeClassifier
(‘Model’, userId);
Select *
Insert into ShoppingPredictionStream;
Online Training
Online Prediction

Business
Rules for Non
Technical
Users
Deﬁne your own
business rules from
scratch
Modify templated
complex business
rules using rule
parameters

Developer Studio
for Streaming
Apps
Supports both drag n
drop & source editor
Editor
Debugger
Simulation
Testing

• High performance
– Process around 100k
events/sec
– Just 2 nodes
– While most others need 5+
• Zero downtime & event Loss
• Incremental state persistence &
recovery
• Simple deployment with RDBMS
– No Zookeeper, Kafka, etc.
• Multi data center support
Minimum HA with 2 Nodes
Stream Processor
Stream Processor
Event Sources
Dashboard
Notiﬁcation
Invocation
Data Source
Siddhi App
Siddhi App
Siddhi App
Siddhi App
Siddhi App
Siddhi App
Event
Store

• Exactly-once processing
• Fault tolerance
• Highly scalable
• No back pressure
• Distributed development conﬁgurations via annotations
• Pluggable distribution options (YARN, K8, etc.)
Distributed Deployment

Distributed Deployment with Kafka
Data
Base
Event
Source
Event
Sink
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Siddhi
App
Kafka Topic
Kafka Topic
Kafka Topic
Kafka
Topic
Kafka
Topic

Status Dashboard
• Understand system performance via
– Throughput
– Latency
– CPU, memory utilizations
• Monitor in various scales
– Node level
– Siddhi app level
– Siddhi query level
Monitor Resource Nodes and Siddhi Apps

● Gives you everything you need to build
streaming analytics
○ Manage data streams
○ Powerful Streaming SQL language
○ Dashboards and more
● Can provide 100K+ events per second with two node
HA (most alternatives need 5+ nodes) and can scale
more on top of Kafka
WSO2 Stream Processor

● Why streaming patterns and when to use them
● How WSO2 Stream Processor can be used to build streaming
patterns
● How to develop streaming patterns with WSO2 Stream
Processor
● The business beneﬁt of manageable rules and dashboards
● Patterns to apply for the deployment of streaming apps
Patterns for Streaming Apps

Patterns for Building Streaming Apps

Recomendados

Recomendados

Más contenido relacionado

Similar a Patterns for Building Streaming Apps

Similar a Patterns for Building Streaming Apps (20)

Último

Último (20)

Patterns for Building Streaming Apps