No Time Like the Present – The Case for Streaming Analytics

Grab some coffee and enjoy
the pre-show banter before
the top of the hour!

No Time Like the Present – The Case for Streaming Analytics

The Briefing Room

Welcome

Host:
Eric Kavanagh
eric.kavanagh@bloorgroup.com

Twitter Tag: #briefr

The Briefing Room

Mission

!   Reveal the essential characteristics of enterprise software,
good and bad
!   Provide a forum for detailed analysis of today s innovative
technologies
!   Give vendors a chance to explain their product to savvy
analysts
!   Allow audience members to pose serious questions... and get
answers!


The Briefing Room

Topics

This Month: ANALYTICS
February: BIG DATA
March: CLOUD
2014 Editorial Calendar at

www.insideanalysis.com/webcasts/the-briefing-room


The Briefing Room

Analytics
What do you
MEAN you
need your
data NOW?


The Briefing Room

Analyst: John Myers

John Myers is Research
Director of Business
Intelligence at
Enterprise Management
Associates


The Briefing Room

SQLstream
! SQLstream is an enterprise software company focused on
making businesses responsive to real-time Big Data assets
!   Its platform provides a relational stream for analyzing large
volumes of service, sensor, and machine and log file data
!   SQL queries in SQLstream generate results continuously as
data becomes available


The Briefing Room

Guests: Damian Black & Christian Lees
Damian Black
CEO, SQLstream
• 

Career in high tech, real-time software sector, with senior
positions at HP, XACCT (now Amdocs) and Followap (now Neustar)

• 

Holds 11 US patents

• 

Finalist in the 1995 International Management Challenge

Christian Lees
CTO, InfoArmor
• 
• 


Over 15 years of information security, network security and
intrusion detection experience
CTO of InfoArmor, with previous experience at Level 3
Communications, Trustwave and owner of Sage Technologies

The Briefing Room

S Q L s t r e a m : Re a l - t i m e B i g D a t a P l a t fo r m

Streaming Analytics
from
High-velocity Machine Data

facts

capabilities

innovations

o  Launched 2009

o  Unstructured and
structured data

o  Massively scalable
streaming data platform

o  Deployments across
many industries

o  Accelerates and extends
Hadoop & RDBMS

o  Only standard SQL
streaming engine

o  Real world benchmarks

o  Not only SQL

o  Five patents for stream
processing

Copyright © 2014 | +1 877 571 5775 | inquiries@sqlstream.com | 10

S e l e c t e d C u s t o m e r s & Pa r t n e r s

Telecommunications

Intelligent Transportation

Security Intelligence

IT Operations

Internet of Things & Sensors

Smarter Internet

Selected Strategic Partners


Bridging The Chasm
Operational Intelligence integrates Operations and BI
“

Operations

Business
Intelligence

Transaction Processing

Post-hoc Analysis

Machine Data

Data Warehousing

Everyday business

Strategic insights

As we move toward a
real-time business
environment, the
capability to process
data flows swiftly and
flexibly will become
increasingly
important. SQLstream
leads the industry in
this kind of
”
capability.
Robin Bloor
Chief Analyst for Bloor Group


Bridging The Chasm
Operational Intelligence integrates Operations and BI
“
Operational Intelligence
Optimizes tactical decisions from real-time actionable insights
Combines operations data with BI data continuously
Provides Real-time integrated view of the business and operations

Operations

Transaction Processing
Machine Data
Everyday business

Security
Compliance
Fraud
Quality
Promotion
Advertising
Cross-selling

Business
Intelligence
Post-hoc Analysis
Data Warehousing
Strategic insights

As we move toward a
real-time business
environment, the
capability to process
data flows swiftly and
flexibly will become
increasingly
important. SQLstream
leads the industry in
this kind of
”
capability.
Robin Bloor
Chief Analyst for Bloor Group


T h e I n f o r m a t i o n Va l u e C h a i n


T h e I n f o r m a t i o n Va l u e C h a i n

Make it happen!
What might happen?
What is happening?
What just happened?

S T R E A M I N G A N A LY T I C S

Analytics previously meant High-latency

Current architectures
o  Multi-stage processing
o  Batch ETL
o  Interim operational data stores

IMPACT
o  High Cost of Ownership
o  Delays to internal customers and consumers
o  Delays to external customers and partners

WAREHOUSE
ETL

PLATFORMS

Near-term
data storage


Streaming Analytics
Massively parallel with incremental evaluation

¤  Continuous queries on unstructured & structured streaming data
¤  Incremental query results
¤  Predictive analytics & automated actions
Operational Intelligence

M2M

Radio
Logs
Wireless
Networks
Mobile
Security
gateways
Sensors
Internet

Enhancing with
historical information

Storage of
intermediate & final
query results


SQL
Where is the intelligence?

Transaction
Log Details

TRANS,2013-02-17-15:30:22,3458783,2347897953,128.56.0.253,STATUS:-15, DE69975, 4157588342

Web Server
Logs

[Sun Feb 17 15:30:49 2013] [notice] srv-sfo-08 caught SIGTERM, shutting down
[Sun Feb 17 15:30:49 2013] [notice] Apache/2.2.21 -- resuming normal operations

CDRs

TERMINATE,ctl09gsx,01299796304,GMT-08:00,02-17-13,15:21:00,9,387,64ms,02-17-13,15:30:55,0005,
IP-TO-IP,4157588342,8775715775,1,0,4157588342,RD_AXY_NN0_001,SFR01AAG34,40.50.245.60,
234.234.60.75,65678,411,399,SIP,SANFRANCISCO,0x4B1698,0x0005E,0x49768,4157588342,0198873465

<id>1597831220</id><deviceid>0198873465</deviceid><lat>lat=47.643957</lat><lon>lon=
-122.3269</lon><time>2013-02-17T15:37:26Z</time><bearing>223.4535</bearing>

Device
Locations

<id>1597865781</id><deviceid>0198873465</deviceid><lat>lat=47.645982</
lat><lon>lon=-122.327500</lon><time>2013-02-17T15:37:26Z</time><bearing>200.6138</bearing>

Twitter

{"created_at:Thu Feb 17 15:30:55 +0000 2013,id:304612775055998976,id_str:
304612775055998976,text:@MyServiceProvider today sucks, keeps dropped!,source:u006ca
href=http:www.url.com rel=nofollow,followers_count:147,friends_count:10142, location: San Francisco,
time_zone: Pacific, geo_enabled:true, location:u00dcT: -6.1987552,106.8661953, screen_name:APerson


SQL

Transaction
Log Details

Timestamp
TRANS,2013-02-17-15:30:22,3458783,2347897953,128.56.0.253,STATUS:-15, DE69975, 4157588342

Timestamp
Web Server
Logs


Timestamp
CDRs



Device
Locations


Timestamp
Timestamp

Twitter



SQL

Transaction
Log Details

Customer

Timestamp

Server


Timestamp

Mobile #

CDRs

Mobile #

TRANS,2013-02-17-15:30:22,3458783,2347897953,128.56.0.253,STATUS:-15, DE69975, 4157588342

Timestamp
Web Server
Logs

Fail Code

Device ID
Term Reason

Device ID

Location


Device
Locations


Timestamp
Timestamp

Twitter

Service Provider

Location

Streaming Analytics Platfor m

Billing

Network

Analysis

Rating

CLEANING &
FILTERING

STREAMING
ANALYTICS

Log

M2M

Mobile

Fraud

Monitoring

STREAMING

AGGREGATION

Networks

Radio
towers

QoE

CONTINUOUS

INTEGRATION

Sensors


Re a l - t i m e A r c h i t e c t u r e
Continuous Raw Data Ingestion, Integration, Analysis and Output of Derived Data in Real-time

Real-time Dashboards & Visualization
Streaming SQL Real-time Applications
SQL
Developer
Tools

Query Planner & Optimizer for MPP Execution

Platform
Administration

Streaming Agent/Adapter Layer + JDBC API
Impala SQL
HBase
Logs

Networks

M2M

Servers

Telematics

Sensors

GPS

Social Media

External Data Warehouses & Systems

HDFS / MR

Data

Warehouse

Hadoop for Stream Persistence,
Enrichment & Replay (Optional)

Geo-Analytics for
Location-based
Applications

s-Analyzer

s-Visualizer

Drag and Drop Application Builder for
Streaming Analytics Applications

Advanced Enterprise
Visualization

s-Server

Dashboards

Data Management Platform for Streaming Big Data

s-Cloud

Fast Start Streaming Apps

s-Transport

StreamApps

Developer & Admin Console

s-Studio

S Q L s t r e a m s - S t r e a m i n g P r o d u c t Po r t f o l i o

s-Server EC2 AMI Deployment

CLOUD INFRASTRUCTURE MONITORING
Cloud infrastr ucture monitoring with Bollinger bands
SELECT STREAM ROWTIME, url, numErrorsLastMinute
FROM ( SELECT STREAM ROWTIME, url, numErrorsLastMinute,
AVG(numErrorsLastMinute) OVER lastMinute AS avgErrorsPerMinute,
STDDEV(numErrorsLastMinute) OVER lastMinute AS stdDevErrorsPerMinute
FROM ServiceRequestsPerMinute
WINDOW lastMinute AS (PARTITION BY url RANGE INTERVAL ‘1’ MINUTE PRECEDING) ) AS S
WHERE S.numErrorsLastMinute > S.avgErrorsPerMinute + 2 * S.stdDevErrorsPerMinute;

BUSINESS NEED:

Detect run-away applications
before resource consumption
becomes an issue.


C u s t o m e r B e n c h m a r ke d Pe r fo r m a n c e
Large network & telecom equipment manufacturer

SYSTEM CHARACTERISTICS

PERFORMANCE STATISTICS

Collection:

Intelligent Remote Agents (Distributed)

System Throughput:

1.35M events / sec

Enrichment:

Streaming data augmentation

Server Conﬁguration:

1 x 4-core CPU

Analytics:

Temporal & spatial pattern detection

Event Size:

~1KB

Output:

Data warehouse + applications (JDBC)

Data Sources:

Many

Network Data

Remote
Agent

Network Data

Remote
Agent

Network Data

Remote
Agent

Network Data

Remote
Agent

Network Data

Remote
Agent

SQLstream

ENRICH

ANALYZE

Data

Warehouse

SHARE

External
Systems

External Data


C a s e s t u dy : C a l l Ra t i n g & Fra u d

Veracity Networks
“SQLstream allows Veracity to provide vital
real-time reports to our customers that
previously took hours to create. SQLstream
also provides real-time monitoring and insight
into network concerns allowing Veracity to
proactively address any such issues.”


C a s e s t u dy : f ra u d p r e ve n t i o n ( c o n t . )

duration

Customer call proﬁle

Mo

Tue

Wed

Thu

Fri

Sat

Destination
①  LA
②  SF
③  NY
④  ….

①  LA
②  Nairobi
③  NY
④  …..

Location
①  LA
②  LA1

IP spooﬁng alerts

①  LA
②  Detroit

Sun

S
T
R
E
A
M
I
N

G

A
N
A
L
Y
T
I
C

S

Alerts

Triggers
•  Call suspension
•  Acct. suspension
•  Emails

Reports


I n f o A r m o r c a s e s t u dy

C a s e s t u dy : C y b e r s e c u r i t y
InfoArmor
¤ Founded by Washington Mutual
to protect 10M credit card
holders
¤ Growing at triple digit rates
¤ Engaged, satisfied subscribers
NEEDS
¤  Decision engine
¤ Consume agnostic data sources
¤ Scalable
¤ Real-time

C a s e s t u dy : C y b e r s e c u r i t y
a g r ow i n g m a r k e t

$207
Billion

Entrepreneur.com

¤  No longer an unorganized hacker world
¤  Innovation and technology
¤  Global economy
In 2012, U.S. Navy databases were hacked and
200,000 sailors’ information was put at risk.
¤  Political support

C y b e r A t t a c k s | D A M AG E S

î  12.6 Million Americans were ID Theft victims last year
î  608,271,950 and growing records have been
compromised due to security breaches since 2005

î  94% of healthcare organizations surveyed had at least
one data breach in the past 2 years

î  1 in 4 data breach notification recipients became a victim
of identity fraud

î  5 times more likely to be a fraud victim if your Social
Security Number has been compromised in a data breach

I N T E R N E T S U RV E I L L A N C E
What is the
Underground
Economy?
An ever-evolving complex of
compromised machines, networks
and web services identified by
InfoArmor and leading cyber
security firms.
InfoArmor Internet Surveillance uses bots to continuously monitor the Underground Economy to uncover compromised,
sensitive information.

Whether it is personal identifying data or a medical insurance card, Internet Surveillance

uncovers breached data and alerts in real time.
What We Monitor:
¤ 

Malicious Command & Control Networks

¤ 

Phishing Networks

¤ 

Black Market Forums

¤ 

Exploited Websites

¤ 

Known Compromised
Machines & Servers


I N T E R N E T S U RV E I L L A N C E

X

INFOARMOR BOTS monitor
UNDERGROUND ECONOMY

COMPROMISED DATA sent
back to INFOARMOR

SENSOR compares
compromised to subscriber data
in secure environment, creating
ALERTS with 100% accuracy

How We Monitor:
¤ 

Proprietary hardware and software solution

¤ 

Unparalleled alert accuracy (minimized false positives)

¤ 

Secure: separate reconnaissance and analysis efforts, plus no refined search queries

What We Monitor:
¤ 

Credentials, SSNs, names, addresses, emails and DOBs

¤ 

Wallet items (i.e. credit cards, medical insurance card)

C a s e s t u dy : S t r e a m i n g a n a l y t i c s
SQLstream BENEFITS
¤ Ability to adapt to many data sources
¤ Real Time analysis and alerting
¤ Offset database load
¤ Data Hygiene prior to data warehousing

RESULTS
¤ Real-time actionable alerts
¤ Unity in Ingress Data points
¤ Dual Purpose solution
•  Helps Compliance
¤ Plans to expand engagement

offline
online


Damian Black

Email | damian.black@sqlstream.com
Website | www.sqlstream.com

DOWNLOADS | http://www.sqlstream.com/downloads/

Perceptions & Questions

Analyst:
John Myers


The Briefing Room

Importance of Speed of Response in Big Data

John L Myers
Enterprise Management Associates
Research Director
JMyers@EnterpriseManagement.com

© 2012 Enterprise Management Associates, Inc.

Speaker

John L Myers
Enterprise Management Associates
Research Director

John Myers joined Enterprise Management Associates in 2011 as senior analyst of the business
intelligence (BI) practice area. John has 10+ years of experience working in areas related to
business analytics in professional services consulting and product development roles, as well as
helping organizations solve their business analytics problems, whether they relate to operational
platforms, such as customer care or billing, or applied analytical applications, such as revenue
assurance or fraud management.

Slide 40

JohnLMyers44

Disruptive Forces in Data Management:
Changing the Speed of Business

Slide 41

75
65
55
45
35
25

Use Cases met with Big Data Implementations
•  Speed of processing response
•  Combining data by structure
•  Pre-processing data
•  Utilization of streaming data
•  Staging structured data
•  Online archiving

Slide 42

Rogers, Myers and Devlin, "Big Data: Operationalizing the Buzz", Enterprise Management,
http://research.enterprisemanagement.com/big-data-2013-webinar-nl.html

Big Data Platforms have Multiple Use Cases

Slide 43


Top 5 Business Challenges Met with Big Data
Projects
•  Risk management
•  Fraud Analysis, Liquidity Risk Assessment

•  Ad-hoc operational queries
•  Customer Relations Management

•  Asset optimization
•  Staff Scheduling, Logistical Asset Planning

•  Operational event and policy processing
•  Billing, Rating

•  Campaign Optimization
•  Market Basket Analysis, Cross-sell/Up-sell Recommendation

•  Clustering, social graph analysis

Slide 44

•  Grouping and Relationship Analysis, Geographic Optimization
Rogers, Myers and Devlin, "Big Data: Operationalizing the Buzz", Enterprise Management,
http://research.enterprisemanagement.com/big-data-2013-webinar-nl.html

Building the Bridge between Operational
Processes and Analytical Results

Slide 45


Hybrid Data Ecosystem 2013:
From Requirements to Consumers

Slide 46


Questions

Slide 47

•  This version of “streaming analytics” sounds a lot like “complex
event processing.” How does SQLstream differentiate from those
solutions?
•  The open source community, such as Apache Hadoop, has been
coming up with solutions to problems like streaming. What
advantages does a proprietary solution like SQLstream have over
these solutions?
•  “Streaming analytics” appears to be well suited for the upcoming
trends in the “location based services” in mobile telecom and
“telematics” in automotive. Which use cases appear to have the
best chances of success? Marketing activities such as “location
coupons?” Operational optimization such as “managed
highways?”


Questions

Slide 48

•  What are the best types of datasets to be used in the world of
“streaming analytics?” Structured big data or large volumes of
single row event data (i.e., log information)? Formatted multi-row
event data (i.e., JSON)?
•  What types of datasets should be avoided?

•  What types of analytical techniques are best used with “streaming
analytics?” Advanced analytical models associated with predictive
or clustering algorithms? Rules-based, policy techniques (i.e.,
decision trees)? Simple descriptive analytics?
•  What types of analytics techniques should be avoided?



The Briefing Room

Upcoming Topics

This Month: ANALYTICS
February: BIG DATA
March: CLOUD
2014 Editorial Calendar at

www.insideanalysis.com/webcasts/the-briefing-room

www.insideanalysis.com


The Briefing Room

Thank You
for Your
Attention


The Briefing Room

No Time Like the Present – The Case for Streaming Analytics

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (6)

Similar to No Time Like the Present – The Case for Streaming Analytics

Similar to No Time Like the Present – The Case for Streaming Analytics (20)

More from Inside Analysis

More from Inside Analysis (20)

Recently uploaded

Recently uploaded (20)

No Time Like the Present – The Case for Streaming Analytics