1. @Kognitio #SparkEvent
Hadoop meets Mature BI:
Where the rubber meets the road for
the Modern Data Platform
Michael Hiskey
Futurist, Product Evangelist
(and VP, Marketing & Business Development)

3. @Kognitio #SparkEvent
Today, and the Future
Big DataAdvanced Analytics
In-memory
Modern Data Platform
Hybrid Data Ecosystem ‘Logical Data Warehouse’
Predictive Analytics
Data Scientists
Data

4. @Kognitio #SparkEvent
The Data Scientist
Sexiest job of the 21st Century?

5. @Kognitio #SparkEvent
Data
Scientist
The Analytical Enterprise
Business
Analyst
Systems
Admin

6. @Kognitio #SparkEvent
Remember: Decision Support Systems?
…accessed with ease
and simplicity
Historical information, latency
BI tools have plateaued
0 1 2 3 4 5 6 7 8 9
Advanced analytics &
data science
More math…a lot more math

7. select Trans_Year, Num_Trans,
count(distinct Account_ID) Num_Accts,
sum(count( distinct Account_ID)) over (partition by Trans_Year order by Num_Trans) Total_Accts,
cast(sum(total_spend)/1000 as int) Total_Spend,
cast(sum(total_spend)/1000 as int) / count(distinct Account_ID) Avg_Yearly_Spend,
rank() over (partition by Trans_Year order by count(distinct Account_ID) desc) Rank_by_Num_Accts,
rank() over (partition by Trans_Year order by sum(total_spend) desc) Rank_by_Total_Spend
from( select Account_ID,
Extract(Year from Effective_Date) Trans_Year,
count(Transaction_ID) Num_Trans,
sum(Transaction_Amount) Total_Spend,
avg(Transaction_Amount) Avg_Spend
from Transaction_fact
where extract(year from Effective_Date)<2009
and Trans_Type='D'
and Account_ID<>9025011
and actionid in (select actionid from DEMO_FS.V_FIN_actions
where actionoriginid =1)
group by Account_ID, Extract(Year from Effective_Date)
) Acc_Summary
group by Trans_Year, Num_Trans
order by Trans Year desc Num Trans;
Behind the
numbers

8. @Kognitio #SparkEvent
What has changed?
More
connected-users?
More-connected
users?

9. @Kognitio #SparkEvent
Don’t be a Railroad Stoker!
Highly skilled engineering required …
but the world innovated around them.

10. @Kognitio #SparkEvent
Machine learning
algorithms Dynamic
Simulation
Statistical
Analysis
Clustering
Behavior
modelling
The drive for deeper understanding
Reporting & BPM
Fraud detection
Dynamic
Interaction
Technology/Automation
Analytical Complexity
Campaign
Management

11. @Kognitio #SparkEvent
Key: “Graduation”
Projects will need
to Graduate
from the
Data Science Lab
and become part
of
Business as Usual

12. @Kognitio #SparkEvent
Your goal:
PRESS HERE
…and really cool Big Data stuff happens!

13. @Kognitio #SparkEvent
Data flow

14. @Kognitio #SparkEvent
© 20th Century Fox

15. @Kognitio #SparkEvent
 No need to pre‐process
 No need to align to schema
 No need to triage
Null storage concerns

16. @Kognitio #SparkEvent
Hadoop just too
slow for interactive
BI!
…loss of train‐
of‐thought
“while Hadoop shines as a processing
platform, it is painfully slow as a query tool”

17. @Kognitio #SparkEvent
Lots of these
Not so many of these
inherently disk oriented
typically low ratio of CPU to Disk
Hadoop is…

18. @Kognitio #SparkEvent
Analytics needs
low latency, no I/O wait
High speed in‐memory processing

19. A* Modern Data Platform
Reference Architecture
Analytical
Platform Near‐line
Storage
(optional)
Access
Application &
Client Layer
All BI Tools All OLAP Clients Excel
Persistence
Layer Hadoop
Clusters
Enterprise Data
Warehouses
Legacy
Systems
…
Reporting
Cloud
Storage
*(not THE)

20. © Hortonworks Inc. 2013
(another) Next-Generation Data Architecture
Page 20
APPLICATIONSDATA SYSTEMS
Microsoft Applications
DATA SOURCES
Traditional Sources
(RDBMS, OLTP, OLAP)
In‐memory MPP Accelerator
BI Tools & OLAP Clients
TRADITIONAL REPOS
RDBMS EDW MPP
OPERATIONAL
TOOLS
MANAGE &
MONITOR
DEV & DATA
TOOLS
BUILD &
TEST
New Sources
(web logs, email, sensors, social media)
HORTONWORKS
DATA PLATFORM

21. Analytical Platform

22. @Kognitio #SparkEvent
It’s all about getting work done
Used to be simple fetch of value
Tasks evolving:
Then was compute dynamic aggregate
Now complex algorithms!
Now complex algorithms!