Architectural patterns and Azure Data Services for Big Data projects

1. Azure Big Data Story
@LynnLangit

2. I love to…Learn & Build

4. Azure Big Data by the V’s
Value
Volume Velocity Variety Veracity

5. Big Data = Business Value ?
60% of Big Data projects
FAIL
to go beyond pilot and will be abandoned
(through 2017) - Gartner

6. Volume
How big is Big Data?

7. Variety
Is my Data rectangular?

8. Persistence Choices
Files
Hadoop
NoSQL
Relational

10. Azure Persistence Choices
• Storage
• Store Simple
Files
• Data Lake
• HDInsight
• Cloudera
Hadoop
• MANY…
NoSQL
• SQL Azure
• SQL Azure DW
• SQL Server on Azure
VM
Relational

11. Drilling In: Relational AND NoSQL

12. Azure Persistence Choices Detailed
• Storage
• Store Simple
Files
• Data Lake
• HDInsight
• Cloudera
Hadoop • Redis Caching
• DataStax Enterprise
• Document DB
• Mongo Labs
• Graph Engine
NoSQL
• SQL Azure
• SQL Azure DW
• SQL Server on Azure VM
Relational

13. Velocity
How fast is my Data?

14. Veracity
How clean is my Data?

15. Load Choices
Load
Stream
Batch

16. Azure Load Choices
Load Libraries
StreamEvent Hub
Batch
Stream
Analytics

17. Data Cleaning Choices
ETL
Client
Machine
Learning

18. Azure Data Cleaning Choices
ETL
• SQL Server VM
• Data Pipeline
Client
• Power BI
• Power Query
ML
• Azure ML
• Data Marketplace
• SQL Server DQS

19. Data Pipelines

20. Azure Data Factory

22. Public Cloud
or
Hybrid Cloud

23. Data
Model
On Premise
SQL Server+
Cloud
Azure+
Key-Value
Queues
None
Windows Queues
Azure Redis Cache
Azure Queues
Wide Sparse
Columns
Columnstore Index
SSAS Tabular Models
Azure Tables
DataStax Enterprise (Cassandra)
Files FileTable, Filestream
XML data type
Azure BLOB Store
StoreSimple
JSON or
Graph
SQL Server 2016
None
Azure DocumentDB / Graph Engine (beta)
Hosted MongoDB or Neo4J
Large
Relational
SQL Server Enterprise
PDW
SQL Analysis Services
SQL Database (basic, standard, premium)
APS
SQL Data Warehouse
Hadoop Hortonworks HDInsight/ Data Lake,
Hosted Cloudera
Other Stream Insight Event Hub, StreamAnalytics, ML
Marketplace

25. Value
How useful is my Big Data?

26. Big Data = Business Value ?
60% of Big Data projects
FAIL
to go beyond pilot and will be abandoned
(through 2017) - Gartner

28. Architectural
Patterns

29. Architecture 1- File Storage / Backup

30. Architecture 2- Data Warehouse

31. Architecture 3 – Operational Database

32. Old
becomes
New

33. Architecture 4 – Small Big Data

34. MORE DATA

35. Architecture 5 – Big Data

36. www.TeachingKidsProgramming.org

37. Azure Big Data Story
@LynnLangit