.NET Fest 2019. Андрей Антиликаторов. Проектирование и разработка Big Data решений на базе Microsoft Azure

Тема доклада
KYIV 2019
Проектирование и разработка Big Data
решений на базе Microsoft Azure
Андрей Антиликаторов
.NET CONFERENCE #1 IN UKRAINE

.NET LEVEL UP
Big Data :: Mobile Phones Usage
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Text

.NET LEVEL UP
Big Data :: Connected Car
Text

.NET LEVEL UP
Big Data :: Smart City
Text

.NET LEVEL UP
Big Data :: Around the World
Text

KYIV 2019
Big Data Solutions :: Getting Started

.NET LEVEL UP
Big Data Pipeline

.NET LEVEL UP
Big Data Reference Architecture for IoT

.NET LEVEL UP
Lambda Architecture

.NET LEVEL UP
Batch Processing

.NET LEVEL UP
Stream Processing

KYIV 2019
Big Data Solutions :: Best Practices

.NET LEVEL UP
Big Data Solutions :: Best Practices

.NET LEVEL UP
Best Practices :: Think Big. Start Small
• Scalable architecture.
• Prototyping is a must.
• Azure Stream Analytics is NOT a silver bullet.
Data Source On-Boarding

.NET LEVEL UP
Best Practices :: Data First
• Data formats are true headache.
• Hard to extract valuable data.
• Diagnostics telemetry must be processed.

.NET LEVEL UP
Best Practices :: Don’t Interrupt Fast Path
• Don’t process between high-scale components
unless you know what you’re doing.
• Pass only valuable data which really needs to
be processed in real-time.

.NET LEVEL UP
Best Practices :: Security as a Culture
• Think about security, identity and management
from the very beginning.
• Security is a shared responsibility between
Azure and the customer.

.NET LEVEL UP
Best Practices :: Understand Business Model
• Data architecture more important than solution
architecture
• Plan and budget appropriately through simple
predictable scenarios
• Remember that you need to support millions of
assets (for IoT)

KYIV 2019
Big Data Solutions :: Data Collection

.NET LEVEL UP
Data Collection :: How To
Transactional data (data sync)
One-Time data migration
IoT data

.NET LEVEL UP
Data Collection :: Data Factory

.NET LEVEL UP
Data Collection :: Data Hubs
Event Hub IoT Hub

.NET LEVEL UP
Data Hubs :: Comparison
Capability Standard Basic Event Hub
Cloud-to-device Yes Yes Yes
HTTPS, AMQP Yes Yes Yes
MQTT Yes Yes
File upload from devices Yes Yes
File upload from devices Yes Yes
Device Provisioning Service Yes Yes
Cloud-to-device messaging Yes
Device twin and management Yes
IoT Edge Yes

.NET LEVEL UP
Data Hubs :: Main Problems
• Strict quotas & limits
• Limited amount of partitions (32)
• Need to plan additional 20% - 30% capacity
• Limited number of concurrent connections
• Works great if resources located in single region
• Cannot provide latency < 0.1 sec

.NET LEVEL UP
Data Connection :: Kafka on HDInsight

.NET LEVEL UP
Kafka on HDInsight :: Pros & Cons
• Open-source
• Managed (SLA 99.9%)
• Zookeeper out of the box
• Not cost effective
• Requires a lot of tuning

.NET LEVEL UP
Data Connection :: What to Choose?
Capability
Cloud-to-device
communications
Yes No No
Device-initiated file
upload
Yes No No
Device state
information
Device Twins No No
Protocol support HTTPS, MQTT. AMQP AMQP, HTTPS Kafka Protocol
Security Per-device identity Shared access
policies
Through external
authentication services

.NET LEVEL UP
Data Connection :: Custom Solution
Queue
Device registry
Data Connector

.NET LEVEL UP
Custom Solution :: Connector

.NET LEVEL UP
Custom Solution :: Connector & Queue
Device
Registry
Queue
Connector

KYIV 2019
Big Data Solutions :: Processing

.NET LEVEL UP
Processing :: Azure Stream Analytics
• Easy to use
• Easy to scale
• Good integration with Azure Services
• Relatively cheap
• HIPAA, HITRUST, CSA STAR, ISO, GDRP
• Limited SAQL
• Vendor lock
• Possible data loss and crash on invalid data

.NET LEVEL UP
Processing :: Stream Analytics :: Repartitioning

.NET LEVEL UP
Processing :: HDInsight

.NET LEVEL UP
HDInsight :: Pros & Cons
• Not cost effective
• Software is outdated
• Software requires a lot of tuning
• Provides managed versions of 1st class frameworks
• Seamless integration
• Frameworks are open-source

.NET LEVEL UP
Processing :: HDInsight :: Storm
• Low latency. True streaming. High throughput
• Excellent for non-complicated use cases
• No state management
• “At least once” guarantee

.NET LEVEL UP
Processing :: HDInsight :: Spark
• High throughput. Complete fault tolerance
• Exactly once guarantees
• No true streaming (micro batches). High latency.
• Too complex to tune.

.NET LEVEL UP
HDInsight :: Spark & Storm Benchmarking
• Demo JSON files { 100 users; 10 events per user }
• 1 Redis node. 5 Kafka nodes. 10 worker nodes.

.NET LEVEL UP
HDInsight :: Spark Benchmarking
70 sec
0
Latency Percentiles (130 000 events/s; 2-second micro-batches)
50 sec
60 sec
40 sec
10050 90
30 sec
25
10 sec
20 sec

.NET LEVEL UP
16 sec
0
Latency Percentiles (100 000 events/s; 10 second micro-batches)
8 sec
12 sec
4 sec
10050 90

.NET LEVEL UP
5 sec
0
Latency Percentiles (100 000 events/s; 3-second micro-batches)
3 sec
4 sec
2 sec
10050 90
1 sec
25

.NET LEVEL UP
HDInsight :: Storm Benchmarking
3 sec
0
Latency Percentiles (up to 150 000 events/s)
2 sec
2.5 sec
1.5 sec
10050 90
1 sec
25

.NET LEVEL UP
Processing :: What to Choose?
NOT Avro, JSON, or CSV
Scale beyond 1 Gb/s
Low latency is a must
Zero data loss

KYIV 2019
One more thing…

.NET LEVEL UP
Big Data Solutions :: Cross-Cutting Concerns

KYIV 2019
Андрей Антиликаторов
Solution Architect

.NET Fest 2019. Андрей Антиликаторов. Проектирование и разработка Big Data решений на базе Microsoft Azure

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a .NET Fest 2019. Андрей Антиликаторов. Проектирование и разработка Big Data решений на базе Microsoft Azure

Similar a .NET Fest 2019. Андрей Антиликаторов. Проектирование и разработка Big Data решений на базе Microsoft Azure (20)

Más de NETFest

Más de NETFest (20)

Último

Último (20)

.NET Fest 2019. Андрей Антиликаторов. Проектирование и разработка Big Data решений на базе Microsoft Azure