1.
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
Проектирование и разработка Big Data
решений на базе Microsoft Azure
Андрей Антиликаторов
.NET CONFERENCE #1 IN UKRAINE

2.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Big Data :: Mobile Phones Usage
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Text

3.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Big Data :: Connected Car
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Text

4.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Big Data :: Smart City
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Text

5.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Big Data :: Around the World
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Text

6.
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
Big Data Solutions :: Getting Started
.NET CONFERENCE #1 IN UKRAINE

7.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Big Data Pipeline
.NET CONFERENCE #1 IN UKRAINE KYIV 2019

8.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Big Data Reference Architecture for IoT
.NET CONFERENCE #1 IN UKRAINE KYIV 2019

9.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Lambda Architecture
.NET CONFERENCE #1 IN UKRAINE KYIV 2019

10.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Batch Processing
.NET CONFERENCE #1 IN UKRAINE KYIV 2019

11.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Stream Processing
.NET CONFERENCE #1 IN UKRAINE KYIV 2019

12.
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
Big Data Solutions :: Best Practices
.NET CONFERENCE #1 IN UKRAINE

13.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Big Data Solutions :: Best Practices
.NET CONFERENCE #1 IN UKRAINE KYIV 2019

14.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Best Practices :: Think Big. Start Small
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• Scalable architecture.
• Prototyping is a must.
• Azure Stream Analytics is NOT a silver bullet.
Data Source On-Boarding

15.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Best Practices :: Data First
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• Data formats are true headache.
• Hard to extract valuable data.
• Diagnostics telemetry must be processed.

16.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Best Practices :: Don’t Interrupt Fast Path
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• Don’t process between high-scale components
unless you know what you’re doing.
• Pass only valuable data which really needs to
be processed in real-time.

17.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Best Practices :: Security as a Culture
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• Think about security, identity and management
from the very beginning.
• Security is a shared responsibility between
Azure and the customer.

18.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Best Practices :: Understand Business Model
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• Data architecture more important than solution
architecture
• Plan and budget appropriately through simple
predictable scenarios
• Remember that you need to support millions of
assets (for IoT)

19.
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
Big Data Solutions :: Data Collection
.NET CONFERENCE #1 IN UKRAINE

20.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Collection :: How To
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Transactional data (data sync)
One-Time data migration
IoT data

21.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Collection :: Data Factory
.NET CONFERENCE #1 IN UKRAINE KYIV 2019

22.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Collection :: Data Hubs
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Event Hub IoT Hub

23.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Hubs :: Comparison
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Capability Standard Basic Event Hub
Cloud-to-device Yes Yes Yes
HTTPS, AMQP Yes Yes Yes
MQTT Yes Yes
File upload from devices Yes Yes
File upload from devices Yes Yes
Device Provisioning Service Yes Yes
Cloud-to-device messaging Yes
Device twin and management Yes
IoT Edge Yes

24.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Hubs :: Main Problems
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• Strict quotas & limits
• Limited amount of partitions (32)
• Need to plan additional 20% - 30% capacity
• Limited number of concurrent connections
• Works great if resources located in single region
• Cannot provide latency < 0.1 sec

25.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Connection :: Kafka on HDInsight
.NET CONFERENCE #1 IN UKRAINE KYIV 2019

26.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Kafka on HDInsight :: Pros & Cons
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• Open-source
• Managed (SLA 99.9%)
• Zookeeper out of the box
• Not cost effective
• Requires a lot of tuning

27.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Connection :: What to Choose?
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Capability
Cloud-to-device
communications
Yes No No
Device-initiated file
upload
Yes No No
Device state
information
Device Twins No No
Protocol support HTTPS, MQTT. AMQP AMQP, HTTPS Kafka Protocol
Security Per-device identity Shared access
policies
Through external
authentication services

28.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Data Connection :: Custom Solution
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Queue
Device registry
Data Connector

29.
.NET LEVEL UP
Custom Solution :: Connector
.NET CONFERENCE #1 IN UKRAINE KYIV 2019

30.
.NET LEVEL UP
Custom Solution :: Connector & Queue
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
Device
Registry
Queue
Connector

31.
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
Big Data Solutions :: Processing
.NET CONFERENCE #1 IN UKRAINE

32.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Processing :: Azure Stream Analytics
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• Easy to use
• Easy to scale
• Good integration with Azure Services
• Relatively cheap
• HIPAA, HITRUST, CSA STAR, ISO, GDRP
• Limited SAQL
• Vendor lock
• Possible data loss and crash on invalid data

33.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Processing :: Stream Analytics :: Repartitioning
.NET CONFERENCE #1 IN UKRAINE KYIV 2019

34.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
Processing :: HDInsight
.NET CONFERENCE #1 IN UKRAINE KYIV 2019

35.
Тема доклада
Тема доклада
Тема доклада
.NET LEVEL UP
HDInsight :: Pros & Cons
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• Not cost effective
• Software is outdated
• Software requires a lot of tuning
• Provides managed versions of 1st class frameworks
• Seamless integration
• Frameworks are open-source

36.
.NET LEVEL UP
Processing :: HDInsight :: Storm
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• Low latency. True streaming. High throughput
• Excellent for non-complicated use cases
• No state management
• “At least once” guarantee

37.
.NET LEVEL UP
Processing :: HDInsight :: Spark
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• High throughput. Complete fault tolerance
• Exactly once guarantees
• No true streaming (micro batches). High latency.
• Too complex to tune.

38.
.NET LEVEL UP
HDInsight :: Spark & Storm Benchmarking
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
• Demo JSON files { 100 users; 10 events per user }
• 1 Redis node. 5 Kafka nodes. 10 worker nodes.

39.
.NET LEVEL UP
HDInsight :: Spark Benchmarking
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
70 sec
0
Latency Percentiles (130 000 events/s; 2-second micro-batches)
50 sec
60 sec
40 sec
10050 90
30 sec
25
10 sec
20 sec

40.
.NET LEVEL UP
HDInsight :: Spark Benchmarking
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
16 sec
0
Latency Percentiles (100 000 events/s; 10 second micro-batches)
8 sec
12 sec
4 sec
10050 90

41.
.NET LEVEL UP
HDInsight :: Spark Benchmarking
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
5 sec
0
Latency Percentiles (100 000 events/s; 3-second micro-batches)
3 sec
4 sec
2 sec
10050 90
1 sec
25

42.
.NET LEVEL UP
HDInsight :: Storm Benchmarking
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
3 sec
0
Latency Percentiles (up to 150 000 events/s)
2 sec
2.5 sec
1.5 sec
10050 90
1 sec
25

43.
.NET LEVEL UP
Processing :: What to Choose?
.NET CONFERENCE #1 IN UKRAINE KYIV 2019
NOT Avro, JSON, or CSV
Scale beyond 1 Gb/s
Low latency is a must
Zero data loss

44.
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
One more thing…
.NET CONFERENCE #1 IN UKRAINE

45.
.NET LEVEL UP
Big Data Solutions :: Cross-Cutting Concerns
.NET CONFERENCE #1 IN UKRAINE KYIV 2019

46.
Тема доклада
Тема доклада
Тема доклада
KYIV 2019
.NET CONFERENCE #1 IN UKRAINE
Андрей Антиликаторов
Solution Architect