Wzrost ilości danych w postaci strumieni danych spowodował potrzebę analizy danych w czasie rzecyzwistych będących strumieniami. W czasie sesji pokazano połączenie: - event hub/Iot hub - Azure Stream Analytics - Azure Machine Learning

1. Azure Stream Analytics i Azure Machine
Learning w analizie strumieni danych
Łukasz Grala
Architect Data Platform & Advanced Analytics & BI Solutions
Data Platform MVP

2. Łukasz Grala
• Senior architekt rozwiązań Platformy Danych & Business Intelligence &
Zaawansowanej Analityki w TIDK
• Certyfikowany trener Microsoft i wykładowca na wyższych uczelniach
• Autor zaawansowanych szkoleń i warsztatów, oraz licznych publikacji i
webcastów
• Od 2010 roku wyróżniany nagrodą Microsoft Data Platform MVP
• Doktorant Politechnika Poznańska – Wydział Informatyki (obszar bazy danych,
eksploracja danych, uczenie maszynowe)
• Prelegent na licznych konferencjach w kraju i na świecie
• Posiada liczne certyfikaty (MCT, MCSE, MCSA, MCITP,…)
• Członek Polskiego Towarzystwa Informatycznego
• Członek i lider Polish SQL Server User Group (PLSSUG)
• Pasjonat analizy, przechowywania i przetwarzania danych, miłośnik Jazzu
Data Scientist
asaService
lukasz@tidk.pl

3. Data (Big Data)
• 72 hours of video are uploaded per minute on YouTube (1 terabyte
every 4 minutes)
• 500 terabytes of new data per day are ingested in Facebook
databases
• Sensors from a Boeing jet engine create 20 terabytes
of data every hour
• The proposed Square Kilometer Array telescope will generate “a few
Exabytes of data per day” (single beam)
lukasz@tidk.pl

4. The Four V’ of Big Data http://www.ibmbigdatahub.com/infographic/four-vs-big-data

5. Internet of Things (IoT)
lukasz@tidk.pl

6. Example Automotive
Connected Car - Generator of Big Data:
http://www.slideshare.net/AditiTechnologies/how-internet-of-things-iot-is-reshaping-the-automotive-sector-infographic

7. Example AutomotiveConnected Car - Use Cases http://gelookahead.economist.com/infograph/car-os/

8. New BI Solutions
ETL Tool
(SSIS, etc) EDW
(SQL Server, Teradata, etc)
Extract
Original Data
Load
Transformed
Data
Transform
BI Tools
Ingest (EL)
Original Data
Scale-out
Storage &
Compute
(HDFS, Blob Storage,
etc)
Transform & Load
Data Marts
Data Lake(s)
Dashboards
Apps
Streaming data
lukasz@tidk.pl

9. Dashboard
lukasz@tidk.pl

10. Spark Streaming

12. lukasz@tidk.pl
Azure Stream Analytics
Point of
Service Devices
Self Checkout
Stations
Kiosks
Smart
Phones
Slates/
Tablets
PCs/
Laptops
Servers
Digital
Signs
Diagnostic
EquipmentRemote Medical
Monitors
Logic
Controllers
Specialized
DevicesThin
Clients
Handhelds
Security
POS
Terminals
Automation
Devices
Vending
Machines
Kinect
ATM

13. lukasz@tidk.pl
SELECT count(*), Topic FROM Tweets
GROUP BY Topic, TumblingWindow(second, 5)
Let’s count tweets by topic…

14. Stream Analytics Query Language
Built-in
Functions
Data Types
Query
Language
Elements
Time
Management

15. Stream Analytics QL – Data Types
• Bigint
• Float
• Nvarchar(max)
• Datetime
• Record
• Array

16. Stream Analytics QL – Built-in Functions
Aggregate
•AVG
•COUNT
•CollectTop
•MAX
•MIN
•STDEV
•STDEVP
•SUM
•TopOne
•VAR
•VARP
Analytic
•ISFIRST
•LAG
•LAST
Record
•GetRecordProperties
•GetRecordPropertyValue
Scalar
•Conversion
•Data and Time
•Mathematical
•String

17. Stream Analytics QL – Scalar Functions
Conversion
• CAST
• TRY_CAST
• GetType
Data and Time
• DATENAME
• DATEPART
• DAY
• MONTH
• YEAR
• DATETIMEFROMPARTS
• DATEDIFF
• DATEADD
Mathematical
• ABS
• CEILING
• EXP
• FLOOR
• POWER
• SIGN
• SQUARE
• SQRT
String
• LEN
• CONCAT
• CHARINDEX
• LOWER
• SUBSTRING
• PATINDEX
• UPPER

18. Group By - Windowing
TUMBLING WINDOW
HOPPING WINDOW
SLIDING WINDOW

19. Group By – Windowing - Tumbling
SELECT count(*), Topic FROM Tweets BY EntryTime
GROUP BY Topic, TumblingWindow(second, 5)
TUMBLINGWINDOW ( timeunit , windowsize, [offsetsize] )
TUMBLINGWINDOW ( Duration( timeunit , windowsize ), [Offset(timeunit , offsetsize)] )

20. Group By – Windowing - Hoppingwindow
SELECT System.TimeStamp, Topic, COUNT(*)
FROM Tweets BY EntryTime
GROUP BY Topic, HoppingWindow(second, 10, 5)
HOPPINGWINDOW ( timeunit , windowsize , hopsize, [offsetsize] )
HOPPINGWINDOW ( Duration( timeunit , windowsize ) , Hop (timeunit , windowsize ), [Offset(timeunit , offsetsize)])

21. lukasz@tidk.pl
Canonical Event-driven Scenario

22. Key Concept – Machine Learning
Data
Model
Parameters
Learning Prediction
Decision Making
Utility Function
lukasz@tidk.pl

23. Class Learning Problems
• Classification: Assign a category to each item (Chinese | French
| Indian | Italian | Japanese restaurant).
• Regression: Predict a real value for each item
(stock/currency value, temperature).
• Ranking: Order items according to some criterion
(web search results relevant to a user query).
• Clustering: Partition items into homogeneous groups
(clustering twitter posts by topic).
• Dimensionality reduction: Transform an initial representation of items
into a lower-dimensional representation while preserving some
properties (preprocessing of digital images).
lukasz@tidk.pl

24. Steps to Build Machine Learning Solution
lukasz@tidk.pl

25. Azure Machine Learning
lukasz@tidk.pl

26. Azure Machine Learning
lukasz@tidk.pl

27. Azure Machine Learning & Stream Analytics

28. lukasz@tidk.pl
Question?
lukasz@tidk.pl

29. lukasz@tidk.pl
Thank You!
lukasz@tidk.pl

30. • Zapisz się na następne spotkanie
• http://www.meetup.com/PLSSUG/
• Przyjdź na najbliższą konferencję
• SQL Saturday Kraków – 1.10.2016
• Polub nas na FB
• https://www.facebook.com/PLSSUG
• https://www.facebook.com/SQLDay
• Wstąp w szeregi
• http://plssug.org.pl/
• Napisz lokalnym liderom, o czym będzie Twoja sesja
• Lublin@plssug.org.pl