1. DATA SATURDAY #10
Sofia, Oct 09th
Deep Dive Time Series Anomaly
Detection with different Azure
data services
Marco Parenzan

2. Marco Parenzan
• Senion Solution Architect @ beanTech
• 1nn0va Community Lead (Pordenone)
• Microsoft Azure MVP
• Profiles
o Linkedin: https://www.linkedin.com/in/marcoparenzan/
o Slideshare: https://www.slideshare.net/marco.parenzan
o GitHub: https://github.com/marcoparenzan

3. This is the journey of…
• …a .NET developer…
• …or an IoT developer…
• …a one-man band (sometimes )…
• …facing typical data science world topics…
• …that wants to use .NET everywhere!

4. Thanks to our Sponsors

5. A typical scenario

6. Scenario
• In an industrial fridge, you monitor temperatures to check not the
temperature «per se», but to check the healthy of the plant
From real industrial fridges 

7. With no any specific request...
what is IoT all about?
Efficiency Anomalies
Batch Streaming

8. Storage
Account
IoT Hub
Devices
Events
Ingest
The batch point of view...

9. How can we implement
processing?
Ingest Process
Storage
Account
Azure
IoT Hub-Related
Services
Devices
Events
?
We explore some of them,
probably the most Microsoft and Azure oriented

10. But….

11. I’m not a data scientist!
Or a BI Analyst!

12. I’m a .NET Developer!

13. Make me think and work as a Data
Scientist!

14. A (very little) bit of theory
for Anomaly Detection
in Time Series

15. Threshold anomalies?
• Threshold alarms are not enough
o Anomalies cannot be just «over a threshold for
some time»...
o Condenser or Evaporator with difficulties starting
o Distinguish from Opening a door (that is also an
anomaly)
o Or also counting the number of times that there
are peaks (too many times)
• You can considering each of these
events as anomalies that alter the
temperature you measure in
different part of the fridge

16. Anomaly Detection
• Anomaly detection is the process of identifying unexpected items or events in
data sets, which differ from the norm.
• And anomaly detection is often applied on unlabeled data which is known as
unsupervised anomaly detection.
• Anomaly is not just a matter of time and scalar values. It can also be a matter
of visual anomalies!

17. Time Series
o Trend
• is a general direction in which something is
developing or changing. A trend can be
upward(uptrend) or downward(downtrend).
It is not always necessary that the increase or
decrease is consistently in the same direction
in a given period.
o Seasonality
• Predictable pattern that recurs or repeats
over regular intervals. Seasonality is often
observed within a year or less.
o Irregular fluctuation
• These are variations that occur due to
sudden causes and are unpredictable. For
example the rise in prices of food due to war,
flood, earthquakes, farmers striking etc.
• Definition
o Time series is a sequence of data points recorded
in time order, often taken at successive equally
paced points in time.
• Examples
o Stock prices, Sales demand, website traffic, daily
temperatures, quarterly sales
• Time series is different from
regression analysis because of its
time-dependent nature.

18. Anomaly Detection in Time Series
• In time series data, an anomaly or outlier can be termed as a data point
which is not following the common collective trend or seasonal or cyclic
pattern of the entire data and is significantly distinct from rest of the data. By
significant, most data scientists mean statistical significance, which in order
words, signify that the statistical properties of the data point is not in
alignment with the rest of the series.
• Anomaly detection has two basic assumptions:
o Anomalies only occur very rarely in the data.
o Their features differ from the normal instances significantly.

19. Classic ML Process

20. Helping no-data scientits developers (all! )
• Unsupervised Machine
LearningNo labelling
• Automated Training Set for
Anomaly Detection Algorithms
• the algorithms automatically
generates a simulated training set
based non your input data
• Auto(mated) MLfind the best
tuning for you with parameters
and algorithms
https://docs.microsoft.com/en-us/azure/machine-learning/algorithm-cheat-sheet

21. Spectrum Residual Cnn (SrCnn)
• To monitor the time-series continuously and alert for potential incidents on time
• The algorithm first computes the Fourier Transform of the original data. Then it computes
the spectral residual of the log amplitude of the transformed signal before applying the
Inverse Fourier Transform to map the sequence back from the frequency to the time domain.
This sequence is called the saliency map. The anomaly score is then computed as the relative
difference between the saliency map values and their moving averages. If the score is above
a threshold, the value at a specific timestep is flagged as an outlier.
• There are several parameters for SR algorithm. To obtain a model with good performance, we
suggest to tune windowSize and threshold at first, these are the most important parameters
to SR. Then you could search for an appropriate judgementWindowSize which is no larger
than windowSize. And for the remaining parameters, you could use the default value directly.
• Time-Series Anomaly Detection Service at Microsoft [https://arxiv.org/pdf/1906.03821.pdf]

22. Data Science for the .NET developer

23. Data Science and AI for the .NET developer
• ML.NET is first and foremost a framework that you can use to
create your own custom ML models. This custom approach
contrasts with “pre-built AI,” where you use pre-designed general
AI services from the cloud (like many of the offerings from Azure
Cognitive Services). This can work great for many scenarios, but
it might not always fit your specific business needs due to the
nature of the machine learning problem or to the deployment
context (cloud vs. on-premises).
• ML.NET enables developers to use their existing .NET skills to
easily integrate machine learning into almost any .NET
application. This means that if C# (or F# or VB) is your
programming language of choice, you no longer have to learn a
new programming language, like Python or R, in order to
develop your own ML models and infuse custom machine
learning into your .NET apps.

24. ML.NET Components
Anomaly Detection

25. Some tools required
• .NET 5 + WPF + ML.NET
• Mandatory , the platform where we try to make experiments
• Xplot.Ploty (soon you will understand I use this) https://fslab.org/XPlot/
• XPlot is a cross-platform data visualization package for the F# programming language
powered by popular JavaScript charting libraries Plotly and Google Charts. The library
provides a complete mapping for the configuration options of the underlying libraries and so
you get a nice F# interface that gives you access to the full power of Plotly and Google
Charts. The XPlot library can be used interactively from F# Interactive, but charts can equally
easy be embedded in F# applications and in HTML reports.
• WebView2 https://docs.microsoft.com/en-us/microsoft-edge/webview2/gettingstarted/wpf
• The Microsoft Edge WebView2 control enables you to embed web technologies (HTML, CSS,
and JavaScript) in your native apps. The WebView2 control uses Microsoft Edge (Chromium)
as the rendering engine to display the web content in native apps. With WebView2, you may
embed web code in different parts of your native app. Build all of the native app within a
single WebView instance.

26. Experimenting with .NET and WPF

27. How Data Scientists work

28. Batch vs. Notebooks
• Batch
o Work on slow data stored into a Datalake
o Submit a complete app in one single deploy
o Receive the entire output
• Notebook
o «sketching» the code
o Write/delete/rewrite continuously
o Run cell by cell (but also all at once) interactive
• In a world of Mathematica

29. Jupyter
• Evolution and generalization of the seminal role of Mathematica
• In web standards way
o Web (HTTP+Markdown)
o Python adoption (ipynb)
• Written in Java
• Python has an interop bridge...not native (if ever important)Python is a
kernel for Jupyter

30. Python!
• Simple to start (that why C# is pythonizing…)
• “Open Source”
• TensorFlow, Scikit-learn, Keras, Pandas, PyTorch
• Remember one thing:
o Often behind a Data Science framework there is a native library and Python binds that library
o Spark is written in Java and there is a bridge for Python to Spark
o Jupyter is written in Java and there is a bridge (kernel) for Python

31. Spark Unifies:
 Batch Processing
 Interactive SQL
 Real-time processing
 Machine Learning
 Deep Learning
 Graph Processing
An unified, open source, parallel, data processing framework for Big Data Analytics
Spark Core Engine
Spark SQL
Batch processing
Spark Structured
Streaming
Stream processing
Spark MLlib
Machine
Learning
Yarn
Spark MLlib
Machine
Learning
Spark
Streaming
Stream processing
GraphX
Graph
Computation
http://spark.apache.org
Apache Spark

32. .NET Interactive and Jupyter
and Visual Studio Code
• .NET Interactive gives C# and F# kernels to Jupyter
• .NET Interactive gives all tools to create your hosting application
independently from Jupyter
• In Visual Studio Code, you have two different notebooks (looking similar but
developed in parallel by different teams)
o .NET Interactive Notebook (by the .NET Interactive Team) that can run also Python
o Jupyter Notebook (by the Azure Data Studio Team – probably) that can run also C# and F#
• There is a little confusion on that 
• .NET Interactive has a strong C#/F# Kernel...
o ...a less mature infrastructure (compared to Jupiter)

33. .NET for Apache Spark 1.1.1
• .NET bindings (C# e F#) to Spark
o Written on the Spark interop layer, designed to
provide high performance bindings to multiple
languages
• Re-use knowledge, skills, code you
have as a .NET developer
o Compliant with .NET Standard
• You can use .NET for Apache
Spark anywhere you write .NET
code
• Original project Moebius
o https://github.com/microsoft/Mobius

34. Experimenting .NET Interactive

35. The Azure toolbox

36. .NET (5) hosting in Azure
Existing apps
.NET web apps (on-premises)
Cloud-Optimized
PaaS
Cloud-Native
PaaS for microservices and serverless
Monolithic / N-Tier
architectures
Monolithic / N-Tier
architectures
Microservices and serverless architectures
Cloud
Infrastructure-Ready
Monolithic / N-Tier
architectures
Relational
Database
VMs
Managed services
On-premises Azure
PaaS for containerized microservices
+ Serverless computing
+ Managed services
And Windows Containers
IaaS
(Infrastructure as a Service)
Azure Azure

37. Functions everywhere
Platform
App delivery
OS
On-premises
Code
App Service on Azure Stack
Windows
●●●
Non-Azure hosts
●●●
●●●
+
Azure Functions
host runtime
Azure Functions
Core Tools
Azure Functions
base Docker image
Azure Functions
.NET Docker image
Azure Functions
Node Docker image
●●●

38. Logic Apps
o Visually design workflows in the cloud
o Express logic through powerful control flow
o Connect disparate functions and APIs
o Utilize declarative definition to work with CI/CD

39. Experimenting with
Azure Functions

40. Anomaly Detection
As A Service in Azure

41. Azure Cognitive Services
• Cognitive Services brings AI within reach of every developer—without
requiring machine-learning expertise. All it takes is an API call to embed the
ability to see, hear, speak, search, understand, and accelerate decision-
making into your apps. Enable developers of all skill levels to easily add AI
capabilities to their apps.
• Five areas:
• Decision
• Language
• Speech
• Vision
• Web search
Anomaly Detector
Identify potential problems early on.
Content Moderator
Detect potentially offensive or unwanted
content.
Metrics Advisor PREVIEW
Monitor metrics and diagnose issues.
Personalizer
Create rich, personalized experiences for every
user.

42. Anomaly Detector
• Through an API, Anomaly Detector ingests time-series data of all types and
selects the best-fitting detection model for your data to ensure high accuracy.
Customize the service to detect any level of anomaly and deploy it where you
need it most -- from the cloud to the intelligent edge with containers. Azure
is the only major cloud provider that offers anomaly detection as an AI
service.

43. Experimenting
with Anomaly Detector

44. Anomaly Detection
with Big Data

45. Fully managed big data analytics service
• Fully managed
for efficiency
Focus on insights, not the
infra-structure for fast time to
value
• No infrastructure to manage;
provision the service, choose
the SKU for your workload,
and create database.
• Optimized for
streaming data
Get near-instant insights
from fast-flowing data
• Scale linearly up to 200 MB per
second per node with highly
performant, low latency
ingestion.
• Designed for
data exploration
• Run ad-hoc queries using the
intuitive query language
• Returns results from 1 Billion
records < 1 second without
modifying the data or
metadata

46. •seconds freshness, days retention
•in-mem aggregated data
•pre-defined standing queries
•split-seconds query performance
•data viewing
Hot
•minutes freshness, months retention
•raw data
•ad-hoc queries
•seconds-minutes query perf
•data exploration
Warm
•hours freshness, years retention
•raw data
•programmatic batch processing
•minutes-hours query perf
•data manipulation
Cold
• in-mem cube
• stream analytics
• …
• column store
• Indexing
• …
• distributed file
system
• map reduce
• …
Multi-temperature data processing paths

47. Experimenting with
Azure Data Explorer

48. Upcoming Events
jsTalks (Bulgaria), 2021
Nov 19-20
http://jstalks.net/
Azure MVP Unplugged, 5th Edition
October 21
Global AI Back Together Bulgaria 2021
19 of October

49. Thanks to our Sponsors