Deep dive time series anomaly detection with different Azure Data Services

Senior Solutions Architect @ beanTech, Microsoft MVP
16 de Jun de 2021
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
Deep dive time series anomaly detection with different Azure Data Services
1 de 50

Más contenido relacionado

La actualidad más candente

A Fast Decision Rule Engine for Anomaly DetectionA Fast Decision Rule Engine for Anomaly Detection
A Fast Decision Rule Engine for Anomaly DetectionDatabricks
FunctionalConf '16 Robert Virding Erlang EcosystemFunctionalConf '16 Robert Virding Erlang Ecosystem
FunctionalConf '16 Robert Virding Erlang EcosystemRobert Virding
Continuous Delivery Amsterdam - Microservices in action at the Dutch National...Continuous Delivery Amsterdam - Microservices in action at the Dutch National...
Continuous Delivery Amsterdam - Microservices in action at the Dutch National...Bert Jan Schrijver
J-Spring 2017 - Microservices in action at the Dutch National PoliceJ-Spring 2017 - Microservices in action at the Dutch National Police
J-Spring 2017 - Microservices in action at the Dutch National PoliceBert Jan Schrijver
AWS Loft Talk: Behind the Scenes with SignalFxAWS Loft Talk: Behind the Scenes with SignalFx
AWS Loft Talk: Behind the Scenes with SignalFxSignalFx
Solving Cyber at ScaleSolving Cyber at Scale
Solving Cyber at ScaleDataWorks Summit/Hadoop Summit

La actualidad más candente(20)

Similar a Deep dive time series anomaly detection with different Azure Data Services

Deep Dive Time Series Anomaly Detection in Azure with dotnetDeep Dive Time Series Anomaly Detection in Azure with dotnet
Deep Dive Time Series Anomaly Detection in Azure with dotnetMarco Parenzan
Time Series Anomaly Detection with .net and AzureTime Series Anomaly Detection with .net and Azure
Time Series Anomaly Detection with .net and AzureMarco Parenzan
Time Series Anomaly Detection for .net and AzureTime Series Anomaly Detection for .net and Azure
Time Series Anomaly Detection for .net and AzureMarco Parenzan
Consolidating MLOps at One of Europe’s Biggest AirportsConsolidating MLOps at One of Europe’s Biggest Airports
Consolidating MLOps at One of Europe’s Biggest AirportsDatabricks
 Machine Learning for Capacity Management Machine Learning for Capacity Management
Machine Learning for Capacity ManagementEDB
Streamlio and IoT analytics with Apache PulsarStreamlio and IoT analytics with Apache Pulsar
Streamlio and IoT analytics with Apache PulsarStreamlio

Similar a Deep dive time series anomaly detection with different Azure Data Services(20)

Más de Marco Parenzan

Azure IoT Central per lo SCADA engineerAzure IoT Central per lo SCADA engineer
Azure IoT Central per lo SCADA engineerMarco Parenzan
Azure Hybrid @ HomeAzure Hybrid @ Home
Azure Hybrid @ HomeMarco Parenzan
Static abstract members nelle interfacce di C# 11 e dintorni di .NET 7.pptxStatic abstract members nelle interfacce di C# 11 e dintorni di .NET 7.pptx
Static abstract members nelle interfacce di C# 11 e dintorni di .NET 7.pptxMarco Parenzan
Azure Synapse Analytics for your IoT SolutionsAzure Synapse Analytics for your IoT Solutions
Azure Synapse Analytics for your IoT SolutionsMarco Parenzan
Power BI Streaming Data Flow e Azure IoT Central Power BI Streaming Data Flow e Azure IoT Central
Power BI Streaming Data Flow e Azure IoT Central Marco Parenzan
Power BI Streaming Data Flow e Azure IoT CentralPower BI Streaming Data Flow e Azure IoT Central
Power BI Streaming Data Flow e Azure IoT CentralMarco Parenzan

Último

Empowering Advanced Users: Extending OutSystems UI Framework with Openness an...Empowering Advanced Users: Extending OutSystems UI Framework with Openness an...
Empowering Advanced Users: Extending OutSystems UI Framework with Openness an...Bernardo Cardoso
DevOps and SF.pdfDevOps and SF.pdf
DevOps and SF.pdfPatrickYANG48
Salesforce @AXA.pdfSalesforce @AXA.pdf
Salesforce @AXA.pdfPatrickYANG48
[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...
[DPE Summit] How Improving the Testing Experience Goes Beyond Quality: A Deve...Roberto Pérez Alcolea
Semantic Search_ NLP_ ML.pdfSemantic Search_ NLP_ ML.pdf
Semantic Search_ NLP_ ML.pdfPlamenaDzharadat
Taming Cloud Sprawl - XConf Europe 2023 - Kief.pdfTaming Cloud Sprawl - XConf Europe 2023 - Kief.pdf
Taming Cloud Sprawl - XConf Europe 2023 - Kief.pdfKief Morris

Deep dive time series anomaly detection with different Azure Data Services

Notas del editor

  1. https://towardsdatascience.com/effective-approaches-for-time-series-anomaly-detection-9485b40077f1 https://towardsdatascience.com/anomaly-detection-for-dummies-15f148e559c1
  2. https://towardsdatascience.com/time-series-analysis-for-beginners-8a200552e332
  3. Anomaly detection is the process of identifying unexpected items or events in data sets, which differ from the norm. And anomaly detection is often applied on unlabeled data which is known as unsupervised anomaly detection. https://towardsdatascience.com/effective-approaches-for-time-series-anomaly-detection-9485b40077f1
  4. SSA works by decomposing a time-series into a set of principal components. These components can be interpreted as the parts of a signal that correspond to trends, noise, seasonality, and many other factors. Then, these components are reconstructed and used to forecast values some time in the future.
  5. The Spectral Residual outlier detector is based on the paper Time-Series Anomaly Detection Service at Microsoft and is suitable for unsupervised online anomaly detection in univariate time series data. The algorithm first computes the Fourier Transform of the original data. Then it computes the spectral residual of the log amplitude of the transformed signal before applying the Inverse Fourier Transform to map the sequence back from the frequency to the time domain. This sequence is called the saliency map. The anomaly score is then computed as the relative difference between the saliency map values and their moving averages. If the score is above a threshold, the value at a specific timestep is flagged as an outlier. For more details, please check out the paper.
  6. What’s next? Modernize applications with .NET Core Today we focused on Cloud-optimized .NET Framework apps. However, many applications will benefit from modern architecture built on .NET Core – a much faster, modular, cross-platform, open source .NET. Websites can be modernized with ASP.NET Core to bring in better security, compliance, and much better performance than ASP.NET on .NET Framework. .NET Core also provides code patterns for building resilient, high-performance microservices on Linux and Windows.
  7. Build 2015
  8. What specific approach would you say is the most efficient way for moving flat file data from the ingest storage locations to the data lake? Follow the pattern of landing data in the data lake first, then ingest from the flat files into relational tables within the data warehouse. Then create pipelines that extract the source data and store in Azure Data Lake Store Gen2 as Parquet files. What storage service would you recommend to use? They should use Azure Data Lake Store (ADLS) Gen2 (Azure Storage with hierarchical file systems).
  9. How would you recommend to structure the folder to manage the data at the various levels of refinement? They should use Azure Data Lake Store (ADLS) Gen2 (Azure Storage with hierarchical file systems). In ADLS, it is a best practice to have a dedicated Storage Account for production, and a separate Storage Account for dev and test workloads. This will ensure that dev or test workloads never interfere with production. One common folder structure is to organize the data in separate folders by degree of refinement. For example a bronze folder contains the raw data, silver contains the cleaned, prepared and integrated data and gold contains data ready to support analytics, which might include final refinements such as pre-computed aggregates.
  10. Use external tables with Synapse SQL - Azure Synapse Analytics | Microsoft Docs