OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Khavronenko - VictoriaMetrics.pdf

•

1 recomendación•41 vistas

OSA Con 2022: Specifics of data analysis in Time Series Databases Roman Khavronenko - VictoriaMetrics Time series data is special. Not only its nature but also the ways that we store and interact with it. In this talk, we'll cover the differences between storing time series data in classic relational databases and a new generation of time series databases like VictoriaMetrics and Prometheus.

Datos y análisis

© 2022 VictoriaMetrics
Speciﬁcs of data analysis in
Time Series Databases

Co-founder of VictoriaMetrics
Software engineer with experience in distributed systems,
monitoring and high-performance services.
https://github.com/hagen1778
https://twitter.com/hagen1778
Roman Khavronenko

The High Performance
Open Source Time Series Database & Monitoring Solution
70M+ downloads
7.4k starts
180+ contributors
Grammarly, CERN,
StitchFix, Wix, Ably,
Brandwatch, Semrush

It is an observation!
up 1
blood_pressure_syst 120
blood_pressure_diast 80
How are you today?

What is time series data
up 1
blood_pressure_syst 120
blood_pressure_diast 80
Mon
up 1
blood_pressure_syst 135
blood_pressure_diast 82
Tue
up 1
blood_pressure_syst 140
blood_pressure_diast 90
Wed
up 1
blood_pressure_syst 180
blood_pressure_diast 99
Thu
up 0
blood_pressure_syst 0
blood_pressure_diast 0
Fri

Time
Series
Data
Mon, Tue, Wed…
120 => 135 => 140…
blood_pressure_syst 120
What is time series data

What kubernetes changed?
● Split application into microservices

What kubernetes changed?
● Split application into microservices
● Deploy multiple replicas of each microservice

What kubernetes changed?
● Split application into microservices
● Deploy multiple replicas of each microservice
● Each cluster, instance, pod, container exposes metrics

What is special about time series data?
● <value> is always associated with <timestamp>
● Series are continuous
● Time series data is unlikely to be changed in past

What is special about time series data?
<value> is always associated with <timestamp>
● Data can be structured in tuples of <timestamp:value> pairs
● Read API should expect <timestamp> ﬁlter
● <values> are always sorted by <timestamp>

What is special about time series data?
Series are continuous:
● Appends are more likely than registering of the new series
● Compression for sorted list of <timestamps> and <values> can be very efﬁcient
● Each data tuple is connected with previous one, so you plot lines not dots

What is special about time series data?
Time series data is unlikely to be changed in past:
● UPDATE/UPSERT/DELETE are not important
● Focus on ingestion and reading

TimeSeries specialized solution focuses on
● Providing max write throughput
● Prioritize writes over reads
● Use optimal data layout on disk
● Provide best compression based on data speciﬁcs
● Support optimal query language for time series

Prometheus Query Language
query language for selecting and aggregating
time series data in real time

CREATE TABLE
CREATE TABLE electricity (
datetime DateTime,
address String,
value Float64
) INSERT INTO electricity VALUES
('2022-10-01 00:00:00', 'Sheen Court 150', 0)
('2022-10-01 00:00:00', 'Richmond Hill 2', 0)
('2022-10-01 00:00:00', 'Belvedere Road 1', 0)
…

SQL=> $columns(address, sum(value) c) FROM electricity

Ephemeral data point always repeats the left closest real point

Counter reset detection
Point N
Point N+1
Point N+1 < Point N ?

Node Exporter: most popular Grafana dashboard

Node Exporter: most popular Grafana dashboard
Has 20 Mil downloads and 60 reviews!
Contains 230 PromQL queries in it:
● ~120 queries are selectors, e.g. node_textﬁle_scrape_total;
● ~80 queries calculate rate, e.g. rate(node_textﬁle_scrape_total);
● the rest are aggregation functions like sum or count.
It is easier than you think!

Time Series analysis speciﬁcs
● Queries need to be simple to write and understand
● It is natural to have time params in Query API
● Query step helps to align time series resolution
● Counter reset detection makes life easier

● Simpler to use query language MetricsQL
● More than 100 additional functions in MetricsQL
● Query API enhancements
● Automatic step and interval detection
● Anomaly detection
● And many more!
Looking for more? Check VictoriaMetrics!

Questions?
Links:
● https://github.com/VictoriaMetrics/VictoriaMetrics
● https://www.youtube.com/watch?v=oJ-RIkBWzAM
● The SRE Report 2023
● ESG Research Report
● VictoriaMetrics blog

Más contenido relacionado

Similar a OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Khavronenko - VictoriaMetrics.pdf

SQL is the lingua franca of data processing, and everybody working with data knows SQL. Apache Flink provides SQL support for querying and processing batch and streaming data. Flink's SQL support powers large-scale production systems at Alibaba, Huawei, and Uber. Based on Flink SQL, these companies have built systems for their internal users as well as publicly offered services for paying customers. In our talk, we will discuss why you should and how you can (not being Alibaba or Uber) leverage the simplicity and power of SQL on Flink. We will start exploring the use cases that Flink SQL was designed for and present real-world problems that it can solve. In particular, you will learn why unified batch and stream processing is important and what it means to run SQL queries on streams of data. After we explored why you should use Flink SQL, we will show how you can leverage its full potential. Since recently, the Flink community is working on a service that integrates a query interface, (external) table catalogs, and result serving functionality for static, appending, and updating result sets. We will discuss the design and feature set of this query service and how it can be used for exploratory batch and streaming queries, ETL pipelines, and live updating query results that serve applications, such as real-time dashboards. The talk concludes with a brief demo of a client running queries against the service. Speaker Timo Walther, Software Engineer, Data Artisans

Why and how to leverage the simplicity and power of SQL on Flink

DataWorks Summit

FINRA’s Data Lake unlocks the value in its data to accelerate analytics and machine learning at scale. FINRA's Technology group has changed its customer's relationship with data by creating a Managed Data Lake that enables discovery on Petabytes of capital markets data, while saving time and money over traditional analytics solutions. FINRA’s Managed Data Lake includes a centralized data catalog and separates storage from compute, allowing users to query from petabytes of data in seconds. Learn how FINRA uses Spot instances and services such as Amazon S3, Amazon EMR, Amazon Redshift, and AWS Lambda to provide the 'right tool for the right job' at each step in the data processing pipeline. All of this is done while meeting FINRA’s security and compliance responsibilities as a financial regulator.

FSI201 FINRA’s Managed Data Lake – Next Gen Analytics in the Cloud

Amazon Web Services

https://youtu.be/MRuCslYiQE4 (full meetup) https://go.dok.community/slack https://dok.community ABSTRACT OF THE TALK This talk is about how I use several tools, technologies and processes to troubleshoot ClicHouse Performance. I will be talking about multiple Linux Toolkits, Trace Profilers like DTrace, BPF etc. and also ClickHouse System Tables. This talk also covers best practices/checklist / run-book for building "High Performance ClickHouse Infrastructure Operations" BIO Open Source Database Systems Geek in MySQL, MariaDB, PostgreSQL and ClickHouse with core expertise in performance, scalability, high availability and database reliability engineering, Shiv currently is the founder and principal of MinervaDB Inc., an enterprise-class 24*7 Consultative Support and Managed Services Provider for MySQL, MariaDB and PostgreSQL. Shiv also is the Founder and Principal of ChistaDATA Inc., an independent 24*7 Consultative Support and Managed Service Provider for ClickHouse. Shiv in the past worked for companies like MySQL AB, SUN Microsystems, AOL, eBay, PayPal, PalominoDB and Percona. Shiv also is a frequent speaker in open source conferences worldwide.

Troubleshooting ClickHouse Performance

DoKC

Monitoring is the key to successful operation of any software service, but commercial solutions are complex, expensive, and slow. Let us show you how to build monitoring that is simple, cost-effective, and fast using open source stacks easily accessible to any developer. We’ll start with the elements of monitoring systems: data ingest, query engine, visualization, and alerting. We’ll then explain and contrast two implementation approaches. The first uses VictoriaMetrics, a fast growing, high performance time series database that uses PromQL for queries. The second is based on ClickHouse, a popular real-time analytics database that speaks SQL. Fast, affordable monitoring is within reach. This webinar provides designs and working code to get you there.

Application Monitoring using Open Source: VictoriaMetrics - ClickHouse

VictoriaMetrics

Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHouse Webinar Slides Monitoring is the key to the successful operation of any software service, but commercial solutions are complex, expensive, and slow. Let us show you how to build monitoring that is simple, cost-effective, and fast using open-source stacks easily accessible to any developer. We’ll start with the elements of monitoring systems: data ingest, query engine, visualization, and alerting. We’ll then explain and contrast two implementation approaches. The first uses VictoriaMetrics, a fast-growing, high-performance time series database that uses PromQL for queries. The second is based on ClickHouse, a popular real-time analytics database that speaks SQL. Fast, affordable monitoring is within reach. This webinar provides designs and working code to get you there. Presented by: Roman Khavronenko, Co-Founder at VictoriaMetrics Robert Hodges, CEO at Altinity

Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...

Altinity Ltd

Scaling up uber's real time data analytics

Xiang Fu

Timeseries - data visualization in Grafana

OCoderFest

Resilient Predictive Data Pipelines (QCon London 2016)

Sid Anand

Speakers: Chris Larsen (Limelight Networks) and Benoit Sigoure (Arista Networks) The OpenTSDB community continues to grow and with users looking to store massive amounts of time-series data in a scalable manner. In this talk, we will discuss a number of use cases and best practices around naming schemas and HBase configuration. We will also review OpenTSDB 2.0's new features, including the HTTP API, plugins, annotations, millisecond support, and metadata, as well as what's next in the roadmap.

OpenTSDB 2.0

HBaseCon

When your query execution is slow, a couple of questions arise. Where to look for resources utilization? What tools do you have to analyze CPU, hard drive and RAM bottlenecks? Could you do something to reduce query execution time? MariaDB's Patrick LeBlanc and Roman Nozdrin touch on both Columnstore's query execution introspection tools as well as operating system capabilities that everyone should know about. They go on to discuss a number of real life use cases too. Some called for configuration changes whilst others forced them to make serious changes in the code.

Performance tuning ColumnStore

MariaDB plc

In the French FedEx company we used Prometheus to monitor the infrastructure. It hosts a CQRS Architecture composed with Kafka, Spark, Cassandra, ElasticSearch, and microservices APIs in scala. This presentation is about using Prometheus in production, you will see why we choosed Prometheus, how we integrated it, configured it and what kind of insights we extracted from the whole infrastructure. In addition, you will see how Prometheus changed our way of working, how we implemented self-healing based on Prometheus, how we configured systemd to trigger AlertManager API, integration with slack and other cool stuffs.

Regain Control Thanks To Prometheus

Etienne Coutaud

SQL is the lingua franca of data processing and everybody working with data knows SQL. Apache Flink provides SQL support for querying and processing batch and streaming data. Flink’s SQL support powers large-scale production systems at Alibaba, Huawei, and Uber. Based on Flink SQL, these companies have built systems for their internal users as well as publicly offered services for paying customers. In our talk, we will discuss why you should and how you can (not being Alibaba or Uber) leverage the simplicity and power of SQL on Flink. We will start exploring the use cases that Flink SQL was designed for and present real-world problems that it can solve. In particular, you will learn why unified batch and stream processing is important and what it means to run SQL queries on streams of data. After we explored why you should use Flink SQL, we will show how you can leverage its full potential. Since recently, the Flink community is working on a service that integrates a query interface, (external) table catalogs, and result serving functionality for static, appending, and updating result sets. We will discuss the design and feature set of this query service and how it can be used for exploratory batch and streaming queries, ETL pipelines, and live updating query results that serve applications, such as real-time dashboards. The talk concludes with a brief demo of a client running queries against the service.

Flink Forward San Francisco 2018: Fabian Hueske & Timo Walther - "Why and how...

Flink Forward

Cloud Native Data Pipelines (in Eng & Japanese) - QCon Tokyo

Sid Anand

In this presentation, we are going to talk about the state of the art infrastructure we have established at Walmart Labs for the Search product using Spark Streaming and DataFrames. First, we have been able to successfully use multiple micro batch spark streaming pipelines to update and process information like product availability, pick up today etc. along with updating our product catalog information in our search index to up to 10,000 kafka events per sec in near real-time. Earlier, all the product catalog changes in the index had a 24 hour delay, using Spark Streaming we have made it possible to see these changes in near real-time. This addition has provided a great boost to the business by giving the end-costumers instant access to features likes availability of a product, store pick up, etc. Second, we have built a scalable anomaly detection framework purely using Spark Data Frames that is being used by our data pipelines to detect abnormality in search data. Anomaly detection is an important problem not only in the search domain but also many domains such as performance monitoring, fraud detection, etc. During this, we realized that not only are Spark DataFrames able to process information faster but also are more flexible to work with. One could write hive like queries, pig like code, UDFs, UDAFs, python like code etc. all at the same place very easily and can build DataFrame template which can be used and reused by multiple teams effectively. We believe that if implemented correctly Spark Data Frames can potentially replace hive/pig in big data space and have the potential of becoming unified data language. We conclude that Spark Streaming and Data Frames are the key to processing extremely large streams of data in real-time with ease of use.

Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...

Spark Summit

Tips Tricks and Little known features in SAP ASE

SAP Technology

SnappyData at Spark Summit 2017

Jags Ramnarayan

Apache Spark 2.0 offers many enhancements that make continuous analytics quite simple. In this talk, we will discuss many other things that you can do with your Apache Spark cluster. We explain how a deep integration of Apache Spark 2.0 and in-memory databases can bring you the best of both worlds! In particular, we discuss how to manage mutable data in Apache Spark, run consistent transactions at the same speed as state-the-art in-memory grids, build and use indexes for point lookups, and run 100x more analytics queries at in-memory speeds. No need to bridge multiple products or manage, tune multiple clusters. We explain how one can take regulation Apache Spark SQL OLAP workloads and speed them up by up to 20x using optimizations in SnappyData. We then walk through several use-case examples, including IoT scenarios, where one has to ingest streams from many sources, cleanse it, manage the deluge by pre-aggregating and tracking metrics per minute, store all recent data in a in-memory store along with history in a data lake and permit interactive analytic queries at this constantly growing data. Rather than stitching together multiple clusters as proposed in Lambda, we walk through a design where everything is achieved in a single, horizontally scalable Apache Spark 2.0 cluster. A design that is simpler, a lot more efficient, and let’s you do everything from Machine Learning and Data Science to Transactions and Visual Analytics all in one single cluster.

SnappyData, the Spark Database. A unified cluster for streaming, transactions...

SnappyData

Update on OpenTSDB and AsyncHBase

HBaseCon

Maximizing Database Tuning in SAP SQL Anywhere

SAP Technology

Sql azure cluster dashboard public.ppt

Qingsong Yao

Similar a OSA Con 2022 - Specifics of data analysis in Time Series Databases - Roman Khavronenko - VictoriaMetrics.pdf (20)

Why and how to leverage the simplicity and power of SQL on Flink

FSI201 FINRA’s Managed Data Lake – Next Gen Analytics in the Cloud

Troubleshooting ClickHouse Performance

Application Monitoring using Open Source: VictoriaMetrics - ClickHouse

Application Monitoring using Open Source - VictoriaMetrics & Altinity ClickHo...

Scaling up uber's real time data analytics

Timeseries - data visualization in Grafana

Resilient Predictive Data Pipelines (QCon London 2016)

OpenTSDB 2.0

Performance tuning ColumnStore

Regain Control Thanks To Prometheus

Flink Forward San Francisco 2018: Fabian Hueske & Timo Walther - "Why and how...

Cloud Native Data Pipelines (in Eng & Japanese) - QCon Tokyo

Learnings Using Spark Streaming and DataFrames for Walmart Search: Spark Summ...

Tips Tricks and Little known features in SAP ASE

SnappyData at Spark Summit 2017

SnappyData, the Spark Database. A unified cluster for streaming, transactions...

Update on OpenTSDB and AsyncHBase

Maximizing Database Tuning in SAP SQL Anywhere

Sql azure cluster dashboard public.ppt

Más de Altinity Ltd

Building an Analytic Extension to MySQL with ClickHouse and Open Source In this webinar Percona and Altinity offer suggestions and tips on how to recognize when MySQL is overburdened with analytics and can benefit from ClickHouse’s unique capabilities. Also, they will walk you through important patterns for integrating MySQL and ClickHouse which will enable the building of powerful and cost-efficient applications that leverage the strengths of both databases.

Building an Analytic Extension to MySQL with ClickHouse and Open Source.pptx

Altinity Ltd

Over the last few years Kubernetes has transitioned from an object of curiosity and fear to a robust platform for big data. Watch this webinar and you will learn how the Altinity Kubernetes Operator for ClickHouse enables users to run high performance analytics on ClickHouse. You will see a simple installation and teach you how to scale it into a cluster that can analyze 100s of terabytes of data. Along the way we’ll share our lessons for ClickHouse on Kubernetes in Altinity.Cloud. We built it on Kubernetes using the Altinity Operator and now run hundreds of clusters in the cloud. You can too!

Cloud Native ClickHouse at Scale--Using the Altinity Kubernetes Operator-2022...

Altinity Ltd

This is a joint webinar Percona - Altinity. In this webinar we will discuss suggestions and tips on how to recognize when MySQL is overburdened with analytics and can benefit from ClickHouse’s unique capabilities. We will then walk through important patterns for integrating MySQL and ClickHouse which will enable the building of powerful and cost-efficient applications that leverage the strengths of both databases.

Building an Analytic Extension to MySQL with ClickHouse and Open Source

Altinity Ltd

Fun with ClickHouse Window Functions | Altinity Webinar Window functions have arrived in ClickHouse! Our webinar will start with an introduction to standard window function syntax and show how it is implemented in ClickHouse. We’ll next show you problems that you can now solve easily using window functions. Finally, we’ll compare window functions to arrays, another powerful ClickHouse feature. There will be time for questions with our SQL experts. Join us for a complete overview of this long-awaited feature! Speakers: Robert Hodges, CEO @Altinity Vitaliy Zakaznikov, QA Manager and Architect @Altinity

Fun with ClickHouse Window Functions-2021-08-19.pdf

Altinity Ltd

Cloud Native Data Warehouses: A Gentle Introduction to Running ClickHouse on Kubernetes | Altinity Webinar Kubernetes is a powerful platform for big data and is particularly well-suited for ClickHouse. If you have been wondering about trying Kubernetes, this webinar is for you. The first half introduces Kubernetes basics, building up to operators, which manage cloud-native applications. The second half focuses on ClickHouse and shows how to deploy data warehouses using the ClickHouse Operator. You’ll learn everything you need to start grappling with big data on Kubernetes. Speaker: Robert Hodges, CEO @Altinity

Cloud Native Data Warehouses - Intro to ClickHouse on Kubernetes-2021-07.pdf

Altinity Ltd

Altinity Stable Builds offer a ClickHouse distribution that is ready for production use and with 3 years of maintenance. Our webinar introduces the special features of Stable Builds and describes how we build them from ClickHouse Long-Term Support (LTS) releases. We’ll show you how to find them and install them yourself, then guide you through the important topic of upgrading. We’ll also walk through how to use Altinity Stable Builds in Altinity.Cloud, our managed ClickHouse platform for high-performance analytics.

Building High Performance Apps with Altinity Stable Builds for ClickHouse | A...

Altinity Ltd

Altinity.Cloud is a managed ClickHouse platform for high-performance analytics. But what if you want to run ClickHouse in your own cloud account? Altinity.Cloud Anywhere does exactly that. In this webinar, we’ll explain how Altinity.Cloud Anywhere works, then walk through the simple setup procedure to get full cloud management of ClickHouse clusters in your VPCs. This webinar teaches you how to have cloud management for your real-time analytic stack while meeting requirements for compliance, control of data, and freedom from lock-in. Have your cake and eat it too!

Own your ClickHouse data with Altinity.Cloud Anywhere-2023-01-17.pdf

Altinity Ltd

ClickHouse ReplacingMergeTree in Telecom Apps

Altinity Ltd

Adventures with the ClickHouse ReplacingMergeTree Engine

Altinity Ltd

Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot While the demands for real-time analytics are growing in leaps and bounds, the analytics software must rely on streaming platforms for ingesting high volumes of data that's traveling in lightning speed down the pipeline. We will take a look at 2 powerful open source Apache platforms: Pulsar and Pinot, that work hand-in-hand together to deliver the analytical results which bring great value to your systems. Presenters: Mary Grygleski - Streaming Developer Advocate & Mark Needham - Developer Relations Engineer at StarTree Note: This webinar will be recorded and later posted on our Webinar page (https://altinity.com/webinarspage/) or Altinity official Youtube channel (https://www.youtube.com/@Altinity).

Building a Real-Time Analytics Application with Apache Pulsar and Apache Pinot

Altinity Ltd

Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data - Presentation Slides Altinity.Cloud is a fully automated cloud service for ClickHouse that is optimized for real-time analytics. In this webinar, we’ll explain how Altinity.Cloud works, then show how to set up your first ClickHouse cluster. We’ll then tour important features like scale-up, scale-out, uptime schedules, and DBA tools to analyze your tables. You’ll learn everything necessary to start working on real-time analytics today. Bring your questions! Presenters: Robert Hodges & Alexander Zaitsev Note: This webinar will be recorded and later posted on our Webinar page (https://altinity.com/webinarspage/) or Altinity official Youtube channel (https://www.youtube.com/@Altinity).

Altinity Webinar: Introduction to Altinity.Cloud-Platform for Real-Time Data.pdf

Altinity Ltd

OSA Con 2022 - What Data Engineering Can Learn from Frontend Engineering - Pe...

Altinity Ltd

OSA Con 2022 - Welcome to OSA CON Version 2022 - Robert Hodges - Altinity.pdf

Altinity Ltd

OSA Con 2022: Using ClickHouse Database to Power Analytics and Customer Engagement Platform Prafulla Gupta - Times Internet This talk covers how we empowered Product Managers and Editors at Times Internet by developing an in-house product, GrowthRx, using Clickhouse Open Source Database to track and analyze user behavior to increase user retention and customer engagement. Times Internet is India's largest digital news publisher, which manages leading brands like Times of India, Economic Times, Navbharat Times, etc, where we are tracking more than 10 billion events per month in the ClickHouse Database.

OSA Con 2022 - Using ClickHouse Database to Power Analytics and Customer Enga...

Altinity Ltd

OSA Con 2022 - Tips and Tricks to Keep Your Queries under 100ms with ClickHou...

Altinity Ltd

OSA Con 2022 - The Open Source Analytic Universe, Version 2022 - Robert Hodge...

Altinity Ltd

OSA Con 2022: Switching Jaeger Distributed Tracing to ClickHouse to Enable Advanced Performance Management Satbir Chahal - OpsVerse Our team switched our Jaeger (open source project used for distributed tracing) storage backend to ClickHouse (from Cassandra), which opened the door to a world of advanced analytics that we can run and provide our users. This talk will describe the journey from the switch, the learning curve, the challenges, and the eventual wins.

OSA Con 2022 - Switching Jaeger Distributed Tracing to ClickHouse to Enable A...

Altinity Ltd

OSA Con 2022 - Streaming Data Made Easy - Tim Spann & David Kjerrumgaard - St...

Altinity Ltd

OSA Con 2022 - State of Open Source Databases Peter Zaitsev - Percona It has been an exciting year in the open-source database industry, with more choices, more cloud, and key changes in the industry. We will dive into the key developments over 2022, including the most important open-source database software releases in general, the significance of cloud-native solutions in a multi-vendor multi-cloud world, the new criticality of security challenges, and the evolution of the open-source software industry.

OSA Con 2022 - State of Open Source Databases - Peter Zaitsev - Percona.pdf

Altinity Ltd

OSA Con 2022 - Signal Correlation, the Ho11y Grail - Michael Hausenblas - AWS...

Altinity Ltd

Más de Altinity Ltd (20)