RedisTimeSeries 1.2 by Pieter Cailliau - Redis Day Bangalore 2020

•Descargar como PPTX, PDF•

1 recomendación•73 vistas

Redis Labs

RedisTimeSeries 1.2 - Redis Day Bangalore Bengaluru 2020

Software

RedisTimeSeries 1.2
Pieter Cailliau
Product Manager, Redis Labs

PRESENTED
BY
What is RedisTimeSeries?
A time series is a series of data points indexed (or listed or graphed)
in time order. Most commonly, a time series is a sequence taken at
successive equally spaced points in time. Thus it is a sequence
of discrete-time data.
What you wrote, Wikipedia
*time series data is append-only
** Sample = <time, value>

PRESENTED
BY
Why RedisTimeSeries?
Modern applications produce more time-series data

PRESENTED
BY
● Compression added
○ Reduce memory up to 98%
○ Improves read performance
○ Based upon the Gorilla paper by Facebook
● Stable ingestion time
○ Independent of the number of the data points on a time-series
● Reviewed API
○ Performance improvements
○ Removed ambiguity
● Extended client support
RedisTimeSeries 1.2
Headlines

PRESENTED
BY
Compression
Timestamp - DoubleDelta

PRESENTED
BY
If ΔΔ is zero, then store a single ‘0’ bit
Else If ΔΔ is between [-63, 64], store ‘10’ followed by
the value (7 bits)
Else If ΔΔ is between [-512,511], store ‘110’ followed by
the value (10 bits)
Else if ΔΔ is between [-4096,4095], store ‘1110’ followed
by the value (13 bits)
Else if ΔΔ is between [-32768,32767], store ‘11110’ followed
by the value (16 bits)
Else store ‘11111’ followed by the value using 64 bits
Compression
Timestamp - DoubleDelta - variable-length encoding

PRESENTED
BY
If XOR is zero (same value)
store single ‘0’ bit
Compression
Value - XOR - variable-length encoding
Else
calculate the number of leading and trailing zeros in the XOR,
store bit ‘1’ followed by
If the block of meaningful bits falls within the block of previous
meaningful bits,
store control bit `0`
Else store control bit `1`,
store the length of the number of leading zeros in the next 5 bits,
store the length of the meaningful XOR value in the next 6 bits.
Finally store the meaningful bits of the XOR value.

PRESENTED
BY
Performance v1.0.3 vs v.1.2
Test Case
#Samples
(Millions)
30 days interval for 100 devices
x 10 metrics (cardinality 1K) 259.20
30 days interval for 1K devices x
10 metrics (cardinality 10K) 2,592.00
90 days interval for 100 devices
x 10 metrics (cardinality 1K) 777.60
%diff cardinality 1K vs 10K
No degradation
by compression
No degradation
by cardinality
Datasets and Ingestion overall throughput
v1.0.3 v1.2 % diff
354,812.17 363,562.25 2.47%
349,522.72 361,519.57 3.43%
352,025.35 343,665.92 -2.37%
-1.49% -0.56%

PRESENTED
BY
Performance v1.0.3 vs v.1.2
94%-95%

PRESENTED
BY
Performance v1.0.3 vs v.1.2
(simple queries) 15%-50% (complex queries)

PRESENTED
BY
Performance v1.0.3 vs v.1.2
(simple queries) 15%-70% (complex queries)

PRESENTED
BY
Stable data ingestion
Independent of the number of samples in the series
800m samples ingested
@ sub-millisecond latency
no degradation

PRESENTED
BY
Partners | Integrations | Ecosystem
Telegraf

How can I get started?
redistimeseries.io

Más contenido relacionado

La actualidad más candente

RedisConf18 - Implementing a New Data Structure for Redis

Redis Labs

Tales from Taming the Long Tail

HBaseCon

Imply at Apache Druid Meetup in London 1-15-20

Jelena Zanko

We recently learned about “Fault Tree Analysis” and decided to apply the technique to bulletproof our Apache Kafka deployments. In this talk, learn about fault tree analysis and what you should focus on to make your Apache Kafka clusters resilient. This talk should provide a framework for answers the following common questions a Kafka operator or user might have: -What guarantees can I promise my users? -What should my replication factor? -What should the ISR setting be? -Should I use RAID or not? -Should I use external storage such as EBS or local disks?

Bulletproof Kafka with Fault Tree Analysis (Andrey Falko, Lyft) Kafka Summit ...

confluent

I have a good shard key now what - Advanced Sharding

David Murphy

HBaseCon 2015: OpenTSDB and AsyncHBase Update

HBaseCon

ClickHouse Paris Meetup. Pragma Analytics Software Suite w/ClickHouse, by Mat...

Altinity Ltd

Real-Time Integration Between MongoDB and SQL Databases

MongoDB

Transitioning a legacy monolithic application to microservices is a daunting task by itself and it only gets more complicated as you start to dig through all the libraries and frameworks out there meant to help. In this talk, we'll cover the transition of a real Cassandra-based application to a microservices architecture using Grpc from Google and Falcor from Netflix. (Yes, Falcor is more than just a magical luck dragon from an awesome 80's movie.) We'll talk about why these technologies were a good fit for the project as well as why Cassandra is often a great choice once you go down the path of microservices. And since all the code for the project is open source, you'll have plenty to dig into afterwards. About the Speaker Luke Tillman Technical Evangelist, DataStax Luke is a Technical Evangelist for Apache Cassandra at DataStax. He's spent most of the last 15 years writing code for web applications built on relational databases both large and small. Most recently, before living the glamorous life of an Evangelist at DataStax, he worked as a software engineer at Hobsons on systems used by hundreds of colleges and universities across the U.S. and the World.

From Monolith to Microservices with Cassandra, Grpc, and Falcor (Luke Tillman...

DataStax

In this talk I'll discuss how we can combine the power of PostgreSQL with TensorFlow to perform data analysis. By using the pl/python3 procedural language we can integrate machine learning libraries such as TensorFlow with PostgreSQL, opening the door for powerful data analytics combining SQL with AI. Typical use-cases might involve regression analysis to find relationships in an existing dataset and to predict results based on new inputs, or to analyse time series data and extrapolate future data taking into account general trends and seasonal variability whilst ignoring noise. Python is an ideal language for building custom systems to do this kind of work as it gives us access to a rich ecosystem of libraries such as Pandas and Numpy, in addition to TensorFlow itself.

Data Analysis with TensorFlow in PostgreSQL

EDB

Recently, Apache Phoenix has been integrated with Apache (incubator) Omid transaction processing service, to provide ultra-high system throughput with ultra-low latency overhead. Phoenix has been shown to scale beyond 0.5M transactions per second with sub-5ms latency for short transactions on industry-standard hardware. On the other hand, Omid has been extended to support secondary indexes, multi-snapshot SQL queries, and massive-write transactions. These innovative features make Phoenix an excellent choice for translytics applications, which allow converged transaction processing and analytics. We share the story of building the next-gen data tier for advertising platforms at Verizon Media that exploits Phoenix and Omid to support multi-feed real-time ingestion and AI pipelines in one place, and discuss the lessons learned.

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

DataWorks Summit

Sharding Methods for MongoDB

MongoDB

HBaseCon2017 Highly-Available HBase

HBaseCon

Hitachi datasheet-universal-replicator

Hitachi Vantara

Infinite event streams are the core data abstraction in Apache Pulsar. Pulsar provides two-level reading APIs for accessing events in Pulsar topics, one is pub/sub and the other one is segment readers. The pub/sub API provides a unified messaging API for accessing events in a streaming way. People can choose different subscription modes for consuming events. The segment API provides a way to access events directly from Apache BookKeeper and tiered storage, which is more suitable for batch-oriented workloads. You can combine both pub/sub API and segment API to create a unified data processing experience as well. In the past year, we at StreamNative have been helping with many customers running Pulsar for different use cases from online queuing, event sourcing to stream and batch processing. We also worked on integrating Pulsar with different components in the big data ecosystem. In this talk, we will share our experiences and best practices of choosing the right API for accessing your event streams in Pulsar for different use cases.

Stream or segment : what is the best way to access your events in Pulsar_Neng

StreamNative

Our product presents an aggregated view of metadata collected for billions of objects (files, emails, sharepoint objects etc.). We used Cassandra to store those billions of objects along with aggregated view of that metadata. Customers can analyse the corpus of data in real time by searching in completely flexible way i.e. be able to get summary aggregates for many billions of objects, and then be able to further drill down to items by filtering using various facets of the metadata. We achieve this using a combination of Cassandra and ElasticSearch. This presentation will talk about various data modelling techniques we use to aggregate and then further summarise all that metadata and be able to search the summary in real t

Symantec: Cassandra Data Modelling techniques in action

DataStax Academy

Monitoring and scaling postgres at datadog

Seth Rosenblum

Lyft has built an internal platform to allow the backend to go from a single monolithic service to more than a hundred microservices. Several key components of the platform are available on GitHub. The talk will cover the following projects: - envoy: L7 edge and mesh proxy, enabling transparent networking for microservice architectures - discovery: track infrastructure topology in an eventually consistent manner - ratelimit: protect your systems from bad actors - confidant: store secrets encrypted at rest in DynamoDB

Open-source Infrastructure at Lyft

Daniel Hochman

Using ClickHouse for Experimentation

Gleb Kanterov

Data processing use cases, from transformation to analytics, perform tasks that require various combinations of queuing, streaming & lightweight processing steps. Until now, supporting all of those needs has required different systems for each task--stream processing engines, messaging queuing middleware, & streaming messaging systems. That has led to increased complexity for development & operations. In this session, well discuss the need to unify these capabilities in a single system & how Apache Pulsar was designed to address that. Apache Pulsar is a next generation distributed pub-sub system that was developed & deployed at Yahoo. Streamlios Karthik Ramasamy, will explain how the architecture & design of Pulsar provides the flexibility to support developers & applications needing any combination of queuing, messaging, streaming & lightweight compute.

Apache Pulsar Seattle - Meetup

Karthik Ramasamy

La actualidad más candente (20)

RedisConf18 - Implementing a New Data Structure for Redis

Tales from Taming the Long Tail

Imply at Apache Druid Meetup in London 1-15-20

Bulletproof Kafka with Fault Tree Analysis (Andrey Falko, Lyft) Kafka Summit ...

I have a good shard key now what - Advanced Sharding

HBaseCon 2015: OpenTSDB and AsyncHBase Update

ClickHouse Paris Meetup. Pragma Analytics Software Suite w/ClickHouse, by Mat...

Real-Time Integration Between MongoDB and SQL Databases

From Monolith to Microservices with Cassandra, Grpc, and Falcor (Luke Tillman...

Data Analysis with TensorFlow in PostgreSQL

Scaling Cloud-Scale Translytics Workloads with Omid and Phoenix

Sharding Methods for MongoDB

HBaseCon2017 Highly-Available HBase

Hitachi datasheet-universal-replicator

Stream or segment : what is the best way to access your events in Pulsar_Neng

Symantec: Cassandra Data Modelling techniques in action

Monitoring and scaling postgres at datadog

Open-source Infrastructure at Lyft

Using ClickHouse for Experimentation

Apache Pulsar Seattle - Meetup

Similar a RedisTimeSeries 1.2 by Pieter Cailliau - Redis Day Bangalore 2020

Cloud spanner architecture and use cases

GDG Cloud Bengaluru

Re-Engineering PostgreSQL as a Time-Series Database

All Things Open

Benchmarking Apache Druid

Imply

Benchmarking Apache Druid

Matt Sarrel

Blinkdb

Nitish Upreti

[Cassandra summit Tokyo, 2015] Cassandra 2015 最新情報 by ジョナサン・エリス(Jonathan Ellis)

datastaxjp

The earliest relational databases were monolithic on-premise systems that were powerful and full-featured. Fast forward to the Internet and NoSQL: BigTable, DynamoDB and Cassandra. These distributed systems were built to scale out for ballooning user bases and operations. As more and more companies vied to be the next Google, Amazon, or Facebook, they too "required" horizontal scalability. But in a real way, NoSQL and even NewSQL have forgotten single node performance where scaling out isn't an option. And single node performance is important because it allows you to do more with much less. With a smaller footprint and simpler stack, overhead decreases and your application can still scale. In this talk, we describe TimescaleDB's methods for single node performance. The nature of time-series workloads and how data is partitioned allows users to elastically scale up even on single machines, which provides operational ease and architectural simplicity, especially in cloud environments.

Databases Have Forgotten About Single Node Performance, A Wrongheaded Trade Off

Timescale

RedisConf17 - Doing More With Redis - Ofer Bengal and Yiftach Shoolman

Redis Labs

Deep Turnover Forecast. "It is very difficult to predict - especially the future." [Neils Bohr] At Decathlon we have developed a model to forecast turnover by store, department (or sport) and week for the next 52 weeks. This forecast is used by our department managers to pilot their activity. The model, inspired by DeepMind's WaveNet architecture, uses Deep Learning with a stack of several Dilated Causal Convolution layers.

Deep Turnover Forecast - meetup Lille

Carta Alfonso

Spanner (may 19)

Sultan Ahmed

A story about powering a 1.5 petabyte internal analytics application at Microsoft with 2816 cores and 18.7 TB of memory in the Citus cluster. The internal RQV analytics dashboard at Microsoft helps the Windows team to assess the quality of upcoming Windows releases. The system tracks 20,000 diagnostic and quality metrics, digests data from 800 million Windows devices and currently supports over 6 million queries per day, with hundreds of concurrent users. The RQV analytics dashboard relies on Postgres—along with the Citus extension to Postgres to scale out horizontally—and is deployed on Microsoft Azure.

Architecting peta-byte-scale analytics by scaling out Postgres on Azure with ...

Citus Data

The Ring programming language version 1.7 book - Part 92 of 196

Mahmoud Samir Fayed

The Ring programming language version 1.9 book - Part 100 of 210

Mahmoud Samir Fayed

Low latency microservices in java QCon New York 2016

Peter Lawrey

958 and 959 sales exam prep

Jason Wong

Flex Pod Solution

myazhome

EKON 23 Code_review_checklist

Max Kleiner

Deep learning is making news across the country as one of the most promising techniques in machine learning research. However, these methods are complex to implement, finicky to tune, and state-of-the-art accuracy is only achieved by a few experts in the field. In this session, we give a beginner-friendly explanation of deep learning using neural networks—what it is, what it does, and how; and introduce the concept of deep features, which allows you to obtain great performance with reduced running times and data set sizes. We then show how these methods can easily be deployed on GPU instances (G2) on Amazon EC2.

(CMP305) Deep Learning on AWS Made EasyCmp305

Amazon Web Services

Computação Paralela: Benefícios e Desafios - Intel Software Conference 2013

Intel Software Brasil

Effectiveness and code optimization in Java

Strannik_2013

Similar a RedisTimeSeries 1.2 by Pieter Cailliau - Redis Day Bangalore 2020 (20)

Cloud spanner architecture and use cases

Re-Engineering PostgreSQL as a Time-Series Database

Benchmarking Apache Druid

Blinkdb

[Cassandra summit Tokyo, 2015] Cassandra 2015 最新情報 by ジョナサン・エリス(Jonathan Ellis)

Databases Have Forgotten About Single Node Performance, A Wrongheaded Trade Off

RedisConf17 - Doing More With Redis - Ofer Bengal and Yiftach Shoolman

Deep Turnover Forecast - meetup Lille

Spanner (may 19)

Architecting peta-byte-scale analytics by scaling out Postgres on Azure with ...

The Ring programming language version 1.7 book - Part 92 of 196

The Ring programming language version 1.9 book - Part 100 of 210

Low latency microservices in java QCon New York 2016

958 and 959 sales exam prep

Flex Pod Solution

EKON 23 Code_review_checklist

(CMP305) Deep Learning on AWS Made EasyCmp305

Computação Paralela: Benefícios e Desafios - Intel Software Conference 2013

Effectiveness and code optimization in Java

Más de Redis Labs

Redis Day Bangalore 2020 - Session state caching with redis

Redis Labs

Protecting Your API with Redis by Jane Paek - Redis Day Seattle 2020

Redis Labs

The Happy Marriage of Redis and Protobuf by Scott Haines of Twilio - Redis Da...

Redis Labs

SQL, Redis and Kubernetes by Paul Stanton of Windocks - Redis Day Seattle 2020

Redis Labs

Rust and Redis - Solving Problems for Kubernetes by Ravi Jagannathan of VMwar...

Redis Labs

Redis for Data Science and Engineering by Dmitry Polyakovsky of Oracle

Redis Labs

Practical Use Cases for ACLs in Redis 6 by Jamie Scott - Redis Day Seattle 2020

Redis Labs

Moving Beyond Cache by Yiftach Shoolman Redis Labs - Redis Day Seattle 2020

Redis Labs

Leveraging Redis for System Monitoring by Adam McCormick of SBG - Redis Day S...

Redis Labs

JSON in Redis - When to use RedisJSON by Jay Won of Coupang - Redis Day Seatt...

Redis Labs

Highly Available Persistent Session Management Service by Mohamed Elmergawi o...

Redis Labs

Anatomy of a Redis Command by Madelyn Olson of Amazon Web Services - Redis Da...

Redis Labs

Building a Multi-dimensional Analytics Engine with RedisGraph by Matthew Goos...

Redis Labs

RediSearch 1.6 by Pieter Cailliau - Redis Day Bangalore 2020

Redis Labs

RedisGraph 2.0 by Pieter Cailliau - Redis Day Bangalore 2020

Redis Labs

RedisAI 0.9 by Sherin Thomas of Tensorwerk - Redis Day Bangalore 2020

Redis Labs

Rate-Limiting 30 Million requests by Vijay Lakshminarayanan and Girish Koundi...

Redis Labs

Three Pillars of Observability by Rajalakshmi Raji Srinivasan of Site24x7 Zoh...

Redis Labs

Solving Complex Scaling Problems by Prashant Kumar and Abhishek Jain of Myntr...

Redis Labs

Redis as a High Scale Swiss Army Knife by Rahul Dagar and Abhishek Gupta of G...

Redis Labs

Más de Redis Labs (20)