SlideShare una empresa de Scribd logo
1 de 14
Confidential
Time Series Data Management Evolution at Capital
One
Building a highly resilient Enterprise Influx across multiple regions
Monitoring Intelligence
October 23, 2018
Confidential
Agenda
2
1. Influx at Capital One
2. Architecture
How Influx Architecture evolved
3. Resiliency
Complete protection against entire region failure
4. Performance Metrics
Critical Platform Metrics
5. Q & A
Confidential
Influx at Capital One
3
• Business Transaction Metrics
• Infrastructure Health metrics
• Application Performance Metrics
• Service Adoption Metrics
Confidential
Agenda
4
1. Influx at Capital One
2. Architecture
How Influx Architecture evolved
3. Resiliency
Complete protection against entire region failure
4. Performance Metrics
Critical Platform Metrics
5. Q & A
Confidential
Architecture – Gen1
5
InfluxDB
( Active )
InfluxDB
(Standby)
DR Site
Visualization
https
Primary Site
LB
Backup/Restore
Features
1. Grafana for visualization
Challenges
1. High Data Retention ( > 400 days )
2. Unstable DR Solution
Splunk
Direct API
Telegraf
Confidential
Architecture – Gen2
6
InfluxDB
( Active )
InfluxDB
(Standby)
DR Site
Visualization
Primary Site
LB
AWS S3
Data Lake
ML Model
Execution
Model
Governance
ML Model
Dev/Train
Backup/Restore
Daily batch
Features
1. Grafana for visualization
2. Raw Data Exported Daily to One Lake
3. Raw Data available for ML
Challenges
High Data Retention ( > 400 days )
2. Unstable DR Solution
A
B. ReviewC. Deploy
E
D
Splunk
Direct API
Telegraf
https
Confidential
Architecture – Current
7
InfluxDB
( Active )
InfluxDB
(Passive)
DR Site
Visualization
Primary Site
LB
AWS S3
Data Lake
ML Model
Execution
Model
Governance
ML Model
Dev/Train
AWS S3
ExportImport Mini batch
Features
1. Grafana for visualization
2. Raw Data Exported every 30 minutes
to One Lake
3. Raw Data available for ML
4. Stable DR Solution with Passive
Cluster
Challenges
High Data Retention ( > 400 days )
Unstable DR Solution
A
B. ReviewC. Deploy
E
D
Splunk
Direct API
Telegraf
https
Confidential
Agenda
8
1. Influx at Capital One
2. Architecture
How Influx Architecture evolved
3. Resiliency
Complete protection against entire region failure
4. Performance Metrics
Critical Platform Metrics
5. Q & A
Confidential
Resiliency – Region 1 Active
9
https
D
M
D
M
D
M
Region 1 ( Active)
Zone 1 Zone 2 Zone 3
LB
S3
Cluster 1
D
M
D
M
D
M
Region 2
(DR)
Zone 1 Zone 2 Zone 3
LB
S3
Cluster 2
Route53
(DNS Switch)
Cross Region
Replication
Admin
A
ASGASG
M Meta Node D Data Node A Admin
Node
3
A
Route53
(DNS Switch)
1
4
2
Splunk
Direct API
Telegraf
All Traffic routed to Region 11
Influx Export Script every 15 min2
Data Replicated to Region 23
Influx Import Script every 15 min4
Confidential
Resiliency – Region 2 Active
10
https
D
M
D
M
D
M
Region 1 ( DR)
Zone 1 Zone 2 Zone 3
LB
S3
Cluster 1
D
M
D
M
D
M
Region 2 (Active)
Zone 1 Zone 2 Zone 3
LB
S3
Cluster 2
Route53
(DNS Switch)
Cross Region
Replication
Admin
A
ASGASG
M Meta Node D Data Node A Admin
Node
3
A
Route53
(DNS Switch)
1
2
4
Splunk
Direct API
Telegraf
All Traffic routed to Region 21
Influx Export Script every 15 min2
Data Replicated to Region 13
Influx Import Script every 15 min4
Confidential
Resiliency – Export/Import
11
Export*
Import*
* High level pseudo code , Not the actual code
Confidential
Agenda
12
1. Influx at Capital One
2. Architecture
How Influx Architecture evolved
3. Resiliency
Complete protection against entire region failure
4. Performance Metrics
Critical Platform Metrics
5. Q & A
Confidential
Performance Metrics
Performance Metrics collected using Telegraf
Confidential
Q & A

Más contenido relacionado

La actualidad más candente

Scalable deployment options in WSO2 API Manager
Scalable deployment options in WSO2 API ManagerScalable deployment options in WSO2 API Manager
Scalable deployment options in WSO2 API Manager
WSO2
 
Storage Architectures And Options
Storage Architectures And OptionsStorage Architectures And Options
Storage Architectures And Options
Alan McSweeney
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology Comparison
Kai Wähner
 
Tupperware: Containerized Deployment at FB
Tupperware: Containerized Deployment at FBTupperware: Containerized Deployment at FB
Tupperware: Containerized Deployment at FB
Docker, Inc.
 
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
Lucas Jellema
 
How Cisco Provides World-Class Technology Conference Experiences Using Automa...
How Cisco Provides World-Class Technology Conference Experiences Using Automa...How Cisco Provides World-Class Technology Conference Experiences Using Automa...
How Cisco Provides World-Class Technology Conference Experiences Using Automa...
InfluxData
 

La actualidad más candente (20)

How to Monitor DOCSIS Devices Using SNMP, InfluxDB, and Telegraf
How to Monitor DOCSIS Devices Using SNMP, InfluxDB, and TelegrafHow to Monitor DOCSIS Devices Using SNMP, InfluxDB, and Telegraf
How to Monitor DOCSIS Devices Using SNMP, InfluxDB, and Telegraf
 
How to Manage Your Time Series Data Pipeline at the Edge with InfluxDB
How to Manage Your Time Series Data Pipeline at the Edge with InfluxDBHow to Manage Your Time Series Data Pipeline at the Edge with InfluxDB
How to Manage Your Time Series Data Pipeline at the Edge with InfluxDB
 
Scalable deployment options in WSO2 API Manager
Scalable deployment options in WSO2 API ManagerScalable deployment options in WSO2 API Manager
Scalable deployment options in WSO2 API Manager
 
Introducing JIRA Service Desk
Introducing JIRA Service DeskIntroducing JIRA Service Desk
Introducing JIRA Service Desk
 
Monoliths, Migrations, and Microservices
Monoliths, Migrations, and MicroservicesMonoliths, Migrations, and Microservices
Monoliths, Migrations, and Microservices
 
Sid integration view
Sid integration viewSid integration view
Sid integration view
 
Storage Architectures And Options
Storage Architectures And OptionsStorage Architectures And Options
Storage Architectures And Options
 
Kappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology ComparisonKappa vs Lambda Architectures and Technology Comparison
Kappa vs Lambda Architectures and Technology Comparison
 
Tupperware: Containerized Deployment at FB
Tupperware: Containerized Deployment at FBTupperware: Containerized Deployment at FB
Tupperware: Containerized Deployment at FB
 
Systematic Migration of Monolith to Microservices
Systematic Migration of Monolith to MicroservicesSystematic Migration of Monolith to Microservices
Systematic Migration of Monolith to Microservices
 
Toi uu hoa he thong 30 trieu nguoi dung
Toi uu hoa he thong 30 trieu nguoi dungToi uu hoa he thong 30 trieu nguoi dung
Toi uu hoa he thong 30 trieu nguoi dung
 
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
Microservices, Apache Kafka, Node, Dapr and more - Part Two (Fontys Hogeschoo...
 
Digital Operations Excellence - PagerDuty Summit Series Keynote
Digital Operations Excellence - PagerDuty Summit Series KeynoteDigital Operations Excellence - PagerDuty Summit Series Keynote
Digital Operations Excellence - PagerDuty Summit Series Keynote
 
Best Practices: How to Analyze IoT Sensor Data with InfluxDB
Best Practices: How to Analyze IoT Sensor Data with InfluxDBBest Practices: How to Analyze IoT Sensor Data with InfluxDB
Best Practices: How to Analyze IoT Sensor Data with InfluxDB
 
How Cisco Provides World-Class Technology Conference Experiences Using Automa...
How Cisco Provides World-Class Technology Conference Experiences Using Automa...How Cisco Provides World-Class Technology Conference Experiences Using Automa...
How Cisco Provides World-Class Technology Conference Experiences Using Automa...
 
Whoops, The Numbers Are Wrong! Scaling Data Quality @ Netflix
Whoops, The Numbers Are Wrong! Scaling Data Quality @ NetflixWhoops, The Numbers Are Wrong! Scaling Data Quality @ Netflix
Whoops, The Numbers Are Wrong! Scaling Data Quality @ Netflix
 
Best Practices for Streaming IoT Data with MQTT and Apache Kafka®
Best Practices for Streaming IoT Data with MQTT and Apache Kafka®Best Practices for Streaming IoT Data with MQTT and Apache Kafka®
Best Practices for Streaming IoT Data with MQTT and Apache Kafka®
 
Data streaming fundamentals
Data streaming fundamentalsData streaming fundamentals
Data streaming fundamentals
 
Building modern data lakes
Building modern data lakes Building modern data lakes
Building modern data lakes
 
Platform Strategy to Deliver Digital Experiences on Azure
Platform Strategy to Deliver Digital Experiences on AzurePlatform Strategy to Deliver Digital Experiences on Azure
Platform Strategy to Deliver Digital Experiences on Azure
 

Similar a Why Architecting for Disaster Recovery is Important for Your Time Series Data by Capital One

Similar a Why Architecting for Disaster Recovery is Important for Your Time Series Data by Capital One (20)

Flink at netflix paypal speaker series
Flink at netflix   paypal speaker seriesFlink at netflix   paypal speaker series
Flink at netflix paypal speaker series
 
ScyllaDB V Developer Deep Dive Series: Resiliency and Strong Consistency via ...
ScyllaDB V Developer Deep Dive Series: Resiliency and Strong Consistency via ...ScyllaDB V Developer Deep Dive Series: Resiliency and Strong Consistency via ...
ScyllaDB V Developer Deep Dive Series: Resiliency and Strong Consistency via ...
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to Kubernetes
 
Patterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to KubernetesPatterns and Pains of Migrating Legacy Applications to Kubernetes
Patterns and Pains of Migrating Legacy Applications to Kubernetes
 
Meetup Microservices Commandments
Meetup Microservices CommandmentsMeetup Microservices Commandments
Meetup Microservices Commandments
 
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scaleVelocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
Velocity San Jose 2017: Traffic shifts: Avoiding disasters at scale
 
Architectural Commandments for Building & Running Microservices at Scale
Architectural Commandments for Building & Running Microservices at ScaleArchitectural Commandments for Building & Running Microservices at Scale
Architectural Commandments for Building & Running Microservices at Scale
 
cncf overview and building edge computing using kubernetes
cncf overview and building edge computing using kubernetescncf overview and building edge computing using kubernetes
cncf overview and building edge computing using kubernetes
 
App Modernization with Aerospike as an Intraday System of Record_Venkat Thamm...
App Modernization with Aerospike as an Intraday System of Record_Venkat Thamm...App Modernization with Aerospike as an Intraday System of Record_Venkat Thamm...
App Modernization with Aerospike as an Intraday System of Record_Venkat Thamm...
 
Flink forward-2017-netflix keystones-paas
Flink forward-2017-netflix keystones-paasFlink forward-2017-netflix keystones-paas
Flink forward-2017-netflix keystones-paas
 
Strata Conference + Hadoop World NY 2016: Lessons learned building a scalable...
Strata Conference + Hadoop World NY 2016: Lessons learned building a scalable...Strata Conference + Hadoop World NY 2016: Lessons learned building a scalable...
Strata Conference + Hadoop World NY 2016: Lessons learned building a scalable...
 
Agile integration cloud native developement
Agile integration   cloud native developementAgile integration   cloud native developement
Agile integration cloud native developement
 
QCON 2015: Gearpump, Realtime Streaming on Akka
QCON 2015: Gearpump, Realtime Streaming on AkkaQCON 2015: Gearpump, Realtime Streaming on Akka
QCON 2015: Gearpump, Realtime Streaming on Akka
 
Beaming flink to the cloud @ netflix ff 2016-monal-daxini
Beaming flink to the cloud @ netflix   ff 2016-monal-daxiniBeaming flink to the cloud @ netflix   ff 2016-monal-daxini
Beaming flink to the cloud @ netflix ff 2016-monal-daxini
 
Monal Daxini - Beaming Flink to the Cloud @ Netflix
Monal Daxini - Beaming Flink to the Cloud @ NetflixMonal Daxini - Beaming Flink to the Cloud @ Netflix
Monal Daxini - Beaming Flink to the Cloud @ Netflix
 
Patterns of-streaming-applications-qcon-2018-monal-daxini
Patterns of-streaming-applications-qcon-2018-monal-daxiniPatterns of-streaming-applications-qcon-2018-monal-daxini
Patterns of-streaming-applications-qcon-2018-monal-daxini
 
Spark Streaming and IoT by Mike Freedman
Spark Streaming and IoT by Mike FreedmanSpark Streaming and IoT by Mike Freedman
Spark Streaming and IoT by Mike Freedman
 
dA Platform Overview
dA Platform OverviewdA Platform Overview
dA Platform Overview
 
Flink Forward San Francisco 2018: Robert Metzger & Patrick Lucas - "dA Platfo...
Flink Forward San Francisco 2018: Robert Metzger & Patrick Lucas - "dA Platfo...Flink Forward San Francisco 2018: Robert Metzger & Patrick Lucas - "dA Platfo...
Flink Forward San Francisco 2018: Robert Metzger & Patrick Lucas - "dA Platfo...
 
Stream Processing with Flink and Stream Sharing
Stream Processing with Flink and Stream SharingStream Processing with Flink and Stream Sharing
Stream Processing with Flink and Stream Sharing
 

Más de InfluxData

How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base
How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base
How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base
InfluxData
 
How Delft University's Engineering Students Make Their EV Formula-Style Race ...
How Delft University's Engineering Students Make Their EV Formula-Style Race ...How Delft University's Engineering Students Make Their EV Formula-Style Race ...
How Delft University's Engineering Students Make Their EV Formula-Style Race ...
InfluxData
 
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
InfluxData
 
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
InfluxData
 

Más de InfluxData (20)

Announcing InfluxDB Clustered
Announcing InfluxDB ClusteredAnnouncing InfluxDB Clustered
Announcing InfluxDB Clustered
 
Best Practices for Leveraging the Apache Arrow Ecosystem
Best Practices for Leveraging the Apache Arrow EcosystemBest Practices for Leveraging the Apache Arrow Ecosystem
Best Practices for Leveraging the Apache Arrow Ecosystem
 
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...
How Bevi Uses InfluxDB and Grafana to Improve Predictive Maintenance and Redu...
 
Power Your Predictive Analytics with InfluxDB
Power Your Predictive Analytics with InfluxDBPower Your Predictive Analytics with InfluxDB
Power Your Predictive Analytics with InfluxDB
 
How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base
How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base
How Teréga Replaces Legacy Data Historians with InfluxDB, AWS and IO-Base
 
Build an Edge-to-Cloud Solution with the MING Stack
Build an Edge-to-Cloud Solution with the MING StackBuild an Edge-to-Cloud Solution with the MING Stack
Build an Edge-to-Cloud Solution with the MING Stack
 
Meet the Founders: An Open Discussion About Rewriting Using Rust
Meet the Founders: An Open Discussion About Rewriting Using RustMeet the Founders: An Open Discussion About Rewriting Using Rust
Meet the Founders: An Open Discussion About Rewriting Using Rust
 
Introducing InfluxDB Cloud Dedicated
Introducing InfluxDB Cloud DedicatedIntroducing InfluxDB Cloud Dedicated
Introducing InfluxDB Cloud Dedicated
 
Gain Better Observability with OpenTelemetry and InfluxDB
Gain Better Observability with OpenTelemetry and InfluxDB Gain Better Observability with OpenTelemetry and InfluxDB
Gain Better Observability with OpenTelemetry and InfluxDB
 
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...
How a Heat Treating Plant Ensures Tight Process Control and Exceptional Quali...
 
How Delft University's Engineering Students Make Their EV Formula-Style Race ...
How Delft University's Engineering Students Make Their EV Formula-Style Race ...How Delft University's Engineering Students Make Their EV Formula-Style Race ...
How Delft University's Engineering Students Make Their EV Formula-Style Race ...
 
Introducing InfluxDB’s New Time Series Database Storage Engine
Introducing InfluxDB’s New Time Series Database Storage EngineIntroducing InfluxDB’s New Time Series Database Storage Engine
Introducing InfluxDB’s New Time Series Database Storage Engine
 
Start Automating InfluxDB Deployments at the Edge with balena
Start Automating InfluxDB Deployments at the Edge with balena Start Automating InfluxDB Deployments at the Edge with balena
Start Automating InfluxDB Deployments at the Edge with balena
 
Understanding InfluxDB’s New Storage Engine
Understanding InfluxDB’s New Storage EngineUnderstanding InfluxDB’s New Storage Engine
Understanding InfluxDB’s New Storage Engine
 
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDB
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDBStreamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDB
Streamline and Scale Out Data Pipelines with Kubernetes, Telegraf, and InfluxDB
 
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...
Ward Bowman [PTC] | ThingWorx Long-Term Data Storage with InfluxDB | InfluxDa...
 
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022
Scott Anderson [InfluxData] | New & Upcoming Flux Features | InfluxDays 2022
 
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts | InfluxDays 2022
 
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
Steinkamp, Clifford [InfluxData] | Welcome to InfluxDays 2022 - Day 2 | Influ...
 
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
Steinkamp, Clifford [InfluxData] | Closing Thoughts Day 1 | InfluxDays 2022
 

Último

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Victor Rentea
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
 

Último (20)

Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUKSpring Boot vs Quarkus the ultimate battle - DevoxxUK
Spring Boot vs Quarkus the ultimate battle - DevoxxUK
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 AmsterdamDEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
DEV meet-up UiPath Document Understanding May 7 2024 Amsterdam
 
Cyberprint. Dark Pink Apt Group [EN].pdf
Cyberprint. Dark Pink Apt Group [EN].pdfCyberprint. Dark Pink Apt Group [EN].pdf
Cyberprint. Dark Pink Apt Group [EN].pdf
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Exploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with MilvusExploring Multimodal Embeddings with Milvus
Exploring Multimodal Embeddings with Milvus
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 

Why Architecting for Disaster Recovery is Important for Your Time Series Data by Capital One

  • 1. Confidential Time Series Data Management Evolution at Capital One Building a highly resilient Enterprise Influx across multiple regions Monitoring Intelligence October 23, 2018
  • 2. Confidential Agenda 2 1. Influx at Capital One 2. Architecture How Influx Architecture evolved 3. Resiliency Complete protection against entire region failure 4. Performance Metrics Critical Platform Metrics 5. Q & A
  • 3. Confidential Influx at Capital One 3 • Business Transaction Metrics • Infrastructure Health metrics • Application Performance Metrics • Service Adoption Metrics
  • 4. Confidential Agenda 4 1. Influx at Capital One 2. Architecture How Influx Architecture evolved 3. Resiliency Complete protection against entire region failure 4. Performance Metrics Critical Platform Metrics 5. Q & A
  • 5. Confidential Architecture – Gen1 5 InfluxDB ( Active ) InfluxDB (Standby) DR Site Visualization https Primary Site LB Backup/Restore Features 1. Grafana for visualization Challenges 1. High Data Retention ( > 400 days ) 2. Unstable DR Solution Splunk Direct API Telegraf
  • 6. Confidential Architecture – Gen2 6 InfluxDB ( Active ) InfluxDB (Standby) DR Site Visualization Primary Site LB AWS S3 Data Lake ML Model Execution Model Governance ML Model Dev/Train Backup/Restore Daily batch Features 1. Grafana for visualization 2. Raw Data Exported Daily to One Lake 3. Raw Data available for ML Challenges High Data Retention ( > 400 days ) 2. Unstable DR Solution A B. ReviewC. Deploy E D Splunk Direct API Telegraf https
  • 7. Confidential Architecture – Current 7 InfluxDB ( Active ) InfluxDB (Passive) DR Site Visualization Primary Site LB AWS S3 Data Lake ML Model Execution Model Governance ML Model Dev/Train AWS S3 ExportImport Mini batch Features 1. Grafana for visualization 2. Raw Data Exported every 30 minutes to One Lake 3. Raw Data available for ML 4. Stable DR Solution with Passive Cluster Challenges High Data Retention ( > 400 days ) Unstable DR Solution A B. ReviewC. Deploy E D Splunk Direct API Telegraf https
  • 8. Confidential Agenda 8 1. Influx at Capital One 2. Architecture How Influx Architecture evolved 3. Resiliency Complete protection against entire region failure 4. Performance Metrics Critical Platform Metrics 5. Q & A
  • 9. Confidential Resiliency – Region 1 Active 9 https D M D M D M Region 1 ( Active) Zone 1 Zone 2 Zone 3 LB S3 Cluster 1 D M D M D M Region 2 (DR) Zone 1 Zone 2 Zone 3 LB S3 Cluster 2 Route53 (DNS Switch) Cross Region Replication Admin A ASGASG M Meta Node D Data Node A Admin Node 3 A Route53 (DNS Switch) 1 4 2 Splunk Direct API Telegraf All Traffic routed to Region 11 Influx Export Script every 15 min2 Data Replicated to Region 23 Influx Import Script every 15 min4
  • 10. Confidential Resiliency – Region 2 Active 10 https D M D M D M Region 1 ( DR) Zone 1 Zone 2 Zone 3 LB S3 Cluster 1 D M D M D M Region 2 (Active) Zone 1 Zone 2 Zone 3 LB S3 Cluster 2 Route53 (DNS Switch) Cross Region Replication Admin A ASGASG M Meta Node D Data Node A Admin Node 3 A Route53 (DNS Switch) 1 2 4 Splunk Direct API Telegraf All Traffic routed to Region 21 Influx Export Script every 15 min2 Data Replicated to Region 13 Influx Import Script every 15 min4
  • 11. Confidential Resiliency – Export/Import 11 Export* Import* * High level pseudo code , Not the actual code
  • 12. Confidential Agenda 12 1. Influx at Capital One 2. Architecture How Influx Architecture evolved 3. Resiliency Complete protection against entire region failure 4. Performance Metrics Critical Platform Metrics 5. Q & A