Where's My Message? with David Navalho & Ricardo Henriques

•

0 recomendaciones•247 vistas

The document discusses common issues encountered with Kafka such as outages, slow performance, empty topics and deleted messages. It provides recommendations for addressing these issues such as implementing dashboards, tests, centralized logging, searchable knowledge bases, tracking topic details and ownership for troubleshooting. The key takeaways are to gain governance and visibility over the Kafka platform, engage stakeholders, continuously update monitoring and automation, and share knowledge to provide faster resolution for issues.

Tecnología

Where’s My Message?
David Navalho and Ricardo Henriques

David Navalho
Head of Core Technologies
Ricardo Henriques
Lead DevOps
Introduction

Incident Report
OMG! Kafka is Down!
Un-Sync Back-and-Forth
Communication
Broken Phone
Time Consuming

Dashboards
OMG! Kafka is Down!
The end of Development and Deployment is not the end of Tests!
Share Knowledge
Incidents Knowledge Base

Kafka is Slow!
Gonna get a cuppa, anyone?

Kafka is Slow!
Dashboards
Tests
Share Knowledge
Incidents Knowledge Base
Reports
Performance Tests

My topic is empty!
Where are my messages?!

My topic is empty!
Topics, Consumers, ACL, Connectors, …
Resources
information
Centralised Logging
Developer Knowledge Base
Searchable
Self-troubleshooting

Kafka deleted my messages!
Topic:
retention: 7 days
created: 4 months

Kafka deleted my messages!
Topic:
retention: 7 days
created: 4 months
Topic:
retention: 7 days
created: 4 months

Kafka deleted my messages!
Topic:
retention: 7 days
created: 4 months
Sink:
created: 4 months
Source:
created: 4 months
Topic:
retention: 7 days
created: 4 months

Kafka deleted my messages!
Replicator:
7 days Topic:
retention: 7 days
created: 4 months
Sink:
created: 4 months
Source:
created: 4 months
Topic:
retention: 7 days
created: 4 months

Kafka deleted my messages!
Topics, Consumers, ACL, Connectors, …
Resources
information + history
Search capabilities
Lineage
Knowledge Bases
Keys, Messages, Headers
Keep updating!
Track connections, ownership

Where’s my message?
When all else fails…

Where’s my message?
Who owns/controls what
Governance
Build Trust
Knowledge Bases
Engage
Keep updating!
New Automations
and Dashboards
Learn the Use Cases to provide faster
results

Recap and QA
Who owns/controls what
Governance
Build Trust
Engage
Keep updating!
Dashboards
Tests Reports
Tailored views
Share Status
Broaden the scope outside
development lifecycle
Share Knowledge
Centralised Logging
Searchable
Labs
Experiment is key!
If we spend all our time troubleshooting, our platform isn't improving but rather decaying!

Thank you
Reach out!
David Navalho
Head of Core Technologies
Ricardo Henriques
Lead DevOps
david-navalho ricardo7aires

Más contenido relacionado

Similar a Where's My Message? with David Navalho & Ricardo Henriques

DevOps-ing the MainframeDevOps.com

Introduction to Cassandra and CQL for Java developersJulien Anguenot

Monkeys in Lab Coats: Applying Failure Testing Research @NetflixC4Media

Open-Source Project Tools for Corporate Projects?Bertrand Delacretaz

NLJUG speaker academy 2023 - session 1Bert Jan Schrijver

Your Goat Anti-Fragiled My Snowflake! Demystifying DevOps Jargon (30 minute v...Clinton Wolfe

Your Goat Anti-Fragiled My SnowFlake! Demystifying DevOps JargonClinton Wolfe

Conferences andcommunityJeff Carouth

Sergio Seabra - Red Hat - OSL19marketingsyone

StackEngine Problem Space DemoBoyd Hemphill

Leaving the Ivory Tower: Research in the Real WorldC4Media

Let’s learn how to use JavaScript responsibly and stay up-to-date. Christian Heilmann

DEVNET-1125 Partner Case Study - “Project Hybrid Engineer”Cisco DevNet

Microservices: State of the UnionC4Media

Devry CIS 246 Full Course LatestAtifkhilji

Introducing Apache Kafka and why it is important to Oracle, Java and IT profe...Lucas Jellema

Module 1 - An Introduction To SWORDSWORD Project

Liveperson DLD 2015 LivePerson

Cassandra Day SV 2014: Spark, Shark, and Apache CassandraDataStax Academy

Curing the Kafka Blindnessgvetticaden

Similar a Where's My Message? with David Navalho & Ricardo Henriques (20)

DevOps-ing the Mainframe

Introduction to Cassandra and CQL for Java developers

Monkeys in Lab Coats: Applying Failure Testing Research @Netflix

Open-Source Project Tools for Corporate Projects?

NLJUG speaker academy 2023 - session 1

Your Goat Anti-Fragiled My Snowflake! Demystifying DevOps Jargon (30 minute v...

Your Goat Anti-Fragiled My SnowFlake! Demystifying DevOps Jargon

Conferences andcommunity

Sergio Seabra - Red Hat - OSL19

StackEngine Problem Space Demo

Leaving the Ivory Tower: Research in the Real World

Let’s learn how to use JavaScript responsibly and stay up-to-date.

DEVNET-1125 Partner Case Study - “Project Hybrid Engineer”

Microservices: State of the Union

Devry CIS 246 Full Course Latest

Introducing Apache Kafka and why it is important to Oracle, Java and IT profe...

Module 1 - An Introduction To SWORD

Liveperson DLD 2015

Cassandra Day SV 2014: Spark, Shark, and Apache Cassandra

Curing the Kafka Blindness

Más de HostedbyConfluent

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...HostedbyConfluent

Renaming a Kafka Topic | Kafka Summit LondonHostedbyConfluent

Evolution of NRT Data Ingestion Pipeline at TrendyolHostedbyConfluent

Ensuring Kafka Service Resilience: A Dive into Health-Checking TechniquesHostedbyConfluent

Exactly-once Stream Processing with Arroyo and KafkaHostedbyConfluent

Fish Plays Pokemon | Kafka Summit LondonHostedbyConfluent

Tiered Storage 101 | Kafla Summit LondonHostedbyConfluent

Building a Self-Service Stream Processing Portal: How And WhyHostedbyConfluent

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...HostedbyConfluent

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...HostedbyConfluent

Navigating Private Network Connectivity Options for Kafka ClustersHostedbyConfluent

Apache Flink: Building a Company-wide Self-service Streaming Data PlatformHostedbyConfluent

Explaining How Real-Time GenAI Works in a Noisy PubHostedbyConfluent

TL;DR Kafka Metrics | Kafka Summit LondonHostedbyConfluent

A Window Into Your Kafka Streams Tasks | KSLHostedbyConfluent

Mastering Kafka Producer Configs: A Guide to Optimizing PerformanceHostedbyConfluent

Data Contracts Management: Schema Registry and BeyondHostedbyConfluent

Code-First Approach: Crafting Efficient Flink AppsHostedbyConfluent

Debezium vs. the World: An Overview of the CDC EcosystemHostedbyConfluent

Beyond Tiered Storage: Serverless Kafka with No Local DisksHostedbyConfluent

Más de HostedbyConfluent (20)

Transforming Data Streams with Kafka Connect: An Introduction to Single Messa...

Renaming a Kafka Topic | Kafka Summit London

Evolution of NRT Data Ingestion Pipeline at Trendyol

Ensuring Kafka Service Resilience: A Dive into Health-Checking Techniques

Exactly-once Stream Processing with Arroyo and Kafka

Fish Plays Pokemon | Kafka Summit London

Tiered Storage 101 | Kafla Summit London

Building a Self-Service Stream Processing Portal: How And Why

From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 ...

Future with Zero Down-Time: End-to-end Resiliency with Chaos Engineering and ...

Navigating Private Network Connectivity Options for Kafka Clusters

Apache Flink: Building a Company-wide Self-service Streaming Data Platform

Explaining How Real-Time GenAI Works in a Noisy Pub

TL;DR Kafka Metrics | Kafka Summit London

A Window Into Your Kafka Streams Tasks | KSL

Mastering Kafka Producer Configs: A Guide to Optimizing Performance

Data Contracts Management: Schema Registry and Beyond

Code-First Approach: Crafting Efficient Flink Apps

Debezium vs. the World: An Overview of the CDC Ecosystem

Beyond Tiered Storage: Serverless Kafka with No Local Disks

Último

Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2

[2024]Digital Global Overview Report 2024 Meltwater.pdfhans926745

Developing An App To Navigate The Roads of BrazilV3cube

Boost PC performance: How more available memory can improve productivityPrincipled Technologies

08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung

How to convert PDF to text with Nanonetsnaman860154

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j

Slack Application Development 101 Slidespraypatel2

A Call to Action for Generative AI in 2024Results

WhatsApp 9892124323 ✓Call Girls In Kalyan ( Mumbai ) secure servicePooja Nehwal

Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdfEnterprise Knowledge

A Domino Admins Adventures (Engage 2024)Gabriella Davis

Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko

GenCyber Cyber Security Day PresentationMichael W. Hawkins

Finology Group – Insurtech Innovation Award 2024The Digital Insurer

Partners Life - Insurer Innovation Award 2024The Digital Insurer

Where's My Message? with David Navalho & Ricardo Henriques

1. Where’s My Message? David Navalho and Ricardo Henriques

2. David Navalho Head of Core Technologies Ricardo Henriques Lead DevOps Introduction

3. Why this Session?

4. Kafka is Down! OMG!

5. Incident Report OMG! Kafka is Down! Un-Sync Back-and-Forth Communication Broken Phone Time Consuming

6. Dashboards OMG! Kafka is Down! The end of Development and Deployment is not the end of Tests! Share Knowledge Incidents Knowledge Base

7. Kafka is Slow! Gonna get a cuppa, anyone?

8. Kafka is Slow!

9. Kafka is Slow! Dashboards Tests Share Knowledge Incidents Knowledge Base Reports Performance Tests

10. My topic is empty! Where are my messages?!

11. My topic is empty!

12. My topic is empty!

13. My topic is empty!

14. My topic is empty! Topics, Consumers, ACL, Connectors, … Resources information Centralised Logging Developer Knowledge Base Searchable Self-troubleshooting

15. Kafka deleted my messages! What?!

16. Kafka deleted my messages!

17. Kafka deleted my messages!

18. Kafka deleted my messages!

19. Kafka deleted my messages!

20. Kafka deleted my messages! Topic: retention: 7 days created: 4 months

21. Kafka deleted my messages! Topic: retention: 7 days created: 4 months Topic: retention: 7 days created: 4 months

22. Kafka deleted my messages! Topic: retention: 7 days created: 4 months Topic: retention: 7 days created: 4 months

23. Kafka deleted my messages! Topic: retention: 7 days created: 4 months Sink: created: 4 months Source: created: 4 months Topic: retention: 7 days created: 4 months

24. Kafka deleted my messages! Replicator: 7 days Topic: retention: 7 days created: 4 months Sink: created: 4 months Source: created: 4 months Topic: retention: 7 days created: 4 months

25. Kafka deleted my messages! Topics, Consumers, ACL, Connectors, … Resources information + history Search capabilities Lineage Knowledge Bases Keys, Messages, Headers Keep updating! Track connections, ownership

26. Where’s my message? When all else fails…

27. Where’s my message?

28. Where’s my message?

29. Where’s my message?

30. Where’s my message?

31. Where’s my message?

32. Where’s my message? Who owns/controls what Governance Build Trust Knowledge Bases Engage Keep updating! New Automations and Dashboards Learn the Use Cases to provide faster results

33. Recap and QA

34. Recap and QA Who owns/controls what Governance Build Trust Engage Keep updating! Dashboards Tests Reports Tailored views Share Status Broaden the scope outside development lifecycle Share Knowledge Centralised Logging Searchable Labs Experiment is key! If we spend all our time troubleshooting, our platform isn't improving but rather decaying!

35. Thank you Reach out! David Navalho Head of Core Technologies Ricardo Henriques Lead DevOps david-navalho ricardo7aires

Where's My Message? with David Navalho & Ricardo Henriques

Recomendados

Recomendados

Más contenido relacionado

Similar a Where's My Message? with David Navalho & Ricardo Henriques

Similar a Where's My Message? with David Navalho & Ricardo Henriques (20)

Más de HostedbyConfluent

Más de HostedbyConfluent (20)

Último

Último (20)

Where's My Message? with David Navalho & Ricardo Henriques