Lessons from the field: Catalog of Kafka Deployments | Joseph Niemiec, Cloudera

•

0 recomendaciones•386 vistas

Streaming architectures have been on the rise steadily and as a result, we have seen the adoption of Kafka go up too. With the diverse spread of use cases across multiple industries, we have seen a variety of Kafka deployments across our hundreds of Kafka customers. Along the way, we have learnt some best practices as well as what not to do in mission-critical architectures. Join Joe Niemiec, Sr. Product Manager at Cloudera, as he shares these insights in this session that covers topics such as - The many ways that Kafka has been deployed in the field Standalone clusters, multiple clusters in a single data center and multiple clusters geographically distributed performing replication Clusters of all sizes small and large, few messages to hundreds of thousands per second Discussion about architecture failure domains Configurations tuned and used in specific deployments

Tecnología

Lessons from the field:
Catalog of Kafka Deployments
Joseph Niemiec | Sr. Product Manager

3
© 2021 Cloudera, Inc. All rights reserved.
Failure/Fault Domains
Limiting the state and scope of failure
What is a Failure/Fault Domain?
A set of components that share a
single point of failure and fail in a
correlated manner

4
© 2021 Cloudera, Inc. All rights reserved.
Serial and Parallel Systems Reliability
Serial Systems
• Failure of any component results
in failure of entire system
• Adding more weakens the
system ie - 90% * 90% = 81%
– Rs= R1* R2… Rn
Parallel Systems
• One component online results in
system online
• Adding more strengthens the system
ie - 1-(1-0.90)*(1-0.90) = 99%
– Rs= 1 - (1 -R1) *(1-R2) … (1-Rn)

5
© 2021 Cloudera, Inc. All rights reserved.
Basic Topic Design Theory
Partitions
● Few as needed
● Increased open files pressure
Topic Replicas
● Parallel availability
min-insync.replicas
● Durability vs Availability
● ack=all
○ for all isr with at least the min isr available
● replica.lag.time.max.ms
○ default 30 seconds!!!
Use log.retention.bytes
● Can use together
● Protect from topics growing larger
than brokers
Log Segments need to be
eligible for retention policies
● log.segment.bytes and log.roll.ms

7
© 2021 Cloudera, Inc. All rights reserved.
Single Node Dev Cluster
• Everything is Serial
• Failure/Fault Domain
– The Cluster
– The Broker & Zookeeper
– The Laptop
– Everything!
• Failure/Fault Tolerance
– Deploy over Multiple Servers
Kafka Cluster
1 Broker
1 Zookeeper

8
© 2021 Cloudera, Inc. All rights reserved.
Single Cluster - Single Rack w/wo colocated Zookeeper
• Some things are Serial
• Some things are Parallel
• Failure/Fault Domain
– Rack
– Brokers
• Other Notes
– Shared Log Dirs for Brokers
and Zookeeper

9
© 2021 Cloudera, Inc. All rights reserved.
Single Cluster - VMs
• Some things are Serial
• Some things are Parallel
• Failure/Fault Domain
– VM Hosts
– Replica Placement
• Replica per VM Host
• Other Notes
– Debugging Complexity

10
© 2021 Cloudera, Inc. All rights reserved.
Classic Failover - Two Clusters AcFve / Passive
• Mostly Parallel*
• Failure/Fault Domain
– Mirror Makers
– Cluster
– WAN*
– Data Center
• Other Notes
– Offsets During Failover
– WAN Bandwidth
– Mirror Maker Producer Side

11
© 2021 Cloudera, Inc. All rights reserved.
STREAMS
REPLICATION
MANAGER (SRM)
with Mirror Maker2
• Supports active-active, multi-
cluster, cross DC replication &
other scenarios
• Leverage Kafka Connect for
scalability and HA
• Replicate data and configurations
• Offset translation for easy
failover

12
© 2021 Cloudera, Inc. All rights reserved.
SRM
MONITORING
SERVICES
• SMM Cluster Replications View
provides monitoring
• SRM Service calculates replication
metrics (latency, throughput)
• REST endpoints with Swagger UI
• SMM Replication Flow Alerting

13
© 2021 Cloudera, Inc. All rights reserved.
Triple OnSite Cluster - In/Out/SysOps
• Serial Pipeline
• Parallel Systems
• Failure/Fault Domain
– Cluster
• Isolated Failures in Pipeline
• IN / Out Pipeline
– Producers / Apps / Consumers
– Datacenter
• Other Notes
– Test Configs/Apps on DevOps
– Ingress / Egress Isolation

14
© 2021 Cloudera, Inc. All rights reserved.
MulF-Geo DistribuFon and AggregaFon
• Serial Pipeline
• Parallel Systems
• Failure/Fault Domain
– Data Center
– Mirror Maker
– Cluster
• Other Notes
– Fat Network Pipe

15
© 2021 Cloudera, Inc. All rights reserved.
Dual Aggregation Active Active
• Mostly Parallel*
• Failure/Fault Domain
– Data Center
– Cluster
– Mirror Maker
– Consumers / Producers*
• Other Notes
– Active/Active
– Ingress / Egress Isolation
– Consumers on any side

16
© 2021 Cloudera, Inc. All rights reserved.
Multi-AZ/Cloud Spanning
• Parallel*
• Failure/Fault Domain
– Availability Zone
– Replica Placement*
• Other Notes
– Rack Awareness
– min-insync.replicas

Más contenido relacionado

La actualidad más candente

Data volumes continue to grow, demanding new, more scalable solutions for low-latency data processing. Previously, the default approach to deploying such systems was to throw a ton of hardware at the problem. However, that is no longer necessary, as newer technologies showcase a level of efficiency that enables smaller, more manageable clusters while handling extreme workloads. Processing billions of events per second on Kafka can now be done with a modest investment in compute resources. In this session, you will learn how to architect and build the fastest data processing applications that scale linearly, and combine streaming data and reference data data-in-motion and data-at-rest with machine learning. We will take you through the end-to-end framework and example application, built on the Hazelcast Platform, an open source software engine designed for ultra-fast performance. We will also show how you can leverage SQL to further explore the operational data in the solution including querying Kafka topics and key-value data on the in-memory data store. Attendees will also get access to the Github sample application shown.

Low-latency real-time data processing at giga-scale with Kafka | John DesJard...

HostedbyConfluent

A Look into the Mirror: Patterns and Best Practices for MirrorMaker2 | Cliff ...

HostedbyConfluent

Transaction Banking from Goldman Sachs is a high volume, latency sensitive digital banking platform offering. We have chosen an event driven architecture to build highly decoupled and independent microservices in a cloud native manner and are designed to meet the objectives of Security, Availability Latency and Scalability. Kafka was a natural choice – to decouple producers and consumers and to scale easily for high volume processing. However, there are certain aspects that require careful consideration – handling errors and partial failures, managing downtime of consumers, secure communication between brokers and producers / consumers. In this session, we will present the patterns and best practices that helped us build robust event driven applications. We will also present our solution approach that has been reused across multiple application domains. We hope that by sharing our experience, we can establish a reference implementation that application developers can benefit from.

Kafka error handling patterns and best practices | Hemant Desale and Aruna Ka...

HostedbyConfluent

Many organizations use Apache Kafka to facilitate the flow of data between multiple applications or data sources. Thanks to Kafka’s distributed architecture, it is easy to set up a scalable and reliable broker, but doing the same with producers or consumers is quite often a fine art. This session provides a quick overview of Couchbase, describes the Couchbase Kafka Connector, and showcases a demo of how it can be used as both a source and a sink for building real-time data processing pipelines for mission-critical applications.

Building Scalable Real-Time Data Pipelines with the Couchbase Kafka Connector...

HostedbyConfluent

Do your event streams use connected-data domains such as fraud detection, live logistics routing, or predicting network outages? How can you maintain the analysis and leverage those connections real-time? Graph databases differ from traditional, tabular ones in that they treat connections between data as first class citizens. This means they are optimized for detecting and understanding these relationships – providing insight at speed and at scale. By combining event streams from Kafka along with the power of the Neo4j graph database for interrogating and investigating connections, you make real-time, event-driven intelligent insight a reality. Neo4j Streams integrates Neo4j with Apache Kafka event streams, to serve as a source of data, for instance Change Data Capture or a sink to ingest any kind of Kafka event into your graph. In this session we’ll show you how to get up and running with Neo4j Streams to show you how to sink and source between graphs and streams.

Supercharge Your Real-time Event Processing with Neo4j's Streams Kafka Connec...

HostedbyConfluent

Streaming data systems have been growing rapidly in importance to the modern data stack. Kafka’s kSQL provides an interface for analytic tools that speak SQL. Apache Superset, the most popular modern open-source visualization and analytics solution, plugs into nearly any data source that speaks SQL, including Kafka. Here, we review and compare methods for connecting Kafka to Superset to enable streaming analytics use cases including anomaly detection, operational monitoring, and online data integration.

Streaming Data Analytics with ksqlDB and Superset | Robert Stolz, Preset

HostedbyConfluent

Cloud is changing the world; Kubernetes is changing the world; real-time event streaming is changing the world. In this talk we explore some of best practices to synergistically combine the power of these paradigm shifts to achieve a much greater return on your Kafka investments. From declarative deployments, zero-downtime upgrades, elastic scaling to self-healing and automated governance, learn how you can bring the next level of speed, agility, resilience, and security to your Kafka implementations.

Kafka Excellence at Scale – Cloud, Kubernetes, Infrastructure as Code (Vik Wa...

HostedbyConfluent

Guru Sattanathan, Confluent, Senior Solutions Engineer Enterprise Integration technologies (aka Middleware) are the key enablers when it comes to Real-time data flows or Event Driven Architecture. Starting from real-time payments, e-commerce, travel booking systems, etc, everything is powered by a middleware underneath. It did transform a lot of things but with caveats! Are ESB’s & MQ’s enough for today’s integration needs? Do you know their technical debts? If you are someone looking at integrating your applications or an Integration Architect this session is for you. It's time to refresh yourself and see how organizations are building integrations today. In this session, we will go in this order: -Recap on Enterprise Integration technologies -What are the key flaws & What needs improvement? -What is Apache Kafka? -Rethinking Integration using Apache Kafka https://www.meetup.com/KafkaMelbourne/events/280590162/

Death of the dumb pipes: Using Apache Kafka® for Integration projects

HostedbyConfluent

As cyber threats continuously grow in sophistication and frequency, companies need to quickly acclimate to effectively detect, respond, and protect their environments. At Intel, we’ve addressed this need by implementing a modern, scalable Cyber Intelligence Platform (CIP) based on Splunk and Apache Kafka. We believe that CIP positions us for the best defense against cyber threats well into the future. Our CIP ingests tens of terabytes of data each day and transforms it into actionable insights through streams processing, context-smart applications, and advanced analytics techniques. Kafka serves as a massive data pipeline within the platform. It achieves economies of scale by acquiring data once and consuming it many times. It reduces technical debt by eliminating custom point-to-point connections for producing and consuming data. At the same time, it provides the ability to operate on data in-stream, enabling us to reduce Mean Time to Detect (MTTD) and Mean Time to Respond (MTTR). Faster detection and response ultimately lead to better prevention. In our session, we’ll discuss the details described in the IT@Intel white paper that was published in Nov 2020 with same title. We’ll share some stream processing techniques, such as filtering and enriching in Kafka to deliver contextually rich data to Splunk and many of our security controls.

Building a Modern, Scalable Cyber Intelligence Platform with Apache Kafka | J...

HostedbyConfluent

One of the great things about running applications in the cloud is that you only pay for the resources that you use. But that also makes it more important than ever for our applications to be resource-efficient. This becomes even more critical when we use serverless functions. Micronaut is an application framework that provides dependency injection, developer productivity features, and excellent support for Apache Kafka. By performing dependency injection, AOP, and other productivity-enhancing magic at compile time, Micronaut allows us to build smaller, more efficient microservices and serverless functions. In this session, we'll explore the ways that Apache Kafka and Micronaut work together to enable us to build fast, efficient, event-driven applications. Then we'll see it in action, using the AWS Lambda Sink Connector for Confluent Cloud.

Event-driven Applications with Kafka, Micronaut, and AWS Lambda | Dave Klein,...

HostedbyConfluent

When moving to a cloud native architecture Moogsoft knew they needed more scale than Rabbit could provide. Moogsoft moved into Kafka which is known for quick writing and driving heavy event driven workloads on top of niceties such as replayability. Choosing the tool was easy, finding a vendor that ticked all their boxes was not. They needed to ensure scalability, upgradability, builds via existing IAC pipelines, and observability via existing tools. When Moogsoft found Aiven, they were impressed with their offering and ability to scale on demand. During this presentation we will explore how Moogsoft used Aiven for Kafka to manage and scale their data in the cloud.

The Road Most Traveled: A Kafka Story | Heikki Nousiainen, Aiven

HostedbyConfluent

The business systems of an organization are a continuous source of events. Each system also needs to know about events happening in the other systems. Exchanging these events through direct API calls creates a web of inter-dependencies, is fragile and fails to scale. We examine how this problem can be solved through the use of right integration patterns implemented as a light-weight event hub that leverages the power of Kafka and Confluent to operate at enterprise scale. We demonstrate how JavaScript with its event-driven programming model can be a good fit for implementing an event hub that ensures guaranteed message delivery in the face of failures within the individual subscriber systems. Many organizations having large engineering teams skilled in NodeJS and a multitude of NodeJs applications. We show how these teams can easily leverage the power of Kafka and scale their applications with the right architectural building blocks. We also offer insights from our own experience of building NodeJS based Kafka applications.

Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix

HostedbyConfluent

SIEM platforms are essential to the new cybersecurity paradigm and data collection layer is a very important piece of it. When you deliver a new platform, you can easily get lost in a variety of different vendors and solutions, too many challenges are facing. What if I change vendors, will I keep my data? How to feed multiple tools with the same data? How to collect data from custom apps and services? How to pay less for an expensive platform? How to keep data without a huge cost? Join us if you are looking for the answers. In this session, you will learn how we replaced the vendor-provided data collection layer with kafka connect and the lessons we learnt. After the talk you will know: - architecture and real-life examples of the flexible and highly available data collection platform - custom connectors that do most of the work for us and how to extend the connectors to consume new data, we made them open sourced - easy way to receive data from thousands of servers and many cloud services - how to archive data at low cost You will leave armed with a set of free tools and recipes to build a truly vendor-agnostic data collection platform. It will allow you to take you SIEM costs under control. You will feed your analytics tools with what they need and archive the rest at low cost. You will feed your SIEM smart!

Feed Your SIEM Smart with Kafka Connect (Vitalii Rudenskyi, McKesson Corp) Ka...

HostedbyConfluent

Apache Kafka fundamentally changes how organizations build and deploy a universal data pipeline that is scalable, reliable, and durable enough to meet the needs of digital-first organizations. However, as powerful as Kafka is today, it’s not an event-streaming platform - and getting it there on your own is a long, complicated, and expensive process. Earlier this year Confluent announced Project Metamorphosis - our plan to bring the best characteristics of cloud native systems to Apache Kafka. Since May we’ve begun transforming Confluent Cloud and Confluent Platform to do just that. Join two of our Product Managers, Dan Rosanova and Addison Huddy to: Learn how we’ve evolved Confluent Cloud with the first phase of Project Metamorphosis releases See how Confluent Platform 6.0 brings these transformational, cloud-like qualities to self-managed Kafka Get a sneak peak of our next Metamorphosis theme and how it impacts your Kafka and event-streaming strategy.

How Confluent Completes the Event Streaming Platform (Addison Huddy & Dan Ros...

HostedbyConfluent

To remain competitive, organizations need to democratize access to fast analytics, not only to gain real-time insights on their business but also to power smart apps that need to react in the moment. In this session, you will learn how Kafka and SingleStore enable modern, yet simple data architecture to analyze both fast paced incoming data as well as large historical datasets. In particular, you will understand why SingleStore is well suited process data streams coming from Kafka.

SingleStore & Kafka: Better Together to Power Modern Real-Time Data Architect...

HostedbyConfluent

Event streaming allows companies to build more scalable and loosely coupled real-time applications supporting massive concurrency demands and simplifying the construction of services. At the same time, API management provides capabilities to securely control the upstream services consumption, including the event processing infrastructure. This session shows how Kong Konnect Enterprise can complement Kafka Event Streaming, exposing it to new and external consumers while applying specific and critical policies to control its consumption, including API key, OAuth/OIDC and others for authentication, rate limiting, caching, log processing, etc.

Exposing and Controlling Kafka Event Streaming with Kong Konnect Enterprise |...

HostedbyConfluent

The data team at Cloudflare uses Kafka to process tens of petabytes a day. All this data is moved using the 2 foundational Kafka api calls: Produce (api key 0) and Fetch (api key 1). Understanding the structure of these calls (and of the underlying RecordSet structure) is key to building high throughput clients. The talk describes the basics of the Kafka wire protocol (api keys, correlation id), and the structure of the Produce and Fetch calls. It shows how the asynchronous nature of the wire protocol can combine with the structure of the Produce and Fetch calls to increase latency and reduce client throughput; a solution is offered through use of synchronous single-partition calls. The RecordSet structure, which is used to encode and store sets (batches) of records is described, and its implications on Fetch requests are discussed. The relationship between Fetch api calls and ""consume"" operations is discussed, as is the impact of offset alignment to RecordSet boundaries.

Understanding Kafka Produce and Fetch api calls for high throughtput applicat...

HostedbyConfluent

Qlik is an industry leader across its solution stack, both on the Data Integration side of things with Qlik Replicate (real-time CDC) and Qlik Compose (data warehouse and data lake automation), and on the Analytics side with Qlik Sense. These two “sides” of Qlik are coming together more frequently these days as the need for “always fresh” data increases across organizations. When real-time streaming applications are the topic du jour, those companies are looking to Apache Kafka to provide the architectural backbone those applications require. Those same companies turn to Qlik Replicate to put the data from their enterprise database systems into motion at scale, whether that data resides in “legacy” mainframe databases; traditional relational databases such as Oracle, MySQL, or SQL Server; or applications such as SAP and SalesForce. In this session we will look in depth at how Qlik Replicate can be used to continuously stream changes from a source database into Apache Kafka. From there, we will explore how a purpose-built consumer can be used to provide the bridge between Apache Kafka and an analytics application such as Qlik Sense.

Keeping Analytics Data Fresh in a Streaming Architecture | John Neal, Qlik

HostedbyConfluent

Confluent Cloud makes Devops engineers lives a lot more easier. Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge. In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge. These lessons include: 1. Automation, Automation, Automation - all the process has to be completely automated at such scale 2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad 3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

HostedbyConfluent

Real-time connectivity of databases and systems is critical in enterprises adopting digital transformation to support super-fast decisioning to drive applications like fraud detection, digital payments, recommendation engines. This talk will focus on the many functions that database streaming serves with Kafka, Spark and Aerospike. We will explore how to eliminate the wall between transaction processing and analytics by synthesizing streaming data with system of record data, to gain key insights in real-time.

Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...

HostedbyConfluent

La actualidad más candente (20)