The Cost of Kafka’s High Availability on Cloud with Geetha Anne
The Cost of Kafka’s High Availability on Cloud with Geetha Anne
Shifting kafka workloads to public cloud platforms seems like a common-sense economics, as cloud empowers companies to scale resources as needed and keeps your Kafka high available. You only pay for what you use, thus saving money. Right? Well, not necessarily. Unless you are vigilant and relentless in monitoring, cloud spend can surprise you in unexpected ways through ingress/ egress fees, over provisioning, forgetting to turn off unwanted resources, not picking the right combination of instances, and so on, can quickly make costs go out of control. But public cloud cost optimization does not have to be esoteric or a long arduous road. Needless to say, achieving high availability for Kafka on cloud is both time consuming and expensive, costing a leg and arm, requiring a dedicated team to handle the entire infrastructure. I would like to share the lessons learnt from the field, productionizing cloud based kafka clusters at many enterprise companies which cover topics like techniques to optimize Kafka’s network transfer costs, configuring production & consumption to single & MZ zone, compression tools, tuning the fan-out etc. So In this session let us discuss ways and means to demystify and explore methods to potentially reduce your cloud spend on KAFKA clusters
Recomendados
Más contenido relacionado
Similar a The Cost of Kafka’s High Availability on Cloud with Geetha Anne
Similar a The Cost of Kafka’s High Availability on Cloud with Geetha Anne(20)
Más de HostedbyConfluent
Más de HostedbyConfluent(20)
Último
Último(20)
The Cost of Kafka’s High Availability on Cloud with Geetha Anne
- 1. Cost of running Kafka on cloud Geetha Anne Solutions Engineer
- 2. Market growth forecast for cloud services
- 4. Optimize Cloud Infrastructure Cost: How to do it? ● Analyse ● Optimise ● Evaluate
- 6. Compute Instance type Cost/hr m5.xlarge $0.192 c5.xlarge $0.17 r5.xlarge $0.252 ● For high performance deployment R5 h1, i3, and d2 instances are generally preferred. ● Consider buying reserved instances to lower costs. ● You may also evaluate m5.large and m5.4xlarge that provide balanced compute, memory, and networking resources for a wide range of use cases, and can be leveraged for large kafka workloads.
- 7. Storage ● EBS volumes provide consistent levels of I/O performance (IOPS) and ultimate flexibility in Kafka deployment ($0.08 per GB-month) ● Storage required: (avg-msg-size) x (msgs-per-day) x (retention-period-days) x (replication-factor) ● EBS volume types, such as gp3, io2, and st1 allow you to have varying throughput and IOPS characteristics for existing volumes supporting log processing, large database & data warehouse workloads KIP 405: Extend Kafka's storage beyond the local storage by retaining the older data in an external store
- 8. Networking ● Producer, Consumers and Partition placement ● Network Fanout ● Cross AZ traffic cost $0.02 / GB If it's viable: Produce in-zone and Consume from the same zone
- 9. Compression ● Kafka supports four primary types of compression: Gzip , Snappy, Lz4, Zstd ● Property compression.type ● Slower message delivery - latency expected ● Server logs, XML data, JSON messages. ● CPU to save disk and network bandwidth. ● dump-log-segments tool Type Ratio CPU usage Compressio n speed Network bandwidth usage Snappy Medium Moderate Moderate Medium Lz4 Low Lowest Fastest Highest Zstd Medium Moderate Moderate Medium Gzip High High Slow Low
- 10. Configurations Producer, Topic , Broker metrics Linger.ms (default is 0) Batch.size (default is 16384 bytes) Buffer.memory max.request.size KIP 500: Replace ZK with an internal quorum run inside Kaka KIP 405: Extend Kafka's storage beyond the local storage by retaining the older data in an external store KIP 392: Enable follower fetching for consumers to reduce n/w costs
- 11. Final thoughts ● Introduce Monitoring Regularly monitor for cost anomalies ● Be patient Takes several months to identify patterns ● Do the proactive administration Assess your needs and get estimates ahead of time
- 12. Understanding and optimizing your Kafka costs ● Part 1: A Guide to Mastering Kafka's Infrastructure Costs ● Part 2: The (Hidden) Cost of Kafka Operations ● Part 3: Tackling the Hidden and Unhidden Costs of Kafka ● Part 4: Confluent Will Beat Your Cost of Running Kafka (or $100 on us)
- 13. Thank you