The session discusses on how companies are using Apache Kafka & also covers under the hood details like partitions, brokers, replication.
About apache kafka: Apache Kafka is a distributed a streaming platform, Apache Kafka provides low-latency, high-throughput, fault-tolerant publish and subscribe pipelines and is able to process streams of events. Kafka provides reliable, millisecond responses to support both customer-facing applications and connecting downstream systems with real-time data.
2. Need of Messaging system
What , Why , How of Apache Kafka
Producers, Brokers , Consumers ,Topics and Partitions
Developing Producers and Consumers
Questions
Overview
4. Front end UVData pipe line
It often starts with one data pipe line
5. Front end UV
Ledger systems
Instrument
management system
Reuse of existing providers for new consumers
6. Front end
Ledger systems
Instrument
management system
Security systems
Database server
Data ware house
Account Maintenance
system
Real time monitoring
Overnight chargesPricing system
Eventually the solution becomes the problem
7. Front end
Ledger systems
Instrument
management system
Security systems
Database server
Data ware house
Account Maintenance
system
Real time monitoring
Overnight chargesPricing system
Messaging system
9. Kafka decouples data pipe lines
Data warehouse
Account Maintenance system
Ledger system
Kafka
Real time monitoring Overnight charges
Pricing system Instrument management system Open position system
10. What is Apache Kafka?
Apache Kafka is a distributed publish-subscribe message system
It was originally developed at LinkedIn, later open sourced in 2011 and
became a part of Apache project
Kafka is fast , scalable ,durable , fault-tolerant and distributed by design
11. Kafka Features
High Throughput: Provides support for Hundreds
of thousand of messages
Scalability: Highly scalable distributed systems
with no downtime
Durability: Provides support to persisting
messages on disk
Replication: Messages can be replicated across
clusters , witch supports multiple subscribers
Stream Processing: Kafka can be used along with real time
streaming applications like Spark and storm