SlideShare una empresa de Scribd logo
1 de 14
Descargar para leer sin conexión
Apache kafka
Apache Kafka is a fast, scalable, durable and distributed
messaging system.
Prerequisites&Installation
• We need basic Java programming skills plus access to:
• Apache Kafka 0.9.0
• Apache Maven 3.0 or later
• Git
• Step 1: Download Kafka
Download the Apache Kafka 0.9.0release and un-tar it.
• Kafka is designed for distributed high throughput
systems. Kafka tends to work very well as a
replacement for a more traditional message
broker. In comparison to other messaging
systems, Kafka has better throughput, built-in
partitioning, replication and inherent fault-
tolerance, which makes it a good fit for large-
scale message processing applications.
•
What is a Messaging System?
• A Messaging System is responsible for transferring data from
one application to another, so the applications can focus on
data, but not worry about how to share it. Distributed
messaging is based on the concept of reliable message
queuing. Messages are queued asynchronously between client
applications and messaging system. Two types of messaging
patterns are available − one is point to point and the other is
publish-subscribe (pub-sub) messaging system. Most of the
messaging patterns follow pub-sub.
Point to Point Messaging System
• In a point-to-point system, messages are persisted in a
queue. One or more consumers can consume the
messages in the queue, but a particular message can be
consumed by a maximum of one consumer only. Once a
consumer reads a message in the queue, it disappears
from that queue. The typical example of this system is an
Order Processing System, where each order will be
processed by one Order Processor, but Multiple Order
Processors can work as well at the same time. The
following diagram depicts the structure.
Sender
Message
Queue
Receiver
Publish-Subscribe Messaging
System
• In the publish-subscribe system, messages are persisted in a
topic. Unlike point-to-point system, consumers can subscribe
to one or more topic and consume all the messages in that
topic. In the Publish-Subscribe system, message producers are
called publishers and message consumers are called
subscribers. A real-life example is Dish TV, which publishes
different channels like sports, movies, music, etc., and anyone
can subscribe to their own set of channels and get them
whenever their subscribed channels are available.
Sender
Message
Queue
Receiver
Receiver
Receiver
Benefits
• Following are a few benefits of Kafka −
• Reliability − Kafka is distributed, partitioned, replicated and
fault tolerance.
• Scalability − Kafka messaging system scales easily without
down time..
• Durability − Kafka uses Distributed commit log which means
messages persists on disk as fast as possible, hence it is
durable..
• Performance − Kafka has high throughput for both publishing
and subscribing messages. It maintains stable performance
even many TB of messages are stored.
• Kafka is very fast and guarantees zero downtime and zero data
loss.
Need for Kafka
• Kafka is a unified platform for handling all the real-time data
feeds. Kafka supports low latency message delivery and gives
guarantee for fault tolerance in the presence of machine
failures. It has the ability to handle a large number of diverse
consumers. Kafka is very fast, performs 2 million writes/sec.
Kafka persists all data to the disk, which essentially means that
all the writes go to the page cache of the OS (RAM). This
makes it very efficient to transfer data from page cache to a
network socket.
•
Kafka main terminologies
• such as topics, brokers, producers and consumers.
• Topics: A stream of messages belonging to a particular category is called a
topic. Data is stored in topics
• Partition:Topics may have many partitions, so it can handle an arbitrary
amount of data.
• Partition offset: Each partitioned message has a unique sequence id called
as offset
• Replicas of partition: Replicas are nothing but backups of a partition.
Replicas are never read or write data. They are used to prevent data loss.
• Brokers
• Brokers are simple system responsible for maintaining the pub-lished data.
Each broker may have zero or more partitions per topic. Assume, if there are
N partitions in a topic and N number of brokers, each broker will have one
partition.
Producer
• The sample Producer is a classical Java application with a main()
method, this application must:
• Initialize and configure a producer
• Use the producer to send messages
• 1- Producer Initialization
Create a producer is quite simple, you just need to create an instance of
the org.apache.kafka.clients.producer.KafkaProducer class with a set of
properties, this looks like:
• producer = new KafkaProducer(properties); In this example, the
configuration is externalized in a property file, with the following
entries:
2- Message posting
•
Once you have a producer instance you can post messages to a topic
using the ProducerRecord class. The ProducerRecord class is a
key/value pair where:
• the key is the topic
• the value is the message
• As you can guess sending a message to the topic is straight forward:
• ... producer.send(new ProducerRecord("fast-messages", "This is a
dummy message")); ...
Producer End
Once you are done with the producer use
the producer.close() method that blocks the process until all the
messages are sent to the server. This call is used in a finally block
to guarantee that it is called. A Kafka producer can also be used in
a try with resources construct.
• ... }
• finally { producer.close();
• }
Consumer
• The Consumer class, like the producer is a simple Java class
with a main method.
• This sample consumer uses the Hdr Histogram library to
record and analyze the messages received from the fast-
messages topic, and Jackson to parse JSON messages.
ZooKeeper
• ZooKeeper is used for managing and coordinating Kafka
broker. ZooKeeper service is mainly used to notify
producer and consumer about the presence of any new
broker in the Kafka system or failure of the broker in the
Kafka system. As per the notification received by the
Zookeeper regarding presence or failure of the broker
then pro-ducer and consumer takes decision and starts
coordinating their task with some other broker.
• Kafka stores basic metadata in Zookeeper such as
information about topics, brokers, consumer offsets
(queue readers) and so on.

Más contenido relacionado

La actualidad más candente

Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache KafkaJeff Holoman
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache KafkaShiao-An Yuan
 
Apache Kafka Fundamentals for Architects, Admins and Developers
Apache Kafka Fundamentals for Architects, Admins and DevelopersApache Kafka Fundamentals for Architects, Admins and Developers
Apache Kafka Fundamentals for Architects, Admins and Developersconfluent
 
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...confluent
 
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?confluent
 
Kafka Overview
Kafka OverviewKafka Overview
Kafka Overviewiamtodor
 
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignApache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignMichael Noll
 
Securing Kafka
Securing Kafka Securing Kafka
Securing Kafka confluent
 
Integrating Apache Kafka Into Your Environment
Integrating Apache Kafka Into Your EnvironmentIntegrating Apache Kafka Into Your Environment
Integrating Apache Kafka Into Your Environmentconfluent
 
Kafka At Scale in the Cloud
Kafka At Scale in the CloudKafka At Scale in the Cloud
Kafka At Scale in the Cloudconfluent
 

La actualidad más candente (20)

Kafka basics
Kafka basicsKafka basics
Kafka basics
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
Kafka 101
Kafka 101Kafka 101
Kafka 101
 
Kafka 101
Kafka 101Kafka 101
Kafka 101
 
Kafka tutorial
Kafka tutorialKafka tutorial
Kafka tutorial
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
Introduction to Apache Kafka
Introduction to Apache KafkaIntroduction to Apache Kafka
Introduction to Apache Kafka
 
Apache Kafka Fundamentals for Architects, Admins and Developers
Apache Kafka Fundamentals for Architects, Admins and DevelopersApache Kafka Fundamentals for Architects, Admins and Developers
Apache Kafka Fundamentals for Architects, Admins and Developers
 
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
Flexible Authentication Strategies with SASL/OAUTHBEARER (Michael Kaminski, T...
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
Apache Kafka - Overview
Apache Kafka - OverviewApache Kafka - Overview
Apache Kafka - Overview
 
Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?Kafka Streams: What it is, and how to use it?
Kafka Streams: What it is, and how to use it?
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
Kafka Overview
Kafka OverviewKafka Overview
Kafka Overview
 
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignApache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - Verisign
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
Securing Kafka
Securing Kafka Securing Kafka
Securing Kafka
 
Integrating Apache Kafka Into Your Environment
Integrating Apache Kafka Into Your EnvironmentIntegrating Apache Kafka Into Your Environment
Integrating Apache Kafka Into Your Environment
 
Kafka At Scale in the Cloud
Kafka At Scale in the CloudKafka At Scale in the Cloud
Kafka At Scale in the Cloud
 

Destacado

Kafka的设计与实现
Kafka的设计与实现Kafka的设计与实现
Kafka的设计与实现wang xing
 
Kafka - Messaging System
Kafka - Messaging SystemKafka - Messaging System
Kafka - Messaging SystemTanuj Mehta
 
Apache Kafka: Next Generation Distributed Messaging System
Apache Kafka: Next Generation Distributed Messaging SystemApache Kafka: Next Generation Distributed Messaging System
Apache Kafka: Next Generation Distributed Messaging SystemEdureka!
 
Introduction Apache Kafka
Introduction Apache KafkaIntroduction Apache Kafka
Introduction Apache KafkaJoe Stein
 
Introduction to Apache Kafka- Part 2
Introduction to Apache Kafka- Part 2Introduction to Apache Kafka- Part 2
Introduction to Apache Kafka- Part 2Knoldus Inc.
 
Current and Future of Apache Kafka
Current and Future of Apache KafkaCurrent and Future of Apache Kafka
Current and Future of Apache KafkaJoe Stein
 
How Apache Kafka is transforming Hadoop, Spark and Storm
How Apache Kafka is transforming Hadoop, Spark and StormHow Apache Kafka is transforming Hadoop, Spark and Storm
How Apache Kafka is transforming Hadoop, Spark and StormEdureka!
 
Deep dive into Apache Kafka consumption
Deep dive into Apache Kafka consumptionDeep dive into Apache Kafka consumption
Deep dive into Apache Kafka consumptionAlexandre Tamborrino
 
Kafka replication apachecon_2013
Kafka replication apachecon_2013Kafka replication apachecon_2013
Kafka replication apachecon_2013Jun Rao
 
Real time Messages at Scale with Apache Kafka and Couchbase
Real time Messages at Scale with Apache Kafka and CouchbaseReal time Messages at Scale with Apache Kafka and Couchbase
Real time Messages at Scale with Apache Kafka and CouchbaseWill Gardella
 
Stream Processing using Apache Spark and Apache Kafka
Stream Processing using Apache Spark and Apache KafkaStream Processing using Apache Spark and Apache Kafka
Stream Processing using Apache Spark and Apache KafkaAbhinav Singh
 
Discover.hdp2.2.storm and kafka.final
Discover.hdp2.2.storm and kafka.finalDiscover.hdp2.2.storm and kafka.final
Discover.hdp2.2.storm and kafka.finalHortonworks
 
Real time analytics with Netty, Storm, Kafka
Real time analytics with Netty, Storm, KafkaReal time analytics with Netty, Storm, Kafka
Real time analytics with Netty, Storm, KafkaTrieu Nguyen
 
Testing Angular 2 Applications - Rich Web 2016
Testing Angular 2 Applications - Rich Web 2016Testing Angular 2 Applications - Rich Web 2016
Testing Angular 2 Applications - Rich Web 2016Matt Raible
 
Handle Large Messages In Apache Kafka
Handle Large Messages In Apache KafkaHandle Large Messages In Apache Kafka
Handle Large Messages In Apache KafkaJiangjie Qin
 
Apache Kafka
Apache KafkaApache Kafka
Apache KafkaJoe Stein
 
Microservices for the Masses with Spring Boot, JHipster, and JWT - Rich Web 2016
Microservices for the Masses with Spring Boot, JHipster, and JWT - Rich Web 2016Microservices for the Masses with Spring Boot, JHipster, and JWT - Rich Web 2016
Microservices for the Masses with Spring Boot, JHipster, and JWT - Rich Web 2016Matt Raible
 

Destacado (20)

Kafka的设计与实现
Kafka的设计与实现Kafka的设计与实现
Kafka的设计与实现
 
Kafka - Messaging System
Kafka - Messaging SystemKafka - Messaging System
Kafka - Messaging System
 
Apache Kafka: Next Generation Distributed Messaging System
Apache Kafka: Next Generation Distributed Messaging SystemApache Kafka: Next Generation Distributed Messaging System
Apache Kafka: Next Generation Distributed Messaging System
 
Introduction Apache Kafka
Introduction Apache KafkaIntroduction Apache Kafka
Introduction Apache Kafka
 
Introduction to Apache Kafka- Part 2
Introduction to Apache Kafka- Part 2Introduction to Apache Kafka- Part 2
Introduction to Apache Kafka- Part 2
 
Current and Future of Apache Kafka
Current and Future of Apache KafkaCurrent and Future of Apache Kafka
Current and Future of Apache Kafka
 
How Apache Kafka is transforming Hadoop, Spark and Storm
How Apache Kafka is transforming Hadoop, Spark and StormHow Apache Kafka is transforming Hadoop, Spark and Storm
How Apache Kafka is transforming Hadoop, Spark and Storm
 
Deep dive into Apache Kafka consumption
Deep dive into Apache Kafka consumptionDeep dive into Apache Kafka consumption
Deep dive into Apache Kafka consumption
 
Kafka replication apachecon_2013
Kafka replication apachecon_2013Kafka replication apachecon_2013
Kafka replication apachecon_2013
 
Apache Kafka at LinkedIn
Apache Kafka at LinkedInApache Kafka at LinkedIn
Apache Kafka at LinkedIn
 
Real time Messages at Scale with Apache Kafka and Couchbase
Real time Messages at Scale with Apache Kafka and CouchbaseReal time Messages at Scale with Apache Kafka and Couchbase
Real time Messages at Scale with Apache Kafka and Couchbase
 
Stream Processing using Apache Spark and Apache Kafka
Stream Processing using Apache Spark and Apache KafkaStream Processing using Apache Spark and Apache Kafka
Stream Processing using Apache Spark and Apache Kafka
 
Discover.hdp2.2.storm and kafka.final
Discover.hdp2.2.storm and kafka.finalDiscover.hdp2.2.storm and kafka.final
Discover.hdp2.2.storm and kafka.final
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
Real time analytics with Netty, Storm, Kafka
Real time analytics with Netty, Storm, KafkaReal time analytics with Netty, Storm, Kafka
Real time analytics with Netty, Storm, Kafka
 
Testing Angular 2 Applications - Rich Web 2016
Testing Angular 2 Applications - Rich Web 2016Testing Angular 2 Applications - Rich Web 2016
Testing Angular 2 Applications - Rich Web 2016
 
Handle Large Messages In Apache Kafka
Handle Large Messages In Apache KafkaHandle Large Messages In Apache Kafka
Handle Large Messages In Apache Kafka
 
Apache Kafka
Apache KafkaApache Kafka
Apache Kafka
 
Microservices for the Masses with Spring Boot, JHipster, and JWT - Rich Web 2016
Microservices for the Masses with Spring Boot, JHipster, and JWT - Rich Web 2016Microservices for the Masses with Spring Boot, JHipster, and JWT - Rich Web 2016
Microservices for the Masses with Spring Boot, JHipster, and JWT - Rich Web 2016
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 

Similar a Apache kafka

Unleashing Real-time Power with Kafka.pptx
Unleashing Real-time Power with Kafka.pptxUnleashing Real-time Power with Kafka.pptx
Unleashing Real-time Power with Kafka.pptxKnoldus Inc.
 
Fundamentals and Architecture of Apache Kafka
Fundamentals and Architecture of Apache KafkaFundamentals and Architecture of Apache Kafka
Fundamentals and Architecture of Apache KafkaAngelo Cesaro
 
Session 23 - Kafka and Zookeeper
Session 23 - Kafka and ZookeeperSession 23 - Kafka and Zookeeper
Session 23 - Kafka and ZookeeperAnandMHadoop
 
Distributed messaging with Apache Kafka
Distributed messaging with Apache KafkaDistributed messaging with Apache Kafka
Distributed messaging with Apache KafkaSaumitra Srivastav
 
kafka_session_updated.pptx
kafka_session_updated.pptxkafka_session_updated.pptx
kafka_session_updated.pptxKoiuyt1
 
Kafka pub sub demo
Kafka pub sub demoKafka pub sub demo
Kafka pub sub demoSrish Kumar
 
Copy of Kafka-Camus
Copy of Kafka-CamusCopy of Kafka-Camus
Copy of Kafka-CamusDeep Shah
 
Introduction to Kafka Streams Presentation
Introduction to Kafka Streams PresentationIntroduction to Kafka Streams Presentation
Introduction to Kafka Streams PresentationKnoldus Inc.
 
Columbus mule soft_meetup_aug2021_Kafka_Integration
Columbus mule soft_meetup_aug2021_Kafka_IntegrationColumbus mule soft_meetup_aug2021_Kafka_Integration
Columbus mule soft_meetup_aug2021_Kafka_IntegrationMuleSoft Meetup
 
Building an Event Bus at Scale
Building an Event Bus at ScaleBuilding an Event Bus at Scale
Building an Event Bus at Scalejimriecken
 
Kafka 10000 feet view
Kafka 10000 feet viewKafka 10000 feet view
Kafka 10000 feet viewyounessx01
 
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...Lucas Jellema
 
Cluster_Performance_Apache_Kafak_vs_RabbitMQ
Cluster_Performance_Apache_Kafak_vs_RabbitMQCluster_Performance_Apache_Kafak_vs_RabbitMQ
Cluster_Performance_Apache_Kafak_vs_RabbitMQShameera Rathnayaka
 
Apache kafka
Apache kafkaApache kafka
Apache kafkaamarkayam
 

Similar a Apache kafka (20)

Unleashing Real-time Power with Kafka.pptx
Unleashing Real-time Power with Kafka.pptxUnleashing Real-time Power with Kafka.pptx
Unleashing Real-time Power with Kafka.pptx
 
Fundamentals and Architecture of Apache Kafka
Fundamentals and Architecture of Apache KafkaFundamentals and Architecture of Apache Kafka
Fundamentals and Architecture of Apache Kafka
 
Session 23 - Kafka and Zookeeper
Session 23 - Kafka and ZookeeperSession 23 - Kafka and Zookeeper
Session 23 - Kafka and Zookeeper
 
Distributed messaging with Apache Kafka
Distributed messaging with Apache KafkaDistributed messaging with Apache Kafka
Distributed messaging with Apache Kafka
 
kafka_session_updated.pptx
kafka_session_updated.pptxkafka_session_updated.pptx
kafka_session_updated.pptx
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 
Kafka pub sub demo
Kafka pub sub demoKafka pub sub demo
Kafka pub sub demo
 
Kafka
KafkaKafka
Kafka
 
Copy of Kafka-Camus
Copy of Kafka-CamusCopy of Kafka-Camus
Copy of Kafka-Camus
 
Introduction to Kafka Streams Presentation
Introduction to Kafka Streams PresentationIntroduction to Kafka Streams Presentation
Introduction to Kafka Streams Presentation
 
Kafka.pptx
Kafka.pptxKafka.pptx
Kafka.pptx
 
Columbus mule soft_meetup_aug2021_Kafka_Integration
Columbus mule soft_meetup_aug2021_Kafka_IntegrationColumbus mule soft_meetup_aug2021_Kafka_Integration
Columbus mule soft_meetup_aug2021_Kafka_Integration
 
Building an Event Bus at Scale
Building an Event Bus at ScaleBuilding an Event Bus at Scale
Building an Event Bus at Scale
 
Kafka Deep Dive
Kafka Deep DiveKafka Deep Dive
Kafka Deep Dive
 
Kafka 10000 feet view
Kafka 10000 feet viewKafka 10000 feet view
Kafka 10000 feet view
 
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...
 
Cluster_Performance_Apache_Kafak_vs_RabbitMQ
Cluster_Performance_Apache_Kafak_vs_RabbitMQCluster_Performance_Apache_Kafak_vs_RabbitMQ
Cluster_Performance_Apache_Kafak_vs_RabbitMQ
 
Message queues
Message queuesMessage queues
Message queues
 
Microservices deck
Microservices deckMicroservices deck
Microservices deck
 
Apache kafka
Apache kafkaApache kafka
Apache kafka
 

Más de Srikrishna k

Más de Srikrishna k (14)

Android
AndroidAndroid
Android
 
Hsqldb tutorial
Hsqldb tutorialHsqldb tutorial
Hsqldb tutorial
 
S3inmule
S3inmuleS3inmule
S3inmule
 
Mule sqs
Mule sqsMule sqs
Mule sqs
 
Apachepoitutorial
ApachepoitutorialApachepoitutorial
Apachepoitutorial
 
Introduction testingmule
Introduction testingmuleIntroduction testingmule
Introduction testingmule
 
Designpattern
DesignpatternDesignpattern
Designpattern
 
Java util
Java utilJava util
Java util
 
Test ng tutorial
Test ng tutorialTest ng tutorial
Test ng tutorial
 
Webservices intro
Webservices introWebservices intro
Webservices intro
 
Easy mock
Easy mockEasy mock
Easy mock
 
Apachespark 160612140708
Apachespark 160612140708Apachespark 160612140708
Apachespark 160612140708
 
Vmtransport 160723040146
Vmtransport 160723040146Vmtransport 160723040146
Vmtransport 160723040146
 
Groovydemo 160721051742
Groovydemo 160721051742Groovydemo 160721051742
Groovydemo 160721051742
 

Último

Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilVICTOR MAESTRE RAMIREZ
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptkinjal48
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxJoão Esperancinha
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native BuildpacksVish Abrams
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 

Último (20)

Salesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptxSalesforce AI Associate Certification.pptx
Salesforce AI Associate Certification.pptx
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
Generative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-CouncilGenerative AI for Cybersecurity - EC-Council
Generative AI for Cybersecurity - EC-Council
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Webinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.pptWebinar_050417_LeClair12345666777889.ppt
Webinar_050417_LeClair12345666777889.ppt
 
Fields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptxFields in Java and Kotlin and what to expect.pptx
Fields in Java and Kotlin and what to expect.pptx
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
Streamlining Your Application Builds with Cloud Native Buildpacks
Streamlining Your Application Builds  with Cloud Native BuildpacksStreamlining Your Application Builds  with Cloud Native Buildpacks
Streamlining Your Application Builds with Cloud Native Buildpacks
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 

Apache kafka

  • 1. Apache kafka Apache Kafka is a fast, scalable, durable and distributed messaging system.
  • 2. Prerequisites&Installation • We need basic Java programming skills plus access to: • Apache Kafka 0.9.0 • Apache Maven 3.0 or later • Git • Step 1: Download Kafka Download the Apache Kafka 0.9.0release and un-tar it.
  • 3. • Kafka is designed for distributed high throughput systems. Kafka tends to work very well as a replacement for a more traditional message broker. In comparison to other messaging systems, Kafka has better throughput, built-in partitioning, replication and inherent fault- tolerance, which makes it a good fit for large- scale message processing applications. •
  • 4. What is a Messaging System? • A Messaging System is responsible for transferring data from one application to another, so the applications can focus on data, but not worry about how to share it. Distributed messaging is based on the concept of reliable message queuing. Messages are queued asynchronously between client applications and messaging system. Two types of messaging patterns are available − one is point to point and the other is publish-subscribe (pub-sub) messaging system. Most of the messaging patterns follow pub-sub.
  • 5. Point to Point Messaging System • In a point-to-point system, messages are persisted in a queue. One or more consumers can consume the messages in the queue, but a particular message can be consumed by a maximum of one consumer only. Once a consumer reads a message in the queue, it disappears from that queue. The typical example of this system is an Order Processing System, where each order will be processed by one Order Processor, but Multiple Order Processors can work as well at the same time. The following diagram depicts the structure. Sender Message Queue Receiver
  • 6. Publish-Subscribe Messaging System • In the publish-subscribe system, messages are persisted in a topic. Unlike point-to-point system, consumers can subscribe to one or more topic and consume all the messages in that topic. In the Publish-Subscribe system, message producers are called publishers and message consumers are called subscribers. A real-life example is Dish TV, which publishes different channels like sports, movies, music, etc., and anyone can subscribe to their own set of channels and get them whenever their subscribed channels are available. Sender Message Queue Receiver Receiver Receiver
  • 7. Benefits • Following are a few benefits of Kafka − • Reliability − Kafka is distributed, partitioned, replicated and fault tolerance. • Scalability − Kafka messaging system scales easily without down time.. • Durability − Kafka uses Distributed commit log which means messages persists on disk as fast as possible, hence it is durable.. • Performance − Kafka has high throughput for both publishing and subscribing messages. It maintains stable performance even many TB of messages are stored. • Kafka is very fast and guarantees zero downtime and zero data loss.
  • 8. Need for Kafka • Kafka is a unified platform for handling all the real-time data feeds. Kafka supports low latency message delivery and gives guarantee for fault tolerance in the presence of machine failures. It has the ability to handle a large number of diverse consumers. Kafka is very fast, performs 2 million writes/sec. Kafka persists all data to the disk, which essentially means that all the writes go to the page cache of the OS (RAM). This makes it very efficient to transfer data from page cache to a network socket. •
  • 9. Kafka main terminologies • such as topics, brokers, producers and consumers. • Topics: A stream of messages belonging to a particular category is called a topic. Data is stored in topics • Partition:Topics may have many partitions, so it can handle an arbitrary amount of data. • Partition offset: Each partitioned message has a unique sequence id called as offset • Replicas of partition: Replicas are nothing but backups of a partition. Replicas are never read or write data. They are used to prevent data loss. • Brokers • Brokers are simple system responsible for maintaining the pub-lished data. Each broker may have zero or more partitions per topic. Assume, if there are N partitions in a topic and N number of brokers, each broker will have one partition.
  • 10. Producer • The sample Producer is a classical Java application with a main() method, this application must: • Initialize and configure a producer • Use the producer to send messages • 1- Producer Initialization Create a producer is quite simple, you just need to create an instance of the org.apache.kafka.clients.producer.KafkaProducer class with a set of properties, this looks like: • producer = new KafkaProducer(properties); In this example, the configuration is externalized in a property file, with the following entries:
  • 11. 2- Message posting • Once you have a producer instance you can post messages to a topic using the ProducerRecord class. The ProducerRecord class is a key/value pair where: • the key is the topic • the value is the message • As you can guess sending a message to the topic is straight forward: • ... producer.send(new ProducerRecord("fast-messages", "This is a dummy message")); ...
  • 12. Producer End Once you are done with the producer use the producer.close() method that blocks the process until all the messages are sent to the server. This call is used in a finally block to guarantee that it is called. A Kafka producer can also be used in a try with resources construct. • ... } • finally { producer.close(); • }
  • 13. Consumer • The Consumer class, like the producer is a simple Java class with a main method. • This sample consumer uses the Hdr Histogram library to record and analyze the messages received from the fast- messages topic, and Jackson to parse JSON messages.
  • 14. ZooKeeper • ZooKeeper is used for managing and coordinating Kafka broker. ZooKeeper service is mainly used to notify producer and consumer about the presence of any new broker in the Kafka system or failure of the broker in the Kafka system. As per the notification received by the Zookeeper regarding presence or failure of the broker then pro-ducer and consumer takes decision and starts coordinating their task with some other broker. • Kafka stores basic metadata in Zookeeper such as information about topics, brokers, consumer offsets (queue readers) and so on.