Kafka high-throughput persistent streams

•Descargar como PPT, PDF•

16 recomendaciones•6,959 vistas

Yahoo Developer Network

Arte y fotografía Tecnología Educación

Kafka high-throughput, persistent, multi-reader streams http://sna-projects.com/kafka

[object Object],[object Object],[object Object]

Problem How do you model and process stream data for a large website?

Examples ,[object Object],[object Object],[object Object],[object Object],[object Object]

Tracking Basics ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Existing messaging systems ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Ideas ,[object Object],[object Object],[object Object],[object Object]

Performance Test ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Performance Summary ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

How can we get high performance with persistence?

Some tricks ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Implementation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Distribution ,[object Object],[object Object]

Consumer State ,[object Object],[object Object],[object Object],[object Object]

APIs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Stream processing (0.06 release) Data published to persistent topics, and redistributed by primary key between stages

Also coming soon ,[object Object],[object Object],[object Object],[object Object]

The End http://sna-projects.com/kafka https://github.com/kafka-dev/kafka [email_address] [email_address]

Más contenido relacionado

La actualidad más candente

Flurry Analytic Backend - Processing Terabytes of Data in Real-timeTrieu Nguyen

Prestogres, ODBC & JDBC connectivity for PrestoSadayuki Furuhashi

Bullet: A Real Time Data Query EngineDataWorks Summit

Presto at TwitterBill Graham

Apache kafkaDaan Gerits

Apache Drill (ver. 0.1, check ver. 0.2)Camuel Gilyadov

Efficient State Management With Spark 2.0 And Scale-Out DatabasesJen Aman

Understanding Presto - Presto meetup @ Tokyo #1Sadayuki Furuhashi

Architecture of a Kafka camus infrastructuremattlieber

Automatic Scaling Iterative ComputationsGuozhang Wang

HBaseConAsia2018 Track1-5: Improving HBase reliability at PInterest with geo ...Michael Stack

Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem a...Hadoop User Group

PrestoChen Chun

Searching At ScaleHadoop User Group

Building Realtime Data Pipelines with Kafka Connect and Spark StreamingJen Aman

Impala presentation ahad ranaData Con LA

Cassandra Introduction & FeaturesPhil Peace

Marc Schwering – Using Flink with MongoDB to enhance relevancy in personaliza...Flink Forward

HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...Michael Stack

Apache Kafka at LinkedInGuozhang Wang

La actualidad más candente (20)

Flurry Analytic Backend - Processing Terabytes of Data in Real-time

Prestogres, ODBC & JDBC connectivity for Presto

Bullet: A Real Time Data Query Engine

Presto at Twitter

Apache kafka

Apache Drill (ver. 0.1, check ver. 0.2)

Efficient State Management With Spark 2.0 And Scale-Out Databases

Understanding Presto - Presto meetup @ Tokyo #1

Architecture of a Kafka camus infrastructure

Automatic Scaling Iterative Computations

HBaseConAsia2018 Track1-5: Improving HBase reliability at PInterest with geo ...

Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem a...

Presto

Searching At Scale

Building Realtime Data Pipelines with Kafka Connect and Spark Streaming

Impala presentation ahad rana

Cassandra Introduction & Features

Marc Schwering – Using Flink with MongoDB to enhance relevancy in personaliza...

HBaseConAsia2018 Track2-6: Scaling 30TB's of data lake with Apache HBase and ...

Apache Kafka at LinkedIn

Destacado

January 2011 HUG: Pig PresentationYahoo Developer Network

January 2011 HUG: Howl PresentationYahoo Developer Network

Yahoo compares Storm and SparkChicago Hadoop Users Group

August 2016 HUG: Better together: Fast Data with Apache Spark™ and Apache Ign...Yahoo Developer Network

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector Yahoo Developer Network

August 2016 HUG: Recent development in Apache OozieYahoo Developer Network

Destacado (6)

January 2011 HUG: Pig Presentation

January 2011 HUG: Howl Presentation

Yahoo compares Storm and Spark

August 2016 HUG: Better together: Fast Data with Apache Spark™ and Apache Ign...

August 2016 HUG: Open Source Big Data Ingest with StreamSets Data Collector

August 2016 HUG: Recent development in Apache Oozie

Similar a Kafka high-throughput persistent streams

Big Data Streams Architectures. Why? What? How?Anton Nazaruk

Open HFT libraries in @JavaPeter Lawrey

AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...GeeksLab Odessa

Luxun a Persistent Messaging System Tailored for Big Data Collecting & AnalyticsWilliam Yang

Apache ActiveMQ - Enterprise messaging in actiondejanb

Apache Kafka - Scalable Message-Processing and more !Guido Schmutz

Scalable Apache for Beginnerswebhostingguy

In search of the perfect IoT Stack - Scalable IoT Architectures with MQTTDominik Obermaier

Spinnaker VLDB 2011sandeep_tata

NoSQL afternoon in Japan Kumofs & MessagePackSadayuki Furuhashi

NoSQL afternoon in Japan kumofs & MessagePackSadayuki Furuhashi

Apache kafkaCristiano Altmann

Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure FunctionsJohn Staveley

[@NaukriEngineering] Messaging QueuesNaukri.com

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010Bhupesh Bansal

Hadoop and Voldemort @ LinkedInHadoop User Group

Cluster_Performance_Apache_Kafak_vs_RabbitMQShameera Rathnayaka

Distributed Systems: scalability and high availabilityRenato Lucindo

APIs & Underlying Protocols #APICraftSFAndrei Savu

Data Pipelines with Apache KafkaBen Stopford

Similar a Kafka high-throughput persistent streams (20)

Big Data Streams Architectures. Why? What? How?

Open HFT libraries in @Java

AI&BigData Lab 2016. Сарапин Виктор: Размер имеет значение: анализ по требова...

Luxun a Persistent Messaging System Tailored for Big Data Collecting & Analytics

Apache ActiveMQ - Enterprise messaging in action

Apache Kafka - Scalable Message-Processing and more !

Scalable Apache for Beginners

In search of the perfect IoT Stack - Scalable IoT Architectures with MQTT

Spinnaker VLDB 2011

NoSQL afternoon in Japan Kumofs & MessagePack

NoSQL afternoon in Japan kumofs & MessagePack

Apache kafka

Messaging - RabbitMQ, Azure (Service Bus), Docker and Azure Functions

[@NaukriEngineering] Messaging Queues

Voldemort & Hadoop @ Linkedin, Hadoop User Group Jan 2010

Hadoop and Voldemort @ LinkedIn

Cluster_Performance_Apache_Kafak_vs_RabbitMQ

Distributed Systems: scalability and high availability

APIs & Underlying Protocols #APICraftSF

Data Pipelines with Apache Kafka

Más de Yahoo Developer Network

Developing Mobile Apps for Performance - Swapnil Patel, Verizon MediaYahoo Developer Network

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...Yahoo Developer Network

Athenz & SPIFFE, Tatsuya Yano, Yahoo JapanYahoo Developer Network

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...Yahoo Developer Network

CICD at Oath using ScrewdriverYahoo Developer Network

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, OathYahoo Developer Network

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenuYahoo Developer Network

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, AmpoolYahoo Developer Network

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...Yahoo Developer Network

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...Yahoo Developer Network

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, OathYahoo Developer Network

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...Yahoo Developer Network

Moving the Oath Grid to Docker, Eric Badger, OathYahoo Developer Network

Architecting Petabyte Scale AI ApplicationsYahoo Developer Network

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...Yahoo Developer Network

Jun 2017 HUG: YARN Scheduling – A Step BeyondYahoo Developer Network

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies Yahoo Developer Network

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...Yahoo Developer Network

February 2017 HUG: Exactly-once end-to-end processing with Apache ApexYahoo Developer Network

February 2017 HUG: Data Sketches: A required toolkit for Big Data AnalyticsYahoo Developer Network

Más de Yahoo Developer Network (20)

Developing Mobile Apps for Performance - Swapnil Patel, Verizon Media

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...

Athenz & SPIFFE, Tatsuya Yano, Yahoo Japan

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...

CICD at Oath using Screwdriver

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, Oath

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...

Moving the Oath Grid to Docker, Eric Badger, Oath

Architecting Petabyte Scale AI Applications

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...

Jun 2017 HUG: YARN Scheduling – A Step Beyond

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

February 2017 HUG: Exactly-once end-to-end processing with Apache Apex

February 2017 HUG: Data Sketches: A required toolkit for Big Data Analytics

Último

Lucknow 💋 Call Girl in Lucknow Phone No 8923113531 Elite Escort Service Avail...anilsa9823

Editorial sephora annual report design projecttbatkhuu1

Indian Deira Call Girls # 0522916705 # Indian Call Girls In Deira Dubai || (UAE)Business Bay Call Girls || 0529877582 || Call Girls Service in Business Bay Dubai

Jeremy Casson - How Painstaking Restoration Has Revealed the Beauty of an Imp...Jeremy Casson

Jeremy Casson - An Architectural and Historical Journey Around EuropeJeremy Casson

Pakistani Deira Call Girls # 00971589162217 # Pakistani Call Girls In Deira D...Business Bay Call Girls || 0529877582 || Call Girls Service in Business Bay Dubai

Islamabad Escorts # 03080115551 # Escorts in Islamabad || Call Girls in Islam...wdefrd

Jeremy Casson - Top Tips for Pottery Wheel ThrowingJeremy Casson

RAJKOT CALL GIRL 76313*77252 CALL GIRL IN RAJKOTHigh profile service all area

Deconstructing Gendered Language; Feminist World-Making 2024samlnance

Best Call girls in Lucknow - 9548086042 - with hotel roomdiscovermytutordmt

Gomti Nagar & High Profile Call Girls in Lucknow (Adult Only) 8923113531 Esc...gurkirankumar98700

Alex and Chloe by Daniel Johnson Storyboardthephillipta

Lucknow 💋 Call Girls Service Lucknow ₹7.5k Pick Up & Drop With Cash Payment 8...anilsa9823

Bridge Fight Board by Daniel Johnson dtjohnsonart.comthephillipta

Lucknow 💋 Call Girls in Lucknow | Service-oriented sexy call girls 8923113531...anilsa9823

this is a jarvis ppt for jarvis ai assistant lovers and this is for youhigev50580

Hazratganj / Call Girl in Lucknow - Phone 🫗 8923113531 ☛ Escorts Service at 6...akbard9823

Lucknow 💋 Call Girl in Lucknow | Whatsapp No 8923113531 VIP Escorts Service A...anilsa9823

Indira Nagar Lucknow #Call Girls Lucknow ₹7.5k Pick Up & Drop With Cash Payme...akbard9823