Using Apache Kafka from Go

•

2 recomendaciones•548 vistas

At Uber, we use Kafka and Golang a lot. In this talk, I go over the basic concepts in Kafka and the libraries we use to publish and consume messages.

Software

Trillion+
Messages/Day
PBs
Data Volume
Tens of Thousands
Topics

Message Message Message Message Message Message
Topic

Partition - ordered, immutable sequence
Message Message Message Message Message
Message Message Message Message Message
Message Message Message Message Message
Partition 0
Partition 1
Partition 2

Offset
Message Message Message Message MessagePartition 0
Offset: 1

Offset
Message Message Message Message MessagePartition 0
Offset: 2

Offset
Message Message Message Message MessagePartition 0
Offset: 2 Offset: 3
Consumer: 1Consumer: 2

Broker
Broker
Broker
P0
P0
P0
P1
P1
P1
P2
P2
P2
Partitioned and Replicated

Broker
Broker
Broker
P0
P0
P0
P1
P1
P1
P2
P2
P2
Producers
Producer
Producer

Consumer
Consumer
Consumer
Consumer
Consumer
Broker
Broker
Broker
P0
P0
P0
P1
P1
P1
P2
P2
P2
Producers
Producer
Producer
Consumer Group 1
Consumer Group 2

log.Printf("failed to fetch URL %s", url)

https://github.com/confluentinc/kafka-rest

Más contenido relacionado

La actualidad más candente

ApacheCon Berlin 2019: Kongo:Building a Scalable Streaming IoT Application us...Paul Brebner

Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...Natan Silnitsky

RedisConf17 - Pain-free PipeliningRedis Labs

Codeless pipelines with pulsar and flinkTimothy Spann

Securing Your Containerized Applications with NGINXDocker, Inc.

Cilium – Kernel Native Security & DDOS Mitigation for Microservices with BPFCynthia Thomas

Apache kafkaCristiano Altmann

マイクロサービスバックエンドAPIのためのRESTとgRPCdisc99_

Practical tips and tricks for Apache Kafka messages integration | Francesco T...HostedbyConfluent

Introduction To Streaming Data and Stream Processing with Apache Kafkaconfluent

Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, TwitterHostedbyConfluent

A 9 line shop powered by armeriaLINE Corporation

A Kong retrospective: from 0.10 to 0.13Thibault Charbonnier

ApacheCon 2021 Apache Deep Learning 302Timothy Spann

Using the flipn stack for edge ai (flink, nifi, pulsar)Timothy Spann

Exactly Once Delivery with Kafka - Kafka Tel-Aviv MeetupNatan Silnitsky

Kafka on Pulsar StreamNative

Using FLiP with influxdb for edgeai iot at scale 2022Timothy Spann

Robust Stream Processing with Apache FlinkJamie Grier

ApacheCon 2021 - Apache NiFi Deep Dive 300Timothy Spann

La actualidad más candente (20)

ApacheCon Berlin 2019: Kongo:Building a Scalable Streaming IoT Application us...

Polyglot, Fault Tolerant Event-Driven Programming with Kafka, Kubernetes and ...

RedisConf17 - Pain-free Pipelining

Codeless pipelines with pulsar and flink

Securing Your Containerized Applications with NGINX

Cilium – Kernel Native Security & DDOS Mitigation for Microservices with BPF

Apache kafka

マイクロサービスバックエンドAPIのためのRESTとgRPC

Practical tips and tricks for Apache Kafka messages integration | Francesco T...

Introduction To Streaming Data and Stream Processing with Apache Kafka

Connect at Twitter-scale | Jordan Bull and Ryanne Dolan, Twitter

A 9 line shop powered by armeria

A Kong retrospective: from 0.10 to 0.13

ApacheCon 2021 Apache Deep Learning 302

Using the flipn stack for edge ai (flink, nifi, pulsar)

Exactly Once Delivery with Kafka - Kafka Tel-Aviv Meetup

Kafka on Pulsar

Using FLiP with influxdb for edgeai iot at scale 2022

Robust Stream Processing with Apache Flink

ApacheCon 2021 - Apache NiFi Deep Dive 300

Similar a Using Apache Kafka from Go

F_1330_Narkhede_Kafka .pptxNIMITJAIN71

Apache kafkasKaushikNarayanan

Apache kafkaMvkZ

Apache kafkasKaushikNarayanan

Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & PartitioningGuido Schmutz

Apache Kafka - Scalable Message-Processing and more !Guido Schmutz

[March sn meetup] apache pulsar + apache nifi for cloud data lakeTimothy Spann

Apache Kafka from 0.7 to 1.0, History and Lesson LearnedGuozhang Wang

Getting Started with Kafka on k8sVMware Tanzu

Timothy Spann: Apache Pulsar for MLEdunomica

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...Natan Silnitsky

Data Pipelines with Apache KafkaBen Stopford

The Evolution of Trillion-level Real-time Messaging System in BIGO - Puslar ...StreamNative

Introduction and Overview of Apache Kafka, TriHUG July 23, 2013mumrah

JConf.dev 2022 - Apache Pulsar Development 101 with JavaTimothy Spann

Apache Kafka - Scalable Message-Processing and more !Guido Schmutz

Similar a Using Apache Kafka from Go (20)

F_1330_Narkhede_Kafka .pptx

Apache kafka

Apache Kafka - Event Sourcing, Monitoring, Librdkafka, Scaling & Partitioning

Apache Kafka - Scalable Message-Processing and more !

[March sn meetup] apache pulsar + apache nifi for cloud data lake

Apache Kafka from 0.7 to 1.0, History and Lesson Learned

Getting Started with Kafka on k8s

Timothy Spann: Apache Pulsar for ML

Polyglot, fault-tolerant event-driven programming with kafka, kubernetes and ...

Data Pipelines with Apache Kafka

The Evolution of Trillion-level Real-time Messaging System in BIGO - Puslar ...

Introduction and Overview of Apache Kafka, TriHUG July 23, 2013

JConf.dev 2022 - Apache Pulsar Development 101 with Java

Apache Kafka - Scalable Message-Processing and more !

Más de Nikolay Stoitsev

Building vs Buying SoftwareNikolay Stoitsev

How and why to manage your managerNikolay Stoitsev

From programming to managementNikolay Stoitsev

A practical introduction to observabilityNikolay Stoitsev

Everything You Need to Know About NewSQL in 2020Nikolay Stoitsev

3 lessons on effective communication for engineersNikolay Stoitsev

The career path of software engineers and how to navigate itNikolay Stoitsev

Migrating a data intensive microservice from Python to GoNikolay Stoitsev

NewSQL: what, when and howNikolay Stoitsev

How to read the v8 source code?Nikolay Stoitsev

Running in multiple data centersNikolay Stoitsev

Distributed tracing for big systemsNikolay Stoitsev

Reusable patterns for scalable APIs running on Docker @ Java2DaysNikolay Stoitsev

Everyday tools and tricks for scaling Node.jsNikolay Stoitsev

Node.js at UberNikolay Stoitsev

Tracing python applicationsNikolay Stoitsev

Distributed tracing for Node.jsNikolay Stoitsev

Design Patterns for Docker ApplicationsNikolay Stoitsev

From Python to JavaNikolay Stoitsev

OpenFest 2016 - Open Microservice ArchitectureNikolay Stoitsev

Más de Nikolay Stoitsev (20)

Building vs Buying Software

How and why to manage your manager

From programming to management

A practical introduction to observability

Everything You Need to Know About NewSQL in 2020

3 lessons on effective communication for engineers

The career path of software engineers and how to navigate it

Migrating a data intensive microservice from Python to Go

NewSQL: what, when and how

How to read the v8 source code?

Running in multiple data centers

Distributed tracing for big systems

Reusable patterns for scalable APIs running on Docker @ Java2Days

Everyday tools and tricks for scaling Node.js

Node.js at Uber

Tracing python applications

Distributed tracing for Node.js

Design Patterns for Docker Applications

From Python to Java

OpenFest 2016 - Open Microservice Architecture

Último

SensoDat: Simulation-based Sensor Dataset of Self-driving CarsChristian Birchler

Large Language Models for Test Case Evolution and RepairLionel Briand

Unveiling the Future: Sylius 2.0 New FeaturesŁukasz Chruściel

Post Quantum Cryptography – The Impact on Identityteam-WIBU

CRM Contender Series: HubSpot vs. SalesforceBrainSell Technologies

What is Advanced Excel and what are some best practices for designing and cre...Technogeeks

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent

Cyber security and its impact on E commercemanigoyal112

英国UN学位证,北安普顿大学毕业证书1:1制作qr0udbr0

Maximizing Efficiency and Profitability with OnePlan’s Professional Service A...OnePlan Solutions

Xen Safety Embedded OSS Summit April 2024 v4.pdfStefano Stabellini

Precise and Complete Requirements? An Elusive GoalLionel Briand

SuccessFactors 1H 2024 Release - Sneak-Peek by Deloitte GermanyChristoph Pohl

Odoo 14 - eLearning Module In Odoo 14 Enterprisepreethippts

Global Identity Enrolment and Verification Pro Solution - Cizo Technology Ser...Cizo Technology Services

UI5ers live - Custom Controls wrapping 3rd-party libs.pptxAndreas Kunz

Cloud Data Center Network Construction - IEEEVICTOR MAESTRE RAMIREZ

Salesforce Implementation Services PPT By ABSYZABSYZ Inc

20240415 [Container Plumbing Days] Usernetes Gen2 - Kubernetes in Rootless Do...Akihiro Suda

Folding Cheat Sheet #4 - fourth in a seriesPhilip Schwarz