Rust with-kafka-07-02-2019

•

0 recomendaciones•989 vistas

Gerard Klijs

Short presentation about using rust with kafka.

Software

www.openweb.nl
Kafka + Schema Registry
in Rust
By Gerard Klijs
Developer at Openweb
gerard@openweb.nl

www.openweb.nl
Summary
▰ About me
▰ Short intro to Kafka
▰ Short intro to the Confluent Schema Registry
▰ Rust support for Kafka
▰ Using rdkafka
▰ Turning the Schema Registry code into a Crate
222

www.openweb.nl
About me
▰ About 7 years working experience as developer,
mainly java, mainly web.
▰ Some pet projects in Clojure(script)
▰ Became especially interested in rust after
reading Rust for Clojurists
▰ Less than a year since I worked through the
book, no prior C or C++ experience.
33

www.openweb.nl
Short introduction to Apache Kafka
▰ A distributed streaming platform.
▰ Developed by LinkedIn, Open sourced in 2011.
▰ Main parts of the platform run on the JVM.
▰ Librdkafka is a C/C++ library implementing both
the producer and the consumer.
▰ Very scalable.
▰ Large ecosystem.
44

www.openweb.nl
Short introduction to Schema Registry
▰ Part of the Confluent Platform, from the
company created in 2014 by the creators.
▰ Uses Apache Avro for data serialization.
▰ Schema’s are stored in Kafka, each message
contains a key.
▰ The Schema Registry itself is open source.
▰ Versions the data and at the same time makes
the messages smaller.
55

www.openweb.nl
Rust support for Kafka
▰ rdkafka: Rust wrapper for librdkafka.
▰ kafka: Rust client for Apache Kafka.
▰ No support confluent platform, but several
crates working with Apache Avro.
66

www.openweb.nl
Using rdkafka
▰ Started with the provided examples from
rdkafka and get it to work.
▰ Tried several avro and serde libraries till I found
a match that worked.
77

www.openweb.nl
Turning the Schema Registry code into
a Crate
▰ Frame part which is generic and missing in
current crates.
▰ Create new project and pasted in code.
▰ Add unit tests and documentation.
▰ Some CI, code coverage and tags.
▰ Make it more generic.
▰ Improve error handling.
▰ Use as lib in originating project.
▰ Publish as schema_registry_converter to
crates.io.
88

Más contenido relacionado

La actualidad más candente

Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...confluent

Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...Igalia

Developing Secure Scala Applications With Fortify For ScalaLightbend

End to end testing a web application with ClojureGerard Klijs

Event machinealmeidaricardo

RESTEasy Reactive: Why should you care? | DevNation Tech TalkRed Hat Developers

Let the alpakka pull your streamEnno Runne

Integrating microservices with apache camel on kubernetesClaus Ibsen

Ractor's speed is not light-speedSATOSHI TAGOMORI

How to manage large amounts of data with akka streamsIgor Mielientiev

Stateful stream processing with kafka and samzaGeorge Li

Introduction Apache KafkaJoe Stein

Reactive database access with Slick3takezoe

Lessons from managing a Pulsar cluster (Nutanix)StreamNative

Samza at LinkedIn: Taking Stream Processing to the Next LevelMartin Kleppmann

Kafka At Scale in the Cloudconfluent

Apache pulsarKotireddy Sareddy

Apache samzaHumberto Streb

pgWALSyncRumman Iftekhar

Docker and FluentdN Masahiro

La actualidad más candente (20)

Deep Dive Into Kafka Streams (and the Distributed Stream Processing Engine) (...

Snabb Switch: Riding the HPC wave to simpler, better network appliances (FOSD...

Developing Secure Scala Applications With Fortify For Scala

End to end testing a web application with Clojure

Event machine

RESTEasy Reactive: Why should you care? | DevNation Tech Talk

Let the alpakka pull your stream

Integrating microservices with apache camel on kubernetes

Ractor's speed is not light-speed

How to manage large amounts of data with akka streams

Stateful stream processing with kafka and samza

Introduction Apache Kafka

Reactive database access with Slick3

Lessons from managing a Pulsar cluster (Nutanix)

Samza at LinkedIn: Taking Stream Processing to the Next Level

Kafka At Scale in the Cloud

Apache pulsar

Apache samza

pgWALSync

Docker and Fluentd

Similar a Rust with-kafka-07-02-2019

Data Engineer’s Lunch #45: Apache LivyAnant Corporation

Kafka ExplainatonNguyenChiHoangMinh

開放原始碼 Ch1.2 intro - oss - apahce foundry (ver 2.0)My own sweet home!

Introduction to ScalaMohammad Hossein Rimaz

.NET RDF APIsAndrei Iacob

Kafka Summit SF 2017 - Kafka and the Polyglot Programmerconfluent

Apache Kafka 0.8 basic training - VerisignMichael Noll

Polyglot Plugin ProgrammingAtlassian

What's New in AWS Serverless and ContainersAmazon Web Services

Lecture 2_ Intro to laravel.pptxSaziaRahman

7 Apache Process Cloudstack Developer DayKimihiko Kitase

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...Athens Big Data

Building a Messaging Solutions for OVHcloud with Apache Pulsar_Pierre ZembStreamNative

Being Ready for Apache Kafka - Apache: Big Data Europe 2015Michael Noll

Graph ql subscriptions through the looking glassGerard Klijs

Kafka for DBAsGwen (Chen) Shapira

Apache kafkaKumar Shivam

From java to scala at crowd mixStefano Galarraga

Why scala for data scienceGuglielmo Iozzia

A Jupyter kernel for Scala and Apache Spark.pdfLuciano Resende

Similar a Rust with-kafka-07-02-2019 (20)

Data Engineer’s Lunch #45: Apache Livy

Kafka Explainaton

開放原始碼 Ch1.2 intro - oss - apahce foundry (ver 2.0)

Introduction to Scala

.NET RDF APIs

Kafka Summit SF 2017 - Kafka and the Polyglot Programmer

Apache Kafka 0.8 basic training - Verisign

Polyglot Plugin Programming

What's New in AWS Serverless and Containers

Lecture 2_ Intro to laravel.pptx

7 Apache Process Cloudstack Developer Day

14th Athens Big Data Meetup - Landoop Workshop - Apache Kafka Entering The St...

Building a Messaging Solutions for OVHcloud with Apache Pulsar_Pierre Zemb

Being Ready for Apache Kafka - Apache: Big Data Europe 2015

Graph ql subscriptions through the looking glass

Kafka for DBAs

Apache kafka

From java to scala at crowd mix

Why scala for data science

A Jupyter kernel for Scala and Apache Spark.pdf

Último

INGKA DIGITAL: Linked Metadata by DesignNeo4j

AI/ML Infra Meetup | Reducing Prefill for LLM Serving in RAGAlluxio, Inc.

10 Essential Software Testing Tools You Need to Know About.pdfkalichargn70th171

JustNaik Solution Deck (stage bus sector)Max Lee

KLARNA - Language Models and Knowledge Graphs: A Systems ApproachNeo4j

Tree in the Forest - Managing Details in BDD Scenarios (live2test 2024)Gáspár Nagy

How to pick right visual testing tool.pdfTestgrid.io

AI/ML Infra Meetup | ML explainability in MichelangeloAlluxio, Inc.

Implementing KPIs and Right Metrics for Agile Delivery Teams.pdfVictor Lopez

5 Reasons Driving Warehouse Management Systems DemandCanary7-Warehouse Management System

A Python-based approach to data loading in TM1 - Using Airflow as an ETL for TM1KnowledgeSeed

How to install and activate eGrabber JobGrabbereGrabber

APVP,apvp apvp High quality supplier safe spot transport, 98% purityamy56318795

The Impact of PLM Software on Fashion ProductionWave PLM

GraphSummit Stockholm - Neo4j - Knowledge Graphs and Product UpdatesNeo4j

IT Software Development Resume, Vaibhav jha 2024vaibhav130304

OpenChain @ LF Japan Executive Briefing - May 2024Shane Coughlan

Secure Software Ecosystem Teqnation 2024Soroosh Khodami

Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdfQ-Advise

SQL Injection Introduction and PreventionMohammed Fazuluddin

Rust with-kafka-07-02-2019

1. www.openweb.nl Kafka + Schema Registry in Rust By Gerard Klijs Developer at Openweb gerard@openweb.nl

2. www.openweb.nl Summary ▰ About me ▰ Short intro to Kafka ▰ Short intro to the Confluent Schema Registry ▰ Rust support for Kafka ▰ Using rdkafka ▰ Turning the Schema Registry code into a Crate 222

3. www.openweb.nl About me ▰ About 7 years working experience as developer, mainly java, mainly web. ▰ Some pet projects in Clojure(script) ▰ Became especially interested in rust after reading Rust for Clojurists ▰ Less than a year since I worked through the book, no prior C or C++ experience. 33

4. www.openweb.nl Short introduction to Apache Kafka ▰ A distributed streaming platform. ▰ Developed by LinkedIn, Open sourced in 2011. ▰ Main parts of the platform run on the JVM. ▰ Librdkafka is a C/C++ library implementing both the producer and the consumer. ▰ Very scalable. ▰ Large ecosystem. 44

5. www.openweb.nl Short introduction to Schema Registry ▰ Part of the Confluent Platform, from the company created in 2014 by the creators. ▰ Uses Apache Avro for data serialization. ▰ Schema’s are stored in Kafka, each message contains a key. ▰ The Schema Registry itself is open source. ▰ Versions the data and at the same time makes the messages smaller. 55

6. www.openweb.nl Rust support for Kafka ▰ rdkafka: Rust wrapper for librdkafka. ▰ kafka: Rust client for Apache Kafka. ▰ No support confluent platform, but several crates working with Apache Avro. 66

7. www.openweb.nl Using rdkafka ▰ Started with the provided examples from rdkafka and get it to work. ▰ Tried several avro and serde libraries till I found a match that worked. 77

8. www.openweb.nl Turning the Schema Registry code into a Crate ▰ Frame part which is generic and missing in current crates. ▰ Create new project and pasted in code. ▰ Add unit tests and documentation. ▰ Some CI, code coverage and tags. ▰ Make it more generic. ▰ Improve error handling. ▰ Use as lib in originating project. ▰ Publish as schema_registry_converter to crates.io. 88

9. www.openweb.nl Questions?

Rust with-kafka-07-02-2019

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Rust with-kafka-07-02-2019

Similar a Rust with-kafka-07-02-2019 (20)

Último

Último (20)

Rust with-kafka-07-02-2019