Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
1Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Introducing Kafka Streams
Michael Noll
michael@co...
2Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
3Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
4Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
5Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
6Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
7Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Stream processing in Real Life™
…before Kafka Str...
8Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016Abandon all hope, ye who enter here.
9Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
How did this… (#machines == 1)
scala> val input =...
10Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
…turn into stuff like this (#machines > 1)
11Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
12Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
13Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
14Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016Taken at a session at ApacheCon: Big Data, Hungar...
15Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Kafka Streams
stream processing made simple
16Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Kafka Streams
• Powerful yet easy-to use Java li...
17Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Kafka Streams
18Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Kafka Streams
19Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
What is Kafka Streams: Unix analogy
$ cat < in.t...
20Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
What is Kafka Streams: Java analogy
21Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
When to use Kafka Streams (as of Kafka 0.10)
Rec...
22Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Alright, can you show me some code now? 
KStrea...
23Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Alright, can you show me some code now? 
Startu...
24Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
API != everything
API, coding
Operations, debugg...
25Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
API != everything
API, coding
Operations, debugg...
26Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Kafka Streams outsources hard problems to Kafka
27Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
How do I install Kafka Streams?
<dependency>
<gr...
28Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
29Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Do I need to install a CLUSTER to run my apps?
•...
30Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
How do I package and deploy my apps? How do I …?
31Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
32Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
How do I package and deploy my apps? How do I …?...
33Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Kafka
concepts
34Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Kafka concepts
35Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Kafka concepts
36Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Kafka Streams
concepts
37Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Stream
38Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Processor topology
39Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Stream partitions and stream tasks
40Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Streams meet Tables
http://www.confluent.io/blog...
41Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Streams meet Tables – in the Kafka Streams DSL
a...
42Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Streams meet Tables – in the Kafka Streams DSL
a...
43Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Streams meet Tables – in the Kafka Streams DSL
a...
44Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Streams meet Tables – in the Kafka Streams DSL
a...
45Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Streams meet Tables – in the Kafka Streams DSL
46Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Streams meet Tables – in the Kafka Streams DSL
•...
47Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Streams meet Tables – in the Kafka Streams DSL
•...
48Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Streams meet Tables – in the Kafka Streams DSL
•...
49Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Streams meet Tables – in the Kafka Streams DSL
50Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Streams meet Tables – in the Kafka Streams DSL
51Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Kafka Streams
key features
52Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Key features in 0.10
• Native, 100%-compatible K...
53Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Native Kafka integration
• Reading data from Kaf...
54Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Native Kafka integration
• You can configure bot...
55Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Key features in 0.10
• Native, 100%-compatible K...
56Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Execution model
57Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Execution model
58Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Execution model
59Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Execution model
60Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Kafka Streams outsources hard problems to Kafka
61Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Key features in 0.10
• Native, 100%-compatible K...
62Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Stateful computations
• Stateful computations li...
63Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Execution model
State stores
(This is a bit simp...
64Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Remember?
65Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Execution model
State stores
(This is a bit simp...
66Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Execution model
State stores
(This is a bit simp...
67Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Execution model
State stores
(This is a bit simp...
68Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Kafka Streams outsources hard problems to Kafka
69Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Stateful computations
• Kafka Streams DSL: abstr...
70Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Stateful computations
• Use the low-level Proces...
71Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Key features in 0.10
• Native, 100%-compatible K...
72Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Time
73Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Time
74Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Time
• You configure the desired time semantics ...
75Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Key features in 0.10
• Native, 100%-compatible K...
76Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Windowing
TimeWindows.of(3000)
TimeWindows.of(30...
77Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Windowing use case: monitoring (1m/5m/15m
averag...
78Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Key features in 0.10
• Native, 100%-compatible K...
79Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Wrapping up
80Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Where to go from here?
• Kafka Streams is availa...
81Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Some of the things to come
• Exactly-once semant...
82Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
Want to contribute to Kafka and open source?
Joi...
Próxima SlideShare
Cargando en…5
×

Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

11.484 visualizaciones

Publicado el

Video recording: https://www.youtube.com/watch?v=o7zSLNiTZbA

Slides of my talk at Berlin Buzzwords in June 2016.


Abstract:
"In the past few years Apache Kafka has established itself as the world's most popular real-time, large-scale messaging system. It is used across a wide range of industries by thousands of companies such as Netflix, Cisco, PayPal, Twitter, and many others.

In this session I am introducing the audience to Kafka Streams, which is the latest addition to the Apache Kafka project. Kafka Streams is a stream processing library natively integrated with Kafka. It has a very low barrier to entry, easy operationalization, and a high-level DSL for writing stream processing applications. As such it is the most convenient yet scalable option to process and analyze data that is backed by Kafka. We will provide the audience with an overview of Kafka Streams including its design and API, typical use cases, code examples, and an outlook of its upcoming roadmap. We will also compare Kafka Streams' light-weight library approach with heavier, framework-based tools such as Apache Storm and Spark Streaming, which require you to understand and operate a whole different infrastructure for processing real-time data in Kafka."

Publicado en: Datos y análisis
  • DOWNLOAD FULL eBOOK INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookeBOOK Crime, eeBOOK Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    ¿Estás seguro?    No
    Tu mensaje aparecerá aquí
  • DOWNLOAD FULL BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... 1.DOWNLOAD FULL doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    ¿Estás seguro?    No
    Tu mensaje aparecerá aquí

Introducing Kafka Streams, the new stream processing library of Apache Kafka, Berlin Buzzwords 2016

  1. 1. 1Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Introducing Kafka Streams Michael Noll michael@confluent.io Berlin Buzzwords, June 06, 2016 miguno
  2. 2. 2Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
  3. 3. 3Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
  4. 4. 4Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
  5. 5. 5Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
  6. 6. 6Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
  7. 7. 7Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Stream processing in Real Life™ …before Kafka Streams …somewhat exaggerated …but perhaps not that much
  8. 8. 8Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016Abandon all hope, ye who enter here.
  9. 9. 9Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 How did this… (#machines == 1) scala> val input = (1 to 6).toSeq // Stateless computation scala> val doubled = input.map(_ * 2) Vector(2, 4, 6, 8, 10, 12) // Stateful computation scala> val sumOfOdds = input.filter(_ % 2 != 0).reduceLeft(_ + _) res2: Int = 9
  10. 10. 10Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 …turn into stuff like this (#machines > 1)
  11. 11. 11Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
  12. 12. 12Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
  13. 13. 13Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
  14. 14. 14Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016Taken at a session at ApacheCon: Big Data, Hungary, September 2015
  15. 15. 15Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Kafka Streams stream processing made simple
  16. 16. 16Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Kafka Streams • Powerful yet easy-to use Java library • Part of open source Apache Kafka, introduced in v0.10, May 2016 • Source code: https://github.com/apache/kafka/tree/trunk/streams • Build your own stream processing applications that are • highly scalable • fault-tolerant • distributed • stateful • able to handle late-arriving, out-of-order data • <more on this later>
  17. 17. 17Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Kafka Streams
  18. 18. 18Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Kafka Streams
  19. 19. 19Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 What is Kafka Streams: Unix analogy $ cat < in.txt | grep “apache” | tr a-z A-Z > out.txt Kafka Kafka Connect Kafka Streams
  20. 20. 20Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 What is Kafka Streams: Java analogy
  21. 21. 21Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 When to use Kafka Streams (as of Kafka 0.10) Recommended use cases • Application Development • “Fast Data” apps (small or big data) • Reactive and stateful applications • Linear streams • Event-driven systems • Continuous transformations • Continuous queries • Microservices Questionable use cases • Data Science / Data Engineering • “Heavy lifting” • Data mining • Non-linear, branching streams (graphs) • Machine learning, number crunching • What you’d do in a data warehouse
  22. 22. 22Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Alright, can you show me some code now?  KStream<Integer, Integer> input = builder.stream(“numbers-topic”); // Stateless computation KStream<Integer, Integer> doubled = input.mapValues(v -> v * 2); // Stateful computation KTable<Integer, Integer> sumOfOdds = input .filter((k,v) -> v % 2 != 0) .selectKey((k, v) -> 1) .reduceByKey((v1, v2) -> v1 + v2, ”sum-of-odds"); • API option 1: Kafka Streams DSL (declarative)
  23. 23. 23Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Alright, can you show me some code now?  Startup Process a record Periodic action Shutdown • API option 2: low-level Processor API (imperative)
  24. 24. 24Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 API != everything API, coding Operations, debugging, … “Full stack” evaluation
  25. 25. 25Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 API != everything API, coding Operations, debugging, … “Full stack” evaluation
  26. 26. 26Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Kafka Streams outsources hard problems to Kafka
  27. 27. 27Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 How do I install Kafka Streams? <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-streams</artifactId> <version>0.10.0.0</version> </dependency> • There is and there should be no “install”. • It’s a library. Add it to your app like any other library.
  28. 28. 28Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
  29. 29. 29Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Do I need to install a CLUSTER to run my apps? • No, you don’t. Kafka Streams allows you to stay lean and lightweight. • Unlearn bad habits: “do cool stuff with data != must have cluster” Ok. Ok. Ok. Ok.
  30. 30. 30Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 How do I package and deploy my apps? How do I …?
  31. 31. 31Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016
  32. 32. 32Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 How do I package and deploy my apps? How do I …? • Whatever works for you. Stick to what you/your company think is the best way. • Why? Because an app that uses Kafka Streams is…a normal Java app. • Your Ops/SRE/InfoSec teams may finally start to love not hate you.
  33. 33. 33Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Kafka concepts
  34. 34. 34Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Kafka concepts
  35. 35. 35Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Kafka concepts
  36. 36. 36Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Kafka Streams concepts
  37. 37. 37Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Stream
  38. 38. 38Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Processor topology
  39. 39. 39Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Stream partitions and stream tasks
  40. 40. 40Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Streams meet Tables http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simpl http://docs.confluent.io/3.0.0/streams/concepts.html#duality-of-streams-and-tables
  41. 41. 41Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Streams meet Tables – in the Kafka Streams DSL alice 2 bob 10 alice 3 time “Alice clicked 2 times.” “Alice clicked 2 times.”KTable = interprets data as changelog stream ~ is a continuously updated materialized view KStream = interprets data as record stream
  42. 42. 42Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Streams meet Tables – in the Kafka Streams DSL alice 2 bob 10 alice 3 time “Alice clicked 2+3 = 5 times.” “Alice clicked 2 3 times.”KTable = interprets data as changelog stream ~ is a continuously updated materialized view KStream = interprets data as record stream
  43. 43. 43Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Streams meet Tables – in the Kafka Streams DSL alice eggs bob lettuce alice milk alice paris bob zurich alice berlin KStream KTable time “Alice bought eggs.” “Alice is currently in Paris.” User purchase records User profile/location information
  44. 44. 44Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Streams meet Tables – in the Kafka Streams DSL alice eggs bob lettuce alice milk alice paris bob zurich alice berlin KStream KTable User purchase records User profile/location information time “Alice bought eggs and milk.” “Alice is currently in Paris Berlin.” Show me ALL values for a key Show me the LATEST value for a key
  45. 45. 45Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Streams meet Tables – in the Kafka Streams DSL
  46. 46. 46Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Streams meet Tables – in the Kafka Streams DSL • JOIN example: compute user clicks by region via KStream.leftJoin(KTable)
  47. 47. 47Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Streams meet Tables – in the Kafka Streams DSL • JOIN example: compute user clicks by region via KStream.leftJoin(KTable) Even simpler in Scala because, unlike Java, it natively supports tuples:
  48. 48. 48Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Streams meet Tables – in the Kafka Streams DSL • JOIN example: compute user clicks by region via KStream.leftJoin(KTable) alice 13 bob 5Input KStream alice (europe, 13) bob (europe, 5)leftJoin() w/ KTable KStream 13europe 5europemap() KStream KTablereduceByKey(_ + _) 13europe …… …… 18europe …… ……
  49. 49. 49Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Streams meet Tables – in the Kafka Streams DSL
  50. 50. 50Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Streams meet Tables – in the Kafka Streams DSL
  51. 51. 51Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Kafka Streams key features
  52. 52. 52Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Key features in 0.10 • Native, 100%-compatible Kafka integration • Also inherits Kafka’s security model, e.g. to encrypt data-in-transit • Uses Kafka as its internal messaging layer, too
  53. 53. 53Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Native Kafka integration • Reading data from Kafka • Writing data to Kafka
  54. 54. 54Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Native Kafka integration • You can configure both Kafka Streams plus the underlying Kafka clients
  55. 55. 55Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Key features in 0.10 • Native, 100%-compatible Kafka integration • Also inherits Kafka’s security model, e.g. to encrypt data-in-transit • Uses Kafka as its internal messaging layer, too • Highly scalable • Fault-tolerant • Elastic
  56. 56. 56Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Execution model
  57. 57. 57Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Execution model
  58. 58. 58Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Execution model
  59. 59. 59Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Execution model
  60. 60. 60Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Kafka Streams outsources hard problems to Kafka
  61. 61. 61Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Key features in 0.10 • Native, 100%-compatible Kafka integration • Also inherits Kafka’s security model, e.g. to encrypt data-in-transit • Uses Kafka as its internal messaging layer, too • Highly scalable • Fault-tolerant • Elastic • Stateful and stateless computations (e.g. joins, aggregations)
  62. 62. 62Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Stateful computations • Stateful computations like aggregations or joins require state • We already showed a join example in the previous slides. • Windowing a stream is stateful, too, but let’s ignore this for now. • State stores in Kafka Streams • Typically: key-value stores • Pluggable implementation: RocksDB (default), in-memory, your own … • State stores are per stream task for isolation (think: share-nothing) • State stores are local for best performance • State stores are replicated to Kafka for elasticity and for fault-tolerance
  63. 63. 63Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Execution model State stores (This is a bit simplified.)
  64. 64. 64Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Remember?
  65. 65. 65Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Execution model State stores (This is a bit simplified.) charlie 3 bob 1 alice 1 alice 2
  66. 66. 66Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Execution model State stores (This is a bit simplified.)
  67. 67. 67Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Execution model State stores (This is a bit simplified.) alice 1 alice 2
  68. 68. 68Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Kafka Streams outsources hard problems to Kafka
  69. 69. 69Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Stateful computations • Kafka Streams DSL: abstracts state stores away from you • Stateful operations include • count(), reduceByKey(), aggregate(), … • Low-level Processor API: direct access to state stores • Very flexible but more manual work for you
  70. 70. 70Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Stateful computations • Use the low-level Processor API to interact directly with state stores Get the store Use the store
  71. 71. 71Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Key features in 0.10 • Native, 100%-compatible Kafka integration • Also inherits Kafka’s security model, e.g. to encrypt data-in-transit • Uses Kafka as its internal messaging layer, too • Highly scalable • Fault-tolerant • Elastic • Stateful and stateless computations • Time model
  72. 72. 72Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Time
  73. 73. 73Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Time
  74. 74. 74Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Time • You configure the desired time semantics through timestamp extractors • Default extractor yields event-time semantics • Extracts embedded timestamps of Kafka messages (introduced in v0.10)
  75. 75. 75Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Key features in 0.10 • Native, 100%-compatible Kafka integration • Also inherits Kafka’s security model, e.g. to encrypt data-in-transit • Uses Kafka as its internal messaging layer, too • Highly scalable • Fault-tolerant • Elastic • Stateful and stateless computations • Time model • Windowing
  76. 76. 76Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Windowing TimeWindows.of(3000) TimeWindows.of(3000).advanceBy(1000)
  77. 77. 77Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Windowing use case: monitoring (1m/5m/15m averages) Confluent Control Center for Kafka
  78. 78. 78Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Key features in 0.10 • Native, 100%-compatible Kafka integration • Also inherits Kafka’s security model, e.g. to encrypt data-in-transit • Uses Kafka as its internal messaging layer, too • Highly scalable • Fault-tolerant • Elastic • Stateful and stateless computations • Time model • Windowing • Supports late-arriving and out-of-order data • Millisecond processing latency, no micro-batching • At-least-once processing guarantees (exactly-once is in the works)
  79. 79. 79Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Wrapping up
  80. 80. 80Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Where to go from here? • Kafka Streams is available in Apache Kafka 0.10 and Confluent Platform 3.0 • http://kafka.apache.org/ • http://www.confluent.io/download (free + enterprise versions, tar/zip/deb/rpm) • Kafka Streams demos at https://github.com/confluentinc/examples • Java 7, Java 8+ with lambdas, and Scala • WordCount, Joins, Avro integration, Top-N computation, Windowing, … • Apache Kafka documentation: http://kafka.apache.org/documentation.html • Confluent documentation: http://docs.confluent.io/3.0.0/streams/ • Quickstart, Concepts, Architecture, Developer Guide, FAQ • Join our bi-weekly Ask Me Anything sessions on Kafka Streams • Contact me at michael@confluent.io for details
  81. 81. 81Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Some of the things to come • Exactly-once semantics • Queriable state – tap into the state of your applications • SQL interface • Listen to and collaborate with the developer community • Your feedback counts a lot! Share it via users@kafka.apache.org Tomorrow’s keynote (09:30 AM) by Neha Narkhede, co-founder and CTO of Confluent “Application development and data in the emerging world of stream processing”
  82. 82. 82Introducing Kafka Streams, Michael G. Noll, Berlin Buzzwords, June 2016 Want to contribute to Kafka and open source? Join the Kafka community http://kafka.apache.org/ Questions, comments? Tweet with #bbuzz and /cc to @ConfluentInc …in a great team with the creators of Kafka? Confluent is hiring  http://confluent.io/

×