Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Neha Narkhede | Kafka Summit London 2019 Keynote | Event Streaming: Our Cloud-Native Journey Lessons

427 visualizaciones

Publicado el

More info: https://cnfl.io/cloud-native-experience-for-kafka-in-cloud | Neha Narkhede is co-founder and CTO at Confluent, a company backing the popular Apache Kafka messaging system. Prior to founding Confluent, Neha led streams infrastructure at LinkedIn, where she was responsible for LinkedIn’s streaming infrastructure built on top of Apache Kafka and Apache Samza. She is one of the initial authors of Apache Kafka and a committer and PMC member on the project.

Publicado en: Tecnología
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Neha Narkhede | Kafka Summit London 2019 Keynote | Event Streaming: Our Cloud-Native Journey Lessons

  1. 1. 1 Event Streaming Lessons from our cloud-native journey Neha Narkhede Co-Creator, Apache Kafka Co-Founder & CTO, Confluent
  2. 2. 22 The New Business Reality Past Technology was a support function Innovation required for growth Running the business on yesterday’s data was “good enough” Today Technology is the business Innovation required for survival Yesterday’s data = failure. Modern, real-time data infrastructure is required.
  3. 3. 3 The Rise Of Event Streaming
  4. 4. 4 The Rise Of Event Streaming
  5. 5. 5 The Rise Of Event Streaming 60%Fortune 100 Companies Using Apache Kafka
  6. 6. 66 What We Are Building With Event Streaming Online Dating Data-driven Video Games Driver-rider match Digital Publishing Pipelines Digital Banking Assistants
  7. 7. 77 The Central Challenge Connecting applications with data ETL What happened in the world Messaging What is happening in the world
  8. 8. 8
  9. 9. 9 ETL/Data Integration Messaging Batch Expensive Time Consuming Difficult to Scale No Persistence After Consumption No Replay Highly Scalable Durable Persistent Ordered Fast (Low Latency) What is happening in the world What happened in the world
  10. 10. 1010 Highly Scalable Durable Persistent Maintains Order ETL/Data Integration MessagingETL/Data Integration MessagingMessaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay Fast (Low Latency) What happened in the world What is happening in the world Highly Scalable Durable Persistent Ordered Fast (Low Latency) Event Streaming Paradigm
  11. 11. 1111 Highly Scalable Durable Persistent Maintains Order ETL/Data Integration MessagingETL/Data Integration MessagingMessaging Batch Expensive Time Consuming Difficult to Scale No Persistence Data Loss No Replay Fast (Low Latency)Highly Scalable Durable Persistent Ordered Fast (Low Latency) Event Streaming Paradigm What is contextually happening in the world
  12. 12. 1212 Event-Driven App (Location Tracking) Only Real-Time Events Messaging Queues and Event Streaming Platforms can do this Contextual Event-Driven App (ETA) Real-Time combined with stored data Only Event Streaming Platforms can do this Where is my driver? When will my driver get here? Where is my driver? When will my driver get here? Why Combine Real-time With Historical Context? 2 min
  13. 13. 13 Event Streaming Paradigm Highly Scalable Durable Persistent Maintains Order Fast (Low Latency) Event Streaming Paradigm To rethink data as not stored records or transient messages, but instead as a continually updating stream of events
  14. 14. 1414 Event Streaming Paradigm Highly Scalable Durable Persistent Maintains Order Fast (Low Latency) Event Streaming Paradigm
  15. 15. 15 Real-Time Inventory Real-Time Fraud Detection Real-Time Customer 360 Machine Learning Models Real-Time Data Transformation ... Contextual Event-Driven Applications Universal Event Pipeline Data Stores Logs 3rd Party Apps Custom Apps/Microservices STREAMS CONNECT CLIENTS
  16. 16. 1616 The Path To Success With Event Streaming Based on our first-hand experience
  17. 17. 1717 Our Path With Confluent Cloud Today 3 GB/s 2018 Start 3 Months 5 MB/s 6 Months 15 MB/s 9 Months 500 MB/s 12 Months 1 GB/s 15 Months 1.5 GB/s
  18. 18. 1818 Today 3 GB/s Late 2019 25 GB/s 2020 100 GB/s Our Path With Confluent Cloud
  19. 19. 1919 Top 5 Lessons Learned Things we learned that everyone can benefit from
  20. 20. 2020 01 Lesson There Is No Cloud-Native Easy Button 20
  21. 21. 21
  22. 22. 22
  23. 23. 23 On Premises topic_name_01 topic_name_02 topic_name_03 Confluent Replicator Schema Registry topic_name_01 topic_name_02 topic_name_03 Schema Registry Public Cloud
  24. 24. 2424 02 Lesson Schemas Are Service APIs For Event Streaming
  25. 25. 25 A Schema Is A Contract, An API Between Services 1 2 3 4 5 6 7 8 9 { “name”: “received_at”, “date”: “1448928000”, “type”: { “type”: “string”, “avro.jaja.string”: “String”, }, “doc”: “FIT’s time of assignment”, }, 1 2 3 4 5 6 7 8 9 { “name”: “received_at”, “date”: “Dec 1, 2015”, “type”: { “type”: “string”, “avro.jaja.string”: “String”, }, “doc”: “FIT’s time of assignment”, },
  26. 26. 2626 Just Like APIs, Schemas Need To Be 02 Versioned 03 Validated + Enforced 01 Discoverable + Searchable
  27. 27. 27
  28. 28. 28
  29. 29. 29
  30. 30. 3030 03 Lesson Stream Processing Is The Best Way To Observe Stream Processing
  31. 31. 31
  32. 32. 32
  33. 33. 33
  34. 34. 34
  35. 35. 3535 04 Lesson Elasticity Is A Cloud-Native Imperative
  36. 36. 3636 Making Kafka Elastic In The Cloud 01 Instant access to capacity with seamless scalability 02 Work around the myriad limits on cloud infrastructure 03 Balance traffic in a continuous and intelligent fashion 04 Move data out of local storage
  37. 37. 3737 05 Lesson Life Is Better Without Cluster Sizing
  38. 38. 3838 Cloud infrastructure costs are confusing and hard to understand Overprovision OR Outage? Pick One Understanding cost attribution for a service in the cloud is even harder Sizing is a moving target, the available options keep changing
  39. 39. 3939 The Cost Of Cloud Complexity per year if you picked i3.2xlarge on- demand instance vs r5.xlarge reserved instance +$250k per year if you picked Private Link instead of VPC Peering for networking +$600k per year if you picked SSD EBS storage VS optimized EBS +$700k Kafka workload: 100 MB/s writes and reads 30 day retention This adds up to over $1M per year!
  40. 40. 4040 03 Stream processing is the best way to observe stream processing 02 Schemas are service APIs for Event Streaming 01 There is no cloud- native easy button 05 Life is better without cluster sizing 04 Elasticity is a cloud- native imperative Top 5 Lessons Learned
  41. 41. 41 Available Today In Confluent Cloud ● 0 to 100MBps: Self-service, no need to size or provision clusters ● 100 MBps to 10s of GBps: scale with provisioned capacity Scale elastically to 100MBps and down in seconds ● Start at less than $50/month ● No minimums or commitment Pay only for what you actually stream ● The most popular Kafka tools available as a fully- managed service Fully-managed Schema Registry, KSQL, S3 Connector (Preview)
  42. 42. 4242
  43. 43. 4343
  44. 44. 4444
  45. 45. 4545 2 Easy Choices To Get Started Confluent Platform All features are free on 1 node Confluent Cloud Pay as you go: Start at less than $50 / month
  46. 46. 4646 What If We All Succeed With Event Streaming?
  47. 47. 4747 Transportation Without Event Streaming No knowledge of driver arrival Call company for driver availability No sensor diagnostics
  48. 48. 4848 Transportation Real-time ETA Real-time driver-rider match Real-time sensor diagnostics With Event Streaming
  49. 49. 4949 Banking Out of sync account Information Batch fraud checks Batch internal risk reporting Without Event Streaming
  50. 50. 5050 Banking With Event Streaming Real-time omni-channel account updates Real-time fraud detection Real-time internal risk calculations
  51. 51. 5151 Retail Customer receives “out of stock” email after ordering Retailers receive sales reports every 1-2 days No ability to upsell through personalization Without Event Streaming
  52. 52. 5252 Retail Real-time inventory across web and store Real-time point of sales reporting Real-time, contextual recommendations With Event Streaming
  53. 53. 5353 Event Streaming Paradigm Is The Future Of Data Infrastructure as code Data as a continuous stream of events Future of the datacenter Future of data Cloud Event Streaming
  54. 54. 54

×