Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Zero to Insights
Real time analytics with Kafka, C*, and Spark
Peter Bakas
Peter Bakas | @peter_bakas
@ Netflix : Cloud Platform Engineering - Event and Data Pipelines
@ Ooyala : Analytics, Discove...
Let’s get down to business
Netflix is a logging company
that occasionally streams video
● 450 billion events per day
● 8 million events & 17 GB per second during
peak
● Hundreds of event types
By the Numbers
Publish, Collect, Process, Aggregate & Move Data
@ Cloud Scale
What does it take to run @ Cloud Scale?
How did we get here?
EMR
Event
Producer
Chukwa
What are we supposed to do?
Event
Producer
Druid
EMR
Stream
Consumers
Kafka
Router
Suro
Event
Producer
Event
Producer
Druid
EMR
Stream
Consumers
Kafka
Router
Suro
Event
Producer
Where are we going?
Stream
Consumers
Router
EMR
Fronting
Kafka
Event
Producer
Druid
Consumer
Kafka
Keystone
Stream
Consumers
Router
EMR
Fronting
Kafka
Event
Producer
Druid
Consumer
Kafka
Keystone
Stream
Consumers
Router
EMR
Fronting
Kafka
Event
Producer
Druid
Consumer
Kafka
Keystone
Routing Service
++
Stream
Consumers
Router
EMR
Fronting
Kafka
Event
Producer
Druid
Consumer
Kafka
Keystone
Consumer
Kafka
Custom Apps
Real time processing
Consumer
Kafka
Custom Apps
Real time processing
Consumer
Kafka
Custom Apps
Real time processing
Consumer
Kafka
Custom Apps
Real time processing
Consumer
Kafka
Custom Apps
Real time processing
Fronting
Kafka
Ooyala’s experience
About Ooyala
Powering personalized
video experiences
across all screens.
● 5 billion events per day
● 1 billion videos per month
● 200 million unique users per month
● 130 countries
● 25% of U.S....
Where did it all start?
Precomputed Aggregates
What if we need more dynamic queries?
Why not just use C*?
What were the options?
100% Precomputation 100% Dynamic
Where we wanted to be
100% Precomputation 100% Dynamic
Partly dynamic
Our solution
API
loggersloggersloggersloggersloggers
loggersloggersloggersloggersingest
loggersloggersloggersloggersjob server
Delphi -...
Challenges
● Hiring
● Rapidly evolving ecosystem
● Enterprise Service for Enterprise Software
Challenges
Q&A time!
Obligatory...
Everyone is hiring
pbakas@netflix.com
Peter Bakas - Zero to Insights - Real time analytics with Kafka, C*, and Spark - NoSQL matters Dublin 2015
Peter Bakas - Zero to Insights - Real time analytics with Kafka, C*, and Spark - NoSQL matters Dublin 2015
Próxima SlideShare
Cargando en…5
×

Peter Bakas - Zero to Insights - Real time analytics with Kafka, C*, and Spark - NoSQL matters Dublin 2015

1.196 visualizaciones

Publicado el

In this talk, Peter will cover his experience using Spark, Cassandra & Kafka to build a real time analytics platform that processed billions events a day. He will cover the challenges in how to turn all those raw events into actionable insights. He will also cover scaling the platform across multiple regions, as well as across multiple cloud environments.

Publicado en: Datos y análisis
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

Peter Bakas - Zero to Insights - Real time analytics with Kafka, C*, and Spark - NoSQL matters Dublin 2015

  1. 1. Zero to Insights Real time analytics with Kafka, C*, and Spark Peter Bakas
  2. 2. Peter Bakas | @peter_bakas @ Netflix : Cloud Platform Engineering - Event and Data Pipelines @ Ooyala : Analytics, Discovery, Platform Engineering & Infrastructure @ Yahoo : Display Advertising, Behavioral Targeting, Payments @ PayPal : Site Engineering and Architecture @ Play : Advisor to various Startups (Data, Security, Containers) Who is this guy?
  3. 3. Let’s get down to business
  4. 4. Netflix is a logging company
  5. 5. that occasionally streams video
  6. 6. ● 450 billion events per day ● 8 million events & 17 GB per second during peak ● Hundreds of event types By the Numbers
  7. 7. Publish, Collect, Process, Aggregate & Move Data
  8. 8. @ Cloud Scale
  9. 9. What does it take to run @ Cloud Scale?
  10. 10. How did we get here?
  11. 11. EMR Event Producer Chukwa
  12. 12. What are we supposed to do?
  13. 13. Event Producer Druid EMR Stream Consumers Kafka Router Suro Event Producer
  14. 14. Event Producer Druid EMR Stream Consumers Kafka Router Suro Event Producer
  15. 15. Where are we going?
  16. 16. Stream Consumers Router EMR Fronting Kafka Event Producer Druid Consumer Kafka Keystone
  17. 17. Stream Consumers Router EMR Fronting Kafka Event Producer Druid Consumer Kafka Keystone
  18. 18. Stream Consumers Router EMR Fronting Kafka Event Producer Druid Consumer Kafka Keystone
  19. 19. Routing Service ++
  20. 20. Stream Consumers Router EMR Fronting Kafka Event Producer Druid Consumer Kafka Keystone
  21. 21. Consumer Kafka Custom Apps Real time processing
  22. 22. Consumer Kafka Custom Apps Real time processing
  23. 23. Consumer Kafka Custom Apps Real time processing
  24. 24. Consumer Kafka Custom Apps Real time processing
  25. 25. Consumer Kafka Custom Apps Real time processing Fronting Kafka
  26. 26. Ooyala’s experience
  27. 27. About Ooyala Powering personalized video experiences across all screens.
  28. 28. ● 5 billion events per day ● 1 billion videos per month ● 200 million unique users per month ● 130 countries ● 25% of U.S. online viewers watch video powered by Ooyala By the Numbers
  29. 29. Where did it all start?
  30. 30. Precomputed Aggregates
  31. 31. What if we need more dynamic queries?
  32. 32. Why not just use C*?
  33. 33. What were the options? 100% Precomputation 100% Dynamic
  34. 34. Where we wanted to be 100% Precomputation 100% Dynamic Partly dynamic
  35. 35. Our solution
  36. 36. API loggersloggersloggersloggersloggers loggersloggersloggersloggersingest loggersloggersloggersloggersjob server Delphi - Real time Analytics Kafka
  37. 37. Challenges
  38. 38. ● Hiring ● Rapidly evolving ecosystem ● Enterprise Service for Enterprise Software Challenges
  39. 39. Q&A time!
  40. 40. Obligatory... Everyone is hiring pbakas@netflix.com

×