Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Flink Forward San Francisco 2019: The Trade Desk's Year in Flink - Jonathan Miles

215 visualizaciones

Publicado el

The Trade Desk's Year in Flink
At advertising technology leader, The Trade Desk, we built a data pipeline with three distinct large-scale products using Flink. The keynote gives you a peek into our journey, the lessons learned and offers some hard-won tips from the trenches. Most jobs were surprisingly simple to build. However, we'll deep-dive into one particularly challenging Flink job where we learned how to synchronise/union multiple streams, both in terms of asymmetric throughput and differing lateness/time.

Publicado en: Tecnología
  • Sé el primero en comentar

Flink Forward San Francisco 2019: The Trade Desk's Year in Flink - Jonathan Miles

  1. 1. The Trade Desk’s Year with Flink Jonathan Miles
  2. 2. The Crux of (or Intro to) AdTech Publishers Ad Exchange Advertisers and Brands Real-Time Bidding How to be paid for content? How to get value and reach?
  3. 3. The Crux of (or Intro to) AdTech Publishers Ad Exchange Advertisers and Brands Real-Time Bidding How to be paid for content? How to get value and reach?
  4. 4. The Trade Desk Data Systems Bidders Ads Won Vertica MS SQL Data Pipeline Kafka + Flink <Ververica Platform> HBase S3 “INCOMING LOGS” Ads Bid On Batch Reports (Excel / TSV / CSV) Interactive Reports (UI) Transform Data Lake Aggregate ETL Stream Processing
  5. 5. • Apache Flink - High-level/abstract, lower testing surface area - Most of the distributed systems work is done for you - Common sysadmin challenges • Ververica Platform - Orchestration with App Manager on Kubernetes • Ververica Support - When embracing open-source, The Trade Desk needed "enterprise grade“ software (revenues) - Lean on Ververica, allowing us to work on our business - Gateway to Flink community Flink Highlights
  6. 6. Overview of Flink jobs at TTD Data Pipeline Kafka + Flink <Ververica Platform> HBase S3Data Lake Map + Union + Aggregate SQL Map + Transform
  7. 7. • Data Lake - Five jobs, one for each input topic - Scaled independently - Internal Data Science - … also advertising partners • Map + Transform - Two jobs, for different topics - Single input and output to Hbase - More Data Science use cases Anatomy: simpler jobs Data Pipeline HBase S3Data Lake Map + Transform Kafka + Flink <Ververica Platform>
  8. 8. • Map + Aggregate (Real-Time Insights) - Aggregate disparate input topics - Brands can query DB interactively - Stream processing • Reasonable latency • Reduce complexity of management - Synchronise five input topics with • … different data rates • … different time characteristics Anatomy: challenging jobs Data Pipeline Map + Union + Aggregate SQL Kafka + Flink <Ververica Platform>
  9. 9. Anatomy: challenging jobs Data N rows/sec Data Pre-process: - Union - Aggregate Servers Database Stream Processor
  10. 10. Anatomy: challenging jobs Rollup by… Dim1 Dim2 Dim3 Dim4 Dim5 L1 L2 N rows/sec Database Kafka Two Flink applications
  11. 11. Anatomy: challenging jobs Synchronising input streams is hard! Rollup by… Dim1 Dim2 Dim3 Dim4 Dim5 L1 L2 N rows/sec Database Kafka +/- Throughput +/- Time
  12. 12. Anatomy: challenging jobs Beware of hotspots! Synchronising input streams is hard! Rollup by… Dim1 Dim2 Dim3 Dim4 Dim5 L1 L2 N rows/sec Database Kafka
  13. 13. Anatomy: challenging jobs Hotspots… … caused global slow-down during catchup.
  14. 14. Tips: synchronizing input streams is hard Key T I1C1..n I2C1..n I3C1..n I4C1..n I5C1..n K1 T1 Value Value Value Value Value K2 T2 Value Value Value Value Value SQL Table SELECT SUM(I1.C1), SUM(I1.C2)... FROM Input1 I1 JOIN Input2 I2 ON (I2.key=I1.key…) JOIN Input3 I3 ON (I3.key=I1.key…) JOIN Input4 I4 ON (I4.key=I1.key…) JOIN Input5 I5 ON (I5.key=I1.key…) WHERE Input1.key = @key GROUP BY Key, Window Synchronising input streams is hard! L1
  15. 15. Tips: synchronizing input streams is hard THREE CHALLENGES: • Time synchronisation • Throughput synchronisation • Late data Synchronising input streams is hard! L1
  16. 16. Tips: synchronizing input streams is hard Time synchronisation Map +1 hour +/- 15 mins +/- 15 mins +/- 15 mins +/- 15 mins +1 hour Source Source Source Source Source Map Map Map Map Union Window Aggregate Event Time == Slowest Input Buffers to State Store (slowest input)
  17. 17. Tips: synchronizing input streams is hard BoundedOutOfOrdernessTimestampExtractor Time synchronisation (out of order) Map +1 hour +/- 15 mins +/- 15 mins +/- 15 mins +/- 15 mins Source Source Source Source Source Map Map Map Map +1 hour Union Window Aggregate Out of Order == Late Data Dropped records (still slowest)
  18. 18. Tips: synchronizing input streams is hard Time synchronisation (different latency) Map +15 mins Separate DAG +1 hour Source Source Source Source Source Map Map Map Map Union Window Aggregate Union Window Aggregate+1 hour +/- 15 mins +/- 15 mins +/- 15 mins +/- 15 mins (two writes) Less Data in State Store Multiple DAGs == Flexible
  19. 19. Tips: synchronizing input streams is hard THREE CHALLENGES: • Time synchronisation • Throughput synchronisation • Late data Synchronising input streams is hard! L1
  20. 20. Tips: synchronizing input streams is hard Throughput balancing Map >1% rps <1% rps 6% rps 8% rps 84% rps Source Source Source Source Source Map Map Map Map Union Window Aggregate Union Window Aggregate Constipation Unbalanced Reads == Event Time Skew Everything Buffers to State Store
  21. 21. Tips: synchronizing input streams is hard Job Manager Task Manager Task Manager Task Manager Task Manager Task Manager
  22. 22. Tips: synchronizing input streams is hard Job Manager Slot Slot Slot Slot Slot One Slot Per Task Manager Entire Pipeline Inside Every Slot … reading all inputs
  23. 23. Tips: synchronizing input streams is hard Throughput balancing Map Source Source Source Source Source Map Map Map Map Union Window Aggregate Union Window Aggregate Every Slot Has These Two DAGs (e.g.) >1% rps <1% rps 6% rps 8% rps 84% rps 300 instances of every node in the DAG… … leads to some being starved
  24. 24. Tips: synchronizing input streams is hard Slot Sharing Groups Low throughput Medium throughput High throughput 2 slots 14 slots 90 slots 194 slots 300 total slots Window Aggregate 1. Isolation 2. Reads in proportion
  25. 25. Tips: synchronizing input streams is hard Job Manager S S S S S S S S S S S S S S SS S S S S S S S S S S SS S S S S SSS S SS S S S S SSS S SS … ongoing work to make this easier out of the box Cluster of slots, dedicated to parts of DAG
  26. 26. Tips: synchronising input streams is hard Shared Slots (default) Slot Groups Control rate of consumption Even distribution of resource Easy attribution of effects
  27. 27. Tips: synchronizing input streams is hard THREE CHALLENGES: • Time synchronisation • Throughput synchronisation • Late data Synchronising input streams is hard! L1
  28. 28. Tips: synchronizing input streams is hard Map Source Source Source Source Source Map Map Map Map Union Window Aggregate Union Window Aggregate Late Data Late path Late path +15 mins +1 hour Event Time (Processing Time)
  29. 29. • Flink’s flexibility and demanding use-case • Unbalanced time - BoundedOutOfOrdernessTimestampExtractor (+/- 15 minutes) - Separate DAG in same job (+1 hour) - Actual late data shunted to separate path • Unbalanced throughput, Flink won't balance reads for you - TTD: Slot Model → Slot Groups - Future/community: Event time synchronization across sources (FLINK-10886) • Constipation and laxative - Time-based distribution of individual topic - Processing-time eviction • Real-time data is okay, catchup is main problem Tips: synchronizing input streams is hard
  30. 30. • Why was Flink a good choice? - Generalised framework, well defined semantics - Most distributed systems problems under control - Flexibility • CDC project - Generalised/democratised access to snapshots of database tables • What Flink features are we excited about? - Event time synchronization across sources (FLINK-10886) - Ability to take checkpoints during high backpressure (alignment sentinels stuck in same queue as messages) • We can dream… - Generalised solution to hotspots problem Future
  31. 31. • Diagnostic tool, flink-diag.py - GitHub? - Come talk to us? • We’re hiring - Come change the world of advertising with us! • Questions? - Many people from The Trade Desk attending today - Feel free to come talk to us! Plug
  32. 32. Thank you

×