What does Netflix, NTT and Rubicon Project have in common? Apache Druid.
Abstract: Netflix is a media services provider and uses Apache Druid to measure the customer experience of watching videos, anywhere in the world. They have significant investment of Apache Druid that helps them improve their services across the board. NTT is the fourth largest telecommunications company in the world. They use Druid for global traffic visibility for technical, economical and security use cases. Rubicon Project is one of the world’s largest digital advertising exchanges. To achieve accurate data calculations, get great analytical performance and extract intelligence fast. They use Druid as their foundation for their realtime analytics platform. In this presentation, we will discuss what their challenges were and why the moved to Apache Druid to meet their needs. Presenter: Rommel Garcia Bio: Rommel Garcia is currently the Director of Solutions Engineering at Imply, a company founded by the same people that created Druid. He is an author of the book Virtualizing Hadoop - How to install. Deploy, and Optimize Hadoop in Virtualized Infrastructure. In the last 10 years, he had worked on Data Management both on-prem and in the cloud, distributed systems, Big Data analytics, and hardware accelerated analytics using GPUs.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to What does Netflix, NTT and Rubicon Project have in common? Apache Druid.
Similar to What does Netflix, NTT and Rubicon Project have in common? Apache Druid. (20)
More from Rommel Garcia
Recently uploaded
Recently uploaded (20)
What does Netflix, NTT and Rubicon Project have in common? Apache Druid.
- 1. NETFLIX + NTT + RUBICON PROJECT USE CASES
- 2. WHO AM I ▸ Dir. of Solutions Engineering, Imply ▸ Author: Virtualizing Hadoop ▸ 11 years of experience in distributed systems, big data platforms, cloud computing
- 3. AGENDA ▸ The analytics challenges it solves ▸ Use Cases ▸ Architecture
- 5. APACHE DRUID ▸ New class of Operational Data Store ▸ Solves the following analytics challenges ▸ scale ▸ speed ▸ grain complexity ▸ high dimensionality ▸ concurrency ▸ freshness
- 6. USE CASES
- 8. SOME NUMBERS ▸ 160 Billion events per day ▸ 190 countries ▸ 300 million devices ▸ Thousands of Druid users ▸ 100s of Druid nodes
- 9. CHALLENGES ▸ RedShift was used as the backend for their ad-hoc aggregated analytics dashboard ▸ It’s slow ▸ Cannot support longer data retention ▸ Cannot support a lot of dimensions ▸ Apache Druid replaced RedShift
- 10. USE CASES ▸ AWS capacity planning ▸ Payment analysis ▸ Algorithm comparison ▸ Security ▸ Client performance / Quality of Experience(QoE)
- 13. HIGHLIGHTS ▸ 4th largest telecommunication in the world ▸ Provides high speed, high capacity IP communication services for ▸ Europe ▸ North and South America ▸ Asia ▸ Oceana
- 15. CHALLENGES ▸ Legacy netflow analytics system ▸ was a blackbox, difficult to troubleshoot and extend on both frontend and backend ▸ not scaling in a cost performant manner ▸ limited ad-hoc analysis
- 16. USE CASES ▸ Netflow analysis ▸ Capacity planning ▸ Traffic matrix analysis ▸ Inter-domain traffic analysis
- 19. SOME NUMBERS ▸ Thousands of external customers, publishers, DSPs across the globe ▸ Trillions of ad and bid requests quarterly ▸ 1K header bidding connections ▸ 40% growth in mobile ad spend Q2 2018 vs Q2 2017 ▸ 70% growth in video ad spend 1st half 2018 vs 1st half 2017 ▸ 300% growth of audio ad spend in Q2 2018
- 20. CHALLENGES ▸ Advertising traffic exponentially grew ▸ MySQL can only store 10% of data volume daily ▸ Scaling interactive analytics to a wide base of users was tough ▸ Cost, performance, timeliness
- 21. DRUID NUMBERS ▸ >2TB data per hour to Druid ▸ <500ms average response time ▸ >1 Trillion events per day ▸ Thousands of users across the globe
- 22. ARCHITECTURE
- 24. SEGMENT ▸ Highly optimized storage unit ▸ Highly compressed bitmap indexes ▸ 150MB - 700MB size ▸ Determines parallelism ▸ Read in memory ▸ No contentions between read and writes ▸ 10x - 75x storage space savings
- 25. DATA MODEL ▸ Roll ups ▸ Approximation algorithms ▸ Segment granularity ▸ Query granularity ▸ Metrics ▸ Bitmap type (concise vs roaring)
- 26. QUERY OPTIMIZATION ▸ Threads ▸ Heap ▸ Horizontal scaling ▸ topN vs groupby ▸ datasketches ▸ splitting data sources for targeted queries
- 27. JOIN THE COMMUNITY Druid community site (current): http://druid.io/ Druid community site (new): https://druid.apache.org/ Imply distribution: https://imply.io/get-started
- 28. TRY THIS AT HOME