Whoops, The Numbers Are Wrong! Scaling Data Quality @ Netflix
•Descargar como PPTX, PDF•
7 recomendaciones•1,331 vistas
Netflix is a famously data-driven company. Data is used to make informed decisions on everything from content acquisition to content delivery, and everything in-between. As with any data-driven company, it’s critical that data used by the business is accurate. Or, at worst, that the business has visibility into potential quality issues as soon as they arise. But even in the most mature data warehouses, data quality can be hard. How can we ensure high quality in a cloud-based, internet-scale, modern big data warehouse employing a variety of data engineering technologies? In this talk, Michelle Ufford will share how the Data Engineering & Analytics team at Netflix is doing exactly that. We’ll kick things off with a quick overview of Netflix’s analytics environment, then dig into details of our data quality solution. We’ll cover what worked, what didn’t work so well, and what we plan to work on next. We’ll conclude with some tips and lessons learned for ensuring data quality on big data.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Whoops, The Numbers Are Wrong! Scaling Data Quality @ Netflix
Similar a Whoops, The Numbers Are Wrong! Scaling Data Quality @ Netflix (20)
Más de DataWorks Summit
Más de DataWorks Summit (20)
Último
Último (20)
Whoops, The Numbers Are Wrong! Scaling Data Quality @ Netflix
- 1. WHOOPS, THE NUMBERS ARE WRONG! SCALING DATA QUALITY @ MICHELLE UFFORD DATA ENGINEERING & ANALYTICS, NETFLIX HADOOP SUMMIT 2017
- 2. Overview.
- 3. The business. 20170612 100+ million members $6 billion on content 125+ million hours watched launched in 1997 every. day. $
- 4. Anytime. Anywhere.* 20170612 * Well, almost anywhere.
- 6. 300 terabyte DW writes 5 petabyte DW reads The data. 20170612 60+ petabyte data warehouse 700+ billion events written
- 7. 300 terabyte DW writes 5 petabyte DW reads The data. 20170612 60+ petabyte data warehouse 700+ billion events written
- 8. 20170612 data access AWS S3 Amazon Redshift data processing fast storage data viz METACA T data services events data operational data elastic storage Apache Pig Big Data Platform
- 11. Federated metastore & extensible data catalog
- 16. 20170612 Extended table attributes ● primary key(s) ● column types ● lifecycle ● audience ● “valid-thru” timestamp ● … and much more Metacat Federated Metastore
- 22. Write - Audit - Publish ETL pattern for high-quality big data jobs
- 27. 20170612 s3://…/utc_date=20170101/batchid=1483229855 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_f utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... ETL Pattern
- 28. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_f Quint o metric eval behavior result -------------------------------------------------- RowCount >= zero fail job RowCount >= prior value fail job NullCount normal dist warn job Quinto configuration utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... ETL Pattern
- 29. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_f Quint o metric eval behavior result -------------------------------------------------- RowCount >= zero fail job RowCount >= prior value fail job NullCount normal dist warn job utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 30. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_f Quint o metric eval behavior result -------------------------------------------------- RowCount >= zero fail job RowCount >= prior value fail job NullCount normal dist warn job utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 31. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric eval behavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 32. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric eval behavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 33. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric eval behavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 34. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric eval behavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 35. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric eval behavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 36. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric eval behavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 37. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric eval behavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job fail Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 38. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric eval behavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job fail Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 41. 20170612 s3://…/utc_date=20170101/batchid=1483229855 … s3://…/utc_date=20170611/batchid=1497226702 WAPStage-3: Publish s3://…/utc_date=20170612/batchid=1497312541 dw.my_table_f valid_thru_ts = 20170613 00:00:00 ETL Pattern
- 43. 20170612 Quinto evaluations ● intelligent recommendations ● multiple tiers of coverage ● configurable rules Jumpstarter. Python Library
- 44. WAP. Python Library Minimal requirements ● parameterized destination table
- 47. 20170612 What’s Next. ● additional Metacat statistics ● robust anomaly detection (RAD) ● complete migration for all prod tables
- 48. 20170612 Tips & Lessons Learned. ● Query-based solution may be “good enough” for many. ● Not all tables need quality coverage. ● One size rarely fits all tables. ● Build components, not “all-or-nothing” frameworks.
Notas del editor
- https://dataworkssummit.com/san-jose-2017/sessions/whoops-the-numbers-are-wrong-scaling-data-quality-netflix/ WHOOPS, THE NUMBERS ARE WRONG! SCALING DATA QUALITY @ NETFLIX Netflix is a famously data-driven company. Data is used to make informed decisions on everything from content acquisition to content delivery, and everything in-between. As with any data-driven company, it’s critical that data used by the business is accurate. Or, at worst, that the business has visibility into potential quality issues as soon as they arise. But even in the most mature data warehouses, data quality can be hard. How can we ensure high quality in a cloud-based, internet-scale, modern big data warehouse employing a variety of data engineering technologies? In this talk, Michelle Ufford will share how the Data Engineering & Analytics team at Netflix is doing exactly that. We’ll kick things off with a quick overview of Netflix’s analytics environment, then dig into details of our data quality solution. We’ll cover what worked, what didn’t work so well, and what we plan to work on next. We’ll conclude with some tips and lessons learned for ensuring data quality on big data. DETAILS This session is a (Intermediate) talk in our Data Processing and Warehousing track. It focuses on Apache Hadoop, Apache Hive, Apache Pig, Apache Spark and is geared towards Architect, Data Analyst, Developer / Engineer, Operations / IT audiences.
- 1500+ devices as of Q1 2017
- Goal is to provide behind-the-scenes look at how we’re approaching DQ. We’re sharing ideas, not code – no open-source announcement.
- That’s cool but sounds like a lot of work I need to: know what stats are available in Metacat know what quality templates exist figure out which ones I should use figure out a good configuration for each and do everything we just walked through in WAP
- Takes ~5 minutes to enable WAP with 108 audits on a new Spark job. Added 42 seconds to hourly processing time.
- It’s a combination of these solutions that allow us to scale not only the processing time but the engineering time too. Takes ~5 minutes to enable WAP with 108 audits on a new Spark job. Added 42 seconds to hourly processing time.
- Stats Cardinality Histograms Map keys RAD High cardinality dimensions Seasonality beyond week-over-week Atypical data distributions Reduce false positives
- Query-based solution Not efficient but much less complicated to implement Good place to start Works well for small-to-medium datasets and/or nightly batch ETL Transient, experimental, single-user Content vs. Streaming
- 2 main motivations Confidence Notify users when quality issues arise Only make data available after some basic validation Increase confidence for data consumers that data is good to use Efficiency Catch issues faster Less business impact Much easier to simply not update downstream dependencies than to fix it after-the-fact