Hadoop and Pig at Twitter__HadoopSummit2010

•

25 recomendaciones•2,429 vistas

Yahoo Developer Network

Hadoop Summit 2010 - Developers Track Hadoop and Pig at Twitter Kevin Weil, Twitter

Tecnología

Hadoop at Twitter ,[object Object],[object Object]

The Twitter Data Lifecycle ,[object Object],[object Object],[object Object],[object Object]

The Twitter Data Lifecycle ,[object Object],[object Object],[object Object],[object Object],1 Community Open Source 2 Twitter Open Source (or soon)

My Background ,[object Object],[object Object],[object Object],[object Object]

The Twitter Data Lifecycle ,[object Object],[object Object],[object Object],[object Object],1 Community Open Source 2 Twitter Open Source

What Data? ,[object Object],[object Object],[object Object]

Logs ,[object Object],[object Object],[object Object],[object Object]

Scribe ,[object Object],[object Object],[object Object],[object Object],FE FE FE Agg Agg HDFS File

Scribe at Twitter ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Scribe at Twitter ,[object Object],[object Object],[object Object],[object Object],[object Object],1 http://github.com/traviscrawford/scribe

Tabular Data ,[object Object],[object Object],[object Object],[object Object],[object Object]

Crane Driver Configuration/Batch Management Extract Load Transform Protobuf P1 Protobuf P2 Source Sink ZooKeeper Registration

Crane ,[object Object],[object Object],[object Object],[object Object]

Crane ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Crane ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Storage Basics ,[object Object],[object Object],[object Object],[object Object],[object Object],1 http://www.github.com/kevinweil/hadoop-lzo

Elephant Bird http://www.flickr.com/photos/jagadish/3072134867/ 1 http://github.com/kevinweil/elephant-bird

Elephant Bird ,[object Object],[object Object]

Elephant Bird ,[object Object],[object Object],[object Object]

Elephant Bird ,[object Object],[object Object],[object Object],[object Object]

Elephant Bird ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Challenge: Mutable Data ,[object Object],[object Object],[object Object]

Challenge: Mutable Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Challenge: Mutable Data ,[object Object],[object Object],[object Object],[object Object]

HBase ,[object Object],[object Object],[object Object]

HBase at Twitter ,[object Object],[object Object],[object Object],[object Object]

Enter Pig ,[object Object],[object Object],[object Object],[object Object],[object Object]

Pig Democratizes Large-scale Data Analysis ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Pig Examples ,[object Object],[object Object]

Pig Workflow ,[object Object],[object Object],[object Object],[object Object],1 http://github.com/ningliang/piglet

Counting Big Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Correlating Big Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Research on Big Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Research on Big Data ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Data Products ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Questions ? Follow me at twitter.com/kevinweil TM ,[object Object]

Más contenido relacionado

La actualidad más candente

Pig workshopSudar Muthu

Apache Pig: A big data processorTushar B Kute

apache pig performance optimizations talk at apachecon 2010Thejas Nair

Nov HUG 2009: Hadoop Record Reader In PythonYahoo Developer Network

Practical pigtrihug

January 2011 HUG: Howl PresentationYahoo Developer Network

Hands on Hadoop and pigSudar Muthu

Karmasphere hadoop-productivity-toolsHadoop User Group

Apache PIGPrashant Gupta

03 pig introSubhas Kumar Ghosh

Pig and Pig Latin - Module 5Rohit Agrawal

Practical Hadoop using PigDavid Wellman

TriHUG November Pig Talk by Alan Gatestrihug

Apache Pig: Making data transformation easyVictor Sanchez Anguix

Introduction to PigPrashanth Babu

Apache spark sessionknowbigdata

Apache pigSadiq Basha

Hive integration: HBase and Rcfile__HadoopSummit2010Yahoo Developer Network

An Introduction to Apache PigSachin Vakkund

Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twi...Alex Levenson

La actualidad más candente (20)

Pig workshop

Apache Pig: A big data processor

apache pig performance optimizations talk at apachecon 2010

Nov HUG 2009: Hadoop Record Reader In Python

Practical pig

January 2011 HUG: Howl Presentation

Hands on Hadoop and pig

Karmasphere hadoop-productivity-tools

Apache PIG

03 pig intro

Pig and Pig Latin - Module 5

Practical Hadoop using Pig

TriHUG November Pig Talk by Alan Gates

Apache Pig: Making data transformation easy

Introduction to Pig

Apache spark session

Apache pig

Hive integration: HBase and Rcfile__HadoopSummit2010

An Introduction to Apache Pig

Hadoop Summit 2015: Performance Optimization at Scale, Lessons Learned at Twi...

Destacado

Hadoop Pig: MapReduce the easy way!Nathan Bijnens

Hadoop and pig at twitter (oscon 2010)Kevin Weil

Un introduction à PigModern Data Stack France

Apache pigMudassir Khan Pathan

Pig statementsGanesh Sanap

Hadoop, Pig, and Python (PyData NYC 2012)mortardata

Hadoop Ecosystem at Twitter - Kevin Weil - Hadoop World 2010Cloudera, Inc.

High-level Programming Languages: Apache Pig and Pig LatinPietro Michiardi

Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem a...Hadoop User Group

Pig Tutorial | Twitter Case Study | Apache Pig Script and Commands | EdurekaEdureka!

introduction to data processing using Hadoop and PigRicardo Varela

Practical Problem Solving with Apache Hadoop & PigMilind Bhandarkar

Hadoop, Pig, and Twitter (NoSQL East 2009)Kevin Weil

Hadoop et son écosystèmeKhanh Maudoux

Big Data: Concepts, techniques et démonstration de Apache Hadoophajlaoui jaleleddine

Pig and Python to Process Big DataShawn Hermans

Solr + Hadoop - Fouillez facilement dans votre système Big Datafrancelabs

Apache Cassandra - Concepts et fonctionnalitésRomain Hardouin

Destacado (18)

Hadoop Pig: MapReduce the easy way!

Hadoop and pig at twitter (oscon 2010)

Un introduction à Pig

Apache pig

Pig statements

Hadoop, Pig, and Python (PyData NYC 2012)

Hadoop Ecosystem at Twitter - Kevin Weil - Hadoop World 2010

High-level Programming Languages: Apache Pig and Pig Latin

Yahoo! Hadoop User Group - May Meetup - HBase and Pig: The Hadoop ecosystem a...

Pig Tutorial | Twitter Case Study | Apache Pig Script and Commands | Edureka

introduction to data processing using Hadoop and Pig

Practical Problem Solving with Apache Hadoop & Pig

Hadoop, Pig, and Twitter (NoSQL East 2009)

Hadoop et son écosystème

Big Data: Concepts, techniques et démonstration de Apache Hadoop

Pig and Python to Process Big Data

Solr + Hadoop - Fouillez facilement dans votre système Big Data

Apache Cassandra - Concepts et fonctionnalités

Similar a Hadoop and Pig at Twitter__HadoopSummit2010

Hadoop at Twitter (Hadoop Summit 2010)Kevin Weil

Data Infrastructure for a World of MusicLars Albertsson

Eric Baldeschwieler Keynote from Storage Developers ConferenceHortonworks

Hadoop basicsAntonio Silveira

Hive @ Hadoop day seattle_2010nzhang

Data infrastructure at Facebook AhmedDoukh

Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...Cloudera, Inc.

Big data and APIs for PHP developers - SXSW 2011Eli White

Finding the needles in the haystack. An Overview of Analyzing Big Data with H...Chris Baglieri

Bigdata and Hadoop BootcampSpotle.ai

Overview of big data & hadoop version 1 - Tony NguyenThanh Nguyen

Overview of Big data, Hadoop and Microsoft BI - version1Thanh Nguyen

Large Scale Data With Hadoopguest27e6764

Big data-at-detikk4ndar

How Hadoop Revolutionized Data Warehousing at Yahoo and FacebookAmr Awadallah

Hadoop Frameworks Panel__HadoopSummit2010Yahoo Developer Network

Big-Data Hadoop Tutorials - MindScripts Technologies, Pune amrutupre

HBaseCon 2013: Apache Drill - A Community-driven Initiative to Deliver ANSI S...Cloudera, Inc.

Architecting the Future of Big Data and SearchHortonworks

Big data or big dealeduarderwee

Similar a Hadoop and Pig at Twitter__HadoopSummit2010 (20)

Hadoop at Twitter (Hadoop Summit 2010)

Data Infrastructure for a World of Music

Eric Baldeschwieler Keynote from Storage Developers Conference

Hadoop basics

Hive @ Hadoop day seattle_2010

Data infrastructure at Facebook

Hadoop World 2011: Building Web Analytics Processing on Hadoop at CBS Interac...

Big data and APIs for PHP developers - SXSW 2011

Finding the needles in the haystack. An Overview of Analyzing Big Data with H...

Bigdata and Hadoop Bootcamp

Overview of big data & hadoop version 1 - Tony Nguyen

Overview of Big data, Hadoop and Microsoft BI - version1

Large Scale Data With Hadoop

Big data-at-detik

How Hadoop Revolutionized Data Warehousing at Yahoo and Facebook

Hadoop Frameworks Panel__HadoopSummit2010

Big-Data Hadoop Tutorials - MindScripts Technologies, Pune

HBaseCon 2013: Apache Drill - A Community-driven Initiative to Deliver ANSI S...

Architecting the Future of Big Data and Search

Big data or big deal

Más de Yahoo Developer Network

Developing Mobile Apps for Performance - Swapnil Patel, Verizon MediaYahoo Developer Network

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...Yahoo Developer Network

Athenz & SPIFFE, Tatsuya Yano, Yahoo JapanYahoo Developer Network

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...Yahoo Developer Network

CICD at Oath using ScrewdriverYahoo Developer Network

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, OathYahoo Developer Network

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenuYahoo Developer Network

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, AmpoolYahoo Developer Network

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...Yahoo Developer Network

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...Yahoo Developer Network

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, OathYahoo Developer Network

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...Yahoo Developer Network

Moving the Oath Grid to Docker, Eric Badger, OathYahoo Developer Network

Architecting Petabyte Scale AI ApplicationsYahoo Developer Network

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...Yahoo Developer Network

Jun 2017 HUG: YARN Scheduling – A Step BeyondYahoo Developer Network

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies Yahoo Developer Network

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...Yahoo Developer Network

February 2017 HUG: Exactly-once end-to-end processing with Apache ApexYahoo Developer Network

February 2017 HUG: Data Sketches: A required toolkit for Big Data AnalyticsYahoo Developer Network

Más de Yahoo Developer Network (20)

Developing Mobile Apps for Performance - Swapnil Patel, Verizon Media

Athenz - The Open-Source Solution to Provide Access Control in Dynamic Infras...

Athenz & SPIFFE, Tatsuya Yano, Yahoo Japan

Athenz with Istio - Single Access Control Model in Cloud Infrastructures, Tat...

CICD at Oath using Screwdriver

Big Data Serving with Vespa - Jon Bratseth, Distinguished Architect, Oath

How @TwitterHadoop Chose Google Cloud, Joep Rottinghuis, Lohit VijayaRenu

The Future of Hadoop in an AI World, Milind Bhandarkar, CEO, Ampool

Apache YARN Federation and Tez at Microsoft, Anupam Upadhyay, Adrian Nicoara,...

Containerized Services on Apache Hadoop YARN: Past, Present, and Future, Shan...

HDFS Scalability and Security, Daryn Sharp, Senior Engineer, Oath

Hadoop {Submarine} Project: Running deep learning workloads on YARN, Wangda T...

Moving the Oath Grid to Docker, Eric Badger, Oath

Architecting Petabyte Scale AI Applications

Introduction to Vespa – The Open Source Big Data Serving Engine, Jon Bratseth...

Jun 2017 HUG: YARN Scheduling – A Step Beyond

Jun 2017 HUG: Large-Scale Machine Learning: Use Cases and Technologies

February 2017 HUG: Slow, Stuck, or Runaway Apps? Learn How to Quickly Fix Pro...

February 2017 HUG: Exactly-once end-to-end processing with Apache Apex

February 2017 HUG: Data Sketches: A required toolkit for Big Data Analytics

Último

Kalyanpur ) Call Girls in Lucknow Finest Escorts Service 🍸 8923113531 🎰 Avail...gurkirankumar98700

🐬 The future of MySQL is Postgres 🐘RTylerCroy

Neo4j - How KGs are shaping the future of Generative AI at AWS Summit London ...Neo4j

IAC 2024 - IA Fast Track to Search Focused AI SolutionsEnterprise Knowledge

Salesforce Community Group Quito, Salesforce 101Paola De la Torre

08448380779 Call Girls In Civil Lines Women Seeking MenDelhi Call girls

How to convert PDF to text with Nanonetsnaman860154

Exploring the Future Potential of AI-Enabled Smartphone Processorsdebabhi2

CNv6 Instructor Chapter 6 Quality of Servicegiselly40

Finology Group – Insurtech Innovation Award 2024The Digital Insurer

Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsRoshan Dwivedi

Tata AIG General Insurance Company - Insurer Innovation Award 2024The Digital Insurer

Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko

EIS-Webinar-Prompt-Knowledge-Eng-2024-04-08.pptxEarley Information Science

04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG

Boost PC performance: How more available memory can improve productivityPrincipled Technologies

08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls

The Codex of Business Writing Software for Real-World Solutions 2.pptxMalak Abu Hammad

2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong

Histor y of HAM Radio presentation slidevu2urc