Big Data Story - From An Engineer's Perspective

Big Data Story - From An Engineer's Perspective
BIG DATA STORY FROM AN ENGINEER'S PERSPECTIVE Hien Luu
Agenda ¨  Big Data Story ¨  Big Data Technology ¨  Big Data Challenges
About Me ¨  Engineering Manager @LinkedIn ¨  Instructor at UCSC Silicon Valley Extension school ¨  Big data enthusiast
Big Data Story THE BIG DATA REVOLUTION Transform how we live, work and think Create opportunities
Big Data Story 2.5 exabytes of data per day 90% of the data in the world has been created in the last two years World's data doubling every two years
Big Data Story World's largest library •  ~37 millions books •  ~10 terabytes TERABYTE (TB) [1,000 Gigabytes]
Big Data Story 1995 - $1,000,000 2015 - $80 1 TB 1 TB
Big Data Story 50 Petabytes – Entire written works of mankind PETABYTE (PB) [1,000 Terabytes] 5 Exabytes – All the words ever spoken by mankind EXABYTE (XB) [1,000 Petabytes]
Big Data Story 250 billion DVDs ZETTABYTE (ZB) [1,000 Exabytes] Size of the entire World Wide Web – 11 trillion years to download a Yottabyte file YOTTABYTE (YB) [1,000 Zettabytes]
Big Data Story US NSA data center – capable of storing a yottabyte of data
Big Data Story 10 exabytes 300 petabytes 1 exabytes Social Networking Ecommerce Search & others
Big Data Story The next wave of revolution (IoT)
Big Data Story Wrist/arm bands, watches, eyewear Cars, navigation devices Heating/ventilation system, air conditioners Body sensors, body implants, pills Traffic/street lights, traffic sensors and signs Wearable Buildings Transportation Health Technology Cities
Big Data Story Tile is the smart companion for all the things you can't stand to lose
Big Data Story Tile community find
Big Data Story Hardware + Software + Connectivity + Data Intelligent Devices Efficient, Safer, Accessible
Big Data Story Tesla : over the air update
Big Data Story 26 billion devices to be operational by 2020 (Source: IDC)
Big Data Story In the age of big data, algorithms will be doing more of the thinking for people - Economist 2010
Big Data Story Google self-driving car
Big Data Story Google self-driving car
Big Data Story Gather nearly 1GB/second
Big Data Story Fueling knowledge economies, sparking innovation, and unleashing waves of creative destruction Data is the oil of the 21st century
Big Data Story Perfect Storm
Big Data Story Data Information Actionable Insight Business Value
Big Data Story Big Data Prediction Recommendation Detection Personalization Inference
Big Data Story Recommendation
Big Data Story $1,000,000 Netflix Prize – 2009 Data set – 100 million movie ratings
Big Data Story Now-casting: Forecasting in near real time
Big Data Story Prediction – Google Flu Trends
Big Data Story Historical query-based estimates vs official influenza surveillance data Early detection of a disease outbreak can save lives http://www.google.org/flutrends/intl/en_us/about/how.html
Big Data Story US spends 15B hours & $120B productivity + fuel
Big Data Story http://inrix.com/xd-traffic/180 million real-time vehicles and devices Real-time traffic information in 37 countries
Big Data Story The Power Of 1 Percent
Big Data Story ~1TB/flight - $200B/year
Big Data Story "Big data is becoming an effective basis of competition in pretty much every industry" - Michael Chui McKinsey Global Institute
Big Data Story Digitally mapping the global economy to connect talent with opportunity at massive scale Connections between people, jobs, skill, companies, and professional knowledge
Big Data Story Using Big Data To Solve Social Problems USING DATA IN THE SERVICE OF HUMANITY
Big Data Story DJ Patil U.S. Chief Data Scientist Weather, health care, climate, flight Responsibly unleash the power of data
Big Data Story “What we realized is data, when used responsibly, is a force multiplier” DJ Patil U.S. Chief Data Scientist White House Office of Science and Technology Policy
Big Data Technology
Big Data Technology Google File System Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung research.google.com/archive/gfs.html MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean, Sanjay Ghemawat research.google.com/archive/mapreduce.html In the beginning…… 2003
Big Data Technology 2003 Google publish GFS 2004 Google publish MapReduce 2006 Hadoop was born 2008 Top level Apache project Y! - 1TB sort in 209 seconds 900-cluster 2009 Google - 1TB sort in 69 seconds 100-cluster 2012 Hadoop version 1.0 2014 Apache Spark 1.0 2015 Spark 1TB Sorting
Big Data Technology Batch King An M/R application that works on a 10GB of data will also run on 10PB of data Automatic parallelism and fault-tolerant Too low level and limiting
Big Data Technology
Big Data Technology http://www.hortonworks.com
Big Data Technology
Big Data Technology http://hortonworks.com/hdp/ Big Data Solution
Big Data Technology Hadoop Ecosystem
Big Data Technology Spark: Cluster Computing with Working Sets people.csail.mit.edu/matei/papers/2010/hotcloud_spark.pdf Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf In recent days…… 2010 Lightning-fast cluster computing
Big Data Technology Speed In-memory computing Versatile General execution model Ease of use APIs in Scala, Java, Python
Big Data Technology Spark Technology Stack
Big Data Technology Hadoop MR Record Spark Record Data Size 100TB 100TB Elapsed Time 72 mins 23 mins # Nodes 2100 206 # Cores 50400 6592 Sort Rate 1.42 TB/min 4.27 TB/min Sort Rate/ node 0.67 GB/min 20.7 GB/min http://databricks.com/blog/2014/11/05/spark-officiallysets-a-new-record-in-large-scale-sorting.html New record in large-scale sorting
Big Data Technology Lambda Architecture An approach to build big data systems Human fault tolerant Data immutability Re-computation http://lambda-architecture.net/ query = function(all data)
Big Data Technology Lambda Architecture Batch Layer Speed Layer Serving Layer Master dataset Batch View Batch View Speed View Data Query
Big Data Challenges
Big Data Challenges TOP THREE ANALYTICS CHALLENGES Analytical insights into business actions Aggregating multiple data sources Lack of appropriate analytical skills MITSloan – The Talent Divide Research Report
Big Data Challenges Big Data Fast Data Data in motion has equal or greater value than "historical" data
Big Data Challenges Data Driven Culture Data Driven Decisions
Big Data Challenges Data Scientist The sexiest job of the 21 century - Havard Business Review 10/2012 Tend to be better programmers than most statisticians and better statisticians than most programmers - Jeanne Harris
Big Data Challenges SUPPLY DEMAND Current Analytics Talent Shortage of 1.5 million big data experts by 2018
Big Data Challenges Master's Program in Data Science
Big Data Challenges Data Science As A Service
Big Data Challenges Machine learning will be everywhere Prediction Phase "Machine learning is to big data as human learning is to life experience"
Big Data Challenges
Thank You

Check these out next

Big Data Story - From An Engineer's Perspective

Recomendados

Más contenido relacionado

Presentaciones para ti(20)

Similar a Big Data Story - From An Engineer's Perspective(20)

Último(20)

Big Data Story - From An Engineer's Perspective