Big data and data science study

Background © Jim Kaskade: Big Data
BIG DATA AND DATA SCIENCE
study materials and online courses by @dspadawan

WHAT IS DATA SCIENCE
2 Copyright © 2013-2014 by Teradata. All rights reserved.
THE DATA SCIENCE VENN DIAGRAM
@dspadawan

DATA SCIENCE DOMAINS
All links go to Wiki.
If you are not sure
what something
means you can learn.
1. Data Science (Fundamentals)
2. Statistics
3. Programming languages
4. Machine Learning / Data Mining
5. Text Mining / Natural Language Processing
6. Data Visualization
7. Big Data (Hadoop, MapReduce, NoSQL)
8. Data Ingestion
9. Data Munging or Data Wrangling
10. Toolbox (Weka, …, Spark, Storm, …, Sqoop, RHIPE, etc.)
@dspadawan

DATA SCIENCE METRO MAP
BECOMING A DATA SCIENTIST

MASSIVE OPEN ONLINE COURSES (MOOC)
• Aggregator
> http://www.mooc-list.com
• Platforms
> https://www.coursera.org
> https://www.edx.org
> https://www.open2study.com
> https://www.udacity.com
> https://www.udemy.com
> http://online.stanford.edu
• Interactive platforms
> http://www.codecademy.com
> https://www.datacamp.com
@dspadawan

WANT TO WORK AS DATA SCIENTIST?
@dspadawan

DATA SCIENCE & ANALYTICS
• Coursera
> Core Concepts in Data Analysis
https://www.coursera.org/course/datan
> Introduction to Data Science:
https://www.coursera.org/course/datasci
> Data Science Specialization:
https://www.coursera.org/specialization/jhudatascience/1
– 9 courses + 1 capstone project
– Each course or capstone takes 4 weeks
– You can do it for free or you can pay 49 USD for certification
> Welcome To Process Mining: Data science in Action!
https://www.coursera.org/course/procmin
1
@dspadawan

DATA SCIENCE & ANALYTICS 1
• Edx
> The Analytics Edge
http://www.edx.org/course/mitx/mitx-15-071x-analytics-edge-
1416
> Data, Analytics and Learning
http://www.edx.org/course/utarlingtonx/utarlingtonx-link5-10x-data-
analytics-2186
• Udacity
$
> Intro to Data Science
https://www.udacity.com/course/ud359
@dspadawan

MATH DANCE
@dspadawan

STATISTICS COURSES
• Coursera
> Data analysis and statistical inference:
https://www.coursera.org/course/statistics
> Statistical inference and exploratory data analysis:
https://www.coursera.org/specialization/jhudatascience/1/courses
• EdX
> Introduction to Statistics: Descriptive Statistics
http://www.edx.org/course/uc-berkeleyx/uc-berkeleyx-stat2-1x-introduction-
1138
> Introduction to Statistics: Probability
1534
> Introduction to Statistics: Inference
1533
2
@dspadawan

STATISTICS COURSES CONT. 2
• Udacity
$
> Intro to statistics:
https://www.udacity.com/course/st101
> Exploratory data analysis:
> Intro to Inferential Statistics
• Mathematical monk
> https://www.youtube.com/playlist?list=PL17567A1A3F5DB5E4
@dspadawan

PROGRAMMING LANGUAGES
• Analysis/Data mining:
> R language
> Python
> SQL
> (Perl)
> (Octave)
• Big Data (Hadoop)
> Java (!)
> Python
• Visualization
> JavaScript
3
@dspadawan

R LANGUAGE
• Basic info and SW
> R Language:
http://www.r-project.org
> R Studio (IDE):
http://www.rstudio.com
• Courses
> R Programming:
https://www.coursera.org/course/rprog
• Practice
> Interactive courses:
https://www.datacamp.com/courses
> Data mining examples in R:
http://www.rdatamining.com
3
@dspadawan

PYTHON
• Basic info and SW:
> Python language:
https://www.python.org
> Eclipse Python:
http://pydev.org
• Python for Java developers:
> http://www.sthurlow.com/python
• Google's Python Class
> https://developers.google.com/edu/python
• Code Academy Python
> http://www.codecademy.com/tracks/python
3
@dspadawan

OCTAVE
• Basic info and SW:
> http://octave.sourceforge.net
> https://gnu.org/software/octave
> http://en.wikipedia.org/wiki/GNU_Octave
• Coursera:
> Machine learning: https://www.coursera.org/course/ml
3
Octave is mostly
compatible with
MatLab.
@dspadawan

MACHINE LEARNING COURSES
Subfield of computer
science and artificial
intelligence about
learn from data.
• Coursera
> Machine Learning (Stanford):
https://www.coursera.org/course/ml
> Machine Learning: (University of Washington)
https://www.coursera.org/course/machlearning
> Practical Machine Learning (Johns Hopkins):
https://www.coursera.org/course/predmachlearn
– part of Data Science Specialization
• Udacity
> Machine Learning (Supervised, Reinforcement, Unsupervised)
4A
$
@dspadawan

MACHINE LEARNING VIDEOS
• Udemy
> Hilary Mason: An Intro to Machine Learning with Web Data
https://www.udemy.com/hilary-mason-an-intro-to-machine-learning-
with-web-data
> Hilary Mason: Advanced Machine Learning
https://www.udemy.com/hilary-mason-advanced-machine-learning/
• Mathematical monk
> https://www.youtube.com/playlist?list=PLD0F06AA0D2E8FFBA
• Videolectures.net
> http://blog.videolectures.net/100-most-popular-machine-learning-
talks-at-videolectures-net/
4A
$
@dspadawan

DATA MINING COURSES
Process of discovery
patterns in large data
sets via machine
learning or statistics.
• Coursera
> Mining Massive Datasets
(Stanford)
https://www.coursera.org/course/mmds
• Udemy
> Matthew Russell on Mining the Social Web
https://www.udemy.com/matthew-russell-on-mining-the-social-web/
> Data Mining
https://www.udemy.com/data-mining
• Web page
> http://www.rdatamining.com
4B
$
@dspadawan

DATA MINING COURSES & TOOLS
• Courses:
> Data Mining with Weka:
https://weka.waikato.ac.nz/dataminingwithweka/preview
> More Data Mining with Weka:
https://weka.waikato.ac.nz/moredataminingwithweka
• Weka
> SW: http://www.cs.waikato.ac.nz/ml/weka
• Knime
> SW: https://www.knime.org/downloads/overview
• RapidMiner
> Official site: http://rapidminer.com
> SW: http://sourceforge.net/projects/rapidminer
4B
@dspadawan

TEXT MINING 5A
• R Data Mining (Word Cloud)
TOP RECURRING THEMES ABOUT BIG DATA
> http://www.rdatamining.com/examples/text-mining
• Videolectures.net
> http://videolectures.net/Top/Computer_Science/Text_Mining
• Tool (Word Cloud)
> Wordle.net
@dspadawan

NATURAL LANGUAGE PROCESSING COURSES
• Coursera
> Natural Language Processing
Subfield of computer
science and artificial
intelligence and
linguistics.
(Columbia University):
https://www.coursera.org/course/nlangp
> Natural Language Processing (Stanford):
https://www.coursera.org/course/nlp
• Deeper Learning MOOC
> http://dlmooc.deeper-learning.org/
• Wikipedia
> http://en.wikipedia.org/wiki/Natural_language_processing
5B
@dspadawan

VISUALIZATION TOOLS 6
• Tableau
> http://www.tableausoftware.com
> Commercial visualization software
• D3.js
> http://d3js.org
> Data Driven document visualization library
• GraphViz
> http://www.graphviz.org
> Graph visualization tools
• Gephi
> https://gephi.github.io
> Visualization platform
@dspadawan

TABLEAU 6
• Trainings
> http://www.tableausoftware.com/learn/training
> On demand
> Live Online planned for specific topic
• Download
> Tableau Public: http://www.tableausoftware.com/public
> Tableau Trial: http://www.tableausoftware.com/products/trial
• Certification
> Desktop (Qualified associate, Certified Professional)
> Server (Qualified associate, Certified Professional)
> http://www.tableausoftware.com/support/certification
@dspadawan

HOW BIG, IS BIG ENOUGH?
@dspadawan

BIG DATA STUDY 7
• MOOC
> http://bigdatauniversity.com
> http://bigdatacourse.appspot.com
• Coursera
> Web Intelligence and Big Data
https://www.coursera.org/course/bigdata
• Udemy
$
> Big Data and Hadoop Essentials
https://www.udemy.com/big-data-and-hadoop-essentials-free-tutorial
• Open2Study
> Big Data for Better Performance
http://www.open2study.com/courses/big-data-for-better-performance
@dspadawan

BIG DATA TOOLS
• Hadoop – Big Data Framework
• Hive – DWH infrastructure build on top of Hadoop
• HBase – Non-relational, distributed DB
• Pig – Hadoop programming tool
• Storm – Real time computation system for Hadoop
• Solr – Search platform
• Falcon – Data management and processing for Hadoop
• Sqoop – CMD application for transfer data into Hadoop
• Flume – Large scale log aggregation framework
• Oozie – Workflow scheduler for Hadoop
• Ambari – Simpler management for Hadoop clusters
• Mahout – Machine Learning algorithms implemented on Hadoop
• ZooKeeper – Coordination service for distributed applications
• Knox - REST API Gateway for interacting with Hadoop clusters
7
@dspadawan

HADOOP STUDY
• Hadoop providers
> http://www.cloudera.com
> http://hortonworks.com
> http://www.mapr.com
> http://www.teradata.com/aster
• Udacity
> Intro to Hadoop and MapReduce
• Udemy
> Become a Certified Hadoop Developer | Training | Tutorial
https://www.udemy.com/hadoop-tutorial
7
There is more
Hadoop providers:
IBM, Pivotal, etc.
$
$
@dspadawan

NOT ONLY SQL DATABASES
• MongoDB – JSON document store
> http://www.mongodb.com
> https://university.mongodb.com
• CouchDB – JSON document store
> http://couchdb.apache.org
• CasandraDB – High performance column oriented DB
> http://cassandra.apache.org
• VoltDB – In-memory database
> http://voltdb.com
• Redis – High performance column oriented DB
> http://redis.io
• NuoDB – Distributed SQL DB
> http://www.nuodb.com
7
@dspadawan

BIG DATA UNIVERSITY 7
• Big Data Courses path:
> Big Data Fundamentals
> Hadoop Fundamentals
> Moving Data into Hadoop (Sqoop and Flume tools)
> Query languages for Hadoop (Hive, Pig and Jaql)
> SQL Access for Hadoop
> Using HBase for Real-time Access to your Big Data
> Accessing Hadoop Data Using Hive
> Introduction to Pig
> Controlling Hadoop Jobs using Oozie
> Hadoop Reporting and Analysis
> Introduction to MapReduce Programming
• Courses are provided by IBM
@dspadawan

IT IS EVEN BETTER, DON’T YOU THINK?
@dspadawan

CLOUDERA HADOOP 7
• Tutorials
> 8 different paths
> On demand and free
> Lectured together with Udacity (paid on monthly basis)
> http://cloudera.com/content/cloudera/en/training/courses.html
> http://cloudera.com/content/cloudera/en/training/library.html
• Sandbox
> http://cloudera.com/content/support/en/downloads/quickstart_v
ms/cdh-5-1-x1.html
• Certification
> 200 USD per exam
> http://cloudera.com/content/cloudera/en/training/certification.ht
ml
@dspadawan

HORTONWORKS HADOOP 7
• Tutorials
> http://hortonworks.com/tutorials
> 3 paths for
– Developers
– Administrators
– Data Scientists
• Sandbox
> http://hortonworks.com/hdp/downloads
• Certifications
> 200 USD per exam
> http://hortonworks.com/training/certification
@dspadawan

MAPR HADOOP 7
• Tutorials
> https://www.mapr.com/services/mapr-academy/training-videos
> 3 paths for
– Developers
– Administrators
– Business users
• Sandbox
> https://www.mapr.com/products/mapr-sandbox-hadoop
• Certification
> For administrator only
> You must pass Hadoop Cluster Administration on MapR course
> https://www.mapr.com/services/mapr-academy/certification
@dspadawan

STREAMING – NO BIG DEAL
@dspadawan

STREAMING DATA PROCESSING
• Storm (https://storm.incubator.apache.org)
• Open source (ASF) real-time Hadoop
• Twitter project
• Spark (https://spark.apache.org)
• Open source (ASF) in-memory Hadoop
• Apache project
• S4 (http://incubator.apache.org/s4)
• Open source (ASF) processing of stream data
• Yahoo project
• Samza (http://samza.incubator.apache.org)
• Open source processing messagining data
• LinkedIn project
7
@dspadawan

DATA INGESTION 8
• Techniques
Process of obtaining,
importing and
processing data for
later use or storage.
> Data import and export
> Data fusion – integration multiple data
> Data sampling – selection of data subset (rows)
> Data discovery – detection patterns in data
> Exploratory data analysis – summarize main data characteristics
> Feature extraction – selection of data subset (columns)
> Data scrubbing – data error correction
> Missing data values – data correction
> Etc.
@dspadawan

DATA WRANGLING / DATA MUNGING 9
• Coursera
> Getting and Cleaning Data
Converting or
mapping data from
one "raw" form into
another format.
part of Data Science Specialization
https://www.coursera.org/course/getdata
• Udacity
$
> Data Wrangling with MongoDB
• School of Data
> Many different courses http://schoolofdata.org
• Tools
> OpenRefine, DataWrangler – clean up and transform tools
> Talend, Pentaho – integration
@dspadawan

TOOLBOX 10
• Hadoop and realtime
> Apache Scibe
• Machine Learning
> H2O – In memory machine learning
• Data Mining
> Rattle – GUI for DM using R
• Python and NLP
> NLTK = Natural Language ToolKit for Python
• R and Hadoop
> RHIPE = R + Hadoop Integrated Programming Environment
• Visualization
> Many Eyes – Online visualization system from IBM
@dspadawan

ONLINE SOURCES
• Data Science Servers:
> http://www.datasciencecentral.com
> http://www.hadoop360.com
> http://www.datascienceweekly.org
• Aggregators
> https://trello.com/b/rbpEfMld/data-science
• Blogs
• http://datasciencemasters.org
• http://www.kdnuggets.com
• http://www.zipfianacademy.com/blog/post/46864003608/a-practical-intro-
to-data-science
• http://datascience101.wordpress.com
• http://fivethirtyeight.blogs.nytimes.com
@dspadawan

FREE BOOKS
• Data Science
> Doing Data Science
> Agile Data Science
> Data Science for Business
• Statistics
> Think Stats
• Programming
> R language
– 25 Recipes for Getting Started with R
– Learning R
> Python
– Learning Python, 5th Edition
– Think Python
@dspadawan

FREE BOOKS CONTINUED
• Machine Learning / Data Mining
> Machine Learning for Hackers
> Mining the Social Web
• Visualization
> Visualizing Data
> Getting Started with D3
> Communicating Data with Tableau
• Text mining / Natural Language Processing
> 21 Recipes for Mining Twitter
> Natural Language Processing with Python
> Natural Language Annotation for Machine Learning
@dspadawan

FREE BOOKS CONTINUED
• Big Data
> Hadoop: The Definitive Guide, 3rd Edition
> Ethics of Big Data
> Big Data Analytics with R and Hadoop
• Data Ingestion
> Data Analysis with Open Source Tools
> Python for Data Analysis
• Data Wrangling and Munging
> Using OpenRefine
• Toolbox
$
> Getting Started with Storm
> Fast Data Processing with Spark
@dspadawan

QUESTIONS AND ANSWERS
By Tara Laskowski
@dspadawan
Contact me at datasciencepadawan@gmail.com
Follow me at twitter @dspadawan
Read my blog http://datasciencepadawan.blogspot.com

Big data and data science study

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (17)

Similar a Big data and data science study

Similar a Big data and data science study (20)

Big data and data science study

Notas del editor