Data Science Crash Course
•
3 recomendaciones•1,657 vistas
Introduction: This workshop will provide a hands on introduction to basic Machine Learning techniques with Spark ML using a Sandbox on students’ personal machines. Format: A short introductory lecture on a select important supervised and unsupervised Machine Learning techniques followed by a demo, lab exercises and a Q&A session. The lecture will be followed by lab time to work through the lab exercises and ask questions. Objective: To provide a quick and short hands-on introduction to Machine Learning with Spark ML. In the lab, you will use the following components: Apache Zeppelin (a “Modern Data Science Toolbox”) and Apache Spark. You will learn how to analyze the data, structure the data, train Machine Learning models and apply them to answer real-world questions. Pre-requisites: Registrants must bring a laptop that can run the Hortonworks Data Cloud. Speaker: Robert Hryniewicz, Developer Advocate, Hortonworks
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Data Science Crash Course
Similar a Data Science Crash Course (20)
Más de DataWorks Summit
Más de DataWorks Summit (20)
Último
Último (20)
Data Science Crash Course
- 1. Data Science Crash Course - DataWorks Summit – Sydney 2017 Robert Hryniewicz Developer Advocate T: @robH8z E: rhryniewicz@hortonworks.com
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What is Data Science? à Extracting knowledge/insights from data – Data: structured or unstructured à Continuation of – statistics – machine learning – data mining – predictive analytics
- 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Machine Learning Use Cases Healthcare Predict diagnosis Prioritize screenings Reduce re-admittance rates Financial services Fraud Detection/prevention Predict underwriting risk New account risk screens Public Sector Analyze public sentiment Optimize resource allocation Law enforcement & security Retail Product recommendation Inventory management Price optimization Telco/mobile Predict customer churn Predict equipment failure Customer behavior analysis Oil & Gas Predictive maintenance Seismic data management Predict well production levels
- 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Spark SQL Overview à Spark module for structured data processing à Three ways to manipulate data: – DataFrames or Datasets API – SQL queries
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DataFrames à Distributed collection of data organized into named columns à Conceptually equivalent to a table in relational DB or a data frame in R/Python à API available in Scala, Java, Python, and R Col1 Col2 … … ColN DataFrame Column Row Data is described as a DataFrame with rows, columns, and a schema
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DataFrames CSVAvro HIVE Spark SQL Col1 Col2 … … ColN DataFrame Column Row JSON
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What is a ML Model? à Mathematical formula with a number of parameters that need to be learned from the data. And fitting a model to the data is a process known as model training à E.g. linear regression – Goal: fit a line y = mx + c to data points – After model training: y = 2x + 5 Input OutputModel 1, 0, 7, 2, … 7, 5, 19, 9, …
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved START Regression Classification Collaborative Filtering Clustering Dimensionality Reduction • Logistic Regression • Support Vector Machines (SVM) • Random Forest (RF) • Naïve Bayes • Linear Regression • Alternating Least Squares (ALS) • K-Means, LDA • Principal Component Analysis (PCA)
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved CLASSIFICATION Identifying to which category an object belongs to Examples: spam detection, diabetes diagnosis, text labeling Algorithms: à Logistic Regression – Fast training, linear model – Classes expressed in probabilities à Support Vector Machines (SVM) – “Best” supervised learning algorithm, effective – More robust to outliers than Log Regression – Handles non-linearity à Random Forest – Fast training – Handles categorical features – Does not require feature scaling – Captures non-linearity and feature interaction à Naïve Bayes – Good for text classification – Assumes independent variables
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Visual Intro to Decision Trees à http://www.r2d3.us/visual-intro-to-machine-learning-part-1 CLASSIFICATION
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved REGRESSION Predicting a continuous-valued output Example: Predicting house prices based on number of bedrooms and square footage Algorithms: Linear Regression
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved CLUSTERING Automatic grouping of similar objects into sets (clusters) Example: market segmentation – auto group customers into different market segments Algorithms: K-means, LDA
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved COLLABORATIVE FILTERING Fill in the missing entries of a user-item association matrix Applications: Product/movie recommendation Algorithms: Alternating Least Squares (ALS)
- 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DIMENSIONALITY REDUCTION Reducing the number of redundant features/variables Applications: à Removing noise in images by selecting only “important” features à Removing redundant features, e.g. MPH & KPH are linearly dependent Algorithms: Principal Component Analysis (PCA)
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved START Regression Classification Deep Learning Clustering Dimensionality Reduction • XGBoost (Extreme Gradient Boosting) • Recurrent Neural Network (RNN) • Convolutional Neural Network (CNN) • Yinyang K-Means • t-Distributed Stochastic Neighbor Embedding (t-SNE) • Local Regression (LOESS) Collaborative Filtering • Weighted Alternating Least Squares (WALS)
- 24. 24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Data Preparation 1. Data analysis (audit for anomalies/errors) 2. Creating an intuitive workflow (formulate seq. of prep operations) 3. Validation (correctness evaluated against sample representative dataset) 4. Transformation (actual prep process takes place) 5. Backflow of cleaned data (replace original dirty data) Approx. 80% of Data Analyst’s job is Data Preparation! Example of multiple values used for U.S. States è California, CA, Cal., Cal
- 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Asking Relevant Questions à Specific (can you think of a clear answer?) à Measurable (quantifiable? data driven?) à Actionable (if you had an answer, could you do something with it?) à Realistic (can you get an answer with data you have?) à Timely (answer in reasonable timeframe?)
- 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hyperparameters à Define higher-level model properties, e.g. complexity or learning rate à Cannot be learned during training à need to be predefined à Can be decided by – setting different values – training different models – choosing the values that test better à Hyperparameter examples – Number of leaves or depth of a tree – Number of latent factors in a matrix factorization – Learning rate (in many models) – Number of hidden layers in a deep neural network – Number of clusters in a k-means clustering
- 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Feature Selection à Also known as variable or attribute selection à Why important? – simplification of models è easier to interpret by researchers/users – shorter training times – enhanced generalization by reducing overfitting à Dimensionality reduction vs feature selection – Dimensionality reduction: create new combinations of attributes – Feature selection: include/exclude attributes in data without changing them Q: Which features should you use to create a predictive model?
- 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved With that in mind… à No simple formula for “good questions” only general guidelines à The right data is better than lots of data à Understanding relationships matters
- 29. 29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Training Set Learning Algorithm h hypothesis/model input output Ingest / Enrich Data Clean / Transform / Filter Select / Create New Features Evaluate Accuracy / Score
- 30. 30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Building Spark ML pipelines Feature transform 1 Feature transform 2 Combine features Linear Regression Input DataFrame Input DataFrame Output DataFrame Pipeline Pipeline Model Train Predict Export Model
- 31. 31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Spark ML Pipeline à fit() is for training à transform() is for prediction Input DataFrame (TRAIN) Input DataFrame (TEST) Output Dataframe (PREDICTIONS) Pipeline Pipeline Model fit() transform() Train Predict
- 32. 32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Sample Spark ML Pipeline indexer = … parser = … hashingTF = … vecAssembler = … rf = RandomForestClassifier(numTrees=100) pipe = Pipeline(stages=[indexer, parser, hashingTF, vecAssembler, rf]) model = pipe.fit(trainData) # Train model results = model.transform(testData) # Test model
- 33. 33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Exporting ML Models - PMML à Predictive Model Markup Language (PMML) –> XML-based predictive model interchange format à Supported models –K-Means –Linear Regression –Ridge Regression –Lasso –SVM –Binary
- 35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hortonworks Cloud Solutions Microsoft AWS Google Managed Azure HDInsight Non-Managed / Marketplace Hortonworks Data Cloud for AWS Cloud IaaS Hortonworks Data Platform (via Ambari and via Cloudbreak)
- 37. 37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Zeppelin with HDP 2.6 • Data exploration and discovery • Visualization • Interactive snippet-at-a-time experience • “Modern Data Science Studio” Features • Ad-hoc experimentation • Deeply integrated with Spark + Hadoop • Supports multiple language backends • Incubating at Apache Use Case Web-based Notebook for interactive analytics
- 39. 39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Scatter 2D Data Visualized scatterData ç DataFrame +-----+--------+ |label|features| +-----+--------+ |-12.0| [-4.9]| | -6.0| [-4.5]| | -7.2| [-4.1]| | -5.0| [-3.2]| | -2.0| [-3.0]| | -3.1| [-2.1]| | -4.0| [-1.5]| | -2.2| [-1.2]| | -2.0| [-0.7]| | 1.0| [-0.5]| | -0.7| [-0.2]| ... ... ...
- 40. 40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Linear Regression Model Training (one feature) Coefficients: 2.81 Intercept: 3.05 y = 2.81x + 3.05 Training Result
- 41. 41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Linear Regression (two features) Coefficients: [0.464, 0.464] Intercept: 0.0563
- 42. 42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved ML Lab • Residuals • residual of an observed value is the difference between the observed value and the estimated value • R2 (R Squared) – Coefficient of Determination • indicates a goodness of fit • R2 of 1 means regression line perfectly fits data • RMSE (Root Mean Square Error) • measure of differences between values predicted by a model and values actually observed • good measure of accuracy, but only to compare forecasting errors of different models (individual variables are scale-dependent)