1.
Copyright © 2018, edureka and/or its affiliates. All rights reserved.
PySpark Tutorial

2.
www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
Objectives of Today’s Training
Programming
PySpark
RDDs
DataFrame
PySpark SQL
PySpark Streaming
Machine Learning (MLlib)

3.
Copyright © 2018, edureka and/or its affiliates. All rights reserved.
PySpark

4.
www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
PySpark
PythonAPIforSpark
UsesPy4jtolaunch
JVM
EasytoLearn&Use
VisualizationisPossible
SimpleAPI
WideRangeof
Libraries

5.
Copyright © 2018, edureka and/or its affiliates. All rights reserved.
RDDs

6.
www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
Resilient Distributed Dataframe (RDD)
RDD is the abstracted data over the distributed collection
Created using various Spark Context Functions
Follows lazy initialization principle
RDDs are immutable and cacheable in nature
Supports two different types of operations
Transformations
Actions

7.
www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
RDD – Transformations & Actions
Map(func)
flatMap(func)
filter(func)
groupByKey()
reduceByKey(func)
mapValues(func)
take(N)
count()
collect()
reduce()
takeOrdered(N)
top(N)
Transformations Actions

8.
Copyright © 2018, edureka and/or its affiliates. All rights reserved.
DataFrame

9.
www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
Helps in increase in performance of PySpark queries3
DataFrame
Immutable but distributed collection of structured & semi-
structured data
1
Organized into named columns similar to a RDMS table2
Supports a wide range of data formats and sources4
API support for various languages like Python, R, Scala, Java5

10.
Copyright © 2018, edureka and/or its affiliates. All rights reserved.
PySpark SQL

11.
www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
PySpark SQL
01
PySparkSQL module is a
higher-level abstraction over
PySpark Core 02
PySparkSQL is used
for processing structured and
semi-structured datasets
03
Through PySparkSQL, SQL
and HiveQL code can be
used 04
PySparkSQL provides an
optimized API

12.
Copyright © 2018, edureka and/or its affiliates. All rights reserved.
PySpark Streaming

13.
www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
PySpark Streaming
It can efficiently deal with various
fault-tolerance aspects and is
highly scalable
Fault
Tolerant
Discretized Stream or Dstream
is a high-level abstraction
which represents a continuous
stream of data
Discretized
Stream
It is a set of APIs that provide a
wrapper over PySpark Core
APIs
PySpark Streaming is the live
data streaming library of
PySpark
Library
PySpark Streaming is the structured stream processing framework that utilizes Spark DataFrames

14.
www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
PySpark Streaming

15.
www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
PySpark Streaming
Spark Streaming receives live input data streams and divides the data into batches
Engine
Input Stream Data Batches of Input
Data
Batches of
Processed Data

16.
Copyright © 2018, edureka and/or its affiliates. All rights reserved.
Machine Learning

17.
www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
Machine Learning (MLlib)
PySpark facilitates the development of custom ML algorithms
It is a wrapper over PySpark Core to do data analysis using machine-learning algorithms
It works on distributed systems and is scalable
MLlib in PySpark, is a machine-learning library

18.
www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
Machine Learning (MLlib)
01
Data preparation Machine learning
algorithms
Utilities
02 03
MLlib provides three core machine learning functionalities

19.
www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
Machine Learning (MLlib)
01
Data preparation Machine learning
algorithms
Utilities
02 03
MLlib provides three core machine learning functionalities

20.
www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
Machine Learning (MLlib)
01
Data preparation Machine learning
algorithms
Utilities
03
MLlib provides three core machine learning functionalities
02

21.
www.edureka.co/pyspark-certification-trainingPython Spark Certification Training using PySpark
Machine Learning (MLlib)
01
Data preparation Machine learning
algorithms
Utilities
02
MLlib provides three core machine learning functionalities
03

22.
Copyright © 2018, edureka and/or its affiliates. All rights reserved.
@