Talk 1 : Evolution of the GoPro's data platform In this talk, we will share GoPro’s experiences in building Data Analytics Cluster in Cloud. We will discuss: evolution of data platform from fixed-size Hadoop clusters to Cloud-based Spark Cluster with Centralized Hive Metastore +S3: Cost Benefits and DevOp Impact; Configurable, spark-based batch Ingestion/ETL framework; Migration Streaming framework to Cloud + S3; Analytics metrics delivery with Slack integration; BedRock: Data Platform Management, Visualization & Self-Service Portal Visualizing Machine learning Features via Google Facets + Spark Speakers: Chester Chen Chester Chen is the Head of Data Science & Engineering, GoPro. Previously, he was the Director of Engineering at Alpine Data Lab. David Winters David is an Architect in the Data Science and Engineering team at GoPro and the creator of their Spark-Kafka data ingestion pipeline. Previously He worked at Apple & Splice Machines. Hao Zou Hao is a Senior big data engineer at Data Science and Engineering team. Previously He worked as Alpine Data Labs and Pivotal

1. Evolution of Data Platform at GoPro

2. ABOUT SPEAKERS
• Chester Chen
• Head of Data Science & Engineering (DSE) at GoPro
• Previously, Director of Engineering, Alpine Data Labs
• Founder and Organizer of SF Big Analytics meetup
• David Winters
• Data Architect of Data Science & Engineering (DSE) at GoPro
• Previously worked at Splice Machine, Apple
• Hao Zou
• Senior Software Engineer of Data Science & Engineering (DSE) at GoPro
• Previously worked at Alpine Data Labs, Pivotal

3. AGENDA
• Business Use Cases
• Evolution of GoPro Data Platform
• Platform Architecture Transformation & Streaming to S3
• Configurable Spark Batch Framework
• Data Democratization
• Data Management & VIsualization
• Data Metrics Delivery
• Initial exploration in ML feature visualization

4. GROWING DATA NEED FROM GOPRO ECOSYSTEM

5. DATA
Analytics
Platform
Consumer Devices GoPro Apps
E-Commerce Social Media/OTT
3rd party data
Product Insight
User segmentation
CRM/Marketing
/Personalization

6. EXAMPLES OF ANALYTICS USE CASES
• Product Analytics
• Web/E-Commercial Analytics
• Camera Analytics
• Mobile Analytics
• GoPro Plus Analytics
• CRM Analytics
• Digital Marketing Analytics
• Social Media Analytics
• Cloud Media Analysis

7. Evolution of Data Platform

8. EVOLUTION OF DATA PLATFORM

9. EVOLUTION OF DATA PLATFORM

10. FIXED CLUSTER ARCHITECTURE
ETL Cluster
•Aggregations and Joins
•Hive and Spark jobs
•Map/Reduce
•Airflow
Secure Data Mart
Cluster
•End User Query
•Impala / Sentry
•Parquet
•Kerberos & LDAP
Analytics Apps
•Hue
•Tableau
•Plotly
•Python
•R
Streaming Cluster
•Log file streaming
•RESTful service
•Kafka
•Spark Streaming
•HBase
Batch Induction
Framework
•Batch files
•Scheduled downloads
•Pre-processing
•Java App
•Airflow
JSON
JSON
Parquet
DDL
• Rest API
• FTP downloads
• S3 sync
Streaming
Batch
Download

11. STREAMING ENDPOINT
ELBHTTP
Pipeline for processing of streaming logs
To ETL Cluster
events
events
state

12. SPARK STREAMING PIPELINE
/path1/…
/path2/…
/path3/…
ToETL
Cluster
/path4/…
events
state
events
events
events
state
state
state

13. ETL PIPELINE
HDFS
Hive Metastore
To SDM Cluster
From Streaming Cluster
Batch
Induction
Framework
state
snapshot

14. DATA DELIVERY!
HDFS
Hive Metastore
Applications
Thrift
ODBC
Server
User
Studio
Studio - Staging
GDA
Report
SDM Cluster
From ETL Cluster

15. PROS AND CONS OF OLD SYSTEM
• Isolation of workloads
• Fast ingest
• Secure
• Fast delivery/queries
• Loosely coupled clusters
• Multiple copies of data
• Tightly coupled storage and compute
• Lack of elasticity
• Operational overhead of multiple clusters

16. DYNAMIC ELASTIC ARCHITECTURE
Data Files
Streaming Cluster #1
Metastore
Ephemeral
ETL
Cluster #1
Parquet
+
DDL
Aggregates
Events
+
State
Ephemeral
Analytical
Cluster #1
Streaming
State Messages
Streaming Cluster #2
Streaming Cluster #N
Dynamic
DDL
Ephemeral
ETL
Cluster #2
Ephemeral
ETL
Cluster #N
Ephemeral
Analytical
Cluster #2
Ephemeral
Analytical
Cluster #N
Centralized Data Repository
Batch
Induction
Framework
• Rest API
• FTP downloads
• S3 sync
Batch
Download
Improvements
Single copy of data
Separate storage from compute
Elastic clusters
Reduced long running clusters to maintain
Parquet
+
DDL
Notebooks

17. STREAMING PIPELINES
Spark Cluster
Long Running Cluster

18. BATCH JOBS
Job Gateway
Spark ClusterScheduled Jobs
New cluster per Job
Dev
Machines
Spark ClusterDev Jobs
New or existing cluster
Production
Job.conf
Dev
Job.conf

19. INTERACTIVE/NOTEBOOKS
Spark Cluster
Long Running Clusters
Notebooks Scripts
(SQL, Python, Scala)
Scheduled Notebook Jobs
auto-scale
mixed on-demand &
spot Instances

20. TAKEAWAYS
Key Changes
•Centralized Hive meta store
•Leveraged S3 as centralized storage
•Separated compute and storage
•Provided horizontal scalability with
cluster elasticity
•Less time in managing infrastructure

21. TAKEAWAYS
Key Challenges
• Pushing data to S3
• Made use of parallel writes with multipart
uploads
• Moving from Hadoop YARN to Spark Standalone
• Changed from fewer large EC2 instances to many
smaller instances
• Combined Spark Streaming jobs
• Considering a move to containers for further
improved instance utilization.

22. TAKEAWAYS
Key Benefits
• Cost
• Reduce redundant storage, compute cost.
• Use the smaller instance types
• 60% AWS cost saving comparing to 1 year ago
• Operation
• Reduce the complexity of DevOps Support
• Analytics tools
• SQL only => Notebook with (SQL, Python, Scala)

23. CONFIGURABLE SPARK BATCH INGESTION FRAMEWORK
HIVE SQL  Spark

24. EVOLUTION OF DATA PLATFORM

25. BATCH INGESTION
GoPro Product data
3rd Parties Data
3rd Parties Data
3rd Parties Data
Rest APIs
sftp
s3 sync
s3 sync
Batch Data Downloads Input File Formats: CSV, JSON
Spark Cluster
New cluster per Job

26. TABLE WRITER JOBS
SparkJob
HiveTableWriter
JDBCToHiveTableWriter
AbstractCSVHiveTableWriter AbstractJSONHiveTableWriter
CSVTableWriter JSONTableWriter
FileToHiveTableWriter
HBaseToHiveTableWriter TableToHiveTableWriter
HBaseSnapshotJob
TableSnapshotJob
CoreTableWriter
Customized Json JobCustomized CSV Job
mixin
All jobs has the same way of configuration loading,
Job State and error reports
All table writers will have the Dynamic DDL
capabilities, as long as they becomes DataFrames,
they will be behave the same
CSV and JSON have
different loader
Need different Loader to
load HBase Record to
DataFrame
Aggregate Jobs

27. ETL JOB CONFIGURATION
gopro.dse.config.etl {
mobile-job {
conf {}
process {}
input {}
output {}
post.process {}
}
}
include classpath("conf/production/etl_mobile_quik.conf")
include classpath("conf/production/etl_mobile_capture.conf")
include classpath("conf/production/etl_mobile_product_events.conf")
Job-level conf override JobType Conf
Job specifics includes
JobType
JobName
Input & output specification

28. ETL JOB CONFIGURATION
xyz {
process {}
input {
delimiter = ","
inputDirPattern = "s3a://teambucket/xyz/raw/production"
file.ext = "csv"
file.format = "csv"
date.format = "yyyy-MM-dd hh:mm:ss"
table.name.extractor.method.name = "com.gopro.dse.batch.spark.job.FromFileName"
}
output {
database = “mobile",
file.format = "parquet"
date.format = "yyyy-MM-dd hh:mm:ss"
partitions = 2
file.compression.codec.key = "spark.sql.parquet.compression.codec"
file.compression.codec.value = "gzip”
save.mode = ”append"
transformers = [com.gopro.dse.batch.spark.transformer.csv.xyz.XYZColumnTransformer]
}
post.process {
deleteSource = true
}
}
Save Mode
JobName
Input specification
output specification

29. Data Transformation
ETL With SQL & Scala

30. DATA TRANSFORMATION
• HSQL over JDBC via beeline
• Suitable for non-java/scala/python-programmers
• Spark Job
• Requires Spark and Scala knowledge, need to setup job, configurations etc.
• Dynamic Scala Scripts
• Scala as script, compile Scala at Runtime, mixed with Spark SQL

31. SCALA SCRIPTS EXAMPLES -- ONE SCALA SCRIPT FILE
class CameraAggCaptureMainJob extends SparkJob {
def run(sc: SparkContext, jobType: String, jobName: String, config: Config): Unit = {
val sqlContext: SQLContext = HiveContextFactory.getOrCreate(sc)
val cameraCleanDataSchema = … //define DataFrame Schema
val = sqlContext.read.schema(ccameraCleanDataStageDFameraCleanDataSchema)
.json("s3a://databucket/camera/work/production/clean-events/final/*")
cameraCleanDataStageDF.createOrReplaceTempView("camera_clean_data")
sqlContext.sql( ""” set hive.exec.dynamic.partition.mode=nonstrict
set hive.enforce.bucketing=false
set hive.auto.convert.join=false
set hive.merge.mapredfiles=true""")
sqlContext.sql( """insert overwrite table work.camera_setting_shutter_dse_on
select row_number() over (partition by metadata_file_name order by log_ts) , …. “”” )
//rest of code
}
new CameraAggCaptureMainJob

32. Data Democratization,
Visualization and Data
Management

33. EVOLUTION OF DATA PLATFORM

34. DATA DEMOCRATIZATION & MANAGEMENT FOCUS AREAS
• BedRock: Self-Service & Data Management (ongoing project)
• Pipeline Monitoring
• Product Analytics Visualization
• Self-service Ingestion
• Other tool services

35. DSE WEB ARCHITECTURE

36. DEMO
dse.gopro-platform.com

37. EVOLUTION OF DATA PLATFORM

38. Delivery Metrics via Slack

39. SLACK METRICS DELIVERY
xxxxxx
xxxxxxx
xxxxx xxxxxxxxxx
xxxxx
xxxxxxx xxxxxx xxxxxx
xxxxx
xxxxx
xxxx
xxxxxxxxxxxxxxxx
xxxxx xxxxx
xxxxx
xxxxx
xxxxx xxxxx

40. SLACK METRICS DELIVERY
• Why Slack?
• Push vs. Pull -- Easy Access
• Avoid another Login when view metrics
• When Slack Connected, you are already login
• Put metrics generation into software engineering process
• SQL code is under software control
• publishing job is scheduled and performance is monitored
• Discussion/Question/Comments on the specific metrics can be
done directly at the channel with people involved.

41. SLACK DELIVERY FRAMEWORK
• Slack Metrics Delivery Framework
• Configuration Driven
• Multiple private Channels : Mobile/Cloud/Subscription/Web etc.
• Daily/Weekly/Monthly Delivery and comparison
• New metrics can be added easily with new SQL and configurations

42. SLACK METRICS CONCEPTS
• Slack Job 
• Channels (private channels) 
• Metrics Groups 
• Metrics1
• …
• MetricsN
• Main Query
• Compare Query (Optional)
• Chart Query (Options)
• Persistence (optional)
• Hive + S3
• Additional deliveries (Optional)
• Kafka
• Other Cache stores (Http Post)

43. BLACK KPI DELIVERY ARCHITECTURE
Slack message json
HTTP POST Rest API Server
Rest API Server
generate graphMetrics Json
Return Image
HTTP POST
Save/Get Image
Plot.ly json
Save Metrics to Hive Table
Slack Spark Job
Get Image URL
Webhooks

44. SLACK DELIVERY BENEFITS
• Pros:
• Quick and easy access via Slack
• Can quickly deliver to engineering manager, executives, business owner and product
manager
• 100+ members subscribed different channels, since we launch the service
• Cons
• Limited by Slack UI Real-States, can only display key metrics in two-column formats,
only suitable for hive-level summary metrics

45. Machine Learning Feature
Visualization with Facets + Spark

46. EVOLUTION OF DATA PLATFORM

47. FEATURE VISUALIZATION
• Explore Feature Visualization via Google Facets
• Part 1 : Overview
• Part 2: Dive
• What is Facets Overview ?

48. FACETS OVERVIEW INTRODUCTION
• From Facets Home Page
• https://pair-code.github.io/facets/
• "Facets Overview "takes input feature data from any number of datasets, analyzes them feature by
feature and visualizes the analysis.
• Overview can help uncover issues with datasets, including the following:
• Unexpected feature values
• Missing feature values for a large number of examples
• Training/serving skew
• Training/test/validation set skew
• Key aspects of the visualization are outlier detection and distribution comparison across multiple
datasets.
• Interesting values (such as a high proportion of missing data, or very different distributions of a
feature across multiple datasets) are highlighted in red.
• Features can be sorted by values of interest such as the number of missing values or the skew
between the different datasets.

49. FACETS OVERVIEW SAMPLE

50. FACETS OVERVIEW IMPLEMENTATIONS
• The Facets-overview implementation is consists of
• Feature Statistics Protocol Buffer definition
• Feature Statistics Generation
• Visualization
• Visualization
• The visualizations are implemented as Polymer web components, backed
by Typescript code
• It can be embedded into Jupyter notebooks or webpages.
• Feature Statistics Generation
• There are two implementations for the stats generation: Python and Javascripts
• Python : using numpy, pandas to generate stats
• JavaScripts: using javascripts to generate stats
• Both implementations are running stats generation in brower

51. FACETS OVERVIEW

52. FEATURE OVERVIEW SPARK
• Initial exploration attempt
• Is it possible to generate larger datasets with small stats size ?
• can we generate stats leveraging distributed computing capability
of spark instead just using one node ?
• Can we generate the stats in Spark, and then used by Python
and/or Javascripts ?

53. FACETS OVERVIEW + SPARK
ScalaPB

54. PREPARE SPARK DATA FRAME
case class NamedDataFrame(name:String, data: DataFrame)
val features = Array("Age", "Workclass", ….)
val trainData: DataFrame = loadCSVFile(”./adult.data.csv")
val testData = loadCSVFile("./adult.test.txt")
val train = trainData.toDF(features: _*)
val test = testData.toDF(features: _*)
val dataframes = List(NamedDataFrame(name = "train", train),
NamedDataFrame(name = "test", test))

55. SPARK FACETS STATS GENERATOR
val generator = new FeatureStatsGenerator(DatasetFeatureStatisticsList())
val proto = generator.protoFromDataFrames(dataframes)
persistProto(proto)

56. SPARK FACETS STATS GENERATOR
def protoFromDataFrames(dataFrames: List[NamedDataFrame],
features : Set[String] = Set.empty[String],
histgmCatLevelsCount:Option[Int]=None): DatasetFeatureStatisticsList

57. FACET OVERVIEW SPARK

58. FACET OVERVIEW SPARK
DEMO

59. INITIAL FINDINGS
• Implementation
• 1st Pass implementation is not efficient
• We have to go through each feature multiple paths, with increase number of features, the
performance suffers, this limits number of features to be used
• The size of dataset used for generate stats also determines the size of the generated protobuf file
• I haven’t dive deeper into this as to what’s contributing the change of the size
• The combination of data size and feature size can produce a large file, which won’t fit in browser
• With Spark DataFrame, we can’t support Tensorflow Records
• The Base64-encoded protobuf String can be loaded by Python or Javascripts
• Protobuf binary file can also be loaded by Python
• But it somehow not be able to loaded by Javascripts.

60. WHAT’S NEXT?
• Improve implementation performance
• When we have a lot of data and features, what’s the proper size that
generate proper stats size that can be loaded into browser or notebook ?
• For example, One experiments: 300 Features  200MB size
• How do we efficiently partition the features so that can be viewable ?
• Data is changing : how can we incremental update the stats on the regular
basis ?
• How to integrate this into production?

61. PG #
RC Playbook: Your guide to
success at GoPro
FINAL THOUGHTS

62. FINAL THOUGHTS
• We are still in the early stage of Data Platform Evolution.
• We will continue to share our experiences with you along the way.
• Questions?
Thank You
Data Science & Engineering
GoPro