Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a SparkR: Enabling Interactive Data Science at Scale
Similar a SparkR: Enabling Interactive Data Science at Scale (20)
Más de jeykottalam
Más de jeykottalam (8)
Último
Último (20)
SparkR: Enabling Interactive Data Science at Scale
- 1. SparkR: Enabling Interactive Data Science at Scale Shivaram Venkataraman Zongheng Yang
- 2. Fast ! Scalable Flexible
- 3. Statistics ! Packages Plots
- 4. Fast ! Scalable Flexible Statistics ! Plots Packages
- 5. Outline SparkR API Live Demo Design Details
- 6. RDD Parallel Collection Transformations map filter groupBy … Actions count collect saveAsTextFile …
- 7. R + RDD = R2D2
- 8. R + RDD = RRDD lapply lapplyPartition groupByKey reduceByKey sampleRDD collect cache filter … broadcast includePackage textFile parallelize
- 9. SparkR – R package for Spark R RRDD Spark
- 10. Example: word_count.R library(SparkR) lines <-‐ textFile(sc, “hdfs://my_text_file”)
- 11. Example: word_count.R library(SparkR) lines <-‐ textFile(sc, “hdfs://my_text_file”) words <-‐ flatMap(lines, function(line) { strsplit(line, " ")[[1]] }) wordCount <-‐ lapply(words, function(word) { list(word, 1L) })
- 12. Example: word_count.R library(SparkR) lines <-‐ textFile(sc, “hdfs://my_text_file”) words <-‐ flatMap(lines, function(line) { strsplit(line, " ")[[1]] }) wordCount <-‐ lapply(words, function(word) { list(word, 1L) }) counts <-‐ reduceByKey(wordCount, "+", 2L) output <-‐ collect(counts)
- 14. MNIST
- 15. A b || Ax − b ||2 Minimize x = (ATA)−1ATb
- 16. How does this work ?
- 17. Dataflow Local Worker Worker
- 18. Dataflow Local R Worker Worker
- 19. Dataflow Local R Spark Context Java Spark Context JNI Worker Worker
- 20. Dataflow Local Worker R Spark Context Worker Java Spark Context JNI Spark Executor exec R Spark Executor exec R
- 24. Dataflow Local Worker R Spark Context Worker Java Spark Context JNI Spark Executor exec R Spark Executor exec R
- 25. Pipelined RDD words <-‐ flatMap(lines,…) wordCount <-‐ lapply(words,…) Spark Executor exec R Spark Executor R exec
- 26. Pipelined RDD Spark Executor exec R Spark Executor R exec Spark Executor exec R R Spark Executor
- 27. Alpha developer release One line install ! install_github("amplab-‐extras/SparkR-‐pkg", subdir="pkg")
- 28. SparkR Implementation Very similar to PySpark Spark is easy to extend 329 lines of Scala code 2079 lines of R code 693 lines of test code in R
- 29. EC2 setup scripts All Spark examples MNIST demo YARN, Windows support Also on github
- 30. Developer Community 13 contributors (10 from outside AMPLab) Collaboration with Alteryx
- 31. On the Roadmap High level DataFrame API Integrating Spark’s MLLib from R Merge with Apache Spark
- 32. SparkR RDD à distributed lists Run R on clusters Re-use existing packages Combine scalability & utility
- 33. SparkR https://github.com/amplab-extras/SparkR-pkg Shivaram Venkataraman shivaram@cs.berkeley.edu Zongheng Yang zongheng.y@gmail.com SparkR mailing list sparkr-dev@googlegroups.com