Interactive Apache Spark in Your Browser
•
12 recomendaciones•3,388 vistas
Presented by Romain Rigaux at Spark Lightning Night 2015
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Interactive Apache Spark in Your Browser
Similar a Interactive Apache Spark in Your Browser (20)
Más de Cloudera, Inc.
Más de Cloudera, Inc. (20)
Último
Último (20)
Interactive Apache Spark in Your Browser
- 1. INTERACTIVE SPARK IN YOUR BROWSER Spark Lightning Night -‐ Hadoop World September 2015
- 4. GOAL OF HUE WEB INTERFACE FOR ANALYZING DATA WITH APACHE HADOOP SIMPLIFY AND INTEGRATE FREE AND OPEN SOURCE —> “BIG NOTEBOOK” FOR HADOOP
- 8. • Married with full ecosystem • File, Job browsers • Create table wizards • Any language (Hive, Spark...) • Graphing • Export/Import/Sharing • MulT users • ImpersonaTon HADOOP WITH SPARK NOTEBOOK
- 10. • REST Web server in Scala • InteracTve Spark Sessions and Batch Jobs • Type IntrospecTon for VisualizaTon • Running sessions in YARN local • Backends: Scala, Python, R • Open Source: h[ps://github.com/cloudera/hue/ tree/master/apps/spark/java • Play with Curl h[p://gethue.com/how-‐to-‐use-‐the-‐ LIVY SPARK SERVER
- 11. LIVY WEB SERVER ARCHITECTURE YARN Master Spark Client YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker Livy Server Scalatra Session Manager Session
- 12. LIVY WEB SERVER ARCHITECTURE Livy Server YARN Master Scalatra Spark Client Session Manager Session YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker 1
- 13. LIVY WEB SERVER ARCHITECTURE YARN Master Spark Client YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker 1 2 Livy Server Scalatra Session Manager Session
- 14. LIVY WEB SERVER ARCHITECTURE YARN Master Spark Client YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker 1 2 3 Livy Server Scalatra Session Manager Session
- 15. LIVY WEB SERVER ARCHITECTURE YARN Master Spark Client YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker 1 2 3 4 Livy Server Scalatra Session Manager Session
- 16. LIVY WEB SERVER ARCHITECTURE YARN Master Spark Client YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker 1 2 3 4 5 Livy Server Scalatra Session Manager Session
- 17. LIVY WEB SERVER ARCHITECTURE YARN Master Spark Client YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker 1 2 3 4 5 6 Livy Server Scalatra Session Manager Session
- 18. LIVY WEB SERVER ARCHITECTURE YARN Master Spark Client YARN Node Spark Interpreter Spark Context YARN Node Spark Worker YARN Node Spark Worker 1 7 2 3 4 5 6 Livy Server Scalatra Session Manager Session
- 19. SESSION CREATION AND EXECUTION % curl -‐XPOST localhost:8998/sessions -‐d '{"kind": "spark"}' { "id": 0, "kind": "spark", "log": [...], "state": "idle" } % curl -‐XPOST localhost:8998/sessions/0/statements -‐d '{"code": " { "id": 0, "output": { "data": { "text/plain": "res0: Int = 2" }, "execution_count": 0, "status": "ok" }, "state": "available" }
- 20. DEMO TIME
- 21. • Stability and Scaling • Security • iPython/Jupyter interpreter and file format COMING SOON