Publicidad

25 de Apr de 2016•0 recomendaciones## 10 recomendaciones

•4,786 vistas## vistas

Sé el primero en que te guste

ver más

Total de vistas

0

En Slideshare

0

De embebidos

0

Número de embebidos

0

Descargar para leer sin conexión

Denunciar

Tecnología

Exploring Titan and Spark GraphX for Analyzing Time-Varying Electrical Networks

Publicidad

Publicidad

Publicidad

On-boarding with JanusGraph PerformanceChin Huang

Containerizing a Data Warehouse for KubernetesVMware Tanzu

Running Apache Spark on Kubernetes: Best Practices and PitfallsDatabricks

HDFS on Kubernetes—Lessons Learned with Kimoon KimDatabricks

Apache Spark's Built-in File Sources in DepthDatabricks

Arbitrary Stateful Aggregations using Structured Streaming in Apache SparkDatabricks

- Exploring Titan and Spark GraphX for Analyzing Time-Varying Electrical Networks Hadoop Summit 2016, Dublin, April 13th Thomas VIAL, OCTO Technology - tvial@octo.com Guillaume GERMAINE, EDF R&D - guillaume.germaine@edf.fr Marie-Luce Picard, Benoit Grossin, Martin Soppé, Michel Lutz
- 2 Outline 1. CONTEXT AND PROBLEM DESCRIPTION 2. GRAPHS: KEY CONCEPTS AND TECHNICAL INSIGHTS 3. TWO CHALLENGERS: SPARK GRAPHX AND TITAN 4. QUERYING TIME-VARYING GRAPHS 5. SCALING UP ON BIG GRAPHS 6. AS A CONCLUSION Brice Richard - Flickr KC Tan Phoyography - Flickr
- 3 Outline 1. CONTEXT AND PROBLEM DESCRIPTION 2. GRAPHS: KEY CONCEPTS AND TECHNICAL INSIGHTS 3. TWO CHALLENGERS: SPARK GRAPHX AND TITAN 4. QUERYING TIME-VARYING GRAPHS 5. SCALING UP ON BIG GRAPHS 6. AS A CONCLUSION Brice Richard - Flickr KC Tan Phoyography - Flickr
- 4 ELECTRICITY GENERATION 623.5 TWH All electricity-related activities Generation Transmission & Distribution Trading and Sales & Marketing Energy services Key figures* €72.9 billion in sales 38.5 million customers 158,161 employees worldwide 84.7% of generation does not emit CO2 2014 INVESTMENTS €4.5 BILLION EDF: A GLOBAL LEADER IN ELECTRICITY *as of 2015
- 5 Description of the French electrical network 2200 distribution substations 618 000 kilometers of medium voltage network 697 000 kilometers of low voltage network A great diversity of components: lines, transformers, switches, line brakers, meters… High voltage Medium voltage Low voltage Transmission network Distribution network More than 100 million components
- 6 French electrical network: more than 100m equipments to manage Distribution substation Meters … X2200 200+ nodes in depth ~ 50000 nodes 100m+ elements
- 7 Some business cases to tackle BC1 Get a global picture of all powered components, at any given time BC2 Track the physical evolution of the network over time (new meters, reconfiguration of the network…) BC3 Process electrical energy balance for any subpart of the network, over any period of time BC4 If an equipment fails, figure out the best way to restore power supply as quickly as possible, given the state of the network at this specific time
- 8 How to modelize the problem? define relationships between technical equipments … it’s all about graphs associate some properties to each component keep track of events across time explore the network, with complex calculations at hand We have to find a way to:
- 9 Outline 1. CONTEXT AND PROBLEM DESCRIPTION 2. GRAPHS: KEY CONCEPTS AND TECHNICAL INSIGHTS 3. TWO CHALLENGERS: SPARK GRAPHX AND TITAN 4. QUERYING TIME-VARYING GRAPHS 5. SCALING UP ON BIG GRAPHS 6. AS A CONCLUSION Brice Richard - Flickr KC Tan Phoyography - Flickr
- 10 A property graph as a data model Edge Vertex Properties key1:value key2:value key1:value key2:value key1:value key2:value Label
- 11 All components of the network are modelized as vertices: • Distribution substations • Switches • Line brakers • Transformers • Loads • Lines • … Edges only represent symbolic, non-oriented links between components Electrical lines are also modelized as vertices! A property graph as a data model Load curves (metering data) are not stored as properties into the graph structure, but in a separate storage (HBase in this study)
- 12 Graphs and temporal data: not so simple! A unique graph to record every event that has ever occurred All events are recorded as vertex properties, in the form of time intervals: Validity: track equipments life cycle (actual period as a working unit) Failure: track equipments failures SwitchState: track network reconfiguration through switches changes of state (open/close) Each time an event occurs, a time interval is updated, or created The concept: apply graph mutations at each new event, enriching the “maximal” graph
- 13 Graphs and temporal data: not so simple! The concept: apply graph mutations at each new event, enriching the “maximal” graph Line LoadSrc Line Load Switch Line Validity : ]-∞, +∞[ Failure : [] LoadLine Validity : ]-∞, +∞[ Failure : [] Validity : ]-∞, +∞[ Failure : [] Validity : [t1, +∞[ Failure : [] SwitchState : [] DefaultState : closed Validity : ]-∞, +∞[ Failure : [] Validity : ]-∞, +∞[ Failure : [] Validity : ]-∞, +∞[ Failure : [] Validity : ]-∞, t2] Failure : [] Validity : ]-∞, +∞[ Failure : [t3, +∞[ Validity : [t1, +∞[ Failure : [] SwitchState : [t4, +∞[ DefaultState : closed
- 14 Outline 1. CONTEXT AND PROBLEM DESCRIPTION 2. GRAPHS: KEY CONCEPTS AND TECHNICAL INSIGHTS 3. TWO CHALLENGERS: SPARK GRAPHX AND TITAN 4. QUERYING TIME-VARYING GRAPHS 5. SCALING UP ON BIG GRAPHS 6. AS A CONCLUSION Brice Richard - Flickr KC Tan Phoyography - Flickr
- 15 Spark GraphX: global architecture Offers a graph API built on top of Spark Extends RDD to represent property graphs: VertexRDD and EdgeRDD Implements a collection of graph algorithms (e.g., Page Rank, Triangle Counting, Connected Components…) and also some fundamental operations on graphs (e.g., mapVertices, mapEdges, subgraph, collectNeighbors…)
- 16 Spark GraphX: Pregel Implements a variant of Pregel, a very popular graph processing architecture developed at Google in 2010: “Pregel: a system for large-scale graph processing - Malewicz et al.” Based on BSP (Bulk Synchronous Parallel) Vertex-centric: edges don't carry any computation Runs in sequence of iterations (supersteps) For each superstep, every vertex can: • read messages sent to it during the previous superstep • execute a user-defined function and generate some messages • send messages to other vertices (that will be received at the next superstep) • vote to halt
- 17 Spark GraphX: Pregel collecte fusion Vertex Vertex Vertex Vertex Vertex Vertex Superstep n-1 Superstep n Superstep n+1 sendMsg sendMsg sendMsg (other vertices) sendMsg sendMsg (other vertices) mergeMsg vprog haltVertex
- 18 TITAN: global architecture Optimized for storing and querying billions of vertices and edges over a cluster Supports thousands of concurrent users Can execute local queries (OLTP) or distributed queries across a cluster (OLAP)
- 19 TITAN: GREMLIN Gremlin OLTP: for local graphs traversal, queries run in a single process Gremlin Hadoop (OLAP): for large graph analysis, queries are distributed across a cluster Gremlin Hadoop implements BSP-based vertex-centric computing Gremlin is a graph traversal scripting language and is part of Tinkerpop, a widely supported open-source graph computing framework (e.g., Neo4J, OrientDB, Sparksee…) Blueprints is like a JDBC driver for graphs +++ Storage backend Graph Database Tinkerpop API Graph Processing
- 20 Outline 1. CONTEXT AND PROBLEM DESCRIPTION 2. GRAPHS: KEY CONCEPTS AND TECHNICAL INSIGHTS 3. TWO CHALLENGERS: SPARK GRAPHX AND TITAN 4. QUERYING TIME-VARYING GRAPHS 5. SCALING UP ON BIG GRAPHS 6. AS A CONCLUSION Brice Richard - Flickr KC Tan Phoyography - Flickr
- 21 Traversing a time-varying graph We have as inputs • A period of observation [t1, t2] • A particular vertex as a starting point We know, for each vertex in the graph, when it was inactive • For an equipment: taken down, under maintenance, commutated, … We want to traverse the graph, following the paths as they vary according to the states of the vertices encountered We devise a “Trickling” algorithm that can be easily implemented with either paradigm, Pregel or Gremlin
- 22 & = & = & = & = & = & = ? t1 t2 What paths from source S can we follow between t1 and t2? VERTEX ACTIVITY OBSERVATION ACTUAL FLOW S Trickling down
- 23 = = = == = What vertices are reachable from substation S at time t? S CONNECTED CONNECTED CONNECTED CONNECTEDCONNECTED DISCONNECTED t t+ε ACTUAL FLOW BC1 – Get a picture of all powered components ?
- 24 = = = == = t1 t2 For how long were vertices reachable from S actually powered between t1 and t2? ∑ intervals = 100% ∑ intervals = 100% ∑ intervals = 75% ∑ intervals = 55%∑ intervals = 75% ∑ intervals = 0% ACTUAL FLOWS BC2 – Track how long equipments are powered ?
- 25 == = How much aggregated power did flow from substation S to leaves between t1 and t2? ∑ loads = 1,988 MWh 1,132 MWh 856 MWh 0 MWh kW kW kW S t1 t2 BC3 – Manage energy balance ?
- 26 Trickling with Gremlin
- 27 Trickling with GraphX
- 28 From what other substations S’ can vertices below S be reached? Pattern matching S S’ SUBSTATION LINE SWITCH SUBSTATION BC4 – Restore power supply after a failure ? SWITCH (many things)
- 29 ^[d]+.*$! // God created comments, and saw that it was good Pattern matching with Gremlin
- 30 Outline 1. CONTEXT AND PROBLEM DESCRIPTION 2. GRAPHS: KEY CONCEPTS AND TECHNICAL INSIGHTS 3. TWO CHALLENGERS: SPARK GRAPHX AND TITAN 4. QUERYING TIME-VARYING GRAPHS 5. SCALING UP ON BIG GRAPHS 6. AS A CONCLUSION Brice Richard - Flickr KC Tan Phoyography - Flickr
- 31 Titan at scale A set of connected trees adding up to ~ 100 million vertices and ~ 100 million edges, loaded on a 10-node HBase 1.1 cluster • The quasi-tree below each substation S contains roughly 50,000 vertices • The Titan table is made of 3 regions that are equally sollicited • The client machine is on the same network as the HBase servers Unitary execution with Titan’s OLTP interface Query Approximate time BC1 – Get powered components Trickling 1 min BC3 – Get aggregated load Trickling 2 min BC4 – Find backup substations Pattern matching 1 min
- 33 Scaling across substations The execution times above are for a single source substation S To compute over several substations, we run the same query in parallel with a different input Client node Titan + Gremlin Query Query Query Query Query Query Query Query Query … … … HBase Cluster
- 34 Scaling across substations – Results
- 35 What about Titan OLAP? The execution times above are predictible but quickly become impractical for interactive querying We may want to compute some KPIs in advance with OLAP backends, re-using the same Gremlin queries Titan 1.0.0 or 1.1.0-snapshot? TinkerPop 3.0.1 or 3.1.0? Titan + TinkerPop JARs or the other way around? Giraph or Spark as the backend? Be patient! Support for Hadoop 2 OLAP is still limited. Bits are being moved around between the two projects, which are undergoing a big refactoring…
- 36 Next step: GraphX Spark GraphX is another natural candidate for OLAP workloads We have yet to benchmark it against our big graph But wait! GraphFrames for Spark is just being released by Databricks. What’s going on? http://graphframes.github.io/
- 37 Outline 1. CONTEXT AND PROBLEM DESCRIPTION 2. GRAPHS: KEY CONCEPTS AND TECHNICAL INSIGHTS 3. TWO CHALLENGERS: SPARK GRAPHX AND TITAN 4. QUERYING TIME-VARYING GRAPHS 5. SCALING UP ON BIG GRAPHS 6. AS A CONCLUSION Brice Richard - Flickr KC Tan Phoyography - Flickr
- 38 A turning point for graph analytics A lot is happening these days • ThinkAurelius’s acquisition by Datastax (will they continue to support HBase?) • Titan and TinkerPop cross-refactorings • Improvements in the computation backends (Giraph, Spark) • Better support for Hadoop 2 and YARN • Introduction of GraphFrames by Databricks Graph analytics frameworks are becoming commodity, and this is a good thing!
- 39 Which framework to use? A quick sketch about the typical usages of the frameworks: Be sure to test them thoroughly in your environment before making a final decision. Be prepared to go deep into Hadoop and Java/Scala internals! Or maybe wait a little bit until the situation gets clearer? Framework Usages Titan OLTP Interactive querying, for a relatively small number of vertices Titan OLAP Batch computations, KPI (but wait for Hadoop 2 support if you’re using HBase) Spark GraphX Batch computations, KPI GraphFrames ? (it’s too early)
- 40 Appendix Appendix A: The ecosystem of graph-based solutions (Graph Databases) Appendix B: The ecosystem of graph-based solutions (Graph Processing frameworks) Appendix C: Graph traversal Appendix D: Architecture with Spark GraphX Appendix E: Architecture with Titan
- 41 Appendix A: The ecosystem of graph-based solutions Graph Databases (OLTP) Optimized for local graph exploration (traversal), with low latency Optimized for handling multiple concurrent users Data can be distributed across several machines Queries themselves are not distributed: global graph analyses are inefficient
- 42 Appendix B: The ecosystem of graph-based solutions Graph Processing Frameworks (OLAP) Optimized for global graph processing (batch) Queries and data are distributed across a cluster, and can handle very large graphs Has a higher latency than OLTP solutions Cannot handle a lot of concurrent users
- 43 Job 4Job 2 Job 3Job 1 Appendix C: Graph traversal Src Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Src Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Vtx Computation nodes Depth-first: Breadth-first: Hard to parallelize ! Low latency High latency
- 44 Spark Edges Vertices Vertex States Load curves Edge RDD Vertex RDD GraphX processors Use case 1 Use case 2 Use case 3 … Final result Add intervals as properties Combine Appendix D: Architecture with GraphX
- 45 JVM Groovy scripts library Edges Vertices Load curves Titan APIs Transactional Gremlin Add intervals as properties Combine load curves HBase API Use case 1 Use case 2 … Vertex States Appendix E: Architecture with Titan Store load curves as time series

Publicidad