Distributed Graph Analytics with Gradoop

Distributed Graph Analytics with Gradoop
inovex Meetup Munich
Let‘s talk about Graph Databases
July 2016
Martin Junghanns (@kc1s)
University of Leipzig – Database Research Group

Motivation
Extended Property Graph Model
Operators
Benchmark
Implementation

Distributed Graph Analytics with Gradoop – inovex Meetup Munich – July 2016 3
Motivation EPGM Operators BenchmarkImplementation
3
Motivation

4
Motivation
𝑮𝑟𝑎𝑝ℎ = (𝑽𝑒𝑟𝑡𝑖𝑐𝑒𝑠, 𝑬𝑑𝑔𝑒𝑠)
„Graphs are everywhere“

5
Motivation
Alice
Bob
Eve
Dave
Carol
Mallory
Peggy
Trent
𝐺𝑟𝑎𝑝ℎ = (𝐔𝐬𝐞𝐫𝐬, 𝐹𝑜𝑙𝑙𝑜𝑤𝑒𝑟𝑠)

6
Motivation
𝐺𝑟𝑎𝑝ℎ = (𝐔𝐬𝐞𝐫𝐬, 𝐹𝑟𝑖𝑒𝑛𝑑𝑠ℎ𝑖𝑝𝑠)
Alice
Bob
Eve
Dave
Carol
Mallory
Peggy
Trent

7
Motivation
Alice
Bob
AC/DC
Dave
Carol
Mallory
Peggy
Metallica
𝐺𝑟𝑎𝑝ℎ = (𝐔𝐬𝐞𝐫𝐬 ∪ 𝐁𝐚𝐧𝐝𝐬, 𝐹𝑟𝑖𝑒𝑛𝑑𝑠ℎ𝑖𝑝𝑠 ∪ 𝐿𝑖𝑘𝑒𝑠)
„Graphs are heterogeneous“

8
Motivation
Alice
Bob
AC/DC
Dave
Carol
Mallory
Peggy
Metallica
„Graphs can be analyzed“

9
Motivation
0.2
0.28
0.26
0.33
0.25
0.26
Alice
Bob
AC/DC
Dave
Carol
Mallory
Peggy
Metallica
3.6
2.82
„Graphs can be analyzed“

10
Motivation
Assuming a social network

11
Motivation
1. Determine subgraph

12
Motivation

13
Motivation
2. Find communities

14
Motivation
2. Find communities

15
Motivation
2. Find communities
3. Filter communities

16
Motivation
2. Find communities

17
Motivation
2. Find communities
4. Find common subgraph

18
Motivation
2. Find communities

19
Motivation
• Heterogeneous data
• Apply graph transformation
2. Find communities
• Handle collections of graphs
• Aggregation, Selection
• Apply dedicated algorithm

20
Motivation
2. Find communities

21
Motivation
2. Find communities

22
Motivation
2. Find communities

23
Motivation
2. Find communities

24
Motivation
„And let‘s not forget …“

25
Motivation
“...Graphs are large.”

26
Motivation
„An open-source framework and research platform for
efficient, distributed and domain independent
management and analytics of heterogeneous graph data.“

27
Motivation
Data Volume and Problem Complexity
Ease-of-use
Graph Processing Systems
Graph Databases
Graph Dataflow Systems Gelly

28
Motivation
Distributed Graph Store (Apache HBase)
Apache Flink Operator Implementation
Apache Flink Distributed Operator Execution
Extended Property Graph Model (EPGM)
Graph Analytical Language (GrALa)
I/O
Distributed File System (Apache HDFS)

29
Extended Property Graph Model
(EPGM)

Motivation EPGM Operators BenchmarkImplementationEPGM
• Vertices and directed Edges

• Logical Graphs

• Logical Graphs
• Identifiers
1 3
4
5
2
1 2
3
4
5
1
2

• Logical Graphs
• Identifiers
• Type Labels
1 3
4
5
2
1 2
3
4
5
Person Band
Person
Person
Band
likes likes
likes
knows
likes
1|Community
2|Community

• Logical Graphs
• Identifiers
• Type Labels
• Properties
1 3
4
5
2
1 2
3
4
5
Person
name : Alice
born : 1984
Band
name : Metallica
founded : 1981
Person
name : Bob
Person
name : Eve
Band
name : AC/DC
founded : 1973
likes
since : 2014
likes
since : 2013
likes
since : 2015
knows
likes
since : 2014
1|Community|interest:Heavy Metal
2|Community|interest:Hard Rock

35
Operators

36
Operators
Operators
Unary Binary
GraphCollectionLogicalGraph
Algorithms
Aggregation
Pattern Matching
Transformation
Grouping Equality
Call
Combination
Overlap
Exclusion
Equality
Union
Intersection
Difference
Flink Gelly Library
BTG Extraction
Frequent Subgraphs
Limit
Selection
Distinct
Sort
Apply
Reduce
Call
Adaptive Partitioning
Subgraph

37
Operators
1 3
4
5
2
3
1 2
1 3
4
5
2
1
2 4
5
Combination
Overlap
Exclusion
3Basic Binary Operators
LogicalGraph graph3 = graph1.combine(graph2);
LogicalGraph graph4 = graph1.overlap(graph2);
LogicalGraph graph5 = graph1.exclude(graph2);

38
Operators
1 3
4
5
2
3
1 3
4
5
2
3 | vertexCount: 5
UDF
Aggregation
graph3 = graph3.aggregate(“vertexCount”, new AggregateFunction<Long>() {
public DataSet<Long> execute(LogicalGraph g) {
return Count.count(g.getVertices());
}
});

39
Operators
UDF
3 | vertexCount: 5
name:Alice
f_name:Bob1 3
4
5
2
3 | Community| vCount: 5
f_name:Alice
f_name:Bob1 3
4
5
2
Transformation
graph3 = graph3.transformEdges(new TransformationFunction<Edge>() {
public Edge execute(Edge e) {
e.setLabel(e.getLabel().equals(“orange”) ? “red” : e.getLabel());
return e;
}});

40
Operators
3
1 3
4
5
2
3
4
1 2
3
4
1 2
4
3
5
2
UDF
UDF
UDF
Subgraph
LogicalGraph graph4 = graph3.subgraph(
new FilterFunction<Vertex>() {
public boolean execute(Vertex v) { return v.getLabel().equals(“green”); }},
new FilterFunction<Edge>() {
public boolean execute(Edge e) { return e.getLabel().equals(“orange”); }});

41
Operators
3
1 3
4
5
2 Pattern
4 5
1 3
4
2
Graph Collection
Pattern Matching
GraphCollection collection = graph3.match(“(:Green)-[:orange]->(:Orange)”);

42
Operators
Keys
3
1 3
4
5
2
+Aggregate
3
a:23 a:84
a:42
a:12
1 3
4
5
2
a:13
a:21
4
count:2 count:3
max(a):42
max(a):84
max(a):13 max(a):21
6 7
4
6 7
Grouping
LogicalGraph grouped = graph3.groupBy()
.useVertexLabel()
.useEdgeLabel()
.addVertexAggregate(new CountAggregator())
.addEdgeAggregate(new MaxAggregator(“a”));

43
Operators
Operator
1
2
0 2
3
4
1
5 7 86
1 | vertexCount: 5
2 | vertexCount: 4
0 2
3
4
1
5 7 86
Apply (e.g. Aggregation)
collection = collection.apply(new Aggregation<>(“vertexCount”, new VertexCount()));

44
Operators
UDF
vertexCount > 4
1 | vertexCount: 5
2 | vertexCount: 4
0 2
3
4
1
5 7 86
1 | vertexCount: 5
0 2
3
4
1
Selection
GraphCollection filtered = collection.select(new FilterFunction<GraphHead>() {
public boolean filter(GraphHead g) {
return g.getPropertyValue(“vertexCount”).getLong() > 4L;
}
});

45
Operators
Algorithm
1
0 2
3
4
1
5 7 86
2
3
0 2
3
4
1
5 7 86
Call (e.g. Clustering)
GraphCollection clustering = graph.callForCollection(new ClusteringAlgorithm());

46
Operators
Algorithm
2
rank:0.11 rank:0.25
rank:0.11
rank:1.29
rank:1.29
rank:1.58rank:0.11
rank:0.75rank:0.11
0 2
3
4
1
5 7 86
1
0 2
3
4
1
5 7 86
Call (e.g. Page Rank)
LogicalGraph pageRankGraph = graph.callForGraph(new PageRankAlgorithm());

47
Implementation
Apache Flink Gradoop on Flink

48
Implementation
„Streaming Dataflow Engine that provides data distribution, communication and fault
tolerance for distributed computations over data streams.“
https://flink.apache.org/
Streaming Dataflow Runtime
DataSet DataStream
HadoopMR
Table
Gelly
FlinkML
Table
Zeppelin
Cascading
MRQL
Dataflow
Storm
Dataflow
SAMOA
GRADOOP
Cluster (e.g. YARN)Local Cloud (e.g. EC2)
Batch Stream
Data Storage (e.g. Files, HDFS, S3, JDBC, HBase, Kafka, …)

49
Implementation
DataSetDataSetDataSet
• DataSet := Distributed Collection of Data Objects
• Transformation := Operation on DataSets (Higher-order function)
• Flink Programm := Composition of Transformations
DataSet
DataSet
DataSet
Transformation
Transformation
DataSet
DataSet
Transformation DataSet
Flink Program

50
Implementation
Hadoop-like Transformations
• map
• flatMap
• mapPartition
• reduce
• reduceGroup
• coGroup
Special Flink Operations
• iterate
• iterateDelta
SQL-like Transformations
• filter
• project
• cross
• union
• distinct
• first-N (limit)
• groupBy
• aggregate
• join
• leftOuterJoin
• rightOuterJoin
• fullOuterJoin

51
Implementation
1: ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
2:
3: DataSet<String> text = env.fromElements( // or env.readTextFile(„hdfs://…“)
4: „He who controls the past controls the future.“,
5: „He who controls the present controls the past.“);
6:
7: DataSet<Tuple2<String, Integer>> wordCounts = text
8: .flatMap(new LineSplitter()) // splits the line and outputs (word, 1) tuples
9: .groupBy(0)
10: .sum(1);
11:
12: wordCounts.print(); // trigger execution
flatMap
„He who controls the past controls the future.“
„He who controls the present controls the past.“
(He,1)
(who,1)
(controls,1)
(the,1)
(past,1)
// ...
groupBy(0)
[(He,1),(He,1)]
[(who,1),(who,1)]
[(future,1)]
[(past,1),(past,1)]
[(present,1)]
// ...
sum(1)
(He,2)
(who,2)
(future,1)
(past,2)
(present,1)
// ...

52
Implementation
flatMap
(He,1)
(who,1)
(controls,1)
groupBy(0)
[(He,1),(He,1)]
[(who,1),(who,1)]
sum(1)
(He,2)
(who,2)
Source flatMap
(the,1)
(past,1)
groupBy(0)
[(future,1)]
[(past,1),(past,1)]
sum(1)
(future,1)
(past,2)
flatMap
(future,1)
(past,1)
groupBy(0)
[(present,1)]
sum(1)
(present,1)
Sink

53
Implementation
Id Label Properties Graphs
Id Label Properties SourceId TargetId Graphs
EPGMGraphHead
EPGMVertex
EPGMEdge
Id Label Properties POJO
POJO
POJO
DataSet<EPGMGraphHead>
DataSet<EPGMVertex>
DataSet<EPGMEdge>
EPGMVertex
GradoopId := UUID
128-bit
String PropertyList := List<Property>
Property := (String, PropertyValue)
PropertyValue := byte[]
GradoopIdSet := Set<GradoopId>
EPGM Graph Representation

54
Implementation
Id Label Properties
1 Community {interest:Heavy Metal}
2 Community {interest:Hard Rock}
1 Person {name:Alice, born:1984} {1}
2 Band {name:Metallica,founded:1981} {1}
3 Person {name:Bob} {1,2}
4 Band {name:AC/DC,founded:1973} {2}
5 Person {name:Eve} {2}
Id Label Source Target Properties Graphs
1 likes 1 2 {since:2014} {1}
2 likes 3 2 {since:2013} {1}
3 likes 3 4 {since:2015} {2}
4 knows 3 5 {} {2}
5 likes 5 4 {since:2014} {2}
likes
since : 2014
likes
since : 2013
1 3
4
5
2
Person
name : Alice
born : 1984
Band
name : Metallica
founded : 1981
Person
name : Bob
Person
name : Eve
Band
name : AC/DC
founded : 1973likes
since : 2015
knows
likes
since : 2014
1 2
3
4
5
DataSet<EPGMGraphHead>
DataSet<EPGMVertex> DataSet<EPGMEdge>

55
Implementation
LogicalGraph grouped = graph1.combine(graph2).groupBy()
.useVertexLabel()
.useEdgeLabel()
.addVertexAggregate(new CountAggregator())
.addEdgeAggregate(new CountAggregator());
6 7
Person
count : 3
Band
count : 2
likes
count : 4
knows
count : 1
6
7
4
likes
since : 2014
likes
since : 2013
1 3
4
5
2
Person
name : Alice
born : 1984
Band
name : Metallica
founded : 1981
Person
name : Bob
Person
name : Eve
Band
name : AC/DC
founded : 1973likes
since : 2015
knows
likes
since : 2014
1 2
3
4
5

56
Implementation
GroupBy(1,2,3) +
GC + GR* + Map
Assign edges to groups
Compute aggregates
Build super edges
Filter + Map
Extract super vertex tuples
Build super vertices
GroupBy(1) + GroupReduce*
Assign vertices to groups
Compute aggregates
Create super vertex tuples
Forward updated group members
V
E
(1,[Person],[])
(2,[Band],[])
(3,[Person],[])
(4,[Band],[])
(5,[Person],[])
(-,6,[Person],[3])
(1,6,[],[])
(-,7,[Band],[2])
(2,7,[],[])
(3,6,[],[])
(4,7,[],[])
(5,6,[],[])
v6
v7
(1,6)
(2,7)
(3,6)
(4,7)
(5,6)
(1,1,2,[likes],[])
(2,3,2,[likes],[])
(3,3,4,[likes],[])
(4,3,5,[knows],[])
(5,5,4,[likes],[])
(1,6,7,[likes],[])
(2,6,7,[likes],[])
(3,6,7,[likes],[])
(4,6,6,[knows],[])
(5,6,7,[likes],[])
e6
e7
Map
Extract
attributes
Filter + Map
Extract group members
Reduce memory footprint
Join*
Replace Source/TargetId
with corresponding super
vertex id
Map
Extract
attributes
*requires worker communication

57
Implementation
class LogicalGraph<G extends EPGMGraphHead,
V extends EPGMVertex,
E extends EPGMEdge> {
fromCollections(...) : LogicalGraph<G, V, E>
fromDataSets(...) : LogicalGraph<G, V, E>
fromGellyGraph(...) : LogicalGraph<G, V, E>
getGraphHead() : DataSet<G>
getVertices() : DataSet<V>
getEdges() : DataSet<E>
aggregate(...) : LogicalGraph<G, V, E>
match(...) : GraphCollection<G, V, E>
groupBy(...) : LogicalGraph<G, V, E>
subgraph(...) : LogicalGraph<G, V, E>
combine(...) : LogicalGraph<G, V, E>
// ...
}
class GraphCollection<G extends EPGMGraphHead,
E extends EPGMEdge > {
fromCollections(...) : GraphCollection<G, V, E>
fromDataSets(...) : GraphCollection<G, V, E>
getGraphHeads() : DataSet<G>
getVertices() : DataSet<V>
getEdges() : DataSet<E>
select(...) : GraphCollection<G, V, E>
distinct( ) : GraphCollection<G, V, E>
sortBy(...) : GraphCollection<G, V, E>
union(...) : GraphCollection<G, V, E>
difference(...) : GraphCollection<G, V, E>
// ...
}
EPGM API (Operators)

58
Implementation
interface DataSource<G extends EPGMGraphHead,
E extends EPGMEdge> {
getLogicalGraph(...) : LogicalGraph<G, V, E>
getGraphCollection(...) : GraphCollection<G, V, E>
}
interface DataSink<G extends EPGMGraphHead,
E extends EPGMEdge > {
write(LogicalGraph<G, V, E>) : void
write(GraphCollection<G, V, E>) : void
}
class GraphDataSource<...> implements DataSource<...> { }
class HBaseDataSource<...> implements DataSource<...> { }
class JSONDataSource<...> implements DataSource<...> { }
class TLFDataSource<...> implements DataSource<...> { }
class HBaseDataSink<...> implements DataSink<...> { }
class JSONDataSink<...> implements DataSink<...> { }
class TLFDataSink<...> implements DataSource<...> { }
EPGM API (I/O)

59
Benchmark

60
Benchmark
1. Extract subgraph containing only Persons and knows relations
2. Transform Persons to necessary information
3. Find communities using Label Propagation
4. Aggregate vertex count for each community
5. Select communities with more than 50K users
6. Combine large communities to a single graph
7. Group graph by Persons location and gender
8. Aggregate vertex and edge count of grouped graph
http://ldbcouncil.org/

61
Benchmark
1. Extract subgraph containing only Persons and knows relations
2. Transform Persons to necessary information
3. Find communities using Label Propagation
4. Aggregate vertex count for each community
5. Select communities with more than 50K users
6. Combine large communities to a single graph
7. Group graph by Persons location and gender
8. Aggregate vertex and edge count of grouped graph
https://git.io/vgozj

62
Benchmark
Dataset # Vertices # Edges Disk size
Graphalytics.1 61,613 2,026,082 570 MB
Graphalytics.10 260,613 16,600,778 4.5 GB
Graphalytics.100 1,695,613 147,437,275 40.2 GB
Graphalytics.1000 12,775,613 1,363,747,260 372 GB
Graphalytics.10000 90,025,613 10,872,109,028 2.9 TB
• 16x Intel(R) Xeon(R) 2.50GHz 6 (12)
• 16x 48 GB RAM
• 1 Gigabit Ethernet
• Hadoop 2.6.0
• Flink 1.0-SNAPSHOT
• slots (per worker) 12
• jobmanager.heap.mb 2048
• taskmanager.heap.mb 40960

63
Benchmark
Graphalytics.1 61,613 2,026,082 570 MB
Graphalytics.10 260,613 16,600,778 4.5 GB
Graphalytics.100 1,695,613 147,437,275 40.2 GB
Graphalytics.1000 12,775,613 1,363,747,260 372 GB
Graphalytics.10000 90,025,613 10,872,109,028 2.9 TB
• 16x 48 GB RAM
• Hadoop 2.6.0
0
200
400
600
800
1000
1200
1 2 4 8 16
Runtime[s]
Number of workers
Graphalytics.100

64
Benchmark
1
2
4
8
16
1 2 4 8 16
Speedup
Number of workers
Graphalytics.100 Linear
Graphalytics.1 61,613 2,026,082 570 MB
Graphalytics.10 260,613 16,600,778 4.5 GB
Graphalytics.100 1,695,613 147,437,275 40.2 GB
Graphalytics.1000 12,775,613 1,363,747,260 372 GB
Graphalytics.10000 90,025,613 10,872,109,028 2.9 TB
• 16x 48 GB RAM
• Hadoop 2.6.0

65
Benchmark
1
10
100
1000
10000
Runtime[s]
Graphalytics.1 61,613 2,026,082 570 MB
Graphalytics.10 260,613 16,600,778 4.5 GB
Graphalytics.100 1,695,613 147,437,275 40.2 GB
Graphalytics.1000 12,775,613 1,363,747,260 372 GB
Graphalytics.10000 90,025,613 10,872,109,028 2.9 TB
• 16x 48 GB RAM
• Hadoop 2.6.0

Summary
• 0.0.1 First Prototype (May 2015)
– Hadoop MapReduce and Giraph for operator implementations
– Too much complexity
– Performance loss through serialization in HDFS/HBase
• 0.0.2 Using Flink as execution layer (June 2015)
– Basic operators
• 0.1 December 2015
– System-side identifiers (UUID)
– Improved property handling
– More operator implementations (e.g., Equality, Bool operators)
– Code refactoring
• 0.2-SNAPSHOT August 2016
– Graph Pattern Matching 
– Frequent Subgraph Mining 
– Memory optimization (96-bit ID, Dictionary Encoding, …)
– Refactoring
Release History

Summary
Contributions welcome!
• Code
• I/O Formats (GraphML, DOT, …)
• Operators and Algorithms
• Tuning (Memory consumption, serialization, …)
• API improvements
• Use cases and data
• Business Intelligence
• Fraud Detection
• Pattern Mining
• …

• Extended Property Graph Model
• Schema flexible: Type Labels and Properties
• Logical Graphs / Graphs Collection
• Graph and Collection Operators
• Combination to analytical workflows
• Implemented on Apache Flink
• Built-in scalability
• Combine with other libraries
Summary

www.gradoop.com
[1] Junghanns, M.; Petermann, A.; Teichmann, N.; Gomez, K.; Rahm, E.,
„Analyzing Extended Property Graphs with Apache Flink“,
Int. Workshop on Network Data Analytics (NDA), SIGMOD 2016.
[2] Petermann, A.; Junghanns, M.,
„Scalable Business Intelligence with Graph Collections“,
it – Special Issue on Big Data Analytics, 2016.
[3] Petermann, A.; Junghanns, M.; Müller, M.; Rahm, E.,
„Graph-based Data Integration and Business Intelligence with BIIIG“,
Proc. VLDB Conf. (Demo), 2014.

Distributed Graph Analytics with Gradoop

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (8)

Similar to Distributed Graph Analytics with Gradoop

Similar to Distributed Graph Analytics with Gradoop (20)

Recently uploaded

Recently uploaded (20)

Distributed Graph Analytics with Gradoop