研究室での論文紹介のスライド

1. Pregel: A System for Large-
Scale Graph Processing
2014 / 5 /14
Ishikawa Yasutaka

2. About this Paper
• Authers:Malewicz, GrzegorzAustern, Matthew HBik,
Aart J.CDehnert, James CHorn, IlanLeiser,
NatyCzajkowski, Grzegorz
• Google’s paper
• Proceedings of the 2010 international conference
on Management of data - SIGMOD '10
2

3. Outline
• Introduction
• Model of computation
• Pregel’s API
• Implementation
• Application
• Experiments
• conclusion
3

4. Outline
• Introduction
• Model of computation
• Pregel’s API
• Implementation
• Application
• Experiments
• conclusion
4

5. Today’s problems of graph
processing
• Poor locality of memory access
• Very little work ver vertex
5

6. Methods of graph
processing…(1/2)
1. Crafting a custom distributed infrastructure
→typically requiring a substantial implementation effort
2. Relying on an existing distributed computing
platform(e.g.,MapReduce)
→this can lead to suboptimal performance and usability
issues.
6

7. Methods of graph
processing…(2/2)
3. Using a single-computer graph algorithm library
→limiting the scale of problems
4. Using an existing parallel graph system
→do not address fault tolerance or other issues that are
important for very large scale distributed systems
7

8. What is Pregel
• Scalable graph processing model
- Based on BSP(Bulk Synchronous Parallel)
- Designed for efficient,scalable and fault- tolerant
Implementation on clusters
- Distribution-related details are hidden behind an
abstract API
• Not open source software
- Apach Giraph is a open source software
implementation of Pregel
8

9. Bulk Synchronous Parallel
• Bridging model for designing parallel algorithm
• BPS iterates superstep for computing
and synchronize all
processes at
each superstep
superstep
9

10. BSP’s algorithm(1/3)
1. Concurrent computation
2. Communication
3. Barrier synchronisation
Each thread processes their
data
concurrently,independently
10

11. BSP’s algorithm(2/3)
1. Concurrent computation
2. Communication
3. Barrier synchronisation
They pass messages
11

12. BSP’s algorithm(3/3)
1. Concurrent computation
2. Communication
3. Barrier synchronisation
They wait for completion
of message passing of all
other tread
Next superstep…
12

13. Outline
• Introduction
• Model of computation
• Pregel’s API
• Implementation
• Application
• Experiments
• conclusion
13

14. Pregel’s input and output
• Input: graph
• Output: graph
• Iterate superstep,which
consists of user defined function,
message passing
Graph:Input
Graph:output
Superstep
Superstep
Superstep
14

15. Graph component
• Graph of Pregel consists of vertex and edge
• Vertex:
- Consisting of unique identifier, user defined value
- Outgoing edge and value are modifiable
• Edge:
- Consisting of source vertex, target vertex, user defined value
- User defined value is modifiable
- Not first class citizen
A B
Vertex value is modifiable
D
C
B
A
D
C
B
A
Outgoing edge
and edge value
are modifiablea
b c
d
15

16. State of vertex
• Vertex has two states:Active,Inactive
• In case vertex receives message, chage state to
Active
• In case vertex has no message, change state to
Inactive
Active Inactive
Vote to halt
Message received 16

17. Pregel’s Superstep
1. In Superstep S,vertex V, compute user defined fuction
with messages send in Superstep S-1
2. Send messages to other vertices that will be received in
Superstep S+1
3. Modify the state of V
4. If all other vertices finish 1~3, go to Superstep S+1
• When no further vertices change in a superstep, algorithm
terminates with output
17

18. Example: maximum value(1/4)
3 6 2 1
3 6 2 1
:Active
:Inactive
Superstep 0
18

19. Example: maximum value(2/4)
3 6 2 1
6 6 2 6
6 6 2 6
:Active
:Inactive
Superstep 0
Superstep 1
19

20. Example: maximum value(3/4)
3 6 2 1
6 6 2 6
6 6 6 6
6 6 6 6
:Active
:Inactive
Superstep 0
Superstep 1
Superstep 2
20

21. Example: maximum value(4/4)
3 6 2 1
6 6 2 6
6 6 6 6
6 6 6 6
:Active
:Inactive
Superstep 0
Superstep 1
Superstep 2
Superstep 3
21

22. Outline
• Introduction
• Model of computation
• Pregel’s API
• Implementation
• Application
• Experiments
• conclusion
22

23. Vertex class
• Writing Pregel program involves subclassing the predefined Vertex class
• Compute() method will be executed at each active vertex
23

24. Message Passing
• The type of message which sent by vertex is
specified by the user as template parameter of
Vertex class
• There is no guaranteed order of messages in the
iterator, but it is guaranteed that messages will be
delivered
24

25. Combiners
• Sending a message to a vertex on another machine
incurs some overhead
• In some case, using combiners can reduce the
number of messages
• To enable this, user subclass
Conbiner class
Reduction of messages
25

26. Aggregators(1/2)
• Pregel aggregators are a mechanism for global
communication
• Each vertex can provide a value in Superstep S, and
this value is made available to all vertices in
Superstep S+1
Superstep S
4
2
1
Superstep S+1
7
7
7
4+2+1…
Sum aggregator:
number of edges 26

27. Aggregators(2/2)
• To define a new aggregator, a user subclasses the
predefined Aggregator class
Superstep S
4
2
1
Superstep S+1
7
7
7
4+2+1…
Sum aggregator:
number of edges 27

28. Topology Mutations(1/2)
• Some graph algorithms need to change the graph’s
topology
- Clustering algorithm
- Minimum spanning tree algorithm
• User’s Compute() function can issue requests to
add or remove vertices or edges
- it causes conflicts
28

29. Topology Mutations(2/2)
• We can solve this conflict using two mechanisms
- Partial ordering: edge remove → vertex remove → vertex
addition → edge addition
- Handler: This picks one arbitrary. User can define
hundler method in vertex subclass
• Partial ordering yields deterministic for most
conflict
29

30. Input and output
• Pregel adapts to many file format in input and
output
- It decouples the task of interpreting an input file from
task of graph computation
- Library provides readers and writers
- Users can write own by subclassing Reader and Writer
File format A
File format B
R
e
a
d
e
r
C
o
m
p
u
t
e
File format C
File format D
W
r
i
t
e
r
30

31. Outline
• Introduction
• Model of computation
• Pregel’s API
• Implementation
• Application
• Experiments
• conclusion
31

32. Basic architecture(1/2)
• The Pregel library divides a graph into partitions
• Assignment of a vertex to a partition depends
sololy on vertex ID
- Default partitioning function is Hash(ID):mod N
32

33. Basic architecture(2/2)
• The execution of a Pregel program consists of
several stages
1. Many copies of the user program begin executing on a
cluster of machines. One of these acts as the master
2. The master determines how many partitions the graph
will have, and assigns partitions to each worker
3. The master assigns a portion of the user’s input to
each worker
4. The master instructs each worker to perform a
superstep
33

34. Fault tolerance(1/2)
• Fault tolerance is achieved through chechpointing
• The master instructs workers to save the state of
their partitions to persistent storage is
- Including vertex values,edge values,imcoming messages
- Master separately saves the aggregator values
34

35. Fault tolerance(2/2)
• Worker failures are detected using regular “ping”
messages the master issues to workers
• When one or more workers fail, the master
reassigns graph partitions to the workers
- Repeating the missing Supersteps
35

36. Worker implementation
• A worker machine maintains the state of its portion
of the graph in memory
• There are two copies of active flag and incoming
message queue
• One for the current superstep and another for the next
superstep
• In message sending, there are two pattern: remote,
local
36

37. Master implementation
• The master assigns unique identifier to each worker at
the time of registration
• The master maintains a list of all workers known to be
active
• If any worker fails, the master enters recovery mode
• The master runs an HTTP server that display statistics
about the progress of computation
37

38. Outline
• Introduction
• Model of computation
• Pregel’s API
• Implementation
• Application
• Experiments
• conclusion
38

39. [1]Page Rank(1/2)
• Page Rank algorithm decide the importance of web
pages
• This algorithm is based on evaluation of paper
- Good paper might be cited from many other papers
- 「A paper that is cited from papers cited from many
papers」 might be good paper
• This is named from one of Google’s founders,
Larry “Page”
39

40. [1]Page Rank(2/2)
40

41. [2]Shortest Path(1/6)
• Shortest-Path problem: calculate the shortest path
in given two nodes of a weighted graph
• There is several variety of Shortest-Path problem
- The single-source shortest paths problem
- The s-t shortest path problem
- All-pairs shortest paths problem
• In this paper, focusing on single-source shortest
paths problems
41

42. [2]Shortest Path(2/6)
∞ ∞
0 ∞
∞
∞
∞
5
3
1 4
3 2
1
2
4
Superstep 0
42

43. [2]Shortest Path(3/6)
5 ∞
0 3
∞
∞
∞
5
3
1 4
3 2
1
2
4
Superstep 1
43

44. [2]Shortest Path(4/6)
4 6
0 3
6
∞
5
5
3
1 4
3 2
1
2
4
Superstep 2
44

45. [2]Shortest Path(5/6)
4 5
0 3
6
9
5
5
3
1 4
3 2
1
2
4
Superstep 3
45

46. [2]Shortest Path(6/6)
46

47. Outline
• Introduction
• Model of computation
• Pregel’s API
• Implementation
• Application
• Experiments
• conclusion
47

48. Experiment details
• Three experiments with the single-source shortest
paths
• Using a cluster of 300 multicore commodity PCs
• Reporting runtime for binary trees and log-normal
graphs
- Binary tree, varying number of worker tasks
- Binary tree, varying graph sizes
- Log-normal, random graphs: varying graph sizes
48

49. [1]1 billion vertex binary tree:varying
number of worker tasks
• Setting
- A billion vertices, the
number of Pregel
workers varying from
50 to 800
• Result
- Using 16 times as
many as Workers
represents a speedup
of about 10
49

50. [2]Binary tree:varying graph sizes
on 800 worker tasks
• Setting
- Varying in size from
a billion to 50 billion
vertices,using a fixed
numberof 800 worker
tasks
• Result
- tree size varying from
a billion to 50 billion,
the time increase from
17.3 to 702
50

51. [3]Log-normal random graphs:
varying graph sizes on 800 worker
tasks(1/2)
• Binary trees are not representative of graphs
encountered in practice
• Use a log-normal distribution of outdegrees
• In this experiment, μ = 4, σ = 1.3
e
d
d
dp
22
2/)(ln
2
1
)(




51

52. [3]Log-normal random graphs:
varying graph sizes on 800 worker
tasks(2/2)
• Setting
- Varying in size from
10million to a a billion
vertices
• Result
- Largest graph took
a little over 10 minutes
52

53. Outline
• Introduction
• Model of computation
• Pregel’s API
• Implementation
• Application
• Experiments
• conclusion
53

54. Conclusion
• They suggest a computing model that is suitable for
graph processing, and has scalability, fault-
tolerance
• They say that programmers can implement graph
processing algorithm easily with Pregel
54

55. This slide’s sources(1/)
• http://www.slideshare.net/doryokujin/largescale-
graph-processingintroduction
• http://shnya.jp/blog/?p=797
• http://www.slideshare.net/sscdotopen/introducing
-apache-giraph-for-large-scale-graph-processing
• http://teppei.hateblo.jp/entry/2013/11/11/232052
• http://ja.wikipedia.org/wiki/%E5%AF%BE%E6%95%
B0%E6%AD%A3%E8%A6%8F%E5%88%86%E5%B8
%83
55

56. This slide’s sources(2/)
• http://keisan.casio.jp/exec/system/1161228861
• http://www.atmarkit.co.jp/ait/articles/1203/22/ne
ws165_2.html
• http://en.wikipedia.org/wiki/Bulk_synchronous_pa
rallel
• http://research.preferred.jp/2011/06/bsp_piccolo_
spark_introduction/
• http://ja.wikipedia.org/wiki/%E3%83%9A%E3%83%
BC%E3%82%B8%E3%83%A9%E3%83%B3%E3%82%
AF
56

57. This slide’s sources(3/)
• http://ja.wikipedia.org/wiki/%E3%82%B9%E3%83%
91%E3%83%8B%E3%83%B3%E3%82%B0%E3%83%
84%E3%83%AA%E3%83%BC%E3%83%97%E3%83%
AD%E3%83%88%E3%82%B3%E3%83%AB
• http://ja.wikipedia.org/wiki/%E6%9C%80%E7%9F%
AD%E7%B5%8C%E8%B7%AF%E5%95%8F%E9%A1%
8C
• http://matome.naver.jp/odai/21286852451259207
01?&page=1
• http://www.cs.ucsb.edu/~prakash/projects/cs290b
/index.html
57

58. This slide’s sources
• http://homepage2.nifty.com/well/Template.html
• http://ja.wikipedia.org/wiki/%E7%AC%AC%E4%B8
%80%E7%B4%9A%E3%82%AA%E3%83%96%E3%82
%B8%E3%82%A7%E3%82%AF%E3%83%88
• http://ja.wikipedia.org/wiki/%E3%82%AF%E3%83%
AA%E3%83%BC%E3%82%AF_(%E3%82%B0%E3%8
3%A9%E3%83%95%E7%90%86%E8%AB%96)
• http://www.alaxala.com/jp/techinfo/archive/manu
al/AX2000R/HTML/KAISETS2/0078.HTM
58