Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/22vmyrg. Matthew Sackman discusses dependencies between transactions, how to capture these with Vector Clocks, how to treat Vector Clocks as a CRDT, and how GoshawkDB uses all these ideas and more to achieve a scalable distributed data store with strong serializability, general transactions and fault tolerance. Filmed at qconlondon.com. Matthew Sackman has been building and studying concurrent and distributed systems for over ten years. He built the first prototype of RabbitMQ, and contributed heavily to the design and implementation of many of its core features. He worked on the initial implementation of the Weave router in 2014. In 2015, he built GoshawkDB: a distributed, transactional, fault-tolerant object store.

1. Making Time with Vector Clocks
Matthew Sackman
matthew@goshawkdb.io
https://goshawkdb.io/

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
goshawkdb

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon London
www.qconlondon.com

4. 1. Have you played with a NoSQL or NewSQL store?

5. 1. Have you played with a NoSQL or NewSQL store?
2. Have you deployed a NoSQL or NewSQL store?

6. 1. Have you played with a NoSQL or NewSQL store?
2. Have you deployed a NoSQL or NewSQL store?
3. Have you studied and know their semantics?

9. Traditional Database Semantics
ACID
• Atomic: an operation (transaction) either succeeds or aborts
completely - no partial successes
• Consistent: constraints like uniqueness, foreign keys, etc are
honoured
• Durable: flushed to disk before the client can find out the result

10. Traditional Database Semantics
ACID
• Atomic: an operation (transaction) either succeeds or aborts
completely - no partial successes
• Consistent: constraints like uniqueness, foreign keys, etc are
honoured
• Isolation: the degree to which operations in one transaction
can observe actions of concurrent transactions
• Durable: flushed to disk before the client can find out the result

11. Traditional Database Semantics
Default isolation levels
• PostgreSQL:
• Oracle 11g:
• MS SQL Server:
• MySQL InnoDB:

12. Traditional Database Semantics
Default isolation levels
• PostgreSQL: Read Committed
• Oracle 11g: Read Committed
• MS SQL Server: Read Committed
• MySQL InnoDB:

13. Traditional Database Semantics
Default isolation levels
• PostgreSQL: Read Committed
• Oracle 11g: Read Committed
• MS SQL Server: Read Committed
• MySQL InnoDB: Repeatable Read

14. Isolation Levels

15. Snapshot Isolation
as per Wikipedia
“Snapshot isolation is a guarantee that all reads made in a transaction
will see a consistent snapshot of the database and the transaction
itself will successfully commit only if no updates it has made conflict
with any concurrent updates made since that snapshot.”

16. Snapshot Isolation
as per Wikipedia
“Snapshot isolation is a guarantee that all reads made in a transaction
will see a consistent snapshot of the database and the transaction
itself will successfully commit only if no updates it has made conflict
with any concurrent updates made since that snapshot.”
Snapshot isolation is called “serializable” mode in Oracle.

17. Snapshot Isolation
trouble up mill
x, y := 0,0
1 func t1() {
2 if x == 0 {
3 y = 1
4 }
5 }
func t2() {
if y == 0 {
x = 1
}
}

18. Snapshot Isolation
trouble up mill
x, y := 0,0
1 func t1() {
2 if x == 0 {
3 y = 1
4 }
5 }
func t2() {
if y == 0 {
x = 1
}
}
• Serialized:
t1 then t2

19. Snapshot Isolation
trouble up mill
x, y := 0,0
1 func t1() {
2 if x == 0 {
3 y = 1
4 }
5 }
func t2() {
if y == 0 {
x = 1
}
}
• Serialized:
t1 then t2

20. Snapshot Isolation
trouble up mill
x, y := 0,0
1 func t1() {
2 if x == 0 {
3 y = 1
4 }
5 }
func t2() {
if y == 0 {
x = 1
}
}
• Serialized:
t1 then t2

21. Snapshot Isolation
trouble up mill
x, y := 0,0
1 func t1() {
2 if x == 0 {
3 y = 1
4 }
5 }
func t2() {
if y == 0 {
x = 1
}
}
• Serialized:
t1 then t2

22. Snapshot Isolation
trouble up mill
x, y := 0,0
1 func t1() {
2 if x == 0 {
3 y = 1
4 }
5 }
func t2() {
if y == 0 {
x = 1
}
}
• Serialized:
t1 then t2: x:0, y:1

23. Snapshot Isolation
trouble up mill
x, y := 0,0
1 func t1() {
2 if x == 0 {
3 y = 1
4 }
5 }
func t2() {
if y == 0 {
x = 1
}
}
• Serialized:
t1 then t2: x:0, y:1
t2 then t1

24. Snapshot Isolation
trouble up mill
x, y := 0,0
1 func t1() {
2 if x == 0 {
3 y = 1
4 }
5 }
func t2() {
if y == 0 {
x = 1
}
}
• Serialized:
t1 then t2: x:0, y:1
t2 then t1: x:1, y:0

25. Snapshot Isolation
trouble up mill
x, y := 0,0
1 func t1() {
2 if x == 0 {
3 y = 1
4 }
5 }
func t2() {
if y == 0 {
x = 1
}
}
• Serialized:
t1 then t2: x:0, y:1
t2 then t1: x:1, y:0
• Snapshot Isolation:

26. Snapshot Isolation
trouble up mill
x, y := 0,0
1 func t1() {
2 if x == 0 {
3 y = 1
4 }
5 }
func t2() {
if y == 0 {
x = 1
}
}
• Serialized:
t1 then t2: x:0, y:1
t2 then t1: x:1, y:0
• Snapshot Isolation:
t1 || t2

27. Snapshot Isolation
trouble up mill
x, y := 0,0
1 func t1() {
2 if x == 0 {
3 y = 1
4 }
5 }
func t2() {
if y == 0 {
x = 1
}
}
• Serialized:
t1 then t2: x:0, y:1
t2 then t1: x:1, y:0
• Snapshot Isolation:
t1 || t2

28. Snapshot Isolation
trouble up mill
x, y := 0,0
1 func t1() {
2 if x == 0 {
3 y = 1
4 }
5 }
func t2() {
if y == 0 {
x = 1
}
}
• Serialized:
t1 then t2: x:0, y:1
t2 then t1: x:1, y:0
• Snapshot Isolation:
t1 || t2

29. Snapshot Isolation
trouble up mill
x, y := 0,0
1 func t1() {
2 if x == 0 {
3 y = 1
4 }
5 }
func t2() {
if y == 0 {
x = 1
}
}
• Serialized:
t1 then t2: x:0, y:1
t2 then t1: x:1, y:0
• Snapshot Isolation:
t1 || t2: x:1, y:1

30. Snapshot Isolation
trouble up mill
x, y := 0,0
1 func t1() {
2 if x == 0 {
3 y = 1
4 }
5 }
func t2() {
if y == 0 {
x = 1
}
}
• Serialized:
t1 then t2: x:0, y:1
t2 then t1: x:1, y:0
• Snapshot Isolation: Write Skew
t1 || t2: x:1, y:1

31. Desired Features
• General purpose transactions

32. Desired Features
• General purpose transactions
• Strong serializability

33. Desired Features
• General purpose transactions
• Strong serializability
• Distribution
• Automatic sharding
• Horizontal scalability
• ...

35. Isolation Levels

36. Isolation Levels

37. CAP
Possibility of Partitions =) ¬(Consistency ^ Availability)

38. CAP
Possibility of Partitions =) ¬(Consistency ^ Availability)

39. CAP
Possibility of Partitions =) ¬(Consistency ^ Availability)

40. CAP
Possibility of Partitions =) ¬(Consistency ^ Availability)

41. CAP
Possibility of Partitions =) ¬(Consistency ^ Availability)

42. Achieving Consistency
Colours Indicate Connected Nodes

43. Achieving Consistency
Colours Indicate Connected Nodes

44. Achieving Consistency
Colours Indicate Connected Nodes

45. Achieving Consistency
Colours Indicate Connected Nodes

46. Achieving Consistency
Colours Indicate Connected Nodes

47. Achieving Consistency
Colours Indicate Connected Nodes

48. Achieving Consistency
Colours Indicate Connected Nodes

49. Achieving Consistency
Colours Indicate Connected Nodes

50. Achieving Consistency
Colours Indicate Connected Nodes

51. Achieving Consistency
Colours Indicate Connected Nodes

52. Learnings 1
• Strong serializability requires Consistency, so must sacrifice
Availability

53. Learnings 1
• Strong serializability requires Consistency, so must sacrifice
Availability
• To achieve Consistency, only accept operations if connected to
majority

54. Learnings 1
• Strong serializability requires Consistency, so must sacrifice
Availability
• To achieve Consistency, only accept operations if connected to
majority
• If cluster size is 2F + 1 then we can withstand no more than F
failures

55. Modifying Values
Colours Indicate Extent of Txn Write

56. Modifying Values
Colours Indicate Extent of Txn Write

57. Modifying Values
Colours Indicate Extent of Txn Write

58. Modifying Values
Colours Indicate Extent of Txn Write

59. Modifying Values
Colours Indicate Extent of Txn Write

60. Modifying Values
Colours Indicate Extent of Txn Write

61. Learnings 2
• Strong serializability requires Consistency, so must sacrifice
Availability
• To achieve Consistency, only accept operations if connected to
majority
• If cluster size is 2F + 1 then we can withstand no more than F
failures
• Writes must go to F + 1 nodes

62. Reading Values
Colours Indicate Extent of Txn Write

63. Reading Values
Colours Indicate Extent of Txn Write

64. Reading Values
Colours Indicate Extent of Txn Write

65. Reading Values
Colours Indicate Extent of Txn Write

66. Reading Values
Colours Indicate Extent of Txn Write

67. Reading Values
Colours Indicate Extent of Txn Write

68. Learnings 3
• Strong serializability requires Consistency, so must sacrifice
Availability
• To achieve Consistency, only accept operations if connected to
majority
• If cluster size is 2F + 1 then we can withstand no more than F
failures
• Writes must go to F + 1 nodes
• Reads must read from F + 1 nodes and be able to order results

70. Txn Processing in Distributed Databases
1. Client submits txn

71. Txn Processing in Distributed Databases
1. Client submits txn
2. Node(s) vote on txn

72. Txn Processing in Distributed Databases
1. Client submits txn
2. Node(s) vote on txn
3. Node(s) reach consensus on txn outcome

73. Txn Processing in Distributed Databases
1. Client submits txn
2. Node(s) vote on txn
3. Node(s) reach consensus on txn outcome
4. Client is informed of outcome

74. Txn Processing in Distributed Databases
1. Client submits txn
2. Node(s) vote on txn
3. Node(s) reach consensus on txn outcome
4. Client is informed of outcome
Most important thing is all nodes agree on the order of transactions

75. Txn Processing in Distributed Databases
1. Client submits txn
2. Node(s) vote on txn
3. Node(s) reach consensus on txn outcome
4. Client is informed of outcome
Most important thing is all nodes agree on the order of transactions
(focus for the rest of this talk!)

76. Leader Based Ordering

77. Leader Based Ordering

78. Leader Based Ordering

79. Leader Based Ordering
• Only leader votes on whether txn commits or aborts

80. Leader Based Ordering
• Only leader votes on whether txn commits or aborts
• Therefore leader must know everything

81. Leader Based Ordering
• Only leader votes on whether txn commits or aborts
• Therefore leader must know everything
• If leader fails, a new leader will be elected from remaining
nodes

82. Leader Based Ordering
• Only leader votes on whether txn commits or aborts
• Therefore leader must know everything
• If leader fails, a new leader will be elected from remaining
nodes
• Therefore all nodes must know everything

83. Leader Based Ordering
• Only leader votes on whether txn commits or aborts
• Therefore leader must know everything
• If leader fails, a new leader will be elected from remaining
nodes
• Therefore all nodes must know everything
• Fine for small clusters, but scaling issues when clusters get big

84. Client Clock Based Ordering

85. Client Clock Based Ordering

86. Client Clock Based Ordering

87. Client Clock Based Ordering
• Nodes receive txns and must vote on txn outcome and then
consensus must be reached (not shown)

88. Client Clock Based Ordering
• Nodes receive txns and must vote on txn outcome and then
consensus must be reached (not shown)
• Clients are responsible for applying an increasing clock value
to txns

89. Client Clock Based Ordering
• Nodes receive txns and must vote on txn outcome and then
consensus must be reached (not shown)
• Clients are responsible for applying an increasing clock value
to txns
• If a client’s clock races then it can prevent other clients from
getting txns submitted

90. Client Clock Based Ordering
• Nodes receive txns and must vote on txn outcome and then
consensus must be reached (not shown)
• Clients are responsible for applying an increasing clock value
to txns
• If a client’s clock races then it can prevent other clients from
getting txns submitted
• So must be very careful to try and keep clocks running at the
same rate

91. Client Clock Based Ordering
• Nodes receive txns and must vote on txn outcome and then
consensus must be reached (not shown)
• Clients are responsible for applying an increasing clock value
to txns
• If a client’s clock races then it can prevent other clients from
getting txns submitted
• So must be very careful to try and keep clocks running at the
same rate
• No possibility to reorder transactions at all to maximise
commits

92. Syntax
V1 < V2 ,8x 2 dom(V1 [ V2) : V1[x]  V2[x]
^9y 2 dom(V1 [ V2) : V1[y] < V2[y]

93. Simple Transaction

94. Simple Transaction

95. Simple Transaction

96. Simple Transaction

97. Simple Transaction

98. Simple Transaction

99. Simple Transaction

100. Simple Transaction

101. Two Writes

102. Two Writes

103. Two Writes

104. Two Writes

105. Two Writes

106. Two Writes

107. Three Writes

108. Three Writes

109. Three Writes

110. Three Writes

111. Three Writes

112. Three Writes

113. Three Writes

114. Three Writes

115. The Dumb Approach Doesn’t Work
• Changing state when receiving a txn seems to be a very bad
idea

116. The Dumb Approach Doesn’t Work
• Changing state when receiving a txn seems to be a very bad
idea
• Maybe only change state when receiving the outcome of a
vote

117. The Dumb Approach Doesn’t Work
• Changing state when receiving a txn seems to be a very bad
idea
• Maybe only change state when receiving the outcome of a
vote
• And don’t vote on txns until we know it’s safe to do so

118. New Ideas
• Divide time into frames. First half of frame is reads, second half
writes.

119. New Ideas
• Divide time into frames. First half of frame is reads, second half
writes.
• Within a frame, we don’t care about order of reads,
• but all reads must come after writes of previous frame,
• all writes must come after reads of this frame,
• all writes must be totally ordered within the frame - must know
which write comes last.

120. Frames & Dependencies

121. Frames & Dependencies

122. Frames & Dependencies

123. Frames & Dependencies

124. Frames & Dependencies

125. Frames & Dependencies

126. Frames & Dependencies

127. Frames & Dependencies

128. Calculating the Write Clock from Reads
• Merge all read clocks together
• Add 1 to result for every object that was written by txns in our
frame’s reads

129. Calculating the Frame Winner
& Next Frame’s Read Clock
• Partition write results by local clock elem, and within that by
txn id
• Each clock inherits missing clock elems from above
• Then sort each partition first by clock (now all same length),
then by txn id
• Next frame starts with winner’s clock, +1 for all writes

130. Calculating the Frame Winner
& Next Frame’s Read Clock
• Partition write results by local clock elem, and within that by
txn id
• Each clock inherits missing clock elems from above
• Then sort each partition first by clock (now all same length),
then by txn id
• Next frame starts with winner’s clock, +1 for all writes
• Guarantees no concurrent vector clocks (proof in progress!)

131. Calculating the Frame Winner
& Next Frame’s Read Clock
• Partition write results by local clock elem, and within that by
txn id
• Each clock inherits missing clock elems from above
• Then sort each partition first by clock (now all same length),
then by txn id
• Next frame starts with winner’s clock, +1 for all writes
• Guarantees no concurrent vector clocks (proof in progress!)
• Many details elided! (deadlock freedom, etc)

132. Transitive Vector Clocks

133. Transitive Vector Clocks

134. Transitive Vector Clocks

135. Transitive Vector Clocks

136. Transitive Vector Clocks

137. Transitive Vector Clocks

138. Transitive Vector Clocks

139. Transitive Vector Clocks

140. Transitive Vector Clocks

141. Transitive Vector Clocks

142. Shrinking Vector Clocks
• Hardest part of Paxos is garbage collection

143. Shrinking Vector Clocks
• Hardest part of Paxos is garbage collection
• Need additional messages to determine when Paxos instances
can be deleted

144. Shrinking Vector Clocks
• Hardest part of Paxos is garbage collection
• Need additional messages to determine when Paxos instances
can be deleted
• We can use these to also express:
You will never see any of these vector clock elems again

145. Shrinking Vector Clocks
• Hardest part of Paxos is garbage collection
• Need additional messages to determine when Paxos instances
can be deleted
• We can use these to also express:
You will never see any of these vector clock elems again
• Therefore we can remove matching elems from vector clocks!
• Many more details elided!

146. Conclusions
Vector clocks capture dependencies and causal relationship
between transactions

147. Conclusions
Vector clocks capture dependencies and causal relationship
between transactions
Plus we always add transactions into the youngest frame

148. Conclusions
Vector clocks capture dependencies and causal relationship
between transactions
Plus we always add transactions into the youngest frame
Which gets us Strong Serializability

149. Conclusions
No leader, so no potential bottleneck

150. Conclusions
No leader, so no potential bottleneck
No wall clocks, so no issues with clock skews

151. Conclusions
No leader, so no potential bottleneck
No wall clocks, so no issues with clock skews
Can separate F from cluster size,

152. Conclusions
No leader, so no potential bottleneck
No wall clocks, so no issues with clock skews
Can separate F from cluster size,
Which gets us horizontal scalability

153. Conclusions

154. References (1)
• Interval Tree Clocks - Paulo Sérgio Almeida, Carlos Baquero,
Victor Fonte
• Highly available transactions: Virtues and limitations - Bailis et
al
• Coordination avoidance in database systems - Bailis et al
• k-dependency vectors: A scalable causality-tracking protocol -
Baldoni, Melideo
• Multiversion concurrency control-theory and algorithms -
Bernstein, Goodman
• Serializable isolation for snapshot databases - Cahill, Röhm,
Fekete
• Paxos made live: an engineering perspective - Chandra,
Griesemer, Redstone

155. References (2)
• Consensus on transaction commit - Gray, Lamport
• Spanner: Google’s globally distributed database - Corbett et al
• Faster generation of shorthand universal cycles for
permutations - Holroyd, Ruskey, Williams
• s-Overlap Cycles for Permutations - Horan, Hurlbert
• Universal cycles of k-subsets and k-permutations - Jackson
• Zab: High-performance broadcast for primary-backup systems
- Junqueira, Reed, Serafini
• Time, clocks, and the ordering of events in a distributed system
- Lamport

156. References (3)
• The part-time parliament - Lamport
• Paxos made simple - Lamport
• Consistency, Availability, and Convergence - Mahajan, Alvisi,
Dahlin
• Notes on Theory of Distributed Systems - Aspnes
• In search of an understandable consensus algorithm - Ongaro,
Ousterhaut
• Perfect Consistent Hashing - Sackman
• The case for determinism in database systems - Thomson,
Abadi

157. Distributed databases are FUN!
https://goshawkdb.io/

158. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/goshawkdb