Cluster Consensus: when Aeron met Raft by Martin Thompson Want to know what is possible in the field of high performance consensus algorithms for distributed systems? If the answer is yes then come to this talk.

1. Cluster Consensus
When Aeron Met Raft
Martin Thompson - @mjpt777

3. What does “Consensus” mean?

4. con•sen•sus
noun kən-ˈsen(t)-səs
: general agreement : unanimity
Source: http://www.merriam-webster.com/

5. con•sen•sus
noun kən-ˈsen(t)-səs
: general agreement : unanimity
: the judgment arrived at by most of those
concerned
Source: http://www.merriam-webster.com/

6. Consensus on what?

7. Message - 1
Message - 2
Message - 3
Message - 4
Message - 5
Message - 6

8. Message - 1
Message - 2
Message - 3
Message - 4
Message - 5
Message - 6
State
Machine
In Memory
Model

9. Message - 1
Message - 2
Message - 3
Message - 4
Message - 5
Message - 6
State
Machine
In Memory
Model
Actors
Event
Sourcing
CQRS
Command
Sourcing
WAL
Immutable
Log

10. https://raft.github.io/raft.pdf

11. https://www.cl.cam.ac.uk/~ms705/pub/papers/2015-osr-raft.pdf

12. Raft in a Nutshell

13. Roles
CandidateFollower Leader

14. RPCs
1. RequestVote RPC
Invoked by candidates to gather votes

15. RPCs
1. RequestVote RPC
Invoked by candidates to gather votes
2. AppendEntries RPC
Invoked by leader to replicate and heartbeat

16. Safety Guarantees
• Election Safety
• Leader Append-Only
• Log Matching
• Leader Completeness
• State Machine Safety

17. Monotonic Functions

18. Version all the things!

19. Clustering Aeron

20. Is it Guaranteed Delivery™ ???

21. What is the “Architect” really looking for?

22. “Guaranteed Processing™”

23. There are many
design approaches

24. Client Client Client Client Client
Service

25. Client Client Client Client Client
Service

26. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service

27. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

28. NIO Pain!

29. Do servers crash?

30. FileChannel channel = null;
try
{
channel = FileChannel.open(directory.toPath());
}
catch (final IOException ignore)
{
}
if (null != channel)
{
channel.force(true);
}

31. FileChannel channel = null;
try
{
channel = FileChannel.open(directory.toPath());
}
catch (final IOException ignore)
{
}
if (null != channel)
{
channel.force(true);
}

32. FileChannel channel = null;
try
{
channel = FileChannel.open(directory.toPath());
}
catch (final IOException ignore)
{
}
if (null != channel)
{
channel.force(true);
}

33. Directory Sync
Files.force(directory.toPath(), true);

34. Performance

35. Let’s consider an
RPC design approach

36. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

37. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

38. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

39. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

40. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

41. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

42. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

43. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

44. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

45. Concurrency and/or Parallelism?

46. 1. Parallel is the opposite of Serial
2. Concurrent is the opposite of Sequential
3. Vector is the opposite of Scalar
– John Gustafson

47. Fetch
Time
Instruction Pipelining

48. Fetch Decode
Time
Instruction Pipelining

49. Fetch Decode Execute
Time
Instruction Pipelining

50. Fetch Decode Execute Retire
Time
Instruction Pipelining

51. Fetch Decode Execute Retire
Time
Fetch Decode Execute Retire
Instruction Pipelining

52. Fetch Decode Execute Retire
Time
Fetch Decode Execute Retire
Fetch Decode Execute Retire
Instruction Pipelining

53. Fetch Decode Execute Retire
Time
Fetch Decode Execute Retire
Fetch Decode Execute Retire
Fetch Decode Execute Retire
Instruction Pipelining

54. Order
Time
Consensus Pipeline

55. Order Log
Time
Consensus Pipeline

56. Order Log Transmit
Time
Consensus Pipeline

57. Order Log Transmit Commit
Time
Consensus Pipeline

58. Order Log Transmit Commit
Time
Consensus Pipeline
Execute

59. Order Log Transmit Commit
Time
Consensus Pipeline
Execute
Order Log Transmit Commit Execute

60. Order Log Transmit Commit
Time
Consensus Pipeline
Execute
Order Log Transmit Commit Execute
Order Log Transmit Commit Execute

61. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

62. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

63. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

64. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

65. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

66. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

67. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

68. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

69. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

70. NIO Pain!

71. ByteBuffer byte copies
ByteBuffer byteBuffer = ByteBuffer.allocate(64 * 1024);
byteBuffer.putInt(index, value);

72. ByteBuffer byte copies
ByteBuffer byteBuffer = ByteBuffer.allocate(64 * 1024);
byteBuffer.putBytes(index, bytes);

73. ByteBuffer byte copies
ByteBuffer byteBuffer = ByteBuffer.allocate(64 * 1024);
byteBuffer.putBytes(index, bytes);

74. How can Aeron help?

75. Message Index => Byte Index

76. Multicast, MDC, and Spy
based Messaging

77. Counters
=>
Bounded Consumption

78. 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 5 10 15 20
Batching – Amortising Costs
Average overhead
per item or operation
in batch

79. 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 5 10 15 20
Batching – Amortising Costs
• System calls
• Network round trips
• Disk writes
• Expensive computations

80. Interesting Features

81. Timers

82. All state must enter the system
as a message!

83. public void foo()
{
// Decide to schedule a timer
cluster.scheduleTimer(correlationId, cluster.timeMs() + TimeUnit.SECONDS.toMillis(5));
}
public void onTimerEvent(final long correlationId, final long timestampMs)
{
// Look up the correlationId associated with the timer
}
Timers

84. public void foo()
{
// Decide to schedule a timer
cluster.scheduleTimer(correlationId, cluster.timeMs() + TimeUnit.SECONDS.toMillis(5));
}
public void onTimerEvent(final long correlationId, final long timestampMs)
{
// Look up the correlationId associated with the timer
}
Timers

85. public void foo()
{
// Decide to schedule a timer
cluster.scheduleTimer(correlationId, cluster.timeMs() + TimeUnit.SECONDS.toMillis(5));
}
public void onTimerEvent(final long correlationId, final long timestampMs)
{
// Look up the correlationId associated with the timer
}
Timers

86. Back Pressure and Stashed Work

87. public ControlledFragmentAssembler.Action onSessionMessage(
final DirectBuffer buffer,
final int offset,
final int length,
final long clusterSessionId,
final long correlationId)
{
final ClusterSession session = sessionByIdMap.get(clusterSessionId);
if (null == session || session.state() == CLOSED)
{
return ControlledFragmentHandler.Action.CONTINUE;
}
final long nowMs = cachedEpochClock.time();
if (session.state() == OPEN && logPublisher.appendMessage(buffer, offset, length, nowMs))
{
session.lastActivity(nowMs, correlationId);
return ControlledFragmentHandler.Action.CONTINUE;
}
return ControlledFragmentHandler.Action.ABORT;
}
Back Pressure

88. public ControlledFragmentAssembler.Action onSessionMessage(
final DirectBuffer buffer,
final int offset,
final int length,
final long clusterSessionId,
final long correlationId)
{
final ClusterSession session = sessionByIdMap.get(clusterSessionId);
if (null == session || session.state() == CLOSED)
{
return ControlledFragmentHandler.Action.CONTINUE;
}
final long nowMs = cachedEpochClock.time();
if (session.state() == OPEN && logPublisher.appendMessage(buffer, offset, length, nowMs))
{
session.lastActivity(nowMs, correlationId);
return ControlledFragmentHandler.Action.CONTINUE;
}
return ControlledFragmentHandler.Action.ABORT;
}
Back Pressure

89. public ControlledFragmentAssembler.Action onSessionMessage(
final DirectBuffer buffer,
final int offset,
final int length,
final long clusterSessionId,
final long correlationId)
{
final ClusterSession session = sessionByIdMap.get(clusterSessionId);
if (null == session || session.state() == CLOSED)
{
return ControlledFragmentHandler.Action.CONTINUE;
}
final long nowMs = cachedEpochClock.time();
if (session.state() == OPEN && logPublisher.appendMessage(buffer, offset, length, nowMs))
{
session.lastActivity(nowMs, correlationId);
return ControlledFragmentHandler.Action.CONTINUE;
}
return ControlledFragmentHandler.Action.ABORT;
}
Back Pressure

90. public ControlledFragmentAssembler.Action onSessionMessage(
final DirectBuffer buffer,
final int offset,
final int length,
final long clusterSessionId,
final long correlationId)
{
final ClusterSession session = sessionByIdMap.get(clusterSessionId);
if (null == session || session.state() == CLOSED)
{
return ControlledFragmentHandler.Action.CONTINUE;
}
final long nowMs = cachedEpochClock.time();
if (session.state() == OPEN && logPublisher.appendMessage(buffer, offset, length, nowMs))
{
session.lastActivity(nowMs, correlationId);
return ControlledFragmentHandler.Action.CONTINUE;
}
return ControlledFragmentHandler.Action.ABORT;
}
Back Pressure

91. public ControlledFragmentAssembler.Action onSessionMessage(
final DirectBuffer buffer,
final int offset,
final int length,
final long clusterSessionId,
final long correlationId)
{
final ClusterSession session = sessionByIdMap.get(clusterSessionId);
if (null == session || session.state() == CLOSED)
{
return ControlledFragmentHandler.Action.CONTINUE;
}
final long nowMs = cachedEpochClock.time();
if (session.state() == OPEN && logPublisher.appendMessage(buffer, offset, length, nowMs))
{
session.lastActivity(nowMs, correlationId);
return ControlledFragmentHandler.Action.CONTINUE;
}
return ControlledFragmentHandler.Action.ABORT;
}
Back Pressure

92. Log Replay and Snapshots

93. Log Replay and Snapshots
Distributed File System?

94. Log Replay and Snapshots
Distributed File System?
Aeron Archive
Recorded Streams

95. Multiple Services on the
same stream

96. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service

97. Client Client Client Client Client
Consensus
Module
Service
Consensus
Module
Service
Consensus
Module
Service
Service
Service
Service
Service
Service
Service

98. NIO Pain!

99. MappedByteBuffer DirectByteBuffer
1 2

100. DirectByteBuffer MappedByteBuffer
MappedByteBuffer DirectByteBuffer
1 2

101. DirectByteBuffer MappedByteBuffer
MappedByteBuffer DirectByteBuffer
1 2

103. In Closing

104. What’s the Roadmap?

106. https://github.com/real-logic/aeron
Twitter: @mjpt777
“A distributed system is one in which the failure
of a computer you didn't even know existed
can render your own computer unusable.”
- Leslie Lamport
Questions?