See the video here: https://www.youtube.com/watch?v=o6lSeNatVFM A look at the elements required by Elasticsearch to turn a simple inverted index into an auto-clustering, horizontally scalable real time search and analytics engine. The talk will start from first principles, explaining how an inverted index works, how to make an inverted index suitable for real time search, how to scale that out, and how to add reliability and failover to the cluster.

1. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Clinton Gormley
@clintongormley
Scaling real time
search and analytics with
elasticsearch

2. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.

3. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch.org/guide

4. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch

5. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch
• real-time

6. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch
• real-time
• distributed

7. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch
• real-time
• distributed
• search

8. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch
• real-time
• distributed
• search
• analytics

9. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how to use it?

10. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how to use it?

11. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how does it work?

12. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 1:
making text searchable

13. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.

14. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
where content like
“%darling%buds%”

15. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
slow & inflexible

16. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.

17. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc
1 Doc
2 Doc
3
breathe
brings
buds
but
by
can
…
damasked
darling
date
day
deaf
death
declines
delight
sorted list of
unique terms

18. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc
1 Doc
2 Doc
3
breathe
brings
buds
but
by
can
…
damasked
darling
date
day
deaf
death
declines
delight
where
they
occur

19. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc
1 Doc
2 Doc
3
breathe
brings
buds
but
by
can
…
damasked
darling
date
day
deaf
death
declines
delight

20. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Term Doc
1 Doc
2 Doc
3
breathe
brings
buds
but
by
can
…
damasked
darling
date
day
deaf
death
declines
delight

21. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index

22. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
• term frequencies

23. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
• term frequencies » relevance

24. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
• term frequencies
• text length
» relevance

25. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
• term frequencies
• text length
» relevance
» doc weight

26. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
• term frequencies
• text length
• term positions
» relevance
» doc weight

27. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
• term frequencies
• text length
• term positions
» relevance
» doc weight
» word proximity

28. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
• term frequencies
• text length
• term positions
• char offsets
» relevance
» doc weight
» word proximity

29. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
• term frequencies
• text length
• term positions
• char offsets
» relevance
» doc weight
» word proximity
» highlighting

30. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
not just for text

31. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index
numbers, dates, bools, enums
geopoints, geoshapes, etc

32. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 2:
analytics

33. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
for search
map values → doc_ids

34. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
for search
map values → doc_ids
for analytics
map doc_ids → values

35. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
uninvert the index

36. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
uninvert the index
cache values in memory
called “fielddata”

37. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
uninvert the index
data access from RAM
very fast

38. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
on-the-fly analytics
in the context of
a user’s query

39. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
on-the-fly analytics
relevant analytics
for each user

40. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
calculate metrics
count, min, max, sum, avg,
percentiles, cardinality,
stddev, variance, sum of squares
!

41. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
grouped by
popular terms, significant terms,
ranges, dates, geolocation, etc

42. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
grouped by
groups can
… contain subgroups
… which contain subgroups
etc

43. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 3:
building the inverted index

44. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
inverted index

45. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable

46. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable
• cache friendly

47. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable
• cache friendly
• reads from RAM

48. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable
• cache friendly
• reads from RAM
• fielddata never changes

49. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable
• cache friendly
• reads from RAM
• fielddata never changes
• compressible

50. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
immutable
• cache friendly
• reads from RAM
• fielddata never changes
• compressible
• no locking

51. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but, immutable…

52. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 4:
dynamic inverted index

53. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
in-memory buffer
commit
segment

54. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable

55. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
commit

56. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable

57. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable
commit

58. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable

59. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit

60. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
• write new segment

61. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
• write new segment
• write new commit point

62. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
• write new segment
• write new commit point
• fsync

63. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
• write new segment
• write new commit point
• fsync
• clear buffer

64. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
• write new segment
• write new commit point
• fsync
• clear buffer
• reopen index

65. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene commit
• write new segment
• write new commit point
• fsync ← expensive!
• clear buffer
• reopen index

66. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 5:
near real-time search

67. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
in-memory buffer
flush
segment

68. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable

69. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
flush

70. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable

71. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
flush

72. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable

73. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
commit

74. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
commit point
segment
searchable

75. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene flush

76. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene flush
• write new segment
• clear buffer
• reopen index
!

77. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene flush
• write new segment
• clear buffer
• reopen index
• no fsync

78. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lucene flush
• write new segment
• clear buffer
• reopen index
• no fsync → lightweight

79. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…
data not safe until fsync’ed!

80. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 6:
don’t lose data

81. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 6:
don’t lose data
→ transaction log

82. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
in-memory buffer
flush
segment
translog

83. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog

84. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
flush
translog

85. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog

86. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog
commit

87. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
segment
searchable
translog
commit point

88. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
elasticsearch “refresh”
• lucene “flush”
• makes changes searchable
• lightweight
!

89. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• lucene “commit”
• clears transaction log
• persists changes
• heavy
!
elasticsearch “flush”

90. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
refresh every second

91. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
near real-time search!

92. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
near real-time search!
near real-time analytics!

93. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…

94. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• slow searches
• poor term frequencies
• poor compression
!
!
too many segments

95. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 7:
reduce segments

96. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable

97. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable

98. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable

99. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable

100. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable

101. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable

102. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable

103. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable

104. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable

105. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable

106. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable

107. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
searchable

108. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
merge process
• many small → one big
• removes deleted docs
• runs in background
• throttled

109. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…

110. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“Any wonder it broke down” by Brian Snelson is licensed under CC BY 2.0

111. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
sometimes you
need another truck

112. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 8:
scale out, not up

113. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
shard your data

114. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
shard your data
transparent in elasticsearch

115. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
many segments

116. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one shard
ss
many segments →

117. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one shard
ss
many segments
ssssssss
many shards
ss
→

118. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one shard
ss
many segments
one index
IIssssssss
many shards
ss
→
→

119. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“node”
running instance of elasticsearch
≈ one server

120. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“shard”
bucket of data
lives on one node
physical worker unit

121. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“index”
logical namespace
points to one or more shards

122. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“index”
logical namespace
points to one or more shards
shard = hash(_id) % no_of_shards

123. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
PUT doc _id:1
hash(1) % 3 shard_2
node_A
shard_0
node_B
shard_1
node_C
shard_2

124. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
GET doc _id:2
hash(2) % 3 shard_0
node_A
shard_0
node_B
shard_1
node_C
shard_2

125. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
Search all docs
shard = hash(_id) % no_of_shards
node_A
shard_0
node_B
shard_1
node_C
shard_2

126. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 9:
scaling elastically

127. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
start small
node_A
shard_0
shard_1
shard_2

128. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add more nodes
node_A
shard_0
shard_1
shard_2
node_B node_C

129. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
shards migrate
node_A
shard_0
shard_1
shard_2
node_B
shard_1
node_C
shard_2

130. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
rebalanced
node_A
shard_0
node_B
shard_1
node_C
shard_2

131. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add new index
node_A
shard_0
shard_1
node_B
shard_1
shard_2
node_C
shard_0
shard_2

132. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…

133. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…
more hardware?

134. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…
more hardware?
more hardware failure

135. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
at 3am on sunday…
node_A
shard_0
shard_1
node_B
shard_1
shard_2
node_C
shard_0
shard_2

136. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
boom!
node_A
shard_0
shard_1
node_B
shard_1
shard_2
node_C
shard_0
shard_2

137. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 10:
add redundancy

138. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
for every shard
…make a copy

139. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“primary shard”
main shard

140. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“replica shard(s)”
copy of primary shard

141. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
one node
node_A
P0
P1
P2

142. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a node
node_A
P0
P1
P2
node_B

143. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a node
node_A
P0
P1
P2
node_B
R0
R1
R2

144. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
redundancy
node_A
P0
P1
P2
node_B
R0
R1
R2

145. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a node
node_A
P0
P1
P2
node_B
R0
R1
R2
node_C

146. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
add a node
node_A
P0
P1
P2
node_B
R0
R1
R2
R1
node_C
P0

147. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
rebalanced
node_A
P0
P1
P2
node_B
R0
R1
R2
node_C
P0

148. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
lose a node
node_A
P0
P1
P2
node_B
R0
R2
R1
node_C
P0

149. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
replica primary
node_A
P0
P1
P2
node_B
R0
R2

150. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
replica primary
node_A
P0
P1
P2
node_B
P0
R2

151. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
allocate replicas
node_A
P0
P1
P2
node_B
P0
R2
R0
R1

152. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
rebalanced
node_A
P0
P1
P2
node_B
P0
R2
R0
R1

153. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
primary shard
• just a role
• receives doc changes first
• forwards new doc to replicas in parallel
• number of primaries fixed

154. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
replica shard
• copy of primary shard
• serves read/search requests
• number of replicas can be changed
• more replicas → more read throughput
*if you have more hardware*

155. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…

156. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
but…
who controls all this?

157. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
step 11:
the master node

158. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“Master Yoda” by Gonzalo Martín is licensed under CC BY-SA 2.0

159. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“node”
running instance of elastic search

160. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“node”
running instance of elastic search
node_A

161. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“cluster”
one or more nodes
with same cluster name
working together

162. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
“cluster”
node_A node_B

163. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
discover a cluster
with multicast/unicast

164. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
discover a cluster
with multicast/unicast

165. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
request routing
send request to any node

166. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
request routing
forwards to correct node

167. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how?

168. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
how?
every node knows where
every document is

169. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster state
every node knows where
every document is

170. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster state
cluster level information

171. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster state
cluster level information
indices shards nodes

172. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
cluster state
can only be updated by
the master node

173. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A
master node
elected when cluster forms

174. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B
master node

175. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B
master node
node_C

176. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
master node

177. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
master node
just a role

178. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_A node_B node_C
master node
re-elected if master fails

179. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_B node_C
master node
node_A
re-elected if master fails

180. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
node_B node_C
master node
re-elected if master fails

181. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
master node
only manages
cluster level changes

182. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
master node
not doc-level
get/put/search

183. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
the result?

184. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
distributed
real-time
search & analytics

185. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
which works in the same way
on your laptop…

186. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
…as on your
1,000 node cluster

187. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
who is using it?

188. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• full text search
• highlighted search snippets
• search-as-you-type
• did-you-mean suggestions

189. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• combine visitor logs with
social network data
• real-time feedback to editors

190. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• combines full text search with
geolocation
• uses more-like-this to find
related questions and answers

191. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• search repositories, users,
issues, pull requests
• search 130 billion lines of code
• track all alerts, events, logs

192. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
• index and analyse
5TB of log data every day

193. Copyright Elasticsearch 2014. Copying, publishing and/or distributing without written permission is strictly prohibited.
thank you
@clintongormley
elasticsearch.org/downloads
elasticsearch.com/support
elasticsearch.com/jobs