20. What Is Riak?
A flexible storage engine...
...with a REST API...
21. What Is Riak?
A flexible storage engine...
...with a REST API...
...and map/reduce capability...
22. What Is Riak?
A flexible storage engine...
...with a REST API...
...and map/reduce capability...
....designed to be fault-tolerant...
23. What Is Riak?
A flexible storage engine...
...with a REST API...
...and map/reduce capability...
....designed to be fault-tolerant...
...distributed...
24. What Is Riak?
A flexible storage engine...
...with a REST API...
...and map/reduce capability...
....designed to be fault-tolerant...
...distributed...
...and ops friendly
32. CAP Theorem
Consistent Reads and writes reflect a
globally consistent system state
Available System is available for reads and
writes
33. CAP Theorem
Consistent Reads and writes reflect a
globally consistent system state
Available System is available for reads and
writes
34. CAP Theorem
Consistent Reads and writes reflect a
globally consistent system state
Available System is available for reads and
writes
Partition Tolerant System can handle
the failure of individual parts
42. Dynamo
Influences
• N = The number of replicas
• R = The number of replicas needed for a
successful read
43. Dynamo
Influences
• N = The number of replicas
• R = The number of replicas needed for a
successful read
• W = The number of replicas needed for a
successful write
50. Dynamo Math
N = 4, W = 2, R = 1
4 - 2 = 2 hosts can be down and Riak can
still perform writes.
51. Dynamo Math
N = 4, W = 2, R = 1
4 - 2 = 2 hosts can be down and Riak can
still perform writes.
4 - 1 = 3 hosts can be down and Riak can
still perform reads.
80. Linking Objects
• Objects can store pointers, or links, to
other objects
• Doesn’t have to be the same bucket
81. Linking Objects
• Objects can store pointers, or links, to
other objects
• Doesn’t have to be the same bucket
• Object links described in a Link header
82. Link Header Format
Object URL
</riak/demo/test1>; riaktag="userinfo"
Link tag
95. Link Walking Examples
/riak/demo/test1/_,_,0/_,_,1
Start walking at /demo/test1, find any linked objects,
then find and return any objects linked to those
98. Link Walking Examples
/riak/demo/test1/_,child,0/_,_,1
Start walking at /demo/test1, find any linked objects
with the link tag “child”, then find and return any objects
linked to those
100. Map/Reduce
Terms
• Phase: A step within a job
101. Map/Reduce
Terms
• Phase: A step within a job
• Job: A sequence of phases and inputs
102. Map/Reduce
Terms
• Phase: A step within a job
• Job: A sequence of phases and inputs
• Map: Data collection phase
103. Map/Reduce
Terms
• Phase: A step within a job
• Job: A sequence of phases and inputs
• Map: Data collection phase
• Reduce: Data collation or processing
phase
106. Map/Reduce
Overview
• Map phases execute in parallel w/data
locality
• Reduce phases execute in parallel on the
node where job was submitted
107. Map/Reduce
Overview
• Map phases execute in parallel w/data
locality
• Reduce phases execute in parallel on the
node where job was submitted
• Results are not cached or stored
108. Map/Reduce
Overview
• Map phases execute in parallel w/data
locality
• Reduce phases execute in parallel on the
node where job was submitted
• Results are not cached or stored
• Phases can be written in Erlang or
Javascript
119. Erlang Map Phase
• Two types: modfun and qfun
• modfuns reference the module and name
of the Erlang function to call
120. Erlang Map Phase
• Two types: modfun and qfun
• modfuns reference the module and name
of the Erlang function to call
• qfuns are anonymous Erlang functions*
121. Erlang Map Phase
• Two types: modfun and qfun
• modfuns reference the module and name
of the Erlang function to call
• qfuns are anonymous Erlang functions*
*Must be on the server-side codepath
131. Erlang Map
Built-Ins
riak_mapreduce:map_object_value/3
• Returns object value wrapped in a list
132. Erlang Map
Built-Ins
riak_mapreduce:map_object_value/3
• Returns object value wrapped in a list
riak_mapreduce:map_object_value_list/3
133. Erlang Map
Built-Ins
riak_mapreduce:map_object_value/3
• Returns object value wrapped in a list
riak_mapreduce:map_object_value_list/3
• Returns object value. Object value must already
be a list
142. Erlang & Javascript
• Same environment as Firefox minus
browser bits
• Erlang to Javascript data is JSON encoded
143. Erlang & Javascript
• Same environment as Firefox minus
browser bits
• Erlang to Javascript data is JSON encoded
• Javascript to Erlang data is JSON decoded
152. Javascript Map
Built-Ins
Riak.mapValues
• Returns object values. Handles detecting
when/if to use list wrapping.
153. Javascript Map
Built-Ins
Riak.mapValues
• Returns object values. Handles detecting
when/if to use list wrapping.
Riak.mapValuesJson
154. Javascript Map
Built-Ins
Riak.mapValues
• Returns object values. Handles detecting
when/if to use list wrapping.
Riak.mapValuesJson
• Returns JSON parsed object values. Also
performs list wrapping, if needed.
157. Reduce Phase
• Performed on the node coordinating the
map/reduce job
• Two processes per reduce phase to add
minor parallelism
158. Reduce Phase
• Performed on the node coordinating the
map/reduce job
• Two processes per reduce phase to add
minor parallelism
• Must return a list
161. Erlang Reduce
Built-Ins
riak_mapreduce:reduce_set_union/2
• Returns unique set of values
162. Erlang Reduce
Built-Ins
riak_mapreduce:reduce_set_union/2
• Returns unique set of values
riak_mapreduce:reduce_sum/2
163. Erlang Reduce
Built-Ins
riak_mapreduce:reduce_set_union/2
• Returns unique set of values
riak_mapreduce:reduce_sum/2
• Returns the sum of inputs
164. Erlang Reduce
Built-Ins
riak_mapreduce:reduce_set_union/2
• Returns unique set of values
riak_mapreduce:reduce_sum/2
• Returns the sum of inputs
riak_mapreduce:reduce_sort/2
165. Erlang Reduce
Built-Ins
riak_mapreduce:reduce_set_union/2
• Returns unique set of values
riak_mapreduce:reduce_sum/2
• Returns the sum of inputs
riak_mapreduce:reduce_sort/2
• Returns the sorted list of inputs
168. Javascript Reduce
Built-Ins
Riak.reduceMin
• Returns the minimum value of the input set
169. Javascript Reduce
Built-Ins
Riak.reduceMin
• Returns the minimum value of the input set
Riak.reduceMax
170. Javascript Reduce
Built-Ins
Riak.reduceMin
• Returns the minimum value of the input set
Riak.reduceMax
• Returns the maximum value of the input set
171. Javascript Reduce
Built-Ins
Riak.reduceMin
• Returns the minimum value of the input set
Riak.reduceMax
• Returns the maximum value of the input set
Riak.reduceSort
172. Javascript Reduce
Built-Ins
Riak.reduceMin
• Returns the minimum value of the input set
Riak.reduceMax
• Returns the maximum value of the input set
Riak.reduceSort
• Returns a sorted list of the input set
188. Erlang Phase
(JSON)
{Type:{“language”:”erlang”, “module”: Module,
“function”: Function, “keep”:Flag}}
• Type: “map” or “reduce”
• Module: String name of Erlang module
189. Erlang Phase
(JSON)
{Type:{“language”:”erlang”, “module”: Module,
“function”: Function, “keep”:Flag}}
• Type: “map” or “reduce”
• Module: String name of Erlang module
• Function: String name of Erlang function
190. Erlang Phase
(JSON)
{Type:{“language”:”erlang”, “module”: Module,
“function”: Function, “keep”:Flag}}
• Type: “map” or “reduce”
• Module: String name of Erlang module
• Function: String name of Erlang function
• Flag: Boolean accumulation toggle
203. Javascript Phase
(JSON)
{Type:{“language”:”javascript”,
“name”:Name,“keep”:Flag}}
• Type: “map” or “reduce”
• Name: String name of Javascript function
204. Javascript Phase
(JSON)
{Type:{“language”:”javascript”,
“name”:Name,“keep”:Flag}}
• Type: “map” or “reduce”
• Name: String name of Javascript function
• Flag: Boolean accumulation toggle
205. Putting It
Together
{“inputs”: [[“stocks”, “goog”]],
“query”: [{“map”:{“language”:”javascript”,
“name”: “Riak.mapValuesJson”},
“keep”: true}]}
206. Putting It
Together
{“inputs”: [[“stocks”, “goog”],
[“stocks”, “csco”]],
“query”: [{“map”:{“language”:”javascript”,
“name”: “Riak.mapValuesJson”},
“keep”: true}]}
207. Putting It
Together
{“inputs”: “stocks”,
“query”: [{“map”:{“language”:”javascript”,
“name”: “App.extractTickers”,
“arg”: “GOOG”},
“keep”: false},
{“reduce”:{“language”:”javascript,
“name”: “Riak.reduceMin”},
“keep”: true}]}