So, there's this "NoSQL" thing you may have heard of, and this related thing called "eventual consistency". Supposedly, they help you scale, but no one has ever explained why! Well, wonder no more! This talk will demystify NoSQL, eventual consistency, how they might help you scale, and -- most importantly -- why you should care. We'll look closely at how Riak, a linearly-scalable, distributed and fault-tolerant NoSQL datastore, implements eventual consistency, and how you can harness it from Ruby via the slick Ripple client/ORM. When the talk is finished, you'll have the tools both to understand eventual consistency and to handle it like a pro inside your next Ruby application.

1. Embrace NoSQL and
Eventual Consistency
with Ripple
Sean Cribbs
Basho Technologies
@seancribbs
basho

2. Understanding
Eventual Consistency

3. Internet

4. Internet

5. Internet

6. Internet

7. Internet
CDN

8. Internet
CDN

9. Internet
CDN
M S
S

10. Internet
CDN

11. Is this system
consistent?

12. You sacrificed ACID
as soon as you
added caching.

13. How does your
application cope?

14. Inconsistency Creep

15. Inconsistency Creep
Cache staleness

16. Inconsistency Creep
Cache staleness
Master-slave replication lag

17. Inconsistency Creep
Cache staleness
Master-slave replication lag
CDN propagation lag

18. Inconsistency Creep
Cache staleness
Master-slave replication lag
CDN propagation lag
Race-conditions

19. Inconsistency Creep
Cache staleness
Master-slave replication lag
CDN propagation lag
Race-conditions
Service failures

20. Eventual Consistency

21. Eventual Consistency
Tolerates temporary inconsistency

22. Eventual Consistency
Tolerates temporary inconsistency
Moves toward consistency
automatically

23. Eventual Consistency
Tolerates temporary inconsistency
Moves toward consistency
automatically
Resilient, maintains availability

24. Eventual Consistency
Tolerates temporary inconsistency
Moves toward consistency
automatically
Resilient, maintains availability
“Eventual” in seconds, not days

25. Eventual Consistency
Tolerates temporary inconsistency
Moves toward consistency
automatically
Resilient, maintains availability
“Eventual” in seconds, not days

26. Real-World
Eventual Consistency

27. Real-World
Eventual Consistency
DNS

28. Real-World
Eventual Consistency
DNS
LDAP

29. Real-World
Eventual Consistency
DNS
LDAP
The Web

30. Harvest & Yield

31. Harvest & Yield
CAP “For Dummies”

32. Harvest & Yield
CAP “For Dummies”
Harvest: how much of dataset is
reflected in a response (C)

33. Harvest & Yield
CAP “For Dummies”
Harvest: how much of dataset is
reflected in a response (C)
Yield: how likely is the datastore
to complete request (A)

34. Harvest & Yield
http://codahale.com/you-cant-sacrifice-partition-
tolerance/
CAP “For Dummies”
Harvest: how much of dataset is
reflected in a response (C)
Yield: how likely is the datastore
to complete request (A)

36. # Do I need it?
def eventual_consistency?
@money === @uptime
end

37. # Do I need it?
def eventual_consistency?
@money === @uptime
end
# “Any sufficiently large
# system is in a constant
# state of failure.”
# --Justin Sheehy, Basho CTO

38. Ok, I’ll use Eventual
Consistency. How?

39. Oh, those NoSQL
things.

40. NoSQL Flavors

41. NoSQL Flavors
Graph (Neo4J, InfiniteGraph)

42. NoSQL Flavors
Graph (Neo4J, InfiniteGraph)
Document (Mongo, Couch)

43. NoSQL Flavors
Graph (Neo4J, InfiniteGraph)
Document (Mongo, Couch)
Column (HBase, Cassandra)

44. NoSQL Flavors
Graph (Neo4J, InfiniteGraph)
Document (Mongo, Couch)
Column (HBase, Cassandra)
Key-Value (Riak,
Voldemort)

45. NoSQL Flavors
structured
Graph (Neo4J, InfiniteGraph)
Document (Mongo, Couch)
Column (HBase, Cassandra)
Key-Value (Riak,
Voldemort)
unstructured

46. Not all NoSQLs are
Eventually Consistent.

47. Riak implements
Eventual Consistency.

48. What is Riak?

49. Riak is...

50. Riak is...
a scalable

51. Riak is...
a scalable
highly-available

52. Riak is...
a scalable
highly-available
networked/distributed

53. Riak is...
a scalable
highly-available
networked/distributed
key-value store

55. Riak’s Data Model

56. Riak’s Data Model
Stores values against keys

57. Riak’s Data Model
Stores values against keys
Values are opaque

58. Riak’s Data Model
Stores values against keys
Values are opaque
Keys are grouped into buckets

59. Values Have Metadata

60. Values Have Metadata
Content-Type

61. Values Have Metadata
Content-Type
Last-Modified

62. Values Have Metadata
Content-Type
Last-Modified
Link

63. Values Have Metadata
Content-Type
Last-Modified
Link
Secondary index values

64. Values Have Metadata
Content-Type
Last-Modified
Link
Secondary index values
Custom metadata

65. Values Have Metadata
Content-Type
Last-Modified
Link
Secondary index values
Custom metadata
Value + Metadata == “Object”

66. Basic Operations

67. Basic Operations
GET /riak/bucket/key

68. Basic Operations
GET /riak/bucket/key
PUT /riak/bucket/key

69. Basic Operations
GET /riak/bucket/key
PUT /riak/bucket/key
DELETE /riak/bucket/key

70. Basic Operations
GET /riak/bucket/key
PUT /riak/bucket/key
DELETE /riak/bucket/key
HTTP or Protocol Buffers

71. Other Query Methods

72. Other Query Methods
Link-traversal (“walking”)

73. Other Query Methods
Link-traversal (“walking”)
Full-text Search

74. Other Query Methods
Link-traversal (“walking”)
Full-text Search
Secondary Indexes

75. Other Query Methods
Link-traversal (“walking”)
Full-text Search
Secondary Indexes
MapReduce

76. Ripple connects
Ruby to Riak.

77. # gem install riak-client
require ‘riak’
# Connect to Riak on localhost
client = Riak::Client.new
# Fetch something, write it to a file
o = client[‘slide-decks’][‘RubyConfAR’]
File.open(‘slides.key’,‘wb’) do |f|
f.write o.raw_data
end

78. # Create a new key
v = client[‘venues’].new(‘RubyConfAR’)
# Set the value, JSON by default
v.data = {
name: ‘Ciudad Cultural Konex’,
address: {
street: ‘Sarmiento 3131’,
city: ‘Buenos Aires’,
zip: ‘C1196AAG’
}
}
# Write it to Riak
v.store

79. # Create a new key for an image
pic = client[‘pictures’].new(‘bue1.jpg’)
# Read the file, set the content-type
pic.raw_data = File.read(‘bue1.jpg’)
pic.content_type = ‘image/jpeg’
# Set some metadata
pic.meta[‘camera’] = ‘Nikon’
pic.indexes[‘date_int’] << Time.now.to_i
# Write it to Riak
pic.store

80. But I want models!
Ok, no problem.

81. require ‘ripple’ # gem install ripple
class Person
include Ripple::Document
property :name, String, presence: true
key_on :name
many :addresses
end
class Address
include Ripple::EmbeddedDocument
property :street, String
property :city, String
property :zip, String
end

82. # Create my info
me = Person.new(
name: ‘Sean Cribbs’,
addresses: [{
street: ‘929 Market St’,
city: ‘San Francisco’,
zip: ‘94103’
}]
)
me.save! # Store as JSON under
# “people/Sean Cribbs”
me == Person.find(‘Sean Cribbs’) # true

83. That’s kind of cool.
But what about
Eventual Consistency?

84. Riak uses Causal
Eventual Consistency.
also called Vector Clocks

85. Causal Consistency

86. Causal Consistency
Each read includes a token of the
current state (vector clock)

87. Causal Consistency
Each read includes a token of the
current state (vector clock)
Clients send the token back with writes

88. Causal Consistency
Each read includes a token of the
current state (vector clock)
Clients send the token back with writes
Riak updates the token and detects stale
versions and conflicts by comparison

89. Causal Consistency
Each read includes a token of the
current state (vector clock)
Clients send the token back with writes
Riak updates the token and detects stale
versions and conflicts by comparison
Conflicts are exposed to your
application

90. Conflicts? That
sounds bad.

91. It happens frequently without strict
consistency. For example, have
two threads write at the same time.
Riak detects this conflict for you.

92. Your application is also
part of the Eventually
Consistent system. It
makes the hard decisions
that Riak can’t.

93. Ok. So how do I
resolve conflicts?

94. Riak will give you all values
that can’t be resolved
automatically. Just write
back the resolved value.

95. Think of it like merging or
rebasing git branches.
Sometimes you MUST use
git mergetool to fix things.

96. Two Ways to Resolve

97. Two Ways to Resolve
Semantic Resolution

98. Two Ways to Resolve
Semantic Resolution
CRDTs

99. Semantic Resolution

100. Semantic Resolution
Given any number of values,
compute the resolved version
using business logic.

101. Example

102. Example
If your value is a shopping cart,
merge all items and sum quantities
into a single cart.

103. Example
If your value is a shopping cart,
merge all items and sum quantities
into a single cart.
Maybe the customer
buys more that way!

104. Ripple supports
Semantic Resolution.

105. # A cart is { product_id => quantity }
Riak::RObject.on_conflict do |obj|
if obj.bucket.name == ‘carts’
obj.content_type = ‘application/json’
obj.data = {}
obj.siblings.each do |val|
val.data.each do |product,q|
obj.data[product] ||= 0
obj.data[product] += q
end
end
obj
end
end

106. # Documents can be resolved too
Person.on_conflict(:addresses) do |vals|
vals.each do |doc|
self.addresses &= doc.addresses
end
end

107. Commutative
Replicated Data-Types
http://hal.inria.fr/inria-00397981/en/

108. Commutative
Replicated Data-Types
Recent area of research
http://hal.inria.fr/inria-00397981/en/

109. Commutative
Replicated Data-Types
Recent area of research
Data-type defines
commutative operations
http://hal.inria.fr/inria-00397981/en/

110. Commutative
Replicated Data-Types
Recent area of research
Data-type defines
commutative operations
Value contains a state
and limited history of operations
http://hal.inria.fr/inria-00397981/en/

111. Commutative
Replicated Data-Types
Recent area of research
Data-type defines
commutative operations
Value contains a state
and limited history of operations
Roll-back, merge and replay
to resolve
http://hal.inria.fr/inria-00397981/en/

112. Currently, only Erlang
has a CRDT for Riak,
called statebox.

113. We’re researching how
to bring CRDTs to Ruby
and other languages.

114. Conclusion

115. Conclusion
Any sufficiently large system is
inconsistent and constantly failing

116. Conclusion
Any sufficiently large system is
inconsistent and constantly failing
Riak remains available during failures,
progresses toward consistency

117. Conclusion
Any sufficiently large system is
inconsistent and constantly failing
Riak remains available during failures,
progresses toward consistency
Riak & Ripple help your application
recover from inconsistency

118. Abrazámonos Riak