Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1nyhaC6. Nathan Marz discusses building NoSQL-based data systems that are scalable and easy to reason about. Filmed at qconlondon.com. Nathan Marz is the creator of many open source projects which are relied upon by over 50 companies around the world, including Cascalog and Storm. Nathan is also working on a book for Manning publications entitled "Big Data: principles and best practices of scalable realtime data systems". Nathan was previously the lead engineer at BackType before being acquired by Twitter in 2011.

1. A call for sanity in NoSQL
Nathan Marz
@nathanmarz 1

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/nosql-cons
http://www.infoq.com/presentati
ons/nasa-big-data
http://www.infoq.com/presentati
ons/nasa-big-data

3. Presented at QCon London
www.qconlondon.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. “Doofus programmer”

5. You broke the
build!

6. Where are the
tests?

7. The site is down!

8. DELETE FROM
users...

9. Oops, I forgot
the WHERE
clause!

10. =

11. =

14. No problem...

16. Mistakes are guaranteed

17. Create
Read
Update
Delete

18. Mutable
database
Guaranteed
corruption
=

19. Insane

20. Schemaless databases

21. Insane

22. Avoiding complexity

24. counter++

25. Insane

26. Denormalization

27. ID Name Location ID
1 Sally 3
2 George 1
3 Bob 3
Location ID City State Population
1 NewYork NY 8.2M
2 San Diego CA 1.3M
3 Chicago IL 2.7M
Normalized schema

28. Join is too expensive, so
denormalize...

29. ID Name Location ID City State
1 Sally 3 Chicago IL
2 George 1 NewYork NY
3 Bob 3 Chicago IL
Location ID City State Population
1 NewYork NY 8.2M
2 San Diego CA 1.3M
3 Chicago IL 2.7M
Denormalized schema

30. Insane

31. What is the source of the insanity?

32. Code is deterministic.
I understand logic.
Therefore, I can write correct
code.
Programming fallacy

33. Your code is wrong

34. Your code
Dependency 1
Dependency 2
Dependency 3

35. Dependency 1
Dependency 4
Dependency 5

36. Dependency 4
Dependency 6
Dependency 9
Dependency 7
Dependency 8

37. Dependency 3,000,000
Hardware

38. Electronics

39. Chemistry

40. Atomic physics

41. Quantum mechanics

42. I think I can safely say
that nobody understands
quantum mechanics.
Richard Feynman

43. Your code is wrong

44. Your code
...

45. Infinite regress

46. All the software you’ve
used has had bugs in it

47. Including the software
you’ve written

50. - Mutability
- Schemaless databases
- Eventual consistency / read-repair
- Denormalization
Insanity

51. Person Location
Sally NewYork
Bob Chicago
Mutability

52. Person Location
Sally London
Bob Chicago
Mutability

53. Person Location Time
Sally NewYork 1318358351
Bob Chicago 1327928370
Immutability

54. Person Location Time
Sally NewYork 1318358351
Bob Chicago 1327928370
Sally London 1338273801
Immutability

55. All
data
Precomputed
batch view
Query
Precomputed
realtime view
New data stream
Lambda Architecture

56. Relational database

57. NewSQL

58. All data problems?

59. What does a data system do?

60. Retrieve data that you
previously stored?
GetPut

61. Counterexamples
Store location information on people
Where does Sally live?
What are the most populous locations?
How many people live in a particular location?

62. Counterexamples
Store pageview information
How many unique visitors over time?
How many pageviews on September 2nd?

63. Counterexamples
Store transaction history for bank account
How much money do people spend on housing?
How much money does George have?

64. What does a data system do?
Query = Function(All data)

65. Example query
Total number of pageviews to a
URL over a range of time

66. Example query
Implementation

67. On-the-fly computation
All
data
Query

68. Precomputation
All
data
Precomputed
view
Query

69. Precomputed view
Example query
All data
Pageview
Pageview
Pageview
Pageview
Pageview
Query 2930

70. Precomputation
All
data
Precomputed
view
Query
(Immutable)

71. Precomputation
All
data
Precomputed
view
Query
Function Function

72. Computing views
All
data
Precomputed
view
Function

73. Function that takes in
all data as input

74. Batch processing

75. MapReduce

76. MapReduce is a framework for
computing arbitrary functions on
arbitrary data

77. Alldata
Batch view #1
Batch view #2
MapReduce workflow
MapReduce workflow
MapReduce precomputation

78. All
data
Precomputed
view
Query

79. All
data
Precomputed
view
Query
Normalized Denormalized

80. Human fault-tolerant

81. Not quite...
• A batch workflow is too slow
• Views are out of date
Absorbed into batch views Not absorbed
Now
Time
Just a few hours
of data!

82. Precomputation
All
data
Precomputed
batch view
Query
Precomputed
realtime view
New data stream
Compute parallel realtime views

83. NoSQL databases
New data stream
Realtime view #1
Realtime view #2
Stream processor

84. Precomputation
All
data
Precomputed
batch view
Query
Precomputed
realtime view
New data stream
Most complex part of system

85. Precomputation
All
data
Precomputed
batch view
Query
Precomputed
realtime view
New data stream
But only represents few hours of data

86. Precomputation
All
data
Precomputed
batch view
Query
Precomputed
realtime view
New data stream
So can be kept small

87. Precomputation
All
data
Precomputed
batch view
Query
Precomputed
realtime view
New data stream
If anything goes wrong, auto-corrects

88. Precomputation
All
data
Precomputed
batch view
Query
Precomputed
realtime view
New data stream
“Complexity isolation”

89. Precomputation
All
data
Precomputed
batch view
Query
Precomputed
realtime view
New data stream
No random writes needed

90. All
data
Precomputed
batch view
Query
Precomputed
realtime view
New data stream
“Lambda Architecture”

91. Applying the Lambda Architecture

92. Unique visitors over a range of hours
A
B
C
A
B
C
D
E
C
D
E
1 2 3 4
Hours
Pageviews

93. A
B
C
A
B
C
D
E
C
D
E
1 2 3 4
Hours
Pageviews
A = B
B = D
Equivs
Unique visitors over a range of hours

94. What should view look like?

95. Attempt #1
1-1 3
Count
1-2 6
1-3 6
2-2 4
2-3 5
3-3 1
Hour rangeURL
foo.com/blog
1-1 3
1-2 6
1-3 6
2-2 4
2-3 5
3-3 1
bar.com/foo

96. Attempt #2
1 3
Count
2 6
3 6
4 8
5 5
6 1
HourURL
foo.com/blog
1 1
2 11
3 23
4 8
5 9
6 0
bar.com/foo

97. Attempt #3
1 A, B, C
Visitors
2 A
3 D, E
4 A
5 B, C, D
6 F, G
HourURL
foo.com/blog
1 A
2 B, C
3 A, C
4 D
5 A, E
6 A
bar.com/foo

98. Attempt #4
1 <HLL>
HyperLogLog set
2 <HLL>
3 <HLL>
4 <HLL>
5 <HLL>
6 <HLL>
HourURL
foo.com/blog
1 <HLL>
2 <HLL>
3 <HLL>
4 <HLL>
5 <HLL>
6 <HLL>
bar.com/foo

99. Final attempt
1 <HLL>
HyperLogLog set
2 <HLL>
3 <HLL>
4 <HLL>
5 <HLL>
6 <HLL>
HourURL
foo.com/blog
1 <HLL>
2 <HLL>
3 <HLL>
4 <HLL>
5 <HLL>
6 <HLL>
bar.com/foo
1 <HLL>
HyperLogLog set
2 <HLL>
3 <HLL>
4 <HLL>
5 <HLL>
6 <HLL>
MonthURL
foo.com/blog
1 <HLL>
2 <HLL>
3 <HLL>
4 <HLL>
5 <HLL>
6 <HLL>
bar.com/foo
1 <HLL>
HyperLogLog set
2 <HLL>
3 <HLL>
4 <HLL>
5 <HLL>
6 <HLL>
WeekURL
foo.com/blog
1 <HLL>
2 <HLL>
3 <HLL>
4 <HLL>
5 <HLL>
6 <HLL>
bar.com/foo

100. Batch workflow
All data
Normalize
pageview URLs
Equiv connected-
component labeling
Normalize
pageview
userids
Aggregate
HyperLogLog sets
per bucket
Index into batch
view

101. Equiv graph
1 3
4
5
11
2
7
6
9

102. Equiv graph
1 3
4
5
11
2
7
6
9
Person A Person B

103. Equiv graph
1 -> A
2 -> A
3 -> A
4 -> A
5 -> A
11 -> A
6 -> B
7 -> B
9 -> B

104. Batch workflow
All data
Normalize
pageview URLs
Equiv connected-
component labeling
Normalize
pageview
userids
Aggregate
HyperLogLog sets
per bucket
Index into batch
view

105. Realtime workflow
The equiv problem

106. The equiv problem
1 <HLL>
HyperLogLog set
2 <HLL>
3 <HLL>
4 <HLL>
5 <HLL>
6 <HLL>
HourURL
foo.com/blog
1 <HLL>
2 <HLL>
3 <HLL>
4 <HLL>
5 <HLL>
6 <HLL>
bar.com/foo

107. The equiv problem
1 A, B, C
Visitors
2 A
3 D, E
4 A
5 B, C, D
6 F, G
HourURL
foo.com/blog
1 A
2 B, C
3 A, C
4 D
5 A, E
6 A
bar.com/foo

108. Solving the equiv problem
All data
Equiv connected-
component labeling
Index into batch view of
userid -> normalized userid
Batch

109. Realtime workflow
Batch
All data
Equiv connected-
component labeling
Index into batch view of
userid -> normalized userid
Pageviews stream Normalize userid
Aggregate realtime
HyperLogLog buckets
Realtime view

110. Schema

111. Schema

112. Schema

113. Schema

114. Schema

115. Schema

116. Conclusions from example
- Avoids every single insane complexity I talked about
- Powerful to be able to extract more out of a piece of data the longer
you have it
- Eventual accuracy is a super useful technique
- Schemas can be useful and non-painful
- A Lambda Architecture is fundamentally easy to extend with new
views/queries

117. Thank you

118. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/nosql-
cons