Why MongoDB is the perfect solution for the challenges of E-Commerce.

1. MongoDB
E-commerce
and Transactions

2. My name is
Steve Francia
@spf13

3. • 15+ years building the
internet
• Father, husband, skateboarder
• Chief Solutions Architect @
10gen
• Author of upcoming O’Reilly
publication
“MongoDB and PHP”

4. Before 10gen I
worked
for
http://opensky.com

5. OpenSky was the first
e-commerce site built
on MongoDB
... also the first e-commerce site built on
Symfony2

6. Introduction to
MongoDB

7. Why MongoDB?

8. MongoDB Goals
• Open source
• Designed for today
• Today’s hardware / environments
• Today’s challenges
• Great developer experience
• Reliable
• Scalable

9. • Company behind MongoDB
• AGPL license, own copyrights, engineering
team
• support, consulting, commercial license
revenue
• Management
• Google/DoubleClick, Oracle, Apple, NetApp
• Funding: Sequoia, Union Square, Flybridge
• Offices in NYC, Redwood Shores & London
• 60+ employees

10. A bit of history

11. 1974
The relational database is created

14. 1979

15. 1979 1982-1996

16. 1979 1982-1996 1995

17. Computers in 1995
•Pentium 100 mhz
•10base T
•16 MB ram
•200 MB HD

18. Cloud in 1995
(Windows 95 cloud wallpaper)

19. Cell Phones in 2011
•Dual core 1.5 Ghz
•WiFi 802.11n (300+ Mbps)
•1 GB ram
•64GB Solid State

20. How about a DB
designed for today?

21. MongoDB philosophy
• Keep functionality when we can (key/
value stores are great, but we need more)
• Non-relational (no joins) makes scaling
horizontally practical
• Document data models are good
• Database technology should run
anywhere VMs, cloud, metal, etc

22. MongoDB is:
Application Document
Oriented
{ author: “steve”,
High date: new Date(),
text: “About MongoDB...”,
Performance tags: [“tech”, “database”]}
Fully
Consistent
Horizontally Scalable

23. Under the hood
•Written in C++
•Runs on nearly anything
•Data serialized to BSON
•Extensive use of memory-mapped files

24. Database Landscape

25. This has led
some to say
“
MongoDB has the best
features of key/ values
stores, document databases
and relational databases in
one.
John Nunemaker

26. Use Cases

27. CMS / Blog
Needs:
• Business needed modern data store for rapid development and
scale
Solution:
• Use PHP & MongoDB
Results:
• Real time statistics
• All data, images, etc stored together, easy access, easy
deployment, easy high availability
• No need for complex migrations
• Enabled very rapid development and growth

28. Photo Meta-Data
Problem:
• Business needed more flexibility than Oracle could deliver
Solution:
• Use MongoDB instead of Oracle
Results:
• Developed application in one sprint cycle
• 500% cost reduction compared to Oracle
• 900% performance improvement compared to Oracle

29. Customer Analytics
Problem:
• Deal with massive data volume across all customer sites
Solution:
• Use MongoDB to replace Google Analytics / Omniture options
Results:
• Less than one week to build prototype and prove business case
• Rapid deployment of new features

30. Online Dictionary
Problem:
• MySQL could not scale to handle their 5B+ documents
Solution:
• Switched from MySQL to MongoDB
Results:
• Massive simplification of code base
• Eliminated need for external caching system
• 20x performance improvement over MySQL

31. E-commerce
Problem:
• Multi-vertical E-commerce impossible to model (efficiently) in
RDBMS
Solution:
• Switched from MySQL to MongoDB
Results:
• Massive simplification of code base
• Rapidly build, halving time to market (and cost)
• Eliminated need for external caching system
• 50x+ improvement over MySQL

32. Tons more
Pretty much if you can use a RDMBS or Key/Value
MongoDB is a great fit

33. In Good Company

34. Why NoSQL for
e-commerce?
Using the right solution for each situation

35. Data dilemma of
e-commerce
Pick One

36. Data dilemma of
e-commerce
Pick One
•Stick to one vertical (Sane schema)
•Flexibility (Insane schema)

37. Sane schema

38. Sane schema
•Works ... for a while
•Fine for a few types of products
•Not possible when more product types
introduced

39. Let’s Use an Example

40. Let’s Use an Example
How about we start with books

41. Book Product Schema
Product {
id:
sku: General Product
product dimensions:
shipping weight:
attributes
MSRP:
price:
description:
...
author: Orson Scott Card
title: Enders Game
binding: Hardcover
publication date: July 15, 1994 Book Specific
publisher name: Tor Science Fiction attributes
number of pages: 352
ISBN: 0812550706
language: English
...

42. Seems simple enough
What happens when we add another vertical...
say music albums

43. Album Product Schema
Product {
id:
sku: General Product
product dimensions: attributes stay
shipping weight:
MSRP:
the same
price:
description:
...
artist: MxPx
title: Panic Album Specific
release date: June 7, 2005
label: Side One Dummy
attributes are
track listing: [ The Darkest ... different
language: English
format: CD
...

44. Okay, it’s getting
hairy but is still
manageable, right?
Now the business want to sell jeans

45. Jeans Product Schema
Product {
id: General Product
sku: attributes stay the
product dimensions: same
shipping weight:
MSRP:
price:
description:
...
brand: Lucky Jeans specific
gender: Mens attributes are
make: Vintage
totally different ...
style: Straight Cut
length: 34 and not consistent
width: 34 across brands &
color: Hipster
make
material: Cotten Blend
...

46. Now we’re screwed

47. Now we’re screwed

48. We need a flexible
schema in RDBMS

49. We need a flexible
schema in RDBMS
We got this ... right?

50. Many approaches
dealing with unknown
unknowns in RDBMS

51. Many approaches
dealing with unknown
unknowns in RDBMS
None work well

52. EAV
as popularized by Magento
“For purposes of flexibility, the Magento database heavily
utilizes an Entity-Attribute-Value (EAV) data model.
As is often the case, the cost of flexibility is complexity -
Magento is no exception.
The process of manipulating data in Magento is often
more “involved” than that typically experienced using
traditional relational tables.”
- Varien

53. EAV
• Crazy SQL queries
• Hundreds of joins in a
query... or
• Hundreds of queries joined
in the application
• No database enforced
integrity

54. Did I say crazy SQL
(this is a single query)

55. Did I say crazy SQL
(this is a single query)
You may have trouble reading this in the back

56. Selecting a single
product

57. Single Table Inheritance
(insanely wide tables)
• No data integrity enforcement
• Only can use FK for common
elements
• Very wasteful (but disk is
cheap!)
• Can’t effectively index

58. Generic Columns
• No data integrity enforcement
• No data type enforcement
• Only can use FK for common
elements
• Wasteful (but disk is cheap!)
• Can’t index

59. Serialized in Blob
• Not searchable
• No integrity
• All the disadvantages of a
document store, but none of the
advantages
• Never should be used
• One exception is Oracle XML
which operates similar to a
document store

60. Concrete Table Inheritance
(a table for each product attribute set)
• Allows for data integrity
• Querying across attribute
sets quite hard to do (lots
of joins, OR statements
and full table scanning)
• New table needs to be
created for each new
attribute set

61. Class table inheritance
(single product table,
each attribute set in own table)
• Likely best of SQL within the
constraint
solution
• Supports data type enforcement
• No data integrity enforcement
• Easybrowse pages) since
(for
querying across categories
common data in single table
• Every set needs a new table
• Requiresare very complicated
changes
a ton of forsight, as

62. MongoDB to the
Rescue

63. MongoDB to the
Rescue
•Flexible (and sane) Schema

64. MongoDB to the
Rescue
•Flexible (and sane) Schema
•Easily searchable

65. MongoDB to the
Rescue
•Flexible (and sane) Schema
•Easily searchable
•Easily accessible

66. MongoDB to the
Rescue
•Flexible (and sane) Schema
•Easily searchable
•Easily accessible
•Fast

67. Flexible schema

68. { {
sku: "00e8da9c", sku: "00e8da9d",
type: "Audio Album", type: "Film",
title: "Hoss", title: "The Matrix",
description: "by Lagwagon", description: "Set in the 22nd century, Th
asin: "B0000007QG", asin: "B000P0J0AQ",
shipping: { shipping: {
weight: 6, weight: 6,
dimensions: { dimensions: {
width: 10, width: 10,
height: 10, height: 10,
depth: 1 depth: 1
}, },
}, },
pricing: { pricing: {
list: 1000, list: 1200,
retail: 800, retail: 1100,
savings: 200, savings: 100,
pct_savings: 20 pct_savings: 8.5
}, },
details: { details: {
title: "Hoss", title: "The Matrix",

69. pct_savings: 20 pct_savings: 8.5
}, },
details: { details: {
title: "Hoss", title: "The Matrix",
artist: "Lagwagon", director: [ "Andy Wachowski", "Larry Wa
genre: [ "Punk", "Hardcore", "Indie Rock" ], [ "Andy Wachowski", "Larry Wach
writer:
label: "Fat Wreck Chords", actor: [ "Keanu Reeves" , "Lawrence Fis
number_of_discs: 1, genre: [ "Science Fiction", "Action" ],
issue_date: "November 21, 1995", number_of_discs: 1,
format: "CD", issue_date: "May 15 2007",
alternate_formats: [ 'Vinyl', 'MP3' ],original_release_date: "1999",
tracks: [ disc_format: "DVD",
"Kids Don't Like To Share", rating: "R",
"Violins", alternate_formats: [ 'VHS', 'Bluray' ],
"Name Dropping", run_time: "136",
"Bombs Away", studio: "Warner Bros",
"Move The Car", language: "English",
"Sleep", format: [ "AC-3", "Closed-captioned", "
"Sick", aspect_ratio: "1.66:1"
"Rifle", },
"Weak", }
"Black Eye",
"Bro Dependent",
"Razor Burn",
"Shaving Your Head",
"Ride The Snake",
],

70. Queries

71. db.products.find( { 'name': "The Matrix" } );

72. db.products.find( { 'name': "The Matrix" } );
{
"_id": ObjectId("4d8ad78b46b731a22943d3d3"),
"sku": "00e8da9d",
"type": "Film",
"name": "The Matrix",
"description": "Set in the 22nd century, The Matrix...",
"asin": "B000P0J0AQ",
"shipping": {
"weight": 6,
"dimensions": {
"width": 10,
"height": 10,
"depth": 1
}
},
"pricing": {

73. db.products.find( { 'details.actor': "Groucho Marx" } );

74. db.products.find( { 'details.actor': "Groucho Marx" } );
},
"pricing": {
"list": 1000,
"retail": 800,
"savings": 200,
"pct_savings": 20
},
"details": {
"title": "A Night at the Opera",
"director": "Sam Wood",
"actor": ["Groucho Marx", "Chico Marx", "Harpo Marx"],
"genre": "Comedy",
"number_of_discs": 1,
"issue_date": "May 4 2004",
"original_release_date": "1935",
"disc_format": "DVD",

75. db.products.find( {
'details.genre': "Jazz", 'details.format': "CD"
} );

76. db.products.find( {
'details.genre': "Jazz", 'details.format': "CD"
} );
"list": 1200,
"retail": 1100,
"savings": 100,
"pct_savings": 8
},
"details": {
"title": "A Love Supreme [Original Recording Reissued]",
"artist": "John Coltrane",
"genre": ["Jazz", "General"],
"format": "CD",
"label": "Impulse Records",
"number_of_discs": 1,
"issue_date": "December 9, 1964",
"alternate_formats": ["Vinyl", "MP3"],
"tracks": [
"A Love Supreme Part I: Acknowledgement",

77. db.products.find( { 'details.actor':
{ $all: ['James Stewart', 'Donna Reed'] }
} );

78. db.products.find( { 'details.actor':
{ $all: ['James Stewart', 'Donna Reed'] }
} );
},
"details": {
"title": "It's a Wonderful Life",
"director": "Frank Capra",
"actor": ["James Stewart", "Donna Reed", "Lionel Barrymore"],
"writer": [
"Frank Capra",
"Albert Hackett",
"Frances Goodrich",
"Jo Swerling",
"Michael Wilson"
],
"genre": "Drama",
"number_of_discs": 1,
"issue_date": "Oct 31 2006",
"original_release_date": "1947",

79. Wanna Play?
• grab products.js from
http://github.com/spf13/
mongoProducts
• mongo --shell products.js
• > use mongoProducts

80. Embedded documents
are great for orders
•Ordered items need to be fixed at the
time of purchase
•Embed them right in the order
db.order.find( { 'items.sku': '00e8da9f' } );
db.order.find( {
'items.details.actor': 'James Stewart'
} ).count();

81. What about
transactions?
Using the right solution for each situation

82. Data (like people) are
really sensitive when
it comes to money

83. Stricter data
requirements for $$

84. Stricter data
requirements for $$
•For financial systems any data
inconsistency is unacceptable

85. Stricter data
requirements for $$
•For financial systems any data
inconsistency is unacceptable
•Perhaps you’ve heard of ACID?

86. What about ACID?

87. What about ACID?
Q: Is MongoDB ACID?

88. What about ACID?
Q: Is MongoDB ACID?
A: Kinda

89. Atomicity

90. Atomicity
•MongoDB does atomic writes

91. Atomicity
•MongoDB does atomic writes
... for single document changesets

92. Atomicity
•MongoDB does atomic writes
... for single document changesets
• $set, $unset, $inc, $push,
$pushAll, $pull, $pullAll, $bit

93. Consistency

94. Consistency
•MongoDB can enforce unique keys

95. Consistency
•MongoDB can enforce unique keys
•MongoDB can't enforce referential
integrity

96. Isolation

97. Isolation
• // Pseudo-isolated updates
db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );

98. Isolation
• // Pseudo-isolated updates
db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );
• // Isolated updates
db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false ,
true );

99. Isolation
• // Pseudo-isolated updates
db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );
• // Isolated updates
db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false ,
true );
• But there are caveats...

100. Isolation
• // Pseudo-isolated updates
db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );
• // Isolated updates
db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false ,
true );
• But there are caveats...
• Despite the $atomic keyword, this is not an atomic update,
since atomicity implies “all or nothing”

101. Isolation
• // Pseudo-isolated updates
db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );
• // Isolated updates
db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false ,
true );
• But there are caveats...
• Despite the $atomic keyword, this is not an atomic update,
since atomicity implies “all or nothing”
• $atomic here means update is done without an interference
from any other operation (isolated)

102. Isolation
• // Pseudo-isolated updates
db.foo.update( { x : 1 } , { $inc : { y : 1 } } , false , true );
• // Isolated updates
db.foo.update( { x : 1 , $atomic : 1 } , { $inc : { y : 1 } } , false ,
true );
• But there are caveats...
• Despite the $atomic keyword, this is not an atomic update,
since atomicity implies “all or nothing”
• $atomic here means update is done without an interference
from any other operation (isolated)
• An isolated update can only act on a single collection. Multi-
collection updates are not transactional, thus not
isolatable.

103. Durability

104. Durability
•Mongo has this one covered

105. What does
MongoDB Support?

107. • Atomic single document writes
• If you need atomic writes across multi-document
transactions don't use Mongo
• Many if not most e-commerce transactions could be
accomplished within a single document write

108. • Atomic single document writes
• If you need atomic writes across multi-document
transactions don't use Mongo
• Many if not most e-commerce transactions could be
accomplished within a single document write
• Unique indexes
• This only works on keys used by the entire
collection

109. • Atomic single document writes
• If you need atomic writes across multi-document
transactions don't use Mongo
• Many if not most e-commerce transactions could be
accomplished within a single document write
• Unique indexes
• This only works on keys used by the entire
collection
• Isolated (not atomic) single collection updates.
• Mongo does not support locking
• There are ways to work around this

110. • Atomic single document writes
• If you need atomic writes across multi-document
transactions don't use Mongo
• Many if not most e-commerce transactions could be
accomplished within a single document write
• Unique indexes
• This only works on keys used by the entire
collection
• Isolated (not atomic) single collection updates.
• Mongo does not support locking
• There are ways to work around this
• It’s durable

111. There are ways to
guarantee ACID
properties in MongoDB
Here are 2 good approaches useful for
E-commerce transactions

112. Optimistic
Concurrency

113. Optimistic
Concurrency
•Read the current state of a product

114. Optimistic
Concurrency
•Read the current state of a product
•Make your changes with the assertion
that your product has the same state as
it did when you last read it

115. Optimistic concurrency
in MongoDB

116. Optimistic concurrency
in MongoDB
We’ll use an update-if-current strategy.

117. Optimistic concurrency
in MongoDB
We’ll use an update-if-current strategy.
This example is straight from the documentation:

118. Optimistic concurrency
in MongoDB
We’ll use an update-if-current strategy.
This example is straight from the documentation:
> t = db.inventory
> p = t.findOne({sku:'abc'})
> t.update({_id:p._id, qty:p.qty}, {'$inc': {qty: -1}});
> db.$cmd.findOne({getlasterror:1});
{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}
// it worked

119. Optimistic concurrency
in MongoDB
We’ll use an update-if-current strategy.
This example is straight from the documentation:
> t = db.inventory
> p = t.findOne({sku:'abc'})
> t.update({_id:p._id, qty:p.qty}, {'$inc': {qty: -1}});
> db.$cmd.findOne({getlasterror:1});
{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}
// it worked
... If that didn't work, try again until it does.

120. Optimistic
concurrency
•Read the current state of a product.
•Make your changes with the assertion
that your product has the same state as
it did when you last read it.

121. Optimistic
concurrency
•Read the current state of a product.
•Make your changes with the assertion
that your product has the same state as
it did when you last read it.
• It is also possible to use OCC to
bootstrap pessimistic concurrency and
fake row level locking

122. Optimistic concurrency
control assumes an
environment with low
data contention

123. OCC works great for
companies like Amazon
•Amazon has a long-tail catalog
•A long tail catalog lends itself well to
optimistic concurrency, because it has
low data contention

124. OCC fails miserably
for

125. OCC fails miserably
for
•eBay

126. OCC fails miserably
for
•eBay
•Gilt

127. OCC fails miserably
for
•eBay
•Gilt
•Groupon

128. OCC fails miserably
for
•eBay
•Gilt
•Groupon
•OpenSky

129. OCC fails miserably
for
•eBay
•Gilt
•Groupon
•OpenSky
•Living Social

130. OCC fails miserably
for
•eBay
•Gilt
•Groupon
•OpenSky
•Living Social
•InsertFlashSaleSiteOfTheMinute

131. Flash sales and
auctions are defined by
high data contention

132. Flash sales and
auctions are defined by
high data contention
•The model doesn't work otherwise

133. Flash sales and
auctions are defined by
high data contention
•The model doesn't work otherwise
•They can't afford to be optimistic

134. Flash sales and
auctions are defined by
high data contention
•The model doesn't work otherwise
•They can't afford to be optimistic
•Order really matters

135. What about high
contention
environments?

136. If we can avoid
concurrency we’ve
got it made

137. Commerce is ACID
In Real Life

139. 1. I go to Barneys and see a pair of shoes I just have to
buy.
2. I call “dibs” (by grabbing them off the shelf).
3. I take them up to the cash register and purchase
them:
• Store inventory has been manually decremented.
• I pay for them with my trusty AmEx.
4. If all goes according to plan, I walk out of the store.
5. If my card was declined, the shoes are “rolled back”
... out onto the shelves and sold to the next customer
who wants them.

140. All of this is
accomplished
without concurrency

141. Each item can only be
held by a consumer

142. We follow the same
model for e-commerce

144. 1. Select a product.

145. 1. Select a product.
2. Update the document to hold inventory.

146. 1. Select a product.
2. Update the document to hold inventory.
• Store inventory has been
decremented.

147. 1. Select a product.
2. Update the document to hold inventory.
• Store inventory has been
decremented.
3. Purchase the product(s)

148. 1. Select a product.
2. Update the document to hold inventory.
• Store inventory has been
decremented.
3. Purchase the product(s)
• Process payment

149. 1. Select a product.
2. Update the document to hold inventory.
• Store inventory has been
decremented.
3. Purchase the product(s)
• Process payment
4. Roll back if anything went wrong.

150. MongoDB e-commerce
transactions

151. MongoDB e-commerce
transactions
• Each Item (not SKU) has it’s own document
• Document consists of...
• a reference to the SKU (product)
• a state ( available / sold / ... )
• potentially other data (timestamp, order
ref)

152. Transactions
in MongoDB

153. Transactions
in MongoDB
We’ll use a simple update statement
here.

154. Transactions
in MongoDB
We’ll use a simple update statement
here.
> t = db.inventory
> sku = sku.findOne({sku:'abc'})
> t.update({ref_id:sku._id, state: 'available'}, {'$set':
{state: 'ordered'}});
> db.$cmd.findOne({getlasterror:1});
{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}
// it worked

155. Transactions
in MongoDB
We’ll use a simple update statement
here.
> t = db.inventory
> sku = sku.findOne({sku:'abc'})
> t.update({ref_id:sku._id, state: 'available'}, {'$set':
{state: 'ordered'}});
> db.$cmd.findOne({getlasterror:1});
{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}
// it worked
... If that didn't work, no inventory available

156. Cart in Cart Action

157. Cart in Cart Action
An added benefit, it can easily provide
inventory hold in cart.

158. Cart in Cart Action
An added benefit, it can easily provide
inventory hold in cart.
> t = db.inventory
> sku = sku.findOne({sku:'abc'})
> t.update({ref_id:sku._id, state: 'available'}, {'$set':
{state: 'in cart'}});
> db.$cmd.findOne({getlasterror:1});
{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}
// it worked

159. Cart in Cart Action
An added benefit, it can easily provide
inventory hold in cart.
> t = db.inventory
> sku = sku.findOne({sku:'abc'})
> t.update({ref_id:sku._id, state: 'available'}, {'$set':
{state: 'in cart'}});
> db.$cmd.findOne({getlasterror:1});
{"err" : , "updatedExisting" : true , "n" : 1 , "ok" : 1}
// it worked
just like reality, each item is either
available, in a cart, or purchased

160. http://spf13.com
http://github.com/spf13
@spf13
Questions?
download at mongodb.org
PS: We’re hiring!! Contact us at jobs@10gen.com