1. Hybrid MongoDB
Applications
with Relational Databases

2. Today’s Agenda
•Who I am
•Why MongoDB w/intro
•Why Hybrid
•Hybrid Case Studies
•How OpenSky implemented Hybrid
MySQL / MongoDB

3. My name is
Steve Francia
@spf13

4. •15+ years building the
internet (13 years using SQL)
•Father, husband,
skateboarder
•Chief Solutions Architect @
10gen responsible for
drivers, integrations, web &
docs

5. • Company behind MongoDB
• AGPL license, own copyrights, engineering team
• support, consulting, commercial license revenue
• Management
• Google/DoubleClick, Oracle, Apple, NetApp
• Funding: Sequoia, Union Square, Flybridge
• Offices in NYC, Palo Alto, London & Dublin
• 90+ employees

6. Before 10gen I
worked
for
http://opensky.com

7. OpenSky was the first
e-commerce site built
on MongoDB

8. Why MongoDB

9. Why MongoDB
My Top 10 Reasons
10. Great developer experience
9. Speaks your language
8. Scale horizontally
7. Fully consistent data w/atomic operations
1.It’ssource scale
web
6. Memory Caching integrated
5. Open
4. Flexible, rich & structured data format not just K:V
3. Ludicrously fast (without going plaid)
2. Simplify infrastructure & application

10. Why MongoDB
My Top 10 Reasons
10. Great developer experience
9. Speaks your language
8. Scale horizontally
7. Fully consistent data w/atomic operations
1.It’ssource scale
web
6. Memory Caching integrated
5. Open
4. Flexible, rich & structured data format not just K:V
3. Ludicrously fast (without going plaid)
2. Simplify infrastructure & application

11. MongoDB is
Application Document
Oriented
{ author: “steve”,
High date: new Date(),
text: “About MongoDB...”,
Performance tags: [“tech”, “database”]}
Fully
Consistent
Horizontally Scalable

12. Under the hood
• Written in C++
• Runs on nearly anything
• Data serialized to BSON
• Extensive use of memory-mapped files
i.e. read-through write-through memory
caching.

13. Database Landscape
MemCache
Scalability & Performance
MongoDB
RDBMS
Depth of Functionality

14. This has led
some to say
“
MongoDB has the best
features of key/values
stores, document databases
and relational databases in
one.
John Nunemaker

15. Why Hybrid?

16. Reasons to build a
hybrid application
•Friction in existing application caused
by RDBMS
•Transitioning an existing application to
MongoDB
•Using the right tool for the right job
•Need some features not present in
MongoDB

17. Reasons Not to build
a hybrid application
•Aggregation (at least not very soon)
•Lack of clear understanding of needs
•Backups
•MongoDB as cache in front of SQL
•Loads more...

18. Hybrid
Applications...
but I don’t
want to
complicate
things

19. Most
RDMBS
applications
are already
hybrid

20. Typical RDMBS
Application
Memcache
App
RDBMS

21. Typical Hybrid
RDMBS Application
MongoDB
App
RDBMS

22. Most of the same
rules apply
•Application partitions data between
two (or more) systems.
•Model layer tracks what content
resides where.

23. Hybrid is easier than
RDMBS + MemCache
• Always know where to find a piece of data.
• Data never needs expiring.
• Data not duplicated (for the most part)
across systems.
• Always handle a record same way.
• Developer freedom to choose the right tool
for the right reasons.

24. Typical RDBMS
retrieval operation
exists & up to date?
if yes... then done Memcache
if no, query DB for it
Retrieve record(s) RDBMS
App
Replace in cache
Memcache
Repeat

25. Typical Hybrid
Retrieval Operation
find
return MongoDB
App OR
query
return RDBMS

26. Typical RDMBS
write operation
insert or update row
confirm written RDBMS
assemble into object(s)
App write object
Memcache

27. Typical RDMBS
write operation
insert or update row
confirm written RDBMS
assemble into object(s)
App write object
write object
write object Memcache
write object
This goes on for a while doesn’t it?

28. Typical RDMBS
write operation
insert or update row
confirm written RDBMS
assemble into object(s)
App write object
write object
write object Memcache
write object
This goes on for a while doesn’t it?
one row can be in many objects so there’s
a lot of complication in updating everything

29. Typical Hybrid
Write Operation
save document
return MongoDB
App OR
insert or update
return RDBMS

30. Typical Hybrid
Write Operation
save document
return MongoDB
App OR
insert or update
return RDBMS

31. Hybrid Use Cases

32. Archiving
Why Hybrid:
• Existing application built on MySQL
• Lots of friction with RDBMS based archive storage
• Needed more scalable archive storage backend
Solution:
• Keep MySQL for active data (100mil), MongoDB for archive (2+
bil)
Results:
• No more alter table statements taking over 2 months to run
• Sharding fixed vertical scale problem
• Very happily looking at other places to use MongoDB

33. Reporting
Why Hybrid:
• Most of the functionality written in MongoDB
• Reporting team doesn’t want to learn MongoDB
Solution:
• Use MongoDB for active database, replicate to MySQL for
reporting
Results:
• Developers happy
• Business Analysts happy

34. E-commerce
Why Hybrid:
• Multi-vertical product catalogue impossible to model in RDBMS
• Needed transaction support RDBMS provides
Solution:
• MySQL for orders, MongoDB for everything else
Results:
• Massive simplification of code base
• Rapidly build, halving time to market (and cost)
• Eliminated need for external caching system
• 50x+ improvement over MySQL alone

35. How
implemented a
hybrid MongoDB /
MySQL solution
http://opensky.com

36. Doctrine (ORM/ODM)
makes it easy

37. Data to store in SQL
•Order
•Order/Shipment
•Order/Transaction
•Inventory

38. Data to store in
MongoDB

39. Data to store in
MongoDB
• User • Event
• Product • TaxRate
• Product/Sellable • ... and then I got
tired of typing them in
• Address
• Just imagine this list
• Cart has 40 more classes
• CreditCard • ...

40. The most boring SQL
schema ever

41. CREATE TABLE `product_inventory` (
`product_id` char(32) NOT NULL,
`inventory` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`product_id`)
);
CREATE TABLE `sellable_inventory` (
`sellable_id` char(32) NOT NULL,
`inventory` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`sellable_id`)
);
CREATE TABLE `orders` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`userId` char(32) NOT NULL,
`shippingName` varchar(255) DEFAULT NULL,
`shippingAddress1` varchar(255) DEFAULT NULL,
`shippingAddress2` varchar(255) DEFAULT NULL,
`shippingCity` varchar(255) DEFAULT NULL,
`shippingState` varchar(2) DEFAULT NULL,
`shippingZip` varchar(255) DEFAULT NULL,
`billingName` varchar(255) DEFAULT NULL,
`billingAddress1` varchar(255) DEFAULT NULL,
`billingAddress2` varchar(255) DEFAULT NULL,
`billingCity` varchar(255) DEFAULT NULL,

42. Did you notice
Inventory is in SQL
But it’s also property in your Mongo collections?

43. CREATE TABLE `product_inventory` (
`product_id` char(32) NOT NULL,
`inventory` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`product_id`)
);
CREATE TABLE `sellable_inventory` (
`sellable_id` char(32) NOT NULL,
`inventory` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`sellable_id`)
);
CREATE TABLE `orders` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`userId` char(32) NOT NULL,
`shippingName` varchar(255) DEFAULT NULL,
`shippingAddress1` varchar(255) DEFAULT NULL,
`shippingAddress2` varchar(255) DEFAULT NULL,
`shippingCity` varchar(255) DEFAULT NULL,
`shippingState` varchar(2) DEFAULT NULL,
`shippingZip` varchar(255) DEFAULT NULL,
`billingName` varchar(255) DEFAULT NULL,
`billingAddress1` varchar(255) DEFAULT NULL,
`billingAddress2` varchar(255) DEFAULT NULL,
`billingCity` varchar(255) DEFAULT NULL,

44. Inventory is
transient

45. Inventory is
transient
• Product::$inventory is effectively a
transient property
•Note how I said “effectively”? ... we
cheat and persist our transient
property to MongoDB as well
•We can do this because we never really
trust the value stored in Mongo

46. Accuracy is only important
when there’s contention

47. Accuracy is only important
when there’s contention
•For display, sorting and alerts, we can
use the value stashed in MongoDB
•It’s faster
•It’s accurate enough

48. Accuracy is only important
when there’s contention
•For display, sorting and alerts, we can
use the value stashed in MongoDB
•It’s faster
•It’s accurate enough
•For financial transactions, we want the
multi table transactions from our
RDBMS.

49. Inventory kept in
sync with listeners

50. Inventory kept in
sync with listeners
•Every time a new product is created,
its inventory is inserted in SQL

51. Inventory kept in
sync with listeners
•Every time a new product is created,
its inventory is inserted in SQL
•Every time an order is placed,
inventory is verified and decremented

52. Inventory kept in
sync with listeners
•Every time a new product is created,
its inventory is inserted in SQL
•Every time an order is placed,
inventory is verified and decremented
•Whenever the SQL inventory changes,
it is saved to MongoDB as well

53. Be careful what you
lock

54. Be careful what you
lock
1. Acquire inventory row lock and begin
transaction
2. Check current product inventory
3. Decrement product inventory
4. Write the Order to SQL
5. Update affected MongoDB documents
6. Commit the transaction
7. Release product inventory lock

55. Making MongoDB
and RDBMS relations
play nice

56. Products are
documents stored
in MongoDB

57. /** @mongodb:Document(collection="products") */
class Product
{
/** @mongodb:Id */
private $id;
/** @mongodb:String */
private $title;
public function getId()
{
return $this->id;
}
public function getTitle()
{
return $this->title;
}
public function setTitle($title)
{
$this->title = $title;
}
}

58. Orders are entities
stored in an RDBMS

59. /**
* @orm:Entity
* @orm:Table(name="orders")
* @orm:HasLifecycleCallbacks
*/
class Order
{
/**
* @orm:Id @orm:Column(type="integer")
* @orm:GeneratedValue(strategy="AUTO")
*/
private $id;
/**
* @orm:Column(type="string")
*/
private $productId;
/**
* @var DocumentsProduct
*/
private $product;
// ...
}

60. So how does an
RDBMS have a
reference to something
outside the database?

61. Setting the Product
class Order {
// ...
public function setProduct(Product $product)
{
$this->productId = $product->getId();
$this->product = $product;
}
}

62. • $productId is mapped and persisted
• $product which stores the Product
instance is not a persistent entity
property

63. Retrieving our
product later

64. OrderPostLoadListener
use DoctrineORMEventLifecycleEventArgs;
class OrderPostLoadListener
{
public function postLoad(LifecycleEventArgs $eventArgs)
{
// get the order entity
$order = $eventArgs->getEntity();
// get odm reference to order.product_id
$productId = $order->getProductId();
$product = $this->dm->getReference('MyBundle:DocumentProduct', $productId);
// set the product on the order
$em = $eventArgs->getEntityManager();
$productReflProp = $em->getClassMetadata('MyBundle:EntityOrder')
->reflClass->getProperty('product');
$productReflProp->setAccessible(true);
$productReflProp->setValue($order, $product);
}
}

65. All Together Now
// Create a new product and order
$product = new Product();
$product->setTitle('Test Product');
$dm->persist($product);
$dm->flush();
$order = new Order();
$order->setProduct($product);
$em->persist($order);
$em->flush();
// Find the order later
$order = $em->find('Order', $order->getId());
// Instance of an uninitialized product proxy
$product = $order->getProduct();
// Initializes proxy and queries the monogodb database
echo "Order Title: " . $product->getTitle();
print_r($order);

66. Read more about
this technique
Jon Wage, one of OpenSky’s engineers, first
wrote about this technique on his personal
blog: http://jwage.com
You can read the full article here:
http://jwage.com/2010/08/25/blending-the-
doctrine-orm-and-mongodb-odm/

67. http://spf13.com
http://github.com/spf13
@spf13
Questions?
download at mongodb.org
PS: We’re hiring!! Contact us at jobs@10gen.com