MongoDB presentation from Silicon Valley Code Camp 2015. Walkthrough developing, deploying and operating a MongoDB application, avoiding the most common pitfalls.

1. Advanced MongoDB
for Development, Deployment and
Operation - The Sequel
Daniel Coupal
Technical Services Engineer, Palo Alto, CA
#MongoDB
Silicon Valley Code Camp 2015

2. 2
• Making you successful in developing,
deploying and operating an application with
MongoDB
• I do expect you to know the basics of
MongoDB.
• …even better if you already have an
application about to be deployed or
deployed
This presentation is about …

3. 3
I hope you walk out of this presentation and
you make at least one single change in your
application, deployment, configuration, etc
that will prevent one issue from happening.
My Goal

4. 4
1. The Story of MongoDB
2. The Story of your Application
Chapter 1: Prototype and Development
Chapter 2: Deployment
Chapter 3: Operation
3. Wrapping up
4. Q&A
Agenda

5. 1. The Story of MongoDB

6. 6
The Sun was shinning on the land of the Oracle …
Once upon a time

7. 7
Then came the Web

8. 8
We’re gonna need a bigger database

9. 9
• Originaly, 10gen
– Founded in 2007
– Released MongoDB 1.0 in 2009
• MongoDB Inc
– Since 2013
– Acquired WiredTiger in 2014
• MongoDB
– Open source software
– Contributions on tools, drivers, …
– Most popular NoSQL database
MongoDB - Timeline

10. 10
MongoDB - Company Overview
450+ employees 1,000+ customers
Over $300 million in funding30+ offices around the world

11. 11
Positions open in Palo Alto, Austin and NYC
• http://www.mongodb.com/careers/positions
Technical service engineers in Palo Alto
• MongoDB
• MongoDB Tools
• Proactive support
MongoDB - We hire!

12. 2. The Story of your Application

13. 13
1. Schema, schema, schema!
2. Incorporate testability in your application
3. Think about data sizing and growth
4. What happens when a failure is returned
by the database?
5. Index correctly
6. Performance Tuning
Chapter 1 - Prototype and Development

14. 14
• Relational world
1. Model your data
2. Write the application against your data
• NoSQL world
1. Define what you do want to do with the data
Ø What are your queries?
2. Model your data
Schema, schema, schema

15. 15
• Test Driven Development
• Ask yourself, how can I test that this piece is
working
• TIP: MongoDB does not need a schema and it
creates databases and collections (tables) on the
fly
– Incorporate username, hostname, timestamps in
database names for unit tests
Incorporate testability in the application

16. 16
• How much data will you have initially?
• How will your data set grow over time?
• How big is your working set?
• Will you be loading huge bulk inserts, or have a
constant stream of writes?
• How many reads and writes will you need to
service per second?
• What is the peak load you need to provision for?
Think about data sizing and growth

17. 17
• Good model and understanding of latencies, write
concerns
• Catch exceptions
• Retries
• …
What happens when a failure is returned
by the database?

18. 18
• More than 50% of the customer issues
• Collection Scan
– Very bad if you have a large collection
– One of the main performance issue see in our
customers’ application
– Can be identified in the logs with the
‘nscannedObjects’ attribute on slow queries
• Watch out for updates to the Application
Index correctly

19. 19
1. Assess the problem and establish acceptable behavior
2. Measure the current performance
3. Find the bottleneck*
4. Remove the bottleneck
5. Re-test to confirm
6. Repeat
* - (This is often the hard part)
(Adapted from http://en.wikipedia.org/wiki/Performance_tuning )
Performance Tuning

20. 20
1. Deployment topology
2. Have a test/staging environment
– Track slow queries and collection scans
3. MongoDB production notes
– http://docs.mongodb.org/manual/administration/production-notes
4. Storage considerations
5. Host considerations
Chapter 2 - Deploy

21. 21
• Sharding or not?
• 3 data nodes per replica set or 2 data nodes + arbiter?
• Many Data Centers or availability zones
• What is important for you?
– Durability of writes
– Performance
=> can be chosen per operation
Deployment topology

22. 22
• Best if it the capacity matches the production deployment
– Otherwise, if prod is 20 shards x 3 nodes, you can have 2 x 3
nodes, or 20 x 1 node
• Data size should be representative
– Start with simulated data
– Use backup of production data
• Disable table/collection scans or scan the logs for them
Have a test/staging environment

23. 23
• Most important documentation of MongoDB
– http://docs.mongodb.org/v3.0/administration/production-notes/
• Security checklist
– Authentication, limit network exposure, … audit system activity
• Allocate Sufficient RAM and CPU
• MongoDB and NUMA Hardware
• Platform Specific Considerations
– Turn off atime for the storage volume containing the database files.
– Set the file descriptor limit, -n, and the user process limit (ulimit), -u,
above 20,000
– MongoDB on Virtual Environments
MongoDB Production notes

24. 24
• RAID
=> 0+1 or None
• HDD or SSD
=> SSD, if budget permit
• NAS, SAN or Direct Attached?
=> Direct Attached, good news those are the cheapest!
• File System type
MMAPv1 => ext4
WiredTiger => xfs
• Settings
=> ReadAhead
Storage considerations

25. 25
• CPU power?
– MMAPv1 => no
– WiredTiger => yes, needed for compression
• RAM?
– Yes! RAM is always an order of magnitude faster than
disk
Host considerations

26. 26
1. Monitor
2. Upgrade
3. Backup
4. Troubleshoot
Chapter 3 - Operation

27. 27
“Shit will happen!”
• Are you prepared?
• Have backups?
• Have a good picture of your “normal state”
Disaster will strike

28. 28
• iostat, top, vmstat, sar
• mongostat, mongotop
• CloudManager/OpsManager Monitoring
– plus Munin extensions
Monitor

29. 29
• Major versions have same binary format,
same protocol, etc for each new minor
version
• Major versions have upgrade and
downgrade paths
• CloudManager and OpsManager Automation
handles automatic upgrades
Upgrade

30. 30
Mongodump File system CloudManager
Backup
OpsManager
Backup
Initial complexity Medium High Low High
System overhead High Low Low Medium
Point in time
recovery of replica
set
No * No * Yes Yes
Consistent
snapshot of
sharded system
No * No * Yes Yes
Scalable No Yes Yes Yes
Restore time Slow Fast Medium Medium
Comparing MongoDB backup approaches
* Possible, but need to write the tools and go though a lot of testing

31. 31
• mtools
– https://github.com/rueckstiess/mtools/wiki
(just Google: github mtools)
Troubleshoot

32. 32
namespace pattern count min (ms) max (ms) mean (ms) sum (ms)
serverside.scrum_master {"datetime_used": {"$ne": 1}} 20 15753 17083 16434 328692
serverside.django_session {"_id": 1} 562 101 1512 317 178168
serverside.user {"_types": 1, "emails.email": 1} 804 101 1262 201 162311
local.slaves {"_id": 1, "host": 1, "ns": 1} 131 101 1048 310 40738
serverside.email_alerts {"_types": 1, "email": 1, "pp_user_id": 1} 13 153 11639 2465 32053
serverside.sign_up {"_id": 1} 77 103 843 269 20761
serverside.user_credits {"_id": 1} 6 204 900 369 2218
serverside.counters {"_id": 1, "_types": 1} 8 121 500 263 2111
serverside.auth_sessions {"session_key": 1} 7 111 684 277 1940
serverside.credit_card {"_id": 1} 5 145 764 368 1840
serverside.email_alerts {"_types": 1, "request_code": 1} 6 143 459 277 1663
serverside.user {"_id": 1, "_types": 1} 5 153 427 320 1601
serverside.user {"emails.email": 1} 2 218 422 320 640
serverside.user {"_id": 1} 2 139 278 208 417
serverside.auth_sessions {"session_endtime": 1, "session_userid": 1} 1 244 244 244 244
serverside.game_level {"_id": 1} 1 104 104 104 104
Troubleshoot – Slow Queries

33. 33
• Interactive graph!
Troubleshoot – Slow Queries Plot

34. 3. Wrapping up

35. 35
1. Missing indexes
2. Not testing before deploying application changes
3. OS settings
4. Appropriate schema
5. Hardware
6. Not seeking help early enough
Common Mistakes

36. 36
• MongoDB on-line presentations mongodb.com/presentations
• Free MongoDB classes university.mongodb.com
• mtools to analyze MongoDB logs github.com/rueckstiess/mtools
• CloudManager cloud.mongodb.com
Free On-Line Resources

37. 37
• MongoDB Support
– 24x7 support
– the sun never set on MongoDB Customer Support Team
• MongoDB Consulting Days
• MongoDB World (@NYC on June 28-29, 2016)
• MongoDB Days (@SanJose on Dec 3, 2015)
Resources

38. 38
• Use available resources
• Testing
– Plan for it, plan resources for it, do it before deploying
in a Test or Staging environment
Summary

39. 39
I hope you walk out of this presentation and
you make at least one single change in your
application, deployment, configuration, etc
that will prevent one issue from happening.
Take away

40. 4. Q&A

41. 41
Positions open in Palo Alto, Austin and NYC
• http://www.mongodb.com/careers
MongoDB for Giant Ideas