Using Compass to Diagnose Performance Problems in Your Cluster
Speaker: Brian Blevins, Technical Services Engineer, MongoDB
Date/Time: June 20, 1:50 PM
Track: Performance
Since the performance of your application drives engagement and revenue, it can make or break the success of your organization. You can use the Compass graphical client from MongoDB to visualize your database schema, collect information on optimization opportunities and make database changes to improve performance. In this talk, we will briefly introduce Compass and then delve into the features supporting database performance optimization. The talk will combine instruction on the use of Compass with recommendations for performance best practices. We will also review the detection and resolution of slow queries and excessive network utilization. After attending the talk, audience members will have a better understanding of the capabilities of Compass, including how those capabilities can be used to find and correct performance bottlenecks in MongoDB databases. This session is designed for those with limited MongoDB experience. Attendees should have a basic understanding of MongoDB’s schema design, the server/database/collection layout, and how their application accesses and uses the MongoDB database.
What You Will Learn:
- Identify excessive network utilization, adjust queries appropriately and use Compass to confirm results.
- Understand how the Compass graphical client can help you improve performance in your MongoDB deployment.
- Use Compass real time statistics to identify slow queries and recognize when a query is a good candidate for adding an index.
15. CONCEPT INTERACTIONS
Query
Shape
Filter, Projection and Sort
Fields
Query planning and
caching
Working
Set
Pages needed to satisfy a
query
Efficient response to
queries
Index Document access
structure
Improves query
performance
20. WHAT DO WE KNOW ABOUT THE “NAME”
QUERY?
Full Collection Scan
Millions Examined
Only 4% Returned
21. HOW TO IMPROVE THE “NAME” QUERY?
1.Measure
2.Create a Targeted
Index with Compass
3.Measure Again
22.
23. RESULTS OF THE INDEX EXPERIMENT
• Only scanning documents returned.
• Slowest query 7x faster.
• Operations/sec up 3x.
24. WHEN TO INDEX?
Indications
R Read Activity Outweighs Write Activity
R Dataset Does Not Fit in RAM
R Frequent or Time Critical Query
R Big Data, Business Intelligence and Analytics
25. WHEN TO INDEX?
Indications
R Read Activity Outweighs Write Activity
R Dataset Does Not Fit in RAM
R Frequent or Time Critical Query
R Big Data, Business Intelligence and Analytics
Contraindications
V Write Activity Outweighs Read Activity
V Infrequent or Unimportant Query
V Logging, Data Collection
Always test changes outside production.
30. WHAT DO WE KNOW
ABOUT THE “CATEGORY_CODE“ QUERY?
• “tech.companies”, uses the most
resources.
• Queries already have an index.
• High level of network utilization for few
operations.
• Large documents
37. NETWORK OPTIMIZATION SIGNALS
Indications
R High Network Usage
R Many Documents in Results
R Large Documents in Results
Contraindications
V All Fields and All Documents
Required
Always test changes outside
production.
43. WHAT WE KNOW ABOUT THE QUERY
db.companies.find(
{ "age": 38, "membership_status": "ACTIVE" },
{ "_id": 0, "email" : 1, "name": 1 })
• “mongodb.fanclub”, uses the most resources.
• “age” field already has index in use.
• Projection already in use.
• Small result set.
51. COVERED QUERY STRATEGY
• Accelerate a
small, critical
query by
including all
filter, result and
sort fields in the
same index.
52. SIGNALS FOR COVERED QUERIES
Indications
R Frequent Query
R Small Result Set
R Projection in Use
53. SIGNALS FOR COVERED QUERIES
Indications
R Frequent Query
R Small Result Set
R Projection in Use
Contraindications
V Index doesn’t Fit
in RAM
V Indexes Slow
Write Ops
Always test changes outside
production.
55. PERFORMANCE HEURISTICS TO LIVE BY
Indexes Reduce Disk Access for Reads
Indexes Protect Working Set
The Fastest Code is the Code that Does Not Run
Cover Filter, Result and Sort fields for Speed
59. ATTRIBUTIONS A
Who is the clown with
the microphone?
ryan harvey https://www.flickr.com/photos/ryanh/7509976606/in/photolist-crCAph-pUWKYx-pqePxH-ocHgxR-9jnR2B-2yYmQA-bnMxe1-e1UHry-bAfqYx-DZBrcY-
6kskYE-fM1bFg-9AjS9x-aDh5nS-3wM6aD-bcwj7F-bcwiNK-6roFBU-8YqrKr-72WwLe-aBJdHB-oPGYyS-9uz2cU-azaqFx-8v5BgU-F733W-9mWPDq-
aEQLs3-bnkyhm-dkca89-fKYvGK-gNxZxv-eb4cvE-qpZWpC-kKUvmL-8YtH3G-98qbQb-98n1t6-8Yvxm4-jXtqZV-a6DAUD-qtjq5x-a6DzUB-bcwmu2-
a1U4pc-kmAa9p-8YKAJJ-cdhsNf-jXvncG-bYwEeA
Our Agenda US Air Force https://www.flickr.com/photos/usairforce/33048702564/
Why Optimize for
Performance?
TaxCredits.net https://www.flickr.com/photos/76657755@N04/7214596024/in/photostream/
Working Set and
STORAGE
Martin Cathrae https://www.flickr.com/photos/suckamc/4276413497/in/photolist-7vTJKc-2Q32fB-wa2vQ-4A14HM-6DNKtq-fDhvum-fDcfk1-Pnk6Y-uVie7-ezdr-eEUWr5-
4C5faJ-4LiN8U-bo1uK4-4JGZ8X-bo1vmR-4JGZbi-bo1xKz-6cQtxv-7vGrC5-4Dty3o-27N7af-9VHzM1-4bfE5W-4jvZQU-adZXgC-27HGaz-4WDC-3X9P4z-
4a7S-b3pwp6-7kjYPw-98Z2H9-5whRwW-6VwcaJ-9hsRzq-eEUQnC-yFwUH-5yZkKy-6VjSrS-34KHNL-4xrRHJ-4pJ4cU-4KhU8o-4Erhbj-5nUpvY-cyN4N-
6Mp5t-33k6jr-3jMQeW
Working Set and
STORAGE
Joseph North https://www.flickr.com/photos/joephotos/246500309/in/photolist-nMnXk-QK8a24-Ucrcyv-5MGN3p-Ugw4SA-5Hnu69-4HNbFN-65dFqW-dmVz2q-vduCE-
diPfC5-5nsg3x-bxTbDc-5XQ6M4-2Vs7Sm-nNDvR2-dmTKK9-dmTDtP-nkC4Wp-2ZQUSn-dmVegw-dmTLUh-qwukjH-dmTBr9-4F6mnZ-HkYizf-5Hnu3y-
r38RvU-dN8UGD-dmVbMq-bC8emS-ozCZQ9-dmU2qt-9gnYJE-fg82tJ-dmTGUm-oBEJun-898YWc-7eznXU-ajvEgB-dmTB2C-fSzRMo-fwdyPM-c9BtzG-
9moxrU-baqnv6-6hwkGt-7z2hSs-oB2sYM-biC91r
RAM is Six Orders of
Magnitude FASTER
reynermedia https://www.flickr.com/photos/89228431@N06/11080408955/
RAM is Six Orders of
Magnitude FASTER
Eirien https://www.flickr.com/photos/ilweranta/8996280560/in/photolist-eGYhMw-TKqLua-24iYC6-ehCvbY-gmbhFP-6aqfiw-ehwsmM-7Jy3RX-8TCGJU-ehwJiT-
oRL71m-ehwHVv-c7Zvk1-ehwwuk-5sVWLm-oziZ4t-cnyWB-2iVQHc-8fJQ5r-N5A6F-cnyS6-67dSQX-ehwvLV-ehCgJQ-258XF-agyDyv-ehCdhw-ehCxaq-
ehChu3-ehwMea-ehCyPw-6mp62D-mhhPJ-JqN6W-aDKcaM-ehwwSg-cnyLA-cZeRxY-ehCsTu-ehCe5W-82K8xC-2j1bJ7-6aqfhw-2iW2Gv-gJXaz-
82hF4R-2iVTSx-88iceA-99kMKq-6xDNqk
60. ATTRIBUTIONS B
Workload for Scenario One Scott Lewis https://www.flickr.com/photos/99781513@N04/14491021598/in/photolist-9M2QU4-83BuMZ-Qu2Fhh-o5wgRY
Slow Query Investigation brittgow https://www.flickr.com/photos/brittgow/4782569782/in/photolist-8hBVnG-8hyCFt
What Do We Know About the
“name” Query?
Michael Gil https://www.flickr.com/photos/msvg/5143096005/in/photolist-8QtHg4-o5efHJ-8p3E5-fcRoPN-os3UFT-V5oFwZ-ecGiaW-bBrEAg-4UkSjm-
4UJPWC-BsTqmE-7whZWk-7yAvdf-fh7UJg-71kBVK-pSSwtL-cKsFNE-V1UR8h-mKgVDa-xHhGv-nKDHT-4UkSjf-4fQmCt-V5otwg-
UPYAo1-arQCo9-hSgPq-iDiP8b-TQPTer-V5oFFr-ro5X3-nKDKa-BywLa-a7Fixt-fvRxn-bhcLyH-oDYyin-6GSZZ8-4UkSj9-d3vbWs-8WW33y-
N39Y6-V5oFgP-8rrhrk-fPhtZQ-55N7WV-4UkUgY-PrPdag-9L4Bb8-4Xy1RX
How to Improve the “name”
Query?
Max Pixel http://maxpixel.freegreatpicture.com/Aim-Focus-Dartboard-S-Target-Goal-Aiming-Arrow-1551504
Results of the Index
Experiment
DVIDSHUB https://www.flickr.com/photos/dvids/2897817247/in/photolist-5q557k-Hu4yVv-Jq9Sn2-Hu4yyZ-Hu4xvM-Hu4zyp-MFSyFN-Jgc8FA-nDrrYm-
nT2SbE-NEDRdG-UWje3H-UWje9K-UWjehv-UFYn1N-HRh4R3-Lg7rif-JQQXhP-HUwfod-6zwr78-M9Q5xX-9vbTZg-NyChcC-BCrk3e-
Mn5KXD-MxZfFL-MxZeQY-D5gg4j-aFV75p-Hmtvuy-HvPV9o-Jkh8MY-JgBmPv-HXJM9L-wx2WoT-wsbZar-wr3axA-wfESkZ-wxCTKX-
wf5b6i-vzGP3G-wr5Fd3-wfwLrS-wwv5Tm-vuKsb5-wa4FDE-vuKx4S-wprYTN-vuUCxi-wpvpyJ
When To Index? Free Icon Shop https://freeiconshop.com/icon/search-document-icon-outline/
When To Index? Free Icon Shop https://freeiconshop.com/icon/edit-document-icon-outline/
Investigation U.S. Naval Forces
Central
Command/U.S. Fifth
Fleet
https://www.flickr.com/photos/navcent/9303824295/in/photolist-fb9wPH-r8Rrwy-nZG4bk-dXEM3i-qTrTTY-huCynx-qqJajb-r8MDJX-nJnAnf-
dMjwii-9yN2BH-qMwUo3-huDkBQ-dXLnaA-qpkW26-aFvbLx-qDbxej-qH7Zkn-aFvbTz-oWapfU-gxqnJn-iidD7g-b8yvbc-e3gh3s-qYptk6-
oy9Cpp-pLvA9t-ogDCUw-iidnX9-qDjc1z-pXxQi6-qpkX4g-dk6r4o-iidFbM-PaqAVW-aVvrzK-dmGZ1c-ppn5x3-r4ZrhC-qYizBh-5DBPAT-
grhrKt-o7Z1dG-eKkNUq-dknpJz-cpPr5o-atdftB-DKE1zx-Jebhuw-aY7NTn
61. ATTRIBUTIONS C
What Do We Know
About the “category_code“
query?
Brent Moore https://www.flickr.com/photos/brent_nashville/240072751/
How to Improve the
“category_code” Query?
stephendann https://www.instagram.com/p/WT-FPNNwb9/
Network Optimization Signals Catherine https://www.flickr.com/photos/spiritinme/4246498337/in/photolist-7tfq1z-NLGy-bofSDY-bBaMjT-bBaLEX-bofSXN-ofAuJ7-bBaLJt-bohsvw-
bofSJ9-nY7VUu-bofUkj-bofSHf-bBaM1Z-bofTky-bBaL6B-bohsG3-npu3mt-bBaMxR-bofSSf-bBcmkp-bBaLVZ-bofTj1-bBaLGt-bBaL84-
bBaLk6-bofT5C-bBcm6K-nFK9b9-bofSWo-bBcnbT-bBaKZx-bBaM8H-bofT45-bofTHd-nY7Rj7-bBaLYr-nFWUDS-6ojcoc-nWbXnN-
GZbdUh-nptzuJ-nDVApS-o11Tax-nh4ohL-nFFouF-7WKAdx-nY8EtE-nyyPDN-nptFW1
Network Optimization Signals versageek https://www.flickr.com/photos/versageek/6098761546/in/photolist-ahVKk9-8KN9iA-ayDpNj-5rwri-9admbX-ayDq5A-Ge6GhA-7EX141-
ezV6kP-a7e3zZ-6mjKiN-6wAGCq-5vh9cc-au9WUa-nL5g91-Gv3jvG-5p9iyT-a6CAHV-74J24d-eaUVZC-dBZ9Ye-gfJCHs-FHTJoR-
SVvPRY-mB3zD2-5EJhju-boCARN-mkrcYv-2x4pH-AA9sx-24sRPz-bpXyrQ-8T7QME-5UJnXP-4VhE9-6qagst-4vM6at-TXWRb4-9ZKD9Y-
bpUKCa-45DQvn-Ge6H5N-FHTJrg-62bKi6-dizVRY-55AyXL-4Eq6Lo-5dnxaG-4rtEyk-SFSVU3
Workload for Scenario Three goatling https://www.flickr.com/photos/mmmchoco/4654658647/
Investigating Frequent
Queries
Travis Goodspeed https://www.flickr.com/photos/travisgoodspeed/2768232925/
Covered Query Strategy Office of Naval
Research
https://www.flickr.com/photos/usnavyresearch/34063946264/in/photostream/
Signals For Covered Queries Joe Goldberg https://www.flickr.com/photos/goldberg/179511846/in/photostream/
Signals For Covered Queries zoetnet https://www.flickr.com/photos/zoetnet/7929093836/
Notas del editor
Welcome to my talk on using Compass to diagnose performance problems in MongoDB. Thank you for coming to MongoDB World this year and thank you for coming to my talk. I'm excited to have the opportunity to discuss Compass and performance with you today.
Georgia Tech and University of Stuttgart grad. Long-time software developer and tinkerer. Compass Technical Mentor.
I work for MongoDB on the Technical Services in NA.
The worldwide Technical Services team at MongoDB provides guidance, insight and support for all customer use cases and all MongoDB products, including Compass.
Attribution:
ryan harvey -- https://www.flickr.com/photos/ryanh/7509976606/in/photolist-crCAph-pUWKYx-pqePxH-ocHgxR-9jnR2B-2yYmQA-bnMxe1-e1UHry-bAfqYx-DZBrcY-6kskYE-fM1bFg-9AjS9x-aDh5nS-3wM6aD-bcwj7F-bcwiNK-6roFBU-8YqrKr-72WwLe-aBJdHB-oPGYyS-9uz2cU-azaqFx-8v5BgU-F733W-9mWPDq-aEQLs3-bnkyhm-dkca89-fKYvGK-gNxZxv-eb4cvE-qpZWpC-kKUvmL-8YtH3G-98qbQb-98n1t6-8Yvxm4-jXtqZV-a6DAUD-qtjq5x-a6DzUB-bcwmu2-a1U4pc-kmAa9p-8YKAJJ-cdhsNf-jXvncG-bYwEeA
Motivation
Performance Concepts
Three Scenarios, Slow Query, Inefficient Query, Frequent Query
Examine Workloads
Search for Problem Queries
Explain Query Plans
Analyze Collection Schema with Compass
Create Targeted Index with Compass
Attribution:
US Air Force -- https://www.flickr.com/photos/usairforce/33048702564/
To start, I would like to get at the WHY behind this talk.
Why optimize for performance and why use the Compass tool?
Question:
Can I see a show of hands, who thinks that performance is important to their business?
A faster database makes your application faster. A faster application makes your customers and users happier and more engaged, which leads to good things like more revenue.
Many organizations, including the ones shown here, have found a strong positive correlation between speed, engagement and revenue. You can read more at the wpostats website.
Google DoubleClick found 2x more revenue for faster mobile sites.
https://www.doubleclickbygoogle.com/articles/mobile-speed-matters/
Pinterest improved signups and traffic 15% each with 40% wait improvement.
https://medium.com/@Pinterest_Engineering/driving-user-growth-with-performance-improvements-cfc50dafadd7
Walmart found 2% better conversion for 1s performance improvement.
https://www.slideshare.net/devonauerswald/walmart-pagespeedslide
Attribution:
TaxCredits.net -- https://www.flickr.com/photos/76657755@N04/7214596024/in/photostream/
The GUI for MongoDB. Visually explore your data. Run ad hoc queries in seconds. Interact with your data with full CRUD functionality. View and optimize your query performance. Compass empowers you to make smarter decisions about indexing, document validation, and more.
On the surface, Compass is a Desktop Database Client that provides a
Graphical Admin Panel
Schema Explorer
Document Explorer
CRUD operations
and
Real-time Server Stats
However, at a deeper level, the Compass design is based on years of experience working with customers to improve their database deployments. The lead engineer for the Compass team, Thomas Rueckstiess, is a former technical services engineer. This tool allows you to rapidly visualize and interpret MongoD performance statistics and explain plan results which might take hours with at text only tool.
Which concepts are important for MongoDB performance?
Fundamental performance related concepts that apply to databases…
Working Set can be thought of as the list of things you need to get work done during the day.
Your office is similar to RAM and the self-storage unit is like Disk.
How fast and efficient would you be if most things you need are in self-storage?
For a database program, the working set includes the documents, indexes and pages needed to satisfy the most common queries.
Access Time (latency)
Transfer Rate
Attribution:
Martin Cathrae -- https://www.flickr.com/photos/suckamc/4276413497/in/photolist-7vTJKc-2Q32fB-wa2vQ-4A14HM-6DNKtq-fDhvum-fDcfk1-Pnk6Y-uVie7-ezdr-eEUWr5-4C5faJ-4LiN8U-bo1uK4-4JGZ8X-bo1vmR-4JGZbi-bo1xKz-6cQtxv-7vGrC5-4Dty3o-27N7af-9VHzM1-4bfE5W-4jvZQU-adZXgC-27HGaz-4WDC-3X9P4z-4a7S-b3pwp6-7kjYPw-98Z2H9-5whRwW-6VwcaJ-9hsRzq-eEUQnC-yFwUH-5yZkKy-6VjSrS-34KHNL-4xrRHJ-4pJ4cU-4KhU8o-4Erhbj-5nUpvY-cyN4N-6Mp5t-33k6jr-3jMQeW
Joseph North -- https://www.flickr.com/photos/joephotos/246500309/in/photolist-nMnXk-QK8a24-Ucrcyv-5MGN3p-Ugw4SA-5Hnu69-4HNbFN-65dFqW-dmVz2q-vduCE-diPfC5-5nsg3x-bxTbDc-5XQ6M4-2Vs7Sm-nNDvR2-dmTKK9-dmTDtP-nkC4Wp-2ZQUSn-dmVegw-dmTLUh-qwukjH-dmTBr9-4F6mnZ-HkYizf-5Hnu3y-r38RvU-dN8UGD-dmVbMq-bC8emS-ozCZQ9-dmU2qt-9gnYJE-fg82tJ-dmTGUm-oBEJun-898YWc-7eznXU-ajvEgB-dmTB2C-fSzRMo-fwdyPM-c9BtzG-9moxrU-baqnv6-6hwkGt-7z2hSs-oB2sYM-biC91r
What’s good, what’s bad?
Consider two slides.
Discuss/mention all three: access, time cost and transfer rate. Compare on graph.
Attribution:
reynermedia -- https://www.flickr.com/photos/89228431@N06/11080408955/
Eirien -- https://www.flickr.com/photos/ilweranta/8996280560/in/photolist-eGYhMw-TKqLua-24iYC6-ehCvbY-gmbhFP-6aqfiw-ehwsmM-7Jy3RX-8TCGJU-ehwJiT-oRL71m-ehwHVv-c7Zvk1-ehwwuk-5sVWLm-oziZ4t-cnyWB-2iVQHc-8fJQ5r-N5A6F-cnyS6-67dSQX-ehwvLV-ehCgJQ-258XF-agyDyv-ehCdhw-ehCxaq-ehChu3-ehwMea-ehCyPw-6mp62D-mhhPJ-JqN6W-aDKcaM-ehwwSg-cnyLA-cZeRxY-ehCsTu-ehCe5W-82K8xC-2j1bJ7-6aqfhw-2iW2Gv-gJXaz-82hF4R-2iVTSx-88iceA-99kMKq-6xDNqk
Two additional performance concepts are query shape and index.
The Query Shape is a generalization of the structure of a query. Specifically, it is compose of a combination of query predicate, sort, and projection specifications. It represents the outline or shape of different specific queries that can be fulfilled with the same query plan, index and execution path. The MongoDB query planner uses the query shape as a type of query category when evaluating competing query plans as well as or caching the winning query plan and matching that plan against subsequent queries.
Index – Conceptually a database index is similar to the index in the back of a book. It allows for a fast search based on a keyword with a reference to the full page or record. In a database, an index is a data structure similar to the red path through the maze. The index allows fast access to the goal without searching every possibility. Indexes provide:
Fast Read (Search) via B-Tree
Write cost due to B-Tree maintenance
Can prevent replacement (pollution) of working set
Without Index
Working Set Eviction from Cache
Linear Collection Scan
Faster Updates
With Index
Only Index Path in Cache
O(log(N)) Search
Updates Maintain Indexes
Query Shape - A combination of query predicate, sort, and projection specifications. For the query predicate, only the structure of the predicate, including the field names, are significant; the values in the query predicate are insignificant. We can investigate performance characteristics of a query shape using the explain plan feature of Compass.
Working Set – Records, data structures and code needed by the database to make progress over some time interval. Database performance is strongly correlated with the percentage of working set that is available in RAM. Due to caching behavior, any documents or pages loaded from disk into RAM replace previous pages. If different query shapes in your workload have large and heterogeneous working sets, they may cause contention for RAM.
Index – A data structure maintained by the MongoD server that significantly improves query performance at the cost of additional overhead for update operations. An index can drastically reduce the working set for limited number of query shapes.
Query Shape - A combination of query predicate, sort, and projection specifications. For the query predicate, only the structure of the predicate, including the field names, are significant; the values in the query predicate are insignificant. We can investigate performance characteristics of a query shape using the explain plan feature of Compass.
Working Set – Records, data structures and code needed by the database to make progress over some time interval. Database performance is strongly correlated with the percentage of working set that is available in RAM. Due to caching behavior, any documents or pages loaded from disk into RAM replace previous pages. If different query shapes in your workload have large and heterogeneous working sets, they may cause contention for RAM.
Index – A data structure maintained by the MongoD server that significantly improves query performance at the cost of additional overhead for update operations. An index can drastically reduce the working set for limited number of query shapes.
Query Shape - A combination of query predicate, sort, and projection specifications. For the query predicate, only the structure of the predicate, including the field names, are significant; the values in the query predicate are insignificant. We can investigate performance characteristics of a query shape using the explain plan feature of Compass.
Working Set – Records, data structures and code needed by the database to make progress over some time interval. Database performance is strongly correlated with the percentage of working set that is available in RAM. Due to caching behavior, any documents or pages loaded from disk into RAM replace previous pages. If different query shapes in your workload have large and heterogeneous working sets, they may cause contention for RAM.
Index – A data structure maintained by the MongoD server that significantly improves query performance at the cost of additional overhead for update operations. An index can drastically reduce the working set for limited number of query shapes.
In our first scenario, we are going to use Compass to examine a workload with a slow query.
Q: Has anyone in the audience ever encountered a slow query in their application?
The application in scenario one is an e-commerce web site with product data stored in MongoDB. A collection of attributes describes product characteristics and the application searches for attributes by name.
Attribution:
Scott Lewis -- https://www.flickr.com/photos/99781513@N04/14491021598/in/photolist-9M2QU4-83BuMZ-Qu2Fhh-o5wgRY
An experiment requires measurement. Measure the workload both before and after any changes in order to draw a valid conclusion.
Find a Slow Query
Real Time Server Statistics
Hottest Collections
Slowest Queries
Investigate the Slow Query
Explain Plan Tab
How do we do this in Compass?
Documentation
https://docs.mongodb.com/manual/administration/analyzing-mongodb-performance/
https://docs.mongodb.com/manual/tutorial/optimize-query-performance-with-indexes-and-projections/
Attribution:
brittgow -- https://www.flickr.com/photos/brittgow/4782569782/in/photolist-8hBVnG-8hyCFt
Use real time server statistics to look for hot collections and slow queries. Find the query shape and use the explain plan tab to look for bottlenecks.
Ask the audience. Who has seen a slow query scan the entire collection?
“webshop.attributes”, uses the most resources.
Queries for “webshop.attributes.name” are scanning the entire collection.
Only approximately 40 thousand of one million documents returned.
Relatively few updates occur on “webshop.attributes”.
Attribution:
Michael Gil -- https://www.flickr.com/photos/msvg/5143096005/in/photolist-8QtHg4-o5efHJ-8p3E5-fcRoPN-os3UFT-V5oFwZ-ecGiaW-bBrEAg-4UkSjm-4UJPWC-BsTqmE-7whZWk-7yAvdf-fh7UJg-71kBVK-pSSwtL-cKsFNE-V1UR8h-mKgVDa-xHhGv-nKDHT-4UkSjf-4fQmCt-V5otwg-UPYAo1-arQCo9-hSgPq-iDiP8b-TQPTer-V5oFFr-ro5X3-nKDKa-BywLa-a7Fixt-fvRxn-bhcLyH-oDYyin-6GSZZ8-4UkSj9-d3vbWs-8WW33y-N39Y6-V5oFgP-8rrhrk-fPhtZQ-55N7WV-4UkUgY-PrPdag-9L4Bb8-4Xy1RX
Run an experiment!
Carefully measure the workload before the experiment.
Create a Targeted Index with Compass
Measure
Compare Results
Real Time Server Stats
Change in Duration
Operations per Second
Explain Plan Tab
Documents Examined vs. Returned
Index Used
Decide Whether to Keep the Index. If Not Improved, Drop Index.
Attribution:
Max Pixel -- http://maxpixel.freegreatpicture.com/Aim-Focus-Dartboard-S-Target-Goal-Aiming-Arrow-1551504
Create an index that improves query performance of the slow query identified earlier.
Creating an index is a very resource intensive process. Always test index creation in a test environment before production.
Wait for index creation to complete.
Verify that query performance improved.
With index, query on “name” only scans documents returned.
Slowest query execution time improved from ~3500ms to ~500ms.
Operations per second improved from ~5 to ~15.
If the improvement holds for all workloads, keep the index...
Attribution:
DVIDSHUB -- https://www.flickr.com/photos/dvids/2897817247/in/photolist-5q557k-Hu4yVv-Jq9Sn2-Hu4yyZ-Hu4xvM-Hu4zyp-MFSyFN-Jgc8FA-nDrrYm-nT2SbE-NEDRdG-UWje3H-UWje9K-UWjehv-UFYn1N-HRh4R3-Lg7rif-JQQXhP-HUwfod-6zwr78-M9Q5xX-9vbTZg-NyChcC-BCrk3e-Mn5KXD-MxZfFL-MxZeQY-D5gg4j-aFV75p-Hmtvuy-HvPV9o-Jkh8MY-JgBmPv-HXJM9L-wx2WoT-wsbZar-wr3axA-wfESkZ-wxCTKX-wf5b6i-vzGP3G-wr5Fd3-wfwLrS-wwv5Tm-vuKsb5-wa4FDE-vuKx4S-wprYTN-vuUCxi-wpvpyJ
You should add an Index when careful experiments in a test environment under realistic workload indicate that it will improve performance.
Attribution:
https://freeiconshop.com/icon/search-document-icon-outline/
Indications
Read Activity Outweighs Write Activity
Dataset Does Not Fit in RAM
Frequent or Time Critical Query
Big Data, Business Intelligence and Analytics
Contraindications
Write Activity Outweighs Read Activity
Infrequent or Unimportant Query
Logging, Data Collection
Attribution:
https://freeiconshop.com/icon/search-document-icon-outline/
https://freeiconshop.com/icon/edit-document-icon-outline/
Use Compass to Recognize and Fix Queries Using Excessive Bandwidth
Scenario two involves a database of high-tech startup companies.
The application is doing a search for companies in a particular category. A faceted search will sometimes generate this type of query.
The investigative process is similar to the first scenario. However, this time we will need to be more observant and pay closer attention to the details provided by Compass.
Attribution:
U.S. Naval Forces Central Command/U.S. Fifth Fleet -- https://www.flickr.com/photos/navcent/9303824295/in/photolist-fb9wPH-r8Rrwy-nZG4bk-dXEM3i-qTrTTY-huCynx-qqJajb-r8MDJX-nJnAnf-dMjwii-9yN2BH-qMwUo3-huDkBQ-dXLnaA-qpkW26-aFvbLx-qDbxej-qH7Zkn-aFvbTz-oWapfU-gxqnJn-iidD7g-b8yvbc-e3gh3s-qYptk6-oy9Cpp-pLvA9t-ogDCUw-iidnX9-qDjc1z-pXxQi6-qpkX4g-dk6r4o-iidFbM-PaqAVW-aVvrzK-dmGZ1c-ppn5x3-r4ZrhC-qYizBh-5DBPAT-grhrKt-o7Z1dG-eKkNUq-dknpJz-cpPr5o-atdftB-DKE1zx-Jebhuw-aY7NTn
In this workload, there are fewer slowest queries. Clicking a slow query gives a query shape of { "category_code": "___" }. However, there is already an index on "category_code".
What else can we observe about the workload?
Approx. 30 operations/sec using approx. 100,000KB/s or 100MB/s.
We can switch over to the tech.companies collection and select the Schema tab to see the fields present in the sampled documents. There are many fields and some are sub-documents or arrays. The average document size is 3.8KB. If a large number of companies matches a category, then each query may return a lot of data.
The query is using an excessive amount of network bandwidth. It appears this is leading to poor performance.
What can we do to improve the performance of this query?
Attribution:
Brent Moore -- https://www.flickr.com/photos/brent_nashville/240072751/
The default query in MongoDB returns the entire document for all documents matching the filter (predicate) fields.
If the application only requires a subset of these fields, we can update the database client to instruct the server to only return the required fields. If the documents in the result set are large and numerous, then this projection of the required fields can significantly reduce network demands and improve performance.
The next video shows the workload after changing the client application to add this project.
Run an experiment!
Use a Projection to Only Return Required Fields.
Original query: db.companies.find( { "category_code" : "cleantech" } )
Updated query: db.companies.find( { "category_code" : "cleantech" }, { "name" : 1, "email_address" : 1, "category_code" : 1 } )
Compare Results
Real Time Server Stats
Network Utilization
Operations per Second
Decide Whether to Keep the Projection.
Attribution:
Stephendann -- https://www.instagram.com/p/WT-FPNNwb9/
The update for this experiment is implemented in the database application. There is no change to the MondoDB server.
This video clip shows the workload after the projection has been added to the query.
With projection, about 10% of the network utilization (14KB/s vs 140KB/s).
With projection, about 4 times the operations per second (160 vs 40).
If the application does not need the fields and there is measurable improvement, keep the projection.
The Voyager aircraft shown here was the first to fly non-stop around the world in 1986. It was piloted by Dick Rutan and Jeana Yeager and flew for 9 days, 3 minutes and 44 seconds. We can assume this vehicle was finely tuned to be as efficient as possible.
Voyager Round-the-World
On Dec. 23, 1986, the experimental airplane Voyager, piloted by Dick Rutan and Jeana Yeager, completed the first nonstop, around-the-world flight without refueling. It landed safely at Edwards Air Force Base in California.
In this screenshot, the benchmarking script is updated to run the workload first without the projection, then pause and run the workload with the projection.
This side-by-side comparison shows the performance improvement that is sometimes possible using a projection to only return the document fields needed by the application.
There are actually two different network optimization strategies available. The first strategy uses projection to Only Return Required Fields as seen in the video. The relevant fields are shown as red columns in the diagram.
The second strategy limits the documents requested from the server using either a Sort+Limit or a Range Query to return only Relevant Documents. This might be useful when displaying the top-ten items. Make the selection of the top-ten on the server so that other matching documents are not sent back across the network. The relevant documents are shown as blue rows in the diagram on this slide.
The best indication to use a network optimization strategy is the collection of improved performance results from a relevant, controlled test.
This suitcase represents your network. If the network load is a bottleneck, then adjust the application to request less data or provision a bigger network.
Indications
High Network Usage
Result Sets with Many Documents
Result Sets with Large, Complex Documents
Contraindications
Application Requires All Fields and All Documents
Attribution:
Catherine -- https://www.flickr.com/photos/spiritinme/4246498337/in/photolist-7tfq1z-NLGy-bofSDY-bBaMjT-bBaLEX-bofSXN-ofAuJ7-bBaLJt-bohsvw-bofSJ9-nY7VUu-bofUkj-bofSHf-bBaM1Z-bofTky-bBaL6B-bohsG3-npu3mt-bBaMxR-bofSSf-bBcmkp-bBaLVZ-bofTj1-bBaLGt-bBaL84-bBaLk6-bofT5C-bBcm6K-nFK9b9-bofSWo-bBcnbT-bBaKZx-bBaM8H-bofT45-bofTHd-nY7Rj7-bBaLYr-nFWUDS-6ojcoc-nWbXnN-GZbdUh-nptzuJ-nDVApS-o11Tax-nh4ohL-nFFouF-7WKAdx-nY8EtE-nyyPDN-nptFW1
This suitcase represents your network. If the network load is a bottleneck, then adjust the application to request less data or provision a bigger network.
If your use case requires all documents and all fields.
Indications
High Network Usage
Result Sets with Many Documents
Result Sets with Large, Complex Documents
Contraindications
Application Requires All Fields and All Documents
Attribution:
Catherine -- https://www.flickr.com/photos/spiritinme/4246498337/in/photolist-7tfq1z-NLGy-bofSDY-bBaMjT-bBaLEX-bofSXN-ofAuJ7-bBaLJt-bohsvw-bofSJ9-nY7VUu-bofUkj-bofSHf-bBaM1Z-bofTky-bBaL6B-bohsG3-npu3mt-bBaMxR-bofSSf-bBcmkp-bBaLVZ-bofTj1-bBaLGt-bBaL84-bBaLk6-bofT5C-bBcm6K-nFK9b9-bofSWo-bBcnbT-bBaKZx-bBaM8H-bofT45-bofTHd-nY7Rj7-bBaLYr-nFWUDS-6ojcoc-nWbXnN-GZbdUh-nptzuJ-nDVApS-o11Tax-nh4ohL-nFFouF-7WKAdx-nY8EtE-nyyPDN-nptFW1
versageek-- https://www.flickr.com/photos/versageek/6098761546/in/photolist-ahVKk9-8KN9iA-ayDpNj-5rwri-9admbX-ayDq5A-Ge6GhA-7EX141-ezV6kP-a7e3zZ-6mjKiN-6wAGCq-5vh9cc-au9WUa-nL5g91-Gv3jvG-5p9iyT-a6CAHV-74J24d-eaUVZC-dBZ9Ye-gfJCHs-FHTJoR-SVvPRY-mB3zD2-5EJhju-boCARN-mkrcYv-2x4pH-AA9sx-24sRPz-bpXyrQ-8T7QME-5UJnXP-4VhE9-6qagst-4vM6at-TXWRb4-9ZKD9Y-bpUKCa-45DQvn-Ge6H5N-FHTJrg-62bKi6-dizVRY-55AyXL-4Eq6Lo-5dnxaG-4rtEyk-SFSVU3
In our third and final scenario, we will use Compass to find, analyze and accelerate a high-frequency Query.
Question:
Are there any Chicago Fire Soccer fans in the house?
The workload for scenario three involves a fanclub database. The application is performing an affinity search to find fans with the same age and membership status.
Attribution:
goatling -- https://www.flickr.com/photos/mmmchoco/4654658647/
Our investigation in scenario three will be similar to the first two, however, we are going to dig deeper into the Compass tool.
Specifically we will combine the server statistics with the full query shape, including the projection from the application. We will consider how the query is using an existing index and how it may be accelerated even further.
Attribution:
Travis Goodspeed -- https://www.flickr.com/photos/travisgoodspeed/2768232925/
After connecting Compass to the database, we can click on the performance tab to view the real time server statistics. There we see that mongodb.fanclub is the collection using the most resources.
Reviewing one of the slow queries, we can see the query shape includes a search or filter against age and membership status. There is an index scan in use, but the index only covers the age field of the documents.
We can copy the filter portion of the query shape to use in the next video.
Going to the explain plan tab in the mongodb.fanclub collection, we can paste in the query filter predicate from real time server stats.
Next, we will use the new options available in Compass 1.7 to also add the query projection from the client application. Click apply to ask the server how it will execute the query. This time, the server will include the projection part of the query in the results.
The explain plan results show almost 24 thousand documents examined and about 16 thousand returned to the client application. Execution time was about 2.5 seconds.
The index scan on age executed fairly quickly. Most of the time was consumed in the fetch stage that loaded and scanned 24 thousand documents. Since the membership status field is not covered by the available index, the query scans more documents that it returns to the client.
The full query is shown at the top of this slide.
An index is being used, but it does not include both fields in the filter predicate. So, adding the membership status field to the index is one option.
There is already a projection in use to limit the result size.
A covered query uses an index with the entire query shape included in the index. To achieve this goal, all filter, sort and projection fields will be included in the covering index. This allows the MongoDB server to fulfill the request without fetching any documents from the underlying collection. By responding directly from the index content, performance can be enhanced significantly.
For this high frequency query, we will need to add four fields to the index: “age”, “membership_status”, “name”, and “email”.
For optimal performance, we need to verify that the new Index will fit comfortably in RAM on the database server.
Covering indexes can be very large, so this is an important check.
There are a number of rules governing when an index may be used for a specific query. For this reason, we will use explain plan to verify that the query planner is taking advantage of the new index.
As with any careful experiment, we must compare the behavior and results before and after our changes in order to reach a solid conclusion. Ideally, we will analyze database performance under all workloads for the application.
Create a new index covering the age, membership status, name and email fields from the query shape.
Do not create indexes directly on databases with production workloads. Creating indexes is very resource intensive and sometimes leads to temporary, but significant, performance degredation. You should use a tool or follow a manual process to add indexes to production servers in a rolling fashion without production workload present.
The new index is displayed with zero usage and a size of 33.9 MB. Having build the database server, I know that this index will fit easily in RAM on the server.
Returning to the explain plan tab, we past the filter and projection documents in and ask the server to re-evaluate the query with the new index present.
From the results, we can see zero documents examined. This means the database server did not need to load any documents from the collection itself. This is possible because all information needed to answer the query is included in the index structure.
We see that about 16 thousand index keys were examined and the same number of documents returned to the client application. Additionally, the execution time was reduced to only 81ms and the results show that the query was covered by an index.
The database server is able to answer the query by examing zero documents from the full colleciton because the entire contents of the response is available in the covering index.
With the new index in place, return to the real time server stats and we see increased operations per second as well as high network usage.
The fanclub collection is still using the most resources, however, there are fewer slowest operations reported and they are completing faster.
If we click on a slow operation, we can see the same query shape. However, the new index is now reported as in use.
On the left, we have a screenshot of the server stats with only the one "age" field indexed. On the right, the same stats with all four fields indexed to cover the query completely.
The maximum operations per second reached 117 for the covered query, but only 12 for the single field. Active read and writes decreased because they are now completing so much faster. Network bandwidth used increase due to answering more queries per second.
The covered query strategy accelerates a small, critical or frequent query by including the entire query shape in the structure of a custom index.
Electromagnetic railgun belonging to the Office of Naval Research.
Attribution:
Office of Naval Research -- https://www.flickr.com/photos/usnavyresearch/34063946264/in/photostream/
As with the other strategies, the best signal to use the strategy is positive results from a controlled test against relevant workload.
In general, covered queries work best for a small, critical or frequent query that has a relatively small result set. In most cases, it makes the most sense to first test the use of a projection to improve performance before considering a covered query.
Attribution:
Joe Goldberg -- https://www.flickr.com/photos/goldberg/179511846/in/photostream/
If the new index size no longer fits in RAM, the covered query strategy will not deliver optimal performance. Additionally, adding another index may have unacceptable performance costs for write operations. The tradeoff between query speed improvement and update speed impacts must be evaluated on a case-by-case basis.
Possibility of too many indexes…
It is possible for too many indexes to reduce performance, particularly if they do not all fit in RAM. Maybe the workload on a collection has changed and new indexes were added but the indexes supporting the old workload were not removed. In that case, you should review the index usage information on the indexes tab in Compass to determine whether those early indexes are still in use. If they are not, run an experiment where the early indexes are removed and find out whether that improves performance in your environment.
Attribution:
Joe Goldberg -- https://www.flickr.com/photos/goldberg/179511846/in/photostream/
zoetnet -- https://www.flickr.com/photos/zoetnet/7929093836/
What did we learn?
The "The Fastest Code is the Code that Does Not Run" applies to the inefficient query in scenario two.
As useful as these heuristics may be, you will be well served to conduct precise tests that include measuring the before and after scenarios for database updates.
Examine your workload to find hints as to which Indexes are important. This examination may also uncover requests for data that goes unused by the application.
The people listed here provided feedback that was instrumental in getting this presentation to a useful state. I almost certainly missed a name or two and I apologize for that in advance. Thank you so much for your help and candid advice.
BTW, MongoDB is a great place to work. As you can see from all the people who helped me prepare this presentation, we have a very collaborative environment. We are currently hiring strong engineers for many positions. See me afterward if you are interested in finding out more about working for MongoDB.