Speaker: Charlie Swanson Learn how MongoDB answers your queries from a query system engineer. If you've ever had a performance problem with a query but didn't know how to find the cause, or if you've ever needed to confirm that your shiny new index is being put to work, the explain command is an excellent place to start. MongoDB's explain system is a powerful tool for solving this type of problem, but can be intimidating and unwieldy to use. In this talk, we will discuss how the explain command works and break down its output into digestible pieces.

1. O C T O B E R 1 2 , 2 0 1 7 | B E S P O K E | S AN F R AN C I S C O
Deciphering Explain
Output
Charlie Swanson

2. # M D B l o c a l
Understand how
MongoDB answers
queries
Knowledge
Figure out what's
going on
Debugging
Learn some tricks to
optimize your queries &
aggregations
Best Practices
GOALS OF THIS TALK

3. # M D B l o c a l
S E N I O R E N G I N E E R - Q U E R Y T E A M
N Y C
Charlie Swanson

4. # M D B l o c a l
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about plan
selection
Log messages
What is explain? The profile
Other commands

5. # M D B l o c a l
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about plan
selection
Log messages
What is explain? The profile
Other commands

6. # M D B l o c a l
• Is your query using an index? Which one?
Target Questions
🤔

7. # M D B l o c a l
• Is your query using an index? Which one?
• Is your query using an index to provide the sort?
Target Questions
🤔

8. # M D B l o c a l
• Is your query using an index? Which one?
• Is your query using an index to provide the sort?
• How many of the examined documents ended up matching?
Target Questions
🤔

9. # M D B l o c a l
• Is your query using an index? Which one?
• Is your query using an index to provide the sort?
• How many of the examined documents ended up matching?
• Why did the server choose to answer the query the way it did?
Target Questions
🤔

10. # M D B l o c a l
• Is your query using an index? Which one?
• Is your query using an index to provide the sort?
• How many of the examined documents ended up matching?
• Why was your winning plan chosen?
• Can my queries go faster?
Target Questions
🤔

11. # M D B l o c a l
Example Query

12. # M D B l o c a l
How Can The Server Answer This Query?

13. # M D B l o c a l
OPTION 1: Collection scan
TWITTER.TWEETS
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
✅
✅

14. # M D B l o c a l
COLLECTION SCAN:
TWITTER.TWEETS
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
✅
✅
204587
190587
SORT
01
.
02
.

15. # M D B l o c a l
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
✅
✅
(190587,
'@Charlie')
(204587,
'@TaylorSwift')
01.
02.
SORT
PROJECTION
02
01
{nFavorites: 204587,
username: '@TaylorSwift'}
{nFavorites: 109587,
username: '@Charlie'}
COLLECTION SCAN:
TWITTER.TWEETS

16. # M D B l o c a l
(190587, '@Charlie')
(204587, '@TaylorSwift')01.
02.
SORT
PROJECTION
02
01
{nFavorites: 204587,
username: '@TaylorSwift'}
{nFavorites: 109587,
username: '@Charlie'}
COLLECTION SCAN
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
✅
✅

17. # M D B l o c a l
OPTION 1: Collection scan
SORT
PROJECT
COLLECTION SCAN

18. # M D B l o c a l
OPTION 2: INDEX SCAN TWITTER.TWEETS
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
INDEX:
{nFavorites: -1}
204587
190587
87983
83092
76032
29023
…

19. # M D B l o c a l
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
INDEX:
{nFavorites: -1}
204587
190587
87983
83092
76032
29023
…
Stop when
entry is
smaller
than
100,000
TWITTER.TWEETS

20. # M D B l o c a l
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
INDEX:
{nFavorites: -1}
204587
190587
87983
83092
76032
29023
…
SORTED!
TWITTER.TWEETS

21. # M D B l o c a l
TWITTER.TWEET
S{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}
INDEX
204587
190587
87983
83092
123
123
…
INDEX SCAN
SORT

22. # M D B l o c a l
{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}
INDEX
204587
190587
87983
83092
123
123
…
INDEX SCAN
FETCH
204587
190587
{ _id: 400000,
createdAt: ISODate(…),
username: "@TaylorSwift", … }
{ _id: 400001,
createdAt: ISODate(…),
username: "@Charlie", … }
TWITTER.TWEET
S

23. # M D B l o c a l
TWITTER.TWEET
S
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
{
…
}
INDEX
204587
190587
87983
83092
123
123
…
INDEX
SCAN
FETCH
{ _id: 400000,
createdAt: ISODate(…),
username: "@TaylorSwift" }
{ _id: 400001,
createdAt: ISODate(…),
username: "@Charlie" }
PROJECTION
{nFavorites: 204587,
username: '@TaylorSwift'}
{nFavorites: 109587,
username: '@Charlie'}
{ _id: 400000,
createdAt: ISODate(…),
username: "@TaylorSwift" }
{ _id: 400001,
createdAt: ISODate(…),
username: "@Charlie", }

24. # M D B l o c a l
OPTION 2: INDEX scan
FETCH
PROJECT
INDEX SCAN

25. # M D B l o c a l
Many Ways to Answer a Query, Which Was
It?
SORT
PROJECT
COLLECTION SCAN
FETCH
PROJECT
INDEX SCAN

26. # M D B l o c a l
1. Command to explain execution of various other commands
2. Helper on shell cursor object
What is Explain?

27. O C T O B E R 1 2 , 2 0 1 7 | B E S P O K E | S AN F R AN C I S C O
What is Explain? - Command

28. O C T O B E R 1 2 , 2 0 1 7 | B E S P O K E | S AN F R AN C I S C O
What is Explain? - Shell Helper

29. O C T O B E R 1 2 , 2 0 1 7 | B E S P O K E | S AN F R AN C I S C O
What is Explain? - Shell Helper
???

30. # M D B l o c a l
Query Plans
SORT
PROJECT
COLLECTION SCAN
FETCH
PROJECT
INDEX SCAN
…OR
FETCH
INDEX SCAN INDEX SCAN

31. # M D B l o c a l
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about plan
selection
Log messages
What is explain? The profile
Other commands

32. # M D B l o c a l
> db.collection.find(…).explain("queryPlanner")
{
"queryPlanner" : {
…
"winningPlan" : {…},
"rejectedPlans" : […]
},
…
}
Explain Output Optional, this is the
default

33. # M D B l o c a l
> db.collection.find(…).explain()
{queryPlanner: {
winningPlan: {
stage: "SORT",
inputStage: {
stage: "FETCH",
inputStage: {
stage: "IXSCAN"
}
}
}
}
}}
Explain Output
FETCH
INDEX SCAN
{nFavorites: -1}
SORT

34. # M D B l o c a l
> db.collection.find(…).explain()
{
…
stage: "IXSCAN"
keyPattern: {nFavorites: -1},
indexBounds: {
a: [ "[inf.0, 100000]" ]
},
… // Other index scan
// specific stats.
…
}}
Explain Output
FETCH
INDEX SCAN
keyPattern: {
nFavorites: -1
}
indexBounds: […]
…
SORT

35. # M D B l o c a l
> db.collection.find(…).explain()
{
"queryPlanner" : {
…
"winningPlan" : {
// Encodes selected plan.
},
"rejectedPlans" : […]
},
…
}
Explain Output
FETCH
INDEX SCAN
SORT

36. # M D B l o c a l
> db.collection.find(…).explain()
{"queryPlanner" : {
"winningPlan" : {
{"stage" : "SORT",
"inputStage" : {…}
}}
},
"rejectedPlans" : [
{"stage" : "SORT",
"inputStage" : {…}
}}
…
]
}}
Explain Output
FETCH
INDEX SCAN
SORT
COLL_SCAN
SORT

37. # M D B l o c a l
APPLYING THIS INFORMATION

38. # M D B l o c a l
Is Your Query Using the Index You Expect?

39. # M D B l o c a l
FETCH
SORT
✓ COLLECTION SCAN
SORT
✗
INDEX SCAN
keyPattern: {nFollowers: -1}
Is Your Query Using the Index You Expect?

40. # M D B l o c a l
Is Your Query Using the Index You Expect?

41. # M D B l o c a l
db.tweets.explain().find(
{nFavorites: {$gte: 100000}},
{_id: 0, nFavorites: 1, username: 1})
.sort({nFavorites: -1})
Is Your Query Using the Index You Expect?

42. # M D B l o c a l
db.tweets.explain().find(
{nFavorites: {$gte: 100000}},
{_id: 0, nFavorites: 1, username: 1}).sort({nFavorites: -1})
{ "queryPlanner": {
"winningPlan": {
"stage": "PROJECTION",
"inputStage": {
"stage": "FETCH",
"inputStage": {
"stage": "IXSCAN",
"keyPattern": {"nFavorites": -1},
"indexBounds": {
"nFavorites": ["[inf.0, 100000.0]"]
} } } },
"rejectedPlans": [ ] } }
Is Your Query Using the Index You Expect?

43. # M D B l o c a l
Is Your Query Using an Index to Provide the Sort?

44. # M D B l o c a l
FETCH
INDEX SCAN
✓ ✗FETCH
INDEX SCAN
SORT
Is Your Query Using an Index to Provide the Sort?

45. # M D B l o c a l
Is Your Query Using an Index to Provide the Sort?

46. # M D B l o c a l
db.tweets.explain().find(
{nFavorites: {$gte: 100000}},
{_id: 0, nFavorites: 1, username: 1}).sort({nFavorites: -1})
{ "queryPlanner": {
"winningPlan": {
"stage": "PROJECTION",
"inputStage": {
"stage": "FETCH",
"inputStage": {
"stage": "IXSCAN",
"keyPattern": {"nFavorites": -1},
"indexBounds": {
"nFavorites": ["[inf.0, 100000.0]"]
} } } },
"rejectedPlans": [ ] } }
NO SORT STAGE ✅
Is Your Query Using an Index to Provide the Sort?

47. # M D B l o c a l
SORT_MERGE is OK
✓SORT_MERGE
INDEX SCAN
FETCH
INDEX SCAN

48. # M D B l o c a l
BONUS: Is Your Query Using an Index to Provide the
Projection?

49. # M D B l o c a l
Compound Index
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
INDEX:
{username: 1, nFavorites: -
1}
"@Charlie"29023
"@MongoDB"87983
"@MongoDB"60587
"@MongoDB"7983
"@MongoDBEn
g"
83092
"@MongoDBEn
g"
76032
… …

50. # M D B l o c a l
db.tweets.find({
username: {$in: ["@MongoDBEng", "@MongoDB"]},
nFavorites: {$gt: 50000}}})
Compound Index
TWITTER.TWEETS
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
INDEX:
{username: 1, nFavorites: -1}
"@Charlie"29023
"@MongoDB"87983
"@MongoDB"60587
"@MongoDB"7983
"@MongoDBEng"83092
"@MongoDBEng"76032
… …

51. # M D B l o c a l
db.tweets.find(
{username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}},
{_id: 0, username: 1, nFavorites: 1})
Compound Index
TWITTER.TWEETS
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
{…} {…} {…} {…} {…} {…}
INDEX:
{username: 1, nFavorites: -1}
"@Charlie"29023
"@MongoDB"87983
"@MongoDB"60587
"@MongoDB"7983
"@MongoDBEng"83092
"@MongoDBEng"76032
… …

52. # M D B l o c a l
db.tweets.find(
{username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}},
{_id: 0, username: 1, nFavorites: 1})
INDEX SCAN
PROJECTION
{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}
INDEX:
{username: 1,
nFavorites: -1}
"@Charlie
"
29023
"@Mongo
DB" 87983
"@Mongo
DB"
60587
"@Mongo
DB"
7983
"@Mongo
DBEng" 83092
"@Mongo
DBEng" 76032
… …
Compound Index

53. # M D B l o c a l
db.tweets.find(
{username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}},
{_id: 0, username: 1, nFavorites: 1})
Compound Index
INDEX SCAN
PROJECTION
{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}
INDEX:
{username: 1,
nFavorites: -1}
"@Charlie
"
29023
"@Mongo
DB" 87983
"@Mongo
DB"
60587
"@Mongo
DB"
7983
"@Mongo
DBEng" 83092
"@Mongo
DBEng" 76032
… …

54. # M D B l o c a l
db.tweets.find(
{username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}},
{_id: 0, username: 1, nFavorites: 1, other: 1})
Compound Index
INDEX SCAN
PROJECTION
FETCH
{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}
INDEX:
{username: 1,
nFavorites: -1}
"@Charlie
"
29023
"@Mongo
DB" 87983
"@Mongo
DB"
60587
"@Mongo
DB"
7983
"@Mongo
DBEng" 83092
"@Mongo
DBEng" 76032
… …

55. # M D B l o c a l
INDEX SCAN
PROJECTION
db.tweets.find(
{username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}},
{_id: 0, username: 1, nFavorites: 1})
.sort({username: 1})
Compound Index
{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}{…
}
{…
}
{…
}
{…
}
{…
}
{…
}
INDEX:
{username: 1,
nFavorites: -1}
"@Charlie
"
29023
"@Mongo
DB" 87983
"@Mongo
DB"
60587
"@Mongo
DB"
7983
"@Mongo
DBEng" 83092
"@Mongo
DBEng" 76032
… …

56. # M D B l o c a l
PROJECTION
INDEX SCAN
✓ ✗
PROJECTION
INDEX SCAN
FETCH
Is your query using an index to provide the
projection?

57. # M D B l o c a l
• Is your query using an index? Which one?
• Is your query using an index to provide the sort?
• Is your query using an index to provide the projection?
The Power of "QueryPlanner"

58. # M D B l o c a l
db.tweets.explain().find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
Next Up: "It's using an index, so what's taking so
long?"
FETCH
INDEX SCAN
keyPattern: {createdDate:
1}

59. # M D B l o c a l
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
It's using an index, so what's taking so long?
FETCH
INDEX SCAN
keyPattern: {createdDate:
1}
12:02
12:03
12:04
…
INDEX SCAN
INDEX:
{createdAt: 1}

60. # M D B l o c a l
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
{createdDate: 12:02,
favorites: [
"@MongoDB",
"@taylorswift"
]}
FETCH
FETCH
filter: {
favorites:
"@eliothorowitz"
}
INDEX SCAN
keyPattern: {createdDate:
1}
It's using an index, so what's taking so long?
12:02
12:03
12:04
…
INDEX:
{createdAt: 1}

61. # M D B l o c a l
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
{createdDate: 12:02,
favorites: [
"@MongoDB",
"@taylorswift"
]}
FETCH
FETCH
filter: {
favorites: "@eliothorowitz"
}
INDEX SCAN
keyPattern: {createdDate: 1}
❌
It's using an index, so what's taking so long?
12:02
12:03
12:04
…
FETCH
filter: {
favorites:
"@eliothorowitz"
}
INDEX SCAN
keyPattern: {createdDate:
1}
INDEX:
{createdAt: 1}

62. # M D B l o c a l
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
{createdDate: 12:03,
favorites: [
"@eliothorowitz",
"@taylorswift"
]}
FETCH
It's using an index, so what's taking so long?
12:02
12:03
12:04
…
FETCH
filter: {
favorites:
"@eliothorowitz"
}
INDEX SCAN
keyPattern: {createdDate:
1}
INDEX:
{createdAt: 1}

63. # M D B l o c a l
• What percentage of the index keys in the scanned range ended up
matching the predicate?
• What's the selectivity?
So how many of them were thrown out?

64. # M D B l o c a l
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about plan
selection
Log messages
What is explain? The profile
Other commands

65. # M D B l o c a l
> db.collection.find(…).explain()
{"queryPlanner" : {
"winningPlan" : {
{"stage" : "SORT",
"inputStage" : {…}
}}
},
"rejectedPlans" : [
{"stage" : "SORT",
"inputStage" : {…}
}}
…
]
}}
Explain Mode: "queryPlanner"
FETCH
INDEX SCAN
(a, b)
SORT
FETCH
INDEX SCAN (a)
SORT

66. # M D B l o c a l
• "executionStats"
Explain Mode: "ExecutionStats"
FETCH
INDEX SCAN (a)
SORT
created by Mike Ashley from Noun Project
created by Creative Stall from Noun Project

67. # M D B l o c a l
> db.tweets.find(…).explain("executionStats")
{
"queryPlanner" : {
…
"winningPlan" : {…},
"rejectedPlans" : […]
},
"executionStats": { // New!
…,
"executionStages": {…}
}
…
}
Explain Mode: "executionStats"

68. # M D B l o c a l
> db.tweets.find(…).explain("executionStats")
{
"queryPlanner" : { /* Same as before. */ },
"executionStats": {
// Top-level stats.
"executionStages": {
stage: "SORT",
// Sort stats.
inputStage: {
// etc, etc.
}
}
}
…
}
Details: "ExecutionStats"
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

69. # M D B l o c a l
> db.tweets.find(…).explain("executionStats")
{
…,
"executionStats" : {
// Top-level stats.
"nReturned" : 390000,
"executionTimeMillis" : 4431,
"totalKeysExamined" : 390000,
"totalDocsExamined" : 390000,
"executionStages" : {…}
},
}
Details: "ExecutionStats"
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

70. # M D B l o c a l
db.tweets.find(…).explain("executionStats")
{
"executionStats" : {
// Top-level stats.
"executionStages" : {
"stage" : "SORT",
"nReturned" : 390000,
"executionTimeMillisEstimate" : 2030,
…
"sortPattern" : { "nFollowers" : 1 },
"memUsage" : 20280000,
"memLimit" : 33554432,
"inputStage" : {…}
}
}
}
Details: "ExecutionStats"
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

71. # M D B l o c a l
db.tweets.find(…).explain("executionStats")
{
"executionStats" : {
// Top-level stats.
"executionStages" : {
"stage" : "SORT",
"nReturned" : 390000,
"executionTimeMillisEstimate" : 2030,
…
"sortPattern" : { "nFollowers" : -1 },
"memUsage" : 20280000,
"memLimit" : 33554432,
"inputStage" : {…}
}
}
}
Details: "ExecutionStats"
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

72. # M D B l o c a l
db.tweets.find(…).explain("executionStats")
{
"executionStats" : {
// Top-level stats.
"executionStages" : {
"stage" : "SORT",
"nReturned" : 390000,
"executionTimeMillisEstimate" : 2030,
"works" : 780003,
"advanced" : 390000,
"needTime" : 390002,
"isEOF" : 1,
"sortPattern" : { "b" : 1 },
…
"inputStage" : {…}
}
}
}
?
Details: "ExecutionStats"
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

73. # M D B l o c a l
• These are all PlanStages
• SortStage
• FetchStage
• IndexScanStage
Execution Stats: works, advanced, etc.
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

74. # M D B l o c a l
• These are all PlanStages
• Each PlanStage implements work()
Execution Stats: works, advanced, etc.
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

75. # M D B l o c a l
• These are all PlanStages
• Each PlanStage implements work(),
returns one of:
• ADVANCED
• NEED_TIME
• IS_EOF
Execution Stats: works, advanced, etc.
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

76. # M D B l o c a l
EXECUTION STATS: WORKS, ADVANCED, ETC.
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}
work()

77. # M D B l o c a l
EXECUTION STATS: WORKS, ADVANCED, ETC.
work()
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}
work()

78. # M D B l o c a l
EXECUTION STATS: WORKS, ADVANCED, ETC.
work()
work()
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}
work()

79. # M D B l o c a l
EXECUTION STATS: WORKS, ADVANCED, ETC.
work()
ADVANCEDID
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}
work()

80. # M D B l o c a l
EXECUTION STATS: WORKS, ADVANCED, ETC.
ADVANCED{…}
ADVANCED
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}
work()

81. # M D B l o c a l
SORT
EXECUTION STATS: WORKS, ADVANCED, ETC.
NEED_TIME
{
…
}
ADVANCED
ADVANCED
FETCH
INDEX SCAN
keyPattern: {hashtags: 1}
work()

82. # M D B l o c a l
db.tweets.find(…).explain("executionStats")
{
"executionStats" : {
// Top-level stats.
"executionStages" : {
"stage" : "SORT",
"nReturned" : 390000,
"executionTimeMillisEstimate" : 2030,
"works" : 780003,
"advanced" : 390000,
"needTime" : 390002,
"isEOF" : 1,
"sortPattern" : { "b" : 1 },
…
"inputStage" : {…}
}
}
}
Details: "executionStats"
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

83. # M D B l o c a l
> db.tweets.find(…).explain("executionStats")
{"executionStats": {
"executionStages": {
stage: "SORT",
// Sort stats, includes "works", "advanced", …
inputStage: {
stage: "FETCH",
// Fetch stats, includes "works", "advanced", …
inputStage: {
// etc, etc.
}
}
}
}
…
}
Details: "executionStats"
FETCH
SORT
INDEX SCAN
keyPattern: {hashtags: 1}

84. # M D B l o c a l
> db.tweets.find(…).explain("executionStats")
{
"queryPlanner" : {
…
"winningPlan" : {…},
"rejectedPlans" : […]
},
"executionStats": { // New!
…,
"executionStages": {…}
}
…
}
Explain Mode: "executionStats"

85. # M D B l o c a l
Applying This Information

86. # M D B l o c a l
How selective is your index?

87. # M D B l o c a l
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
How selective is your index?
12:02
12:03
12:04
12:06
…
INDEX SCAN
FETCH
filter: {
favorites:
"@eliothorowitz"
}
INDEX SCAN
keyPattern: {createdDate:
1}

88. # M D B l o c a l
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
{
"executionStats" : {
"nReturned" : 314,
"totalKeysExamined" : 2704, // < 12% matched
…
}
How selective is your index?
FETCH
filter: {
favorites:
"@eliothorowitz"
}
INDEX SCAN
keyPattern: {createdDate:
1}

89. # M D B l o c a l
What's the most expensive part of your plan?

90. # M D B l o c a l
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
What's the most expensive part of your plan?
FETCH
filter: {
favorites:
"@eliothorowitz"
}
INDEX SCAN
keyPattern: {createdDate:
1}

91. # M D B l o c a l
db.tweets.explain("executionStats").find(…)
What's the most expensive part of your plan?
FETCH
executionTimeMillisEstimate: 431
INDEX SCAN
executionTimeMillisEstimate: 67

92. # M D B l o c a l
db.tweets.explain("executionStats").find(…)
What's the most expensive part of your plan?
FETCH
works: 2705
advanced: 314
needTime: 2391
// 314/2705 ≈ 8%
INDEX SCAN

93. # M D B l o c a l
• "queryPlanner"
• Is your query using the index you expect?
• Is your query using an index to provide the sort?
• Is your query using an index to provide the projection?
• "executionStats"
• How selective is your index?
• Which part of your plan is the most expensive?
Our Progress

94. # M D B l o c a l
db.tweets.explain("executionStats").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
• We had an index on {favorites: 1}, would that have been
faster?
next up: "Why was This plan chosen?"
🤔

95. # M D B l o c a l
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about plan
selection
Log messages
What is explain? The profile
Other commands

96. # M D B l o c a l
> db.collection.find(…).explain()
{"queryPlanner" : {
"winningPlan" : {
{"stage" : "SORT",
"inputStage" : {…}
}}
},
"rejectedPlans" : [
{"stage" : "SORT",
"inputStage" : {…}
}}
…
]
}}
Explain Mode: "QueryPlanner"
❓
FETCH
INDEX SCAN
(a, b)
SORT
FETCH
INDEX SCAN (a)
SORT

97. # M D B l o c a l
Query Planning
FETCH
INDEX SCAN (a,
b)
SORT
FETCH
INDEX SCAN
(a)
SORT
FETCH
INDEX SCAN (a, b)

98. # M D B l o c a l
• MultiPlanStage::pickBestPlan()
Query Planning
MATCH
COLL SCAN …
MATCH
COLL SCAN
COLL SCAN
FETCH
IX SCAN (ts)
MultiPlanStage
work()
work()
work()

99. # M D B l o c a l
• MultiPlanStage::pickBestPlan()
Query Planning
ADVANCED
NEED_TIME
ADVANCED
MATCH
COLL SCAN …
MATCH
COLL SCAN
COLL SCAN
FETCH
IX SCAN (ts)
MultiPlanStage

100. # M D B l o c a l
• MultiPlanStage::pickBestPlan()
Query Planning
Advances:78 22 50
MATCH
COLL SCAN …
MATCH
COLL SCAN
COLL SCAN
FETCH
IX SCAN (ts)
MultiPlanStage

101. # M D B l o c a l
• MultiPlanStage::pickBestPlan()
Query Planning
Advances:78 22 50
MATCH
COLL SCAN …
MATCH
COLL SCAN
COLL SCAN
FETCH
IX SCAN (ts)
MultiPlanStage

102. # M D B l o c a l
> db.tweets.find(…).explain("allPlansExecution")
{
"queryPlanner" : {
…
"winningPlan" : {…},
"rejectedPlans" : […]
},
"executionStats": {
…,
"executionStages": {…},
"allPlansExecution": […] // New!
}
…
}
Explain Mode: "allPlansExecution"

103. # M D B l o c a l
> db.collection.find(…).explain()
{"queryPlanner" : {
"winningPlan" : {
{"stage" : "SORT",
"inputStage" : {…}
}}
},
"rejectedPlans" : [
{"stage" : "SORT",
"inputStage" : {…}
}}
…
]
}}
Verbosity: "queryPlanner"
FETCH
INDEX SCAN
SORT
COLL_SCAN
SORT

104. # M D B l o c a l
> db.collection.find(…).explain()
{"queryPlanner" : {
"winningPlan" : {…},
"rejectedPlans" : [
{…},
…
]
},
"executionStats": {
executionStages: {…}
}}
Verbosity: "executionStats"
FETCH
INDEX SCAN
SORT
SORT
FETCH
INDEX SCAN
SORT

105. # M D B l o c a l
> db.collection.find(…).explain()
{"queryPlanner" : {
"winningPlan" : {…},
"rejectedPlans" : [
{…},
…
]
},
"executionStats": {
executionStages: {…}
}}
Verbosity: "executionStats"
FETCH
INDEX SCAN
SORT
SORT
FETCH
INDEX SCAN
SORT

106. # M D B l o c a l
> db.collection.find(…).explain()
{"queryPlanner" : {
"winningPlan" : {…},
"rejectedPlans" : [
{…},
…
]
},
"executionStats": {
executionStages: {…},
allPlansExecution: [
{…},
{…},
…
]
}}
Verbosity: "executionStats"
SORT
FETCH
INDEX SCAN
SORT
FETCH
INDEX SCAN
SORT
FETCH
INDEX
SCAN
SORT
FETCH
INDEX
SCAN
SORT

107. # M D B l o c a l
db.tweets.explain("allPlansExecution").find({
createdDate: {$gte: <today>},
favorites: "@eliothorowitz"
})
{
"executionStats": {
"allPlansExecution": [
{nReturned: 34,
executionStages: { /* Index Scan on "favorites" */ }
},
{nReturned: 101,
executionStages: { /* Index Scan on "createdDate" */ }
}
]
}
…
}
Explain Mode: "allPlansExecution"

108. # M D B l o c a l
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about plan
selection
Log messages
What is explain? The profile
Other commands

109. # M D B l o c a l
• Queries with response time >100ms (server side) are logged:
Slow Queries
2017-05-25T10:01:58.917-0400 I COMMAND [conn7] command twitter.tweets appName: "MongoDB Shell" command: find { find: "tweets",
filter: { nFavorites: { $gte: 10000.0 } }, limit: 20.0, singleBatch: false, sort: { nFavorites: -1.0, username: 1.0 }, projection: { _id: 0.0, nFavorites:
1.0, username: 1.0 } } planSummary: IXSCAN { nFavorites: -1 } keysExamined:359907 docsExamined:359907 hasSortStage:1
cursorExhausted:1 numYields:2871 nreturned:20 reslen:1087 locks:{ Global: { acquireCount: { r: 5744 } }, Database: { acquireCount: { r: 2872 }
}, Collection: { acquireCount: { r: 2872 } } } protocol:op_command 1493ms

110. # M D B l o c a l
• Queries with response time >100ms (server side) are logged:
Slow Queries
2017-05-25T10:01:58.917-0400 I COMMAND [conn7] command twitter.tweets appName: "MongoDB Shell" command: find { find: "tweets", filter: {
nFavorites: { $gte: 10000.0 } }, limit: 20.0, singleBatch: false, sort: { nFavorites: -1.0, username: 1.0 }, projection: { _id: 0.0, nFavorites: 1.0, username:
1.0 } } planSummary: IXSCAN { nFavorites: -1 } keysExamined:359907 docsExamined:359907 hasSortStage:1 cursorExhausted:1 numYields:2871
nreturned:20 reslen:1087 locks:{ Global: { acquireCount: { r: 5744 } }, Database: { acquireCount: { r: 2872 } }, Collection: { acquireCount: { r: 2872 } } }
protocol:op_command 1493ms

111. # M D B l o c a l
• Queries with response time >100ms (server side) are logged:
Slow Queries
2017-05-25T10:01:58.917-0400 I COMMAND [conn7] command twitter.tweets appName: "MongoDB Shell" command: find { find:
"tweets", filter: { nFavorites: { $gte: 10000.0 } }, limit: 20.0, singleBatch: false, sort: { nFavorites: -1.0,
username: 1.0 }, projection: { _id: 0.0, nFavorites: 1.0, username: 1.0 } } planSummary: IXSCAN { nFavorites: -1 }
keysExamined:359907 docsExamined:359907 hasSortStage:1 cursorExhausted:1 numYields:2871 nreturned:20 reslen:1087 locks:{ Global: {
acquireCount: { r: 5744 } }, Database: { acquireCount: { r: 2872 } }, Collection: { acquireCount: { r: 2872 } } } protocol:op_command 1493ms

112. # M D B l o c a l
• Queries with response time >100ms (server side) are logged:
Slow Queries
2017-05-25T10:01:58.917-0400 I COMMAND [conn7] command twitter.tweets appName: "MongoDB Shell" command: find { find: "tweets",
filter: { nFavorites: { $gte: 10000.0 } }, limit: 20.0, singleBatch: false, sort: { nFavorites: -1.0, username: 1.0 }, projection: { _id: 0.0, nFavorites:
1.0, username: 1.0 } } planSummary: IXSCAN { nFavorites: -1 } keysExamined:359907
docsExamined:359907 hasSortStage:1 cursorExhausted:1 numYields:2871 nreturned:20 reslen:1087 locks:{ Global: {
acquireCount: { r: 5744 } }, Database: { acquireCount: { r: 2872 } }, Collection: { acquireCount: { r: 2872 } } } protocol:op_command 1493ms

113. # M D B l o c a l
• Queries with response time >100ms (server side) are logged.
• Configurable via profiling parameter 'slowMs'
Slow Queries

114. # M D B l o c a l
• If turned on, queries show up in the system.profile collection
The Profile
{ "op": "query",
"ns": "twitter.tweets",
"query": { "find": "tweets", "filter": { … }, "limit": 20, "sort": { … }, "projection": { … } },
"millis": 1355,
"planSummary": "IXSCAN { nFavorites: -1 }",
"execStats": {
"stage": "PROJECTION",
"inputStage": {
"stage": "SORT",
"inputStage": {
"stage": "FETCH",
"inputStage": {
"stage": "IXSCAN",
} } } } } }

115. # M D B l o c a l
db.runCommand({
explain: {/* command */},
verbosity: <queryPlanner|executionStats|allPlansExecution>
})
Other Commands

116. # M D B l o c a l
db.runCommand({
explain: {findAndModify: {…}},
verbosity: <queryPlanner|executionStats|allPlansExecution>
})
Other Commands

117. # M D B l o c a l
db.runCommand({
explain: {update: {…}},
verbosity: <queryPlanner|executionStats|allPlansExecution>
})
Other Commands

118. # M D B l o c a l
db.runCommand({
explain: {aggregate: {…}},
verbosity: <queryPlanner|executionStats|allPlansExecution>
})
Other Commands

119. # M D B l o c a l
• Aggregation is special…
Aggregation
runCommand({
aggregate: "collection",
pipeline: […],
explain: <true|false>
})

120. # M D B l o c a l
• Aggregation is was special…
• 3.4 and earlier:
• 3.6 and beyond:
Aggregation
runCommand({
aggregate: "collection",
pipeline: […],
explain: <true|false>
})
runCommand({explain: {
aggregate: "collection",
pipeline: […],
},
verbosity: "…" })

121. # M D B l o c a l
db.explain().aggregate([{$group: {…}}, {$project: {…}}])
{
stages: [
{$cursor: {…}},
{$group: {…}},
{$project: {…}},
]
}
Aggregation

122. # M D B l o c a l
db.explain().aggregate([{$group: {…}}, {$project: {…}}])
{
stages: [
{$cursor: {
query: {…},
fields: {…},
queryPlanner: {/* same as query explain! */},
executionStats: {/* 3.6+ only, same as query explain! */}
}},
{$group: {…}},
{$project: {…}} ] }
Aggregation

123. # M D B l o c a l
SUMMARY
01 02 03 04 05
Motivation "queryPlanner"
Verbosity
"executionStats"
Verbosity
"allPlansExecution"
Versbosity
Beyond Queries
Why do you care? Describing considered
plans
More details about
winning plan
More details about plan
selection
Log messages
What is explain? The profile
Other commands

124. # M D B l o c a l
Some Closing Thoughts

125. # M D B l o c a l
01: Output is VERY large

126. # M D B l o c a l
Compass Shoutout

127. # M D B l o c a l
Compass Shoutout

128. # M D B l o c a l
Compass Shoutout

129. # M D B l o c a l
02: explain won't solve all your problems
created by Mike Ashley from Noun Project
Your Application MongoDB

130. # M D B l o c a l
Can I see all the
tweets with hashtag
"#MDBlocal" from
this hour?
02: explain won't solve all your problems
Your Application MongoDB

131. # M D B l o c a l
Hmm… Let
me think
about that…
02: explain won't solve all your problems
Your Application MongoDB

132. # M D B l o c a l
Ah! Here are
your results!
02: explain won't solve all your problems
Your Application MongoDB

133. # M D B l o c a l
02: explain won't solve all your problems
What took
you so
long?!
Your Application MongoDB

134. # M D B l o c a l
- network
latency
02: explain won't solve all your problems
What took
you so
long?!
Your Application MongoDB

135. # M D B l o c a l
- network latency
- large result set
{…
}
02: explain won't solve all your problems
{…
}
{…
}
{…
}
{…
}
{…
}
{…
} {…
}
{…
}
What took
you so
long?!
Your Application MongoDB

136. # M D B l o c a l
02: explain won't solve all your problems
Your Application MongoDB
- network
latency
- large result set
- server
contention
What took
you so
long?!

137. # M D B l o c a l
- network
latency
- large result set
- server
contention
- query planning
problem
02: explain won't solve all your problems
What took
you so
long?!
Your Application MongoDB

138. # M D B l o c a l
• Is your query using an index? Which one?
• Is your query using an index to provide the sort?
• How many of the examined documents ended up matching?
• Why did the server choose to answer the query the way it did?
03: Target Questions
🤔