Tim Frietas gave a presentation on MongoDB Atlas Full-Text Search. He discussed how full-text search works in MongoDB using inverted indices and analyzers. MongoDB Atlas FTS uses Apache Lucene under the hood and integrates a separate process called mongot that handles search queries. Mongot communicates with mongod to index and search document collections and return results. Future roadmap items include expanded data type support, synonyms, improved query syntax, and performance optimizations.

1. @
#MDBlocal
@
#MDBlocal
Tim Frietas
Sr. Product Manager - Cloud
timfrietas
LONDON

2. Why Full-Text Search?

3. #MDBLocal#MDBLocal
Developer: “This result doesn’t
match the most terms.”
Marketing: “Why isn’t the promo on
top?”
Product: “If I type in an exact
product name, just skip the search.”
UX: “Users need categories
instead.”
CEO: “Change the top result right
NOW!”
Customer: “Ich verstehe das
nicht…”
Where’s my search result?

4. #MDBLocal#MDBLocal
Searching with $text
db.movies.find
({$text: {$search: "chikago"}},
{score: {$meta: "textScore"},
_id : 0,
title: 1,
fullplot: 1
}).sort
({score:{$meta:"textScore"}}).pretty()
[] (empty set)

5. #MDBLocal#MDBLocal
Searching with $searchBeta
db.movies.aggregate([
{$searchBeta: { term: {
path: "title",
query: "chikago",
fuzzy: { "maxEdits": 1}},
highlight: { path: "fullplot"}}},
{$project: {
_id : 0,
title: 1,
score: { $meta: 'searchScore' },
highlights: { $meta: "searchHighlights"}}},
{$limit : 1}]).pretty()
{"title" : "Chicago",
"score" : 4.210028171539307,
highlights" : [{
path" : "fullplot",
"texts" : [
{"value" : "Murderess Velma Kelly finds herself on
death row in 1920s ",
"type" : "text"},
{"value" : "Chicago",
"type" : "hit"},
{"value" : ".",
"type" : "text"}],
"score" : 0.863726019859314}]}

6. #MDBLocal#MDBLocal
Query-time Scoring
db.movies.aggregate([
{ $match": {
”title": ”The Godfather”
}},
{ $sort: { title: 1 } }
])
$searchBeta
db.movies.aggregate([
{ $searchBeta": {
”search": { query: “The
Godfather”,
path: “title”,
score: { boost: {value: 3}}}}
])
Sort occurs after filter Sort occurs during filter

7. DEMODEMO
Let’s Search for Movies
Tim Frietas

8. Architecture
How FTS works

9. #MDBLocal#MDBLocal
MongoDB +
• Pre-existing functionality
• Highlights, Fuzzy-matching, Query-time scoring and
more
• Analyzers
• Language support
• Western languages: English, French, etc.
• Eastern Languages: CJK, bigram/unigram support
• Inverted index structure = fast searches

10. #MDBLocal#MDBLocal
Real-world Inverted Index: A Concordance

11. #MDBLocal#MDBLocal
Inverted Indices and Analyzers
“_id”: 3,
“title” : “Planes, Trains & Automobiles”
{
“planes” : [3, …],
“trains” : [3, …],
“automobiles” : [3, … ]
},
lucene.simple lucene.english
{
“plane” : [3, …],
“train” : [3, …],
“automobile” : [3, …],
}
lucene.keyword
{
“Planes, Trains &
Automobiles” : [3, …]
},

12. #MDBLocal#MDBLocal
Querying and Analyzers
search: { query: “planes trains”,
path: “title” }
term(“plane”) OR
term(“train”)
term: { query: “planws”,
path: { value: “title”,
multi: ”simple”},
fuzzy: {maxEdits: 1 } }
Fuzzy(“planws”,
maxEdits: 1)
term: { query: “Planes(.*)”,
path: { value: “title”,
multi: ”keyword”},
regex: true} }
Regex(“Planes(.*)”
lucene.keyword lucene.simple lucene.english

13. #MDBLocal#MDBLocal
MongoDB Atlas FTS components
mongod mongos mongot (NEW!)
● $searchBeta aggregation
pipeline stage
● Talks mongodb wire protocol to
mongot
● Shard aware implementation
● scatter-gather queries
● Based on Apache Lucene 8
● Integrated into MongoDB Atlas
● Separate java process from
mongod
● collocated with mongod

14. #MDBLocal#MDBLocal
FTS Indexing: Initial Sync
MongoDB Atlas
Query
Index Definition
Automation Agent
mongotmongod
Collection Scan
Complete!

15. #MDBLocal#MDBLocal
FTS Indexing: Steady State
Documents
mongotmongod
changestream
MongoDB Atlas
(per node)
mongot watches
the changestream
continuously and
updates the
search index

16. #MDBLocal#MDBLocal
Query Lifecycle
Wire Protocol
(over internet to
MongoDB Atlas) aggregate([ {$searchBeta: {
search: {
path: “name”,
query: “robot vacuum”
}
}}]
mongod
Wire protocol
(localhost)
Lucene booleanQuery:
(should
(term(“name”, “robot”),
term(“name”, vacuum”))
search: {
path: “name”,
query: “robot vacuum”
}
[ { _id: “123”,score:
1.23,highlights: […] },
{…}]
{ “name” :
{ “robot” : [123,124],
“vacuum” :
[123,125,…]}}
Lookup([{_id: “123”],
{…}])
mongot
db.col.aggregate([
{$searchBeta: {
search: {
path: “name”,
query: “robot vacuum”
}
}}])
app
[ { _id: “123”, title:
“the best vacuum”}, {…}]
MongoDB Atlas Host

17. #MDBLocal#MDBLocal
Sharding
• Merge
• Sort by score
mongos
request
result
per shard
per shard
• Scatter/gather
(each shard)
mongot,
mongod
mongot,
mongod
Primary Secondaries
Query

18. What’s next?
Feature Roadmap

19. #MDBLocal#MDBLocal
• Expanded data type support
• Nums, dates, geo
• Synonyms
• Improved operators/syntax
• Architecture/performance
improvements
2019 Roadmap

20. #MDBLocal#MDBLocal
We have a few things in mind…
But we’d love to hear from YOU!
2020 Roadmap

21. THANK YOU

22. #MDBlocal
MongoDB Atlas Full-Text
Search Deep Dive
Tim Frietas
https://www.surveymonkey.com/r/K2R7SFM