Search Basics

224.4.2015 © Sanoma Media
Agenda
• Search Basics
• Features
• Search solutions
»MySQL (Full-Text search and Sphinx)
»Solr
»ElasticSearch
• Sanoma Content Library
• Common gotcha’s

High level components
Filtering Indexing Querying Ranking

Filtering techniques
54/24/2015 © Sanoma Media
• Tokenizing
• Stop Words
• Synonyms
• Stemming
• Term occurrence
• Phonetics

Filtering techniques Filtering Indexing Querying Ranking
• Tokenizing
• Stop Words
• Synonyms
• Stemming
• Term occurrence
• Phonetics
The quick brown fox jumps
over a lazy dog
The
quick
brown
fox
jumps
over
a
lazy
dog

• Tokenizing
• Stop Words
• Synonyms
• Stemming
• Term occurrence
• Phonetics
• Special characters: +/-,.;!@#$%^&
etc.
»I.B.M.
• Case and numeric changes:
»PowerShot, TransAM, SD500, iPod
• Decide what you want to happened
with:
»Canon Power-Shot SD500
(Canon Power shot SD-500, Canon Powershot SD 500)
»O’neill’s

• Remove stop words from being indexed
• No value, since they’re to common
The
quick quick
brown brown
fox fox
jumps jumps
over
a
lazy lazy
dog dog
Stop words
a,able,about,
across,after,al
l,almost,also,
am,among,an
,and,any,are,
as,at,be,beca
use,been,but,
by,can,cannot
,could,dear,di
d,do,does,eith
er,else,ever,e
very,for,from,
,got,had
have,h
• Tokenizing
• Stop Words
• Synonyms
• Stemming
• Term occurrence
• Phonetics

• Tokenizing
• Stop Words
• Synonyms
• Stemming
• Term occurrence
• Phonetics
• De-duplicate various words:
»bicycle, cycle, bike
»i-pod, ipot => iPod

• Tokenizing
• Stop Words
• Synonyms
• Stemming
• Term occurrence
• Phonetics
• Determine the stem of a word:
»Dogs => dog
»Recharging => recharg
»Rechargeable => recharg
• Language specific:
»Porter for English (-s, -ed, -ly, -ing, etc.)
»SnowballPorter or Kraaij-Pohlmann
for Dutch (ge-, -en, etc.)

• Tokenizing
• Stop Words
• Synonyms
• Stemming
• Term occurrence
• Phonetics
• Options for limiting the size of the
index:
»Minimum Term frequency
»Minimum Term Length

• Tokenizing
• Stop Words
• Synonyms
• Stemming
• Term occurrence
• Phonetics
• Handling sounds like queries:
» Robert => R163 <= Rupert
» Smith => (SM0,XMT) ∩ (XMT,SMT) <= Schmith
• Various methods available:
» DoubleMetaphone
» Metaphone
» Soundex
» RefinedSoundex
» Caverphone
» BeiderMorse
• Levenstein can be used during quering

Apply the filters on Filtering and querying
Same filters

Stopwords,stemming,
synonyms,etc.
Filters
Indexing

Querying

DEMO
Stemming, Phonetics

Ranking
TF-IDF
Term Frequency-Inverse Document Frequency
How often does the search
term occur in the text
How many words are
in the entire text

Ranking – TF-IDF
3/12 = 0,25 5/24 = 0,21
More relevant

USER PATTERNS

User patterns
• Features should be adjusted to the user and usage patterns your seeing
• What are users searching for on your site
• How are they searching for it
• Use web analytics to track and improve your search behavior
Image credits: http://www.flickr.com/photos/morville/collections/72157604060564791/

User pattern - Quit

User patterns – Pogosticking

User patterns - Thrashing

User patterns - Narrow

User patterns – Others
• Pearl Growing
• Expand

Search Features
• Faceting
• Autocomplete
• More like this..
• Highlighting
• Spellchecking
did you mean
• Geospatial
“bike repair” in area of [long,lat],[long,lat]
• Boosting
when title is more relevant then content
• Elevation
always get a certain result at position n
get the current weather, current traffic at 1st
position or ingest ads

Search Features - Faceting
From the user perspective, faceted
search (also called faceted
navigation, guided navigation, or
parametric search) breaks up search
results into multiple categories,
typically showing counts for each,
and allows the user to "drill down" or
further restrict their search results
based on those facets.

Search Features - Autocomplete

Search Features - More like this..
• Give you the related items based on a document
• Compares the Term Vectors of various documents
• Creates a query with boosting:
body:pre body:username^.56974 body:column^.57123 body:oracle^.61915 ...
Term
Number of Instances of
Term in Document
Number of Documents
Matching Term
IDF value Score
pre 18 26 4.609916 82.978
username 10 23 4.7276993 47.276
column 9 13 5.266696 47.400264
oracle 9 8 5.7085285 51.376
alter 7 1 7.212606 50.488

Search Features - Highlighting
• Highlighting the search terms
• Includes stemming and other logic

DEMO SOLR

SOLUTIONS

Services
Common search options
• MySQL based
»Native Full-Text search
»Sphinx Search Plugin
• Lucene based (Java)
»Apache Lucene/Solr
»ElasticSearch

Services
Common search options
Ease of
use
Power

MySQL Based
Native Full-Text vs Sphinx
MySQL Full-Text search
• Only for MyISAM tables, and only on
CHAR, VARCHAR and TEXT fields
• Only standard English stop words
• Limited query capabilities
• Slow on large collections (1GB+)
• Building facetting is “hard” and
“expensive”
• No stemming, no synonyms, no
custom flieds, no highlighting
Sphinx
• External plugin
• All storage engines
• Also on numeric field types
• ~3x faster on index and query
• Simple stemming and synonyms
• No custom fields, no highlighting

Querying is easy
• MySQL Full-Text query:
SELECT * FROM articles
WHERE MATCH (title,body)
AGAINST ('database');
• Getting the score:
SELECT id, MATCH (title,body)
AGAINST ('Tutorial')
FROM articles;
• Sphinx query, index is
separate table:
SELECT id, created_time, @weight
FROM my_sphinx_index
WHERE created_time BETWEEN (X AND Y)
AND MATCH ('Android phone’)
ORDER by @weight DESC,
created_time DESC

Lucene based
ElasticSearch
• Simpler Solr
• No need for a schema
• Easy to cluster
• Focus on scaling and realtime
• Go with the defaults
• Configuration = 3 lines
• Percolation!
• Versions and TTLs
Solr
• Exposing all of the lucene
power
• Clustering possible, but
harder
• Focus on complete and
customizable
• Defaults?
• Configuration = 3.000 lines

Solr vs ElasticSearch
Search Fresh Index While Idle
0
10
20
30
40
50
60
Searchtimeinms.
ElasticSearch
Solr
Lower is better

Search Fresh Index While Indexing 1doc/3sec
0
50
100
150
200
250
Searchtimeinms.
ElasticSearch
Solr
Lower is better

Search Full Index While Indexing 1doc/3sec
0
500
1000
1500
2000
2500
Searchtimeinms.
ElasticSearch
Solr
Lower is better

Search Full Index While Indexing 1doc/3sec
0
500
1000
1500
2000
2500
Searchtimeinms.
ElasticSearch
Solr
Lower is better
Idle Indexing Full + Indexing

Lower is better
SOLR ElasticSearch

Querying with Solr and ElasticSearch
Solr
• Normal query
http://../solr?q=field:banana
• Facetting
http://../solr?q=field:banana&facet=
on&facet.field=tags
ElasticSearch
• Normal query
http://../_search?q=field:value
• Advanced queries, via PUT:
POST http://../collection/seach
{
"query": { "query_string" :{"query" : "T*"}
},
"facets" : {
"tags" : { "terms" : {"field" : "tags"} }
}
} 4424.4.2015 © Sanoma Media

ElasticSearch

SANOMA CONTENT LIBRARY

Sanoma Content Library
Search
.. in site
.. in cluster
.. in network
Elevation (ads)
Facetting
Related
More like this
Relevant ads
Products
Reuse
Sharing
Variants
(simple) Drm
Images
Analyse
Sentiment
Named Entities
Tagging
Classificatie
Key phrases

Services: Content Library
Content
Library
Analyse
Pipeline
NER Sentiment
Crawler
Indexer
Search
index
Search
- nu.nl
- wtf
Related
- Vrouwen
- Kieskeurig
Relevant
- Txel
API
Edge
Redir
ects
Loader
Solr
Mongo
Integration
- Vrouwen
- Wordpress
- SAS
CMS
/
JCR
Keyphrase
extractor
Classifier

Common gotcha’s
• Use right settings for your language stopwords and stemming
• Indexing too much or too detailed:
»Timestamps

END

Search Basics

Recomendados

Recomendados

Más contenido relacionado

Similar a Search Basics

Similar a Search Basics (20)

Último

Último (20)

Search Basics

Notas del editor