Improving search experience is a very data driven & iterative process. Measuring the quality of search results and having the tools to measure, make sense and then improve your search ranking algorithms is key to the entire process. We'll talk through some aspects of how to measure quality for given search results and key metrics that the teams need to track to as they make changes to their search to make it better. Later during the talk, we’ll show a very basic example of how one can start to incorporate user behavior as part of the search results to influence the ranking of the results.

1. Making search better by tracking &
utilizing user search behavior
Sameer Maggon

2. Agenda
• About Me
!
• Measuring Search Quality
!
• Quality Metrics based on users interactions
!
• Quality Metrics based on User Tagging (labelers)
!
• Improving Search Results
!
• Using user behavior to improve results ranking -
Example
2

3. Sameer Maggon
Founder of Cloud based Zero Management Solution
for Search
Built relevance based search platform for AT&T
Interactive & properties including yp.com, buzz.com,
yellowpages.com
Consulted for numerous Startups to Fortune 500 companies
around Search & Discovery.
3
Engineering Alumni
@maggon
http://linkedin.com/in/maggon

4. Search seems Easy
1 2 3
Index all content Put a search box Show google
4
like results

5. Is It?
• How do you know that your users are finding what they
5
are looking for?
!
• How do you know what impact your one-off fix has on an
aggregate?
!
• Seemingly good result list to one might be irrelevant to
another (e.g. mosaic)

6. How do we measure search?
Usage Data Editorial Labeling
6
Collect, Analyze & Report on
interactions users are having
with your search functionality.
Get a set of users to mark top
x results with “Relevant vs.
Not” for a pre-determined
sample set of searches.
!
Then compute specific
metrics based on those.

7. Usage Data: Key Metrics to look at
7
• No Result Search %
• Search Exits %
• CTR %
!
• Average Click Position
• MRR (Mean Reciprocal Rank)
• Clicks per Search
• Paging (how deep do people have
to dig?)
!
• Latency (Average, tp90 and tp95)
Aggregate &
Trends
Trends
Aggregate &
Trends

8. Search Analytics: Examples
8

9. 9
Editorial Labeling
• Precision Recall
!
• DCG (Discounted
Cumulative Gain)
!
• nDCG (Normalized DCG)
http://en.wikipedia.org/wiki/Discounted_cumulative_gain
Relevant
Not Relevant
Relevant
Not Relevant
Not Relevant
Relevant
Not Relevant
weight-age decreases as as
you go down on an ordered list

10. Improving Search Results
Attack Low Hanging fruit first
• Popular No Result Searches - Can we use keyword stuffing?
• Popular Search Exits - Eyeball outliers
• Popular Searches with low CTRs
• Generally improve Average Click Position / MRR via
Utilizing Search Behavior to improve ranking
• Utilize Popularity (click stream) to inform search ranking
• Utilizing past search history to offer assistive features
(search suggestions, related searches)
10
identifying patterns
(impacts CTR)
Advanced: Learning Models
• Topic for some other time :)

11. Using Popularity to affect search ranking
DEMO
11

12. www.measuredsearch.com
sameer@measuredsearch.com