When we talk about search and recommendations, we frequently talk about them as polar opposites. In this talk, I will discuss the different technology components we employ when thinking about Search and Recommendations at OpenTable. We will describe how we can use the same technology stack to power the many different modes of decision-making users have, ranging from helping people locate a restaurant they have in mind, to browsing based on a general set of criteria, to suggesting ideas for what might be best for a given context.
4. OpenTable in Numbers
• Our network connects diners with more
than 32,000 restaurants worldwide.
• Our diners have spent approximately $35
billion at our partner restaurants.
• OpenTable seats more than 17 million
diners each month.
• Every month, OpenTable diners write more
than 475,000 restaurant reviews
4
5. OpenTable Data Ecosystem
Search
(Context & Intent)
Restaurant Profile
(Decision Confidence)
Reservation History
(Verifying the Loop)
Reviews
(Verifying the Loop)
User’s Location
Search Location
Date, Time
Query
Reviews,
Ratings (Overall, Food, Noise Level, etc)
Seating Logs
Photos, Reviews, Ratings, Menus
6. OpenTable Data Ecosystem
Search
(Context & Intent)
Restaurant Profile
(Decision Confidence)
Reservation History
(Verifying the Loop)
Reviews
(Verifying the Loop)
User’s Location
Search Location
Date, Time
Query
Reviews,
Ratings (Overall, Food, Noise Level, etc)
Seating Logs
Photos, Reviews, Ratings, Menus
User
Interaction
Logs
9. What’s the Goal
Minimizing Engineering Time to Improve The
Metric that Matters
• Make it Easy to Measure
• Make it Easy to Iterate
• Reduce Iteration Cycle Times
9
10. Pick Your Business Metric
Revenue, Conversions
• OpenTable
• Amazon
Retention, Engagement
• Netflix
• Pandora
• Spotify
10
11. Importance of A/B Testing
• If you don’t measure
it, you can’t improve it
• Metrics Drive Behavior
• Continued Forward
Progress
11
18. Query Logs
• Effective mechanism for
understanding what
users are trying to do
• Reducing 0 result
queries
- Anecdote: should we
support zipcodes next?
19. Search to Recommendations Continuum
• Common Themes
- Ranking always tries to move key metric (like
conversion)
- Always leverage implicit signals (time of day, day
of week, location, etc)
- User Control vs. Paradox of Choice
Advantage Example Stage Item Count
Search User Control $$, French, Takes
Credit Card
Retrieval Many
Browse Use Case Control Great View /
Romantic
Ranking Many
Recommend Data-Driven
Flexibility
Best around me Ranking Few
20. Differences in Recommender Usage
Right now vs. Planning
Cost of Being Wrong
Search vs. Recommendations
20
22. Search & Recommendation Stack
Query Interpretation
Retrieval
Ranking – Item & Explanation
Index
Building
Context for Query & User
Model
Building
Explanation
Content
Visualization
Collaborative
Filters
Item / User
Metadata
22
23. Using Context, Frequency & Sentiment
• Context
- Implicit: Location, Time,
Mobile/Web
- Explicit: Query
• High End Restaurant for Dinner
- Low Frequency, High Sentiment
• Fast, Mediocre Sushi for Lunch
- High Frequency, Moderate
Sentiment
23
24. Offline Models with Limited Data
• Minimize Confusing User Experience
• Little to No Data
- Heuristics
Encoding Product Expectations
• Eg: Romantic Dates are not $. Sushi is not good for
Breakfast
• Limited Data
- Data-Informed
Eg: Analyze what Cuisines Users Click on when they
Query for Lunch
25. Offline Models with Significant Data
• Compensate for Sparseness
• As Signals Improve, Popular ->
Personalized
• OpenTable Example
- Context: User Location, Searched Location,
Query, etc.
• Learning to Rank
- E [ Revenue | Query, Position, Item, User ]
- E [ Engagement | Query, Position, Item, User ]
- Regression, RankSVM, LambdaMart…
31. Online Learning – Overview
• Naïve Online Learning is
A/B testing
- Try different sets of
parameters, pick the winner
• Multi-Arm Bandit
- Exploiting the parameter
sets that do well
- Exploring parameters that
we don’t understand well yet
(high variance)
32. Online Learning – Implementation
• Iteration Loop
- Add Sets of Parameters
- Explore vs. Exploit Current
Parameters
• Validate Online Learning
with A/B testing
• Note: Tradeoff in Time to
Statistical Significance
43. Training DataFlow
Collaborative Filter
Service
(Realtime)
Collaborative Filter
HyperParameter Tuning
(Batch with Spark)
Collaborative Filter
Training
(Batch with Spark)
Search Service
(Realtime)
Search HyperParameter
Tuning
(Batch with Spark)
Search Training
(Batch with Spark)
User Interaction Logs
(Kafka)
Frontends & Backend
Services
44. Training DataFlow
Collaborative Filter
Service
(Realtime)
Collaborative Filter
HyperParameter Tuning
(Batch with Spark)
Collaborative Filter
Training
(Batch with Spark)
Search Service
(Realtime)
Search HyperParameter
Tuning
(Batch with Spark)
Search Training
(Batch with Spark)
User Interaction Logs
(Kafka)
Online Learning
Frontends & Backend
Services
45. Training DataFlow
Collaborative Filter
Service
(Realtime)
Collaborative Filter
HyperParameter Tuning
(Batch with Spark)
Collaborative Filter
Training
(Batch with Spark)
Search Service
(Realtime)
Search HyperParameter
Tuning
(Batch with Spark)
Search Training
(Batch with Spark)
User Interaction Logs
(Kafka)
Online Learning
Frontends & Backend
Services
A/B Validation
48. Summarizing Content
• Essential for Mobile
• Balance Utility With Trust?
- Summarize, but surface raw
data
• Example:
- Initially, read every review
- Later, use average star rating
48