Talk presented at the ACM Recommender Systems Conference in Boston, 2016

1. 11
Past, Present & Future of Recommender Systems:
An Industry Perspective
Xavier Amatriain (Quora)
Justin Basilico (Netflix) RecSys 2016
@xamat @JustinBasilico
DeLorean image by JMortonPhoto.com & OtoGodfrey.com

2. 2
1. Past

3. 3
Netflix Prize
2006

4. 4
For more information ...

5. 5
2. Present

6. 6
Recommender Systems in Industry
Recommender Systems are used pervasively across application
domains

7. 7
Recommender Systems in Industry
click
upvote
downvote
expand
share

8. 8
Beyond explicit feedback
▪ Applications typically oriented around an action: click, buy,
read, listen, watch, …
▪ Implicit Feedback
▪ More data: Implicit feedback comes as part of normal use
▪ Better data: Matches with actions we want to predict
▪ Augment with contextual information
▪ Content for cold-start
▪ Hybrid: Combine together when you can

9. 9
Ranking
▪ Ranking items is central
to recommending
▪ News feeds
▪ Items in catalogs
▪ …
▪ Most recsys can be
assimilated to:
▪ A learning-to-rank approach
▪ A feature engineering
problem

10. 10
Everything is a RecommendationRows
Ranking

11. 11
3. Future

12. 12
Many interesting future directions
1. Indirect feedback
2. Value-awareness
3. Full-page optimization
4. Personalizing the how
▪ Others
▪ Intent/session awareness
▪ Interactive recommendations
▪ Context awareness
▪ Deep learning for
recommendations
▪ Conversational interfaces/bots
for recommendations
▪ …

13. 13
Indirect Feedback
Challenges
▪ User can only click on what you show
▪ But, what you show is the result of what your model
predicted is good
▪ No counterfactuals
▪ Implicit data has no real “negatives”
Potential solutions
▪ Attention models
▪ Context is also indirect/implicit feedback
▪ Explore/exploit approaches and learning across
time
▪ ...
click
upvote
downvote
expand
share

14. 14
Value-aware recommendations
▪ Recsys optimize for probability of action
▪ Not all clicks/actions have the same “reward”
▪ Different margin in ecommerce
▪ Different “quality” of content
▪ Long-term retention vs. short-term clicks (clickbait)
▪ …
▪ In Quora, the value of showing a story to a user is
approximated by weighted sum of actions:
v = ∑a
va
1{ya
= 1}
▪ Extreme application of value-aware recommendations:
suggest items to create that have the highest value
▪ Netflix: Which shows to produce or license
▪ Quora: Answers and questions that are not in the service

15. 15
Full page optimization
▪ Recommendations are rarely displayed in
isolation
▪ Rankings are combined with many other elements
to make a page
▪ Want to optimize the whole page
▪ Means jointly solving for set of items and
their placement
▪ While incorporating
▪ Diversity, freshness, exploration
▪ Depth and coverage of the item set
▪ Non-recommendation elements (navigation,
editorial, etc.)
▪ Needs work hand-in-hand with the UX

16. 16
Personalizing How We Recommend (… not just what we recommend)
▪ Algorithm level: Ideal balance of diversity, novelty,
popularity, freshness, etc. may depend on the person
▪ Display level: How you present items or explain
recommendations can also be personalized
▪ Select the best information and presentation for a user to quickly
decide whether or not they want an item
▪ Interaction level: Balancing the needs of lean-back users and
power users

17. 17
Rows
Example: Rows & Beyond
Hero
Image
Predicted
rating
Evidence
Synopsis
Horizontal
Image
Row Title
Metadata
Ranking

18. 18
4. Conclusions

19. 19
Conclusions
▪ Approaches have evolved a lot in the past 10 years
▪ Looking forward to the next 10
▪ Industry and academia working together has advanced the
field since the beginning, we should make sure that
continues

20. 20
Thank You
Justin Basilico
jbasilico@netflix.com
@JustinBasilico
Xavier Amatriain
xavier@quora.com
@xamat