1) Recommendations should be viewed as a conversation with the user, where asking questions of the user and adapting based on feedback is important.
2) Good recommendations ask good questions of the user initially and over time, make mistakes but provide explanations, and adapt to the user's feedback.
3) The goal is to consider asking the user for information rather than just guessing, provide explanations for recommendations, and allow users to correct mistakes through feedback to improve recommendations over time.
4. Clifford Nass’s secret:
1) Find a conclusion by a social science researcher.
2) Change
“People do X when interacting with other people.”
to
“People do X when interacting with a computer.”
3) Profit!
4
6. Core Message
Recommendations are a conversation with the user.
1) Consider asking vs. guessing.
2) Ask good questions.
3) It's ok to make mistakes…
if you have a good explanation
and adapt to feedback.
6
7. Our goal:
http://www.wilsoninfo.com/computerclipart.shtm
l
7
13. Quality: Above All, the Truth
Xiao, Bo and Benbasat, Izak. 2011. "Product-Related Deception in E-Commerce: A
Theoretical Perspective," MIS Quarterly, (35: 1) pp.169-195.
13
14. Don’t Lie
1) Don’t use “recommended” when you really mean
“sponsored” or “excess inventory”.
2) Optimize for the user’s utility.
3) Apply a standard of evidence (quality, quantity) that
you believe in.
14
16. Right Amount of Information
1) Exchange small units of information.
2) If recommendations supplement other content,
consider overall cognitive load.
3) Provide short, meaningful explanations.
16
22. Human-Computer Information Retrieval
Empower people to explore large-scale information
but demand that
people also take responsibility for this control
by expending cognitive and physical energy.
Marchionini, G., “Toward Human-Computer Information Retrieval” [2006]
22
23. Principles of HCIR
1) Do more than deliver relevant information:
facilitate sensemaking.
2) Increase user responsibility and control:
require and reward effort.
3) Adapt to increasingly knowledgeable users over time.
4) Be engaging and fun to use!
23
28. Applying the theory to…
1) Personalized Recommendations
2) Social Recommendations
3) Item Recommendations
28
29. Personalized Recommendations
1) Be transparent about model so users gain insight.
2) Allow users to modify models to correct mistakes.
3) Solicit just enough information to provide value.
29
30. Social Recommendations
1) Identify the right set of similar users.
2) Allow users to manipulate the social lens.
3) Accommodate users who break your model.
30
31. Item Recommendations
1) Explain recommendations to users.
2) Watch out for non-sequiturs (e.g., diapers -> beer).
3) Play well with user-controlled filtering and sorting.
31
44. Learning from Netflix
1) Ask the user for help up front. But not too much help.
2) Pay attention to what the user tells you!
3) Give users value early and often.
75% of Netflix views result from recommendations
44
54. Learning from Pandora
1) Get meaningful input from user in one step.
2) Explain recommendations to users.
3) Solicit feedback and act on it immediately.
54
60. Learning from Amazon
1) Show the factors that drive your conclusions.
2) Distinguish different kinds of recommendations.
3) Combine recommendations with user control.
Amazon: 35% of sales result from recommendations
60
62. Increase explainability.
Explanations can be even more important than the
recommendations themselves.
Herlocker et al., “Explaining collaborative filtering recommendations” [2000]
Sinha and Swearingen, “The role of transparency in recommender systems”
[2002]
Tintarev and Masthoff, “Effective explanations of recommendations: User-
centered design” [2007]
(via Òscar Celma’s book, Music Recommendation and Discovery: The Long
Tail, Long Fail, and Long Play in the Digital Music Space)
62
63. Some models more explainable than others.
1) Consider decision trees and rule-based systems.
2) Avoid using latent, unlabeled features.
3) If the model is opaque, use examples as surrogates.
63
64. Make a good first impression.
Your user’s first experience is critical.
Use popularity as a default if it makes sense.
Solicit one valuable piece of information as quickly and
painlessly as possible.
“Do you like the taste of beer?”
http://blog.okcupid.com/index.php/the-best-questions-for-first-dates/
64
65. Design feedback into your system.
You can make mistakes, if users can easily fix them.
Challenging if models use offline computation.
Respond instantly; generalize as quickly as possible.
Agarwal and Chen, “Machine Learning for Large Scale Recommender Systems”
[ICML 2011 Tutorial]
65
66. Integrate recommendations with search.
Recommend next steps, not just items.
In a task context, recommendations are just another
source of information scent.
Be careful in integrating offline recommendations with
online features like search and navigation.
Pirolli, Information Foraging Theory: Adaptive Interaction with Information [2007]
66
67. Summary
Recommendations are a conversation with the user.
1) Consider asking vs. guessing.
2) Ask good questions.
3) It's ok to make mistakes…
if you have a good explanation
and adapt to feedback.
67
Herlockerer al: “Explanations provide us with a mechanism for handling errors that come with a recommendation…most users value the explanations and would like to see them added”Sinha and Swearingen: “Meanlikingwas significantly higher for transparent than non-transparent recommendations…Mean Confidence showed a similar trend”Tintarev and Mastoff: “Feature selection in explanations needs to be tailored to the user [and] context…Features can be selected from a relatively short list…Explanation source matters”