Immersive Recommendation Workshop, NYC Media Lab'17

1. Immersive Recommendation Deep User and Content Modeling for Personalization Longqi Yang, Ph.D. student Connected Experiences Lab, Small Data Lab Cornell Tech

2. Collaborators Faculty Interns Industry Collaborators Ph.D. students PostDocs

3. What we will be talking about today Recommendation Systems: Past, Present and Future Immersive Recommendation. OpenRec.

4. Recommendation Systems Research

5. The Matrix From item-based filtering to Netflix Challenge

6. … … … … … … … … … Item-based filtering (WWW 2001) ?4 1 1 4 5 Sarwar, Badrul, et al. "Item-based collaborative filtering recommendation algorithms." Proceedings of the 10th international conference on World Wide Web. ACM, 2001. 1 5 1

7. … … … … … … … … … Item-based filtering (WWW 2001) ?4 1 1 1 5 4 5 4 Sarwar, Badrul, et al. "Item-based collaborative filtering recommendation algorithms." Proceedings of the 10th international conference on World Wide Web. ACM, 2001.

8. … … … … … … … … … Item-based filtering (WWW 2001) ?4 1 1 1 5 4 5 4 0.9 0.1 Sarwar, Badrul, et al. "Item-based collaborative filtering recommendation algorithms." Proceedings of the 10th international conference on World Wide Web. ACM, 2001.

9. … … … … … … … … … Item-based filtering (WWW 2001) ?4 1 1 1 5 4 5 4 0.9 0.1 Sarwar, Badrul, et al. "Item-based collaborative filtering recommendation algorithms." Proceedings of the 10th international conference on World Wide Web. ACM, 2001.

10. … … … … … … … … … Item-based filtering (WWW 2001) 3.74 1 1 1 5 4 5 4 0.9 0.1 Sarwar, Badrul, et al. "Item-based collaborative filtering recommendation algorithms." Proceedings of the 10th international conference on World Wide Web. ACM, 2001.

11. Early Adoption by Amazon.com

12. Netflix Challenge (Prize) We’re quite curious, really. To the tune of one million dollars… … To help customers find movies, we’ve developed our world-class movie recommendation system: Cinematch. Its job is to predict whether someone will enjoy a movie based on how much they liked or disliked other movies … … We provide you with a lot of anonymous rating data, and a prediction accuracy bar that is 10% better than what Cinematch can do on the same training data set… If you develop a system that we judge most beats that bar on the qualifying test set we provide, you get serious money and the bragging rights …

13. Netflix Challenge (Prize) Cinematch score - RMSE = 0.9525 2007 Progress Prize - RMSE = 0.8723 8.42% 2008 Progress Prize - RMSE = 0.8627 9.27% 2009 Grand Prize - RMSE = 0.8567 10.06% (AT&T Research)

14. Rating-based recommendations

15. Beyond Matrix Implicit feedback Deep Content Modeling Beyond “black-box items” Beyond Accuracy Diversity and Fairness Rich Context and Modality Learning preference from auxiliary channels

17. Click-through Upvote/Like View/Watch/Visit Listen/Play Implicit Feedback

18. Main Challenge of Implicit Feedback Only “positive signal” is observed Does “rating estimation” still work?

19. … … … … … … … … … ?1 1 1 1 1 1 1 1 Main Challenge of Implicit Feedback

20. … … … … … … … … … 11 1 1 1 1 1 1 1 Main Challenge of Implicit Feedback

21. … … … … … … … … … 11 1 1 1 1 1 1 1 No matter what this item is! Main Challenge of Implicit Feedback

22. Feature Learning Framework for Implicit Feedback [0.1 -0.2 0.35 … 0.15]𝒖 𝟏 = 𝒗 𝟏 = [-0.05 0.5 0.1 … -0.3] 𝒖 𝟐 𝒖 𝑵 … 𝒗 𝟐 𝒗 𝑴 … Optimization

23. One Example – Bayesian Personalized Ranking (BPR) 𝒖𝒊, 𝒗 𝒑, 𝒗 𝒏For all m𝑎𝑥 ln 𝜎 𝒖𝒊 ∙ 𝒗 𝒑 − 𝒖𝒊 ∙ 𝒗 𝒏 Rendle, Steffen, et al. "BPR: Bayesian personalized ranking from implicit feedback." Proceedings of the twenty-fifth conference on uncertainty in artificial intelligence. AUAI Press, 2009. user i An item that the user “click” An item that the user does not “click”

24. Algorithms (Incomprehensive List) • Weighted Regularized Matrix Factorization (WRMF) • Probabilistic Matrix Factorization (PMF) “Shallow” Models: • Weighted Approximately Ranked Pairwise Loss (WARP) “Deep” Models: Hsieh, Cheng-Kang, et al. "Collaborative metric learning." Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2017.• Collaborative Metric Learning (CML) He, Xiangnan, et al. "Neural collaborative filtering." Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2017.• Neural Collaborative Filtering Mnih, Andriy, and Ruslan R. Salakhutdinov. "Probabilistic matrix factorization." Advances in neural information processing systems. 2008. • Wide and Deep Learning for Recommender Systems Hu, Yifan, Yehuda Koren, and Chris Volinsky. "Collaborative filtering for implicit feedback datasets." Data Mining, 2008. ICDM'08. Eighth IEEE International Conference on. Ieee, 2008. Cheng, Heng-Tze, et al. "Wide & deep learning for recommender systems." Proceedings of the 1st Workshop on Deep Learning for Recommender Systems. ACM, 2016. Weston, Jason, Samy Bengio, and Nicolas Usunier. "Wsabie: Scaling up to large vocabulary image annotation." IJCAI. Vol. 11. 2011.

26. Pure Collaborative Filtering is cool (and maybe accurate), but real world recommendations are far more complex than “likes” Understanding the contents really matters! - Articles Is it appropriate to recommend these two articles together?

27. Pure Collaborative Filtering is cool (and maybe accurate), but real world recommendations are far more complex than “likes” Understanding the contents really matters! - Food Random (Most healthy) Trattner, Christoph, and David Elsweiler. "Investigating the healthiness of internet- sourced recipes: implications for meal planning and recommender systems." Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2017

28. Pure Collaborative Filtering is cool (and maybe accurate), but real world recommendations are far more complex than “likes” Understanding the contents really matters! - Food “Users in general tend to interact most often with the least healthy recipes. Recommender algorithms tend to score popular items highly and thus on average promote unhealthy items.” Trattner, Christoph, and David Elsweiler. "Investigating the healthiness of internet- sourced recipes: implications for meal planning and recommender systems." Proceedings of the 26th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 2017

29. Pure Collaborative Filtering is cool (and maybe accurate), but real world recommendations are far more complex than “likes” Understanding the contents really matters! – Cold Start A new fashion cloth A new online course A new job post Recommendations without user feedback

30. Deep Content Modeling for Recommendations (C/R/Res/Adversarial/Rei nforcement) NN User Item Interaction Optimization

32. Are ratings/clicks/views enough for recommendations? Context matters! – Music Recommendation Schedl, Markus, et al. "Music recommender systems." Recommender Systems Handbook. Springer US, 2015. 453-492. Schedl, Markus, Peter Knees, and Fabien Gouyon. "New Paths in Music Recommender Systems Research." Proceedings of the Eleventh ACM Conference on Recommender Systems. ACM, 2017. time location weather Environmental Context Individual Context emotion activity social context Schedl, Markus, Georg Breitschopf, and Bogdan Ionescu. "Mobile Music Genius: Reggae at the Beach, Metal on a Friday Night?." Proceedings of International Conference on Multimedia Retrieval. ACM, 2014.

33. Recommendations are not always “a list”: Rich modality Sun, Yu, et al. "Contextual intent tracking for personal assistants." Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2016. Kang, Jie, et al. "Understanding How People Use Natural Language to Ask for Recommendations." Proceedings of the Eleventh ACM Conference on Recommender Systems. ACM, 2017.

34. Rich Context and Modality (C/R/Res/Adversarial/Rei nforcement) NN User Item Interaction Optimization (C/R/Res/Adversarial/Rei nforcement) NN (C/R/Res/Adversarial/Rei nforcement) NN

36. Accuracy Popularity & Similarity

37. Diversity – Filtering bubble and echo chamber

38. Fairness – Long tail and Minority # views (attention) popular unpopular

39. Fairness – Long tail and Minority Recommender system better worse Yao, Sirui, and Bert Huang. "Beyond Parity: Fairness Objectives for Collaborative Filtering." arXiv preprint arXiv:1705.08804 (2017).

40. Incorporating diversity and fairness into recommendations (C/R/Res/Adversarial/Rei nforcement) NN User Item Interaction Optimization (C/R/Res/Adversarial/Rei nforcement) NN (C/R/Res/Adversarial/Rei nforcement) NN Penalize homogeneous and unfair recommendations

41. Immersive Recommendation

42. Immersive Recommendation Deep understanding of users’ diverse digital traces Deep modeling of heterogeneous contents + = News Events Food Art Spoken word

43. Immersive Recommendation Deep understanding of users’ diverse digital traces Deep modeling of heterogeneous contents + = News Events Food Art Spoken word

44. Yum-me Bringing healthiness into the recommendation of food Yang, Longqi, et al. "Yum-Me: A Personalized Nutrient-Based Meal Recommender System." ACM Transactions on Information Systems (TOIS) 36.1 (2017): 7.

45. *Number of Americans Living with Diet-and Inactivity-Related Diseases Obesity HBP Diabetes 113M 50M 15M Critical Issue of Food

46. The problem is not awareness, but adherence How can we (efficiently) find meals that are healthy but also cater to people’s tastes? - Bringing the notion of healthiness into recommendations!

47. Yum-me: An interactive healthy meal recommendation system Take a look at the food below and tap all that look delicious to you. http:// http:// Compare the food pair below and tap on whichever looks delicious to you. Press on Yuck if neither of them fits to your taste 2iters + 13iters 2iters + 13iters 2iters + 13iters Browser Mobile Wearable Personal Dietary Profile (Food Preferences) … … Healthy meal recommendations based on dietary restrictions Re-ranking Personalized healthy meal recommendations …... …... Phase I Phase II Take a look at the food below and tap all that look delicious to you. Compare the food pair below and tap on whichever looks delicious to you. Press on Yuck if neither of them fits to your taste Take a look at the food below and tap all that look delicious to you. Compare the food pair below and tap on whichever looks delicious to you. Press on Yuck if neither of them fits to your taste http:// Choose the closest diet type to you. ⌾No restrictions ⌾ Vegetarian⌾ Vegan ⌾ Kosher ⌾ Halal Identify your health goals. ⌾Reduce ⌾ Maintain ⌾ Increase ⌾Reduce ⌾ Maintain ⌾ Increase ⌾Reduce ⌾ Maintain ⌾ Increase Calories Protein Fat + Survey Choose the closest diet type to you. Identify your health goals. ⌾Reduce ⌾ Maintain Calories ⌾No restrictions ⌾ Vegetarian ⌾ Vegan ⌾ Kosher ⌾ Halal + + Choose the closest diet type to you. ⌾No restrictions ⌾ Vegetarian ⌾ Vegan ⌾ Kosher ⌾ Halal

49. Interactive Learning Process propagate

50. Interactive Learning Process selection

51. Similarity between Food Items 0.9151 0.6471 0.9652 1.3484 1.3410 1.3484 1.1476 Siamese Network

52. Similarity between Food Items (a) No restrictions (b) Vegetarian

54. Re-ranking healthy unhealthy Re-ranked by an user’s preference Top N recommendations

55. User study Step 1. Users identify their diet types and health goals. Step 2. Users use visual interace to express their fine-grained food preferences. Step 3. Users identify each of recommended meals as either Yummy or No way. (The order of the items is randomized) Top 500 healthy items that meet users’ diet types and health goals. Select top 10 items ranked by user’s fine-grained dietary preference. Randomly select 10 food items from 500 healthy meal pool. …... …... …...…...

56. Acceptance Rate 51% 72.5% Traditional Approach Yum-me

57. User study Goal: reduce calories (25 users) Goal: maintain calories (21 users) Goal: maintain protein (36 users) Goal: increase protein (12 users) Goal: reduce fat (17 users) Goal: increase calories (2 users) Goal: maintain fat (30 users) users’ 20 favorite meals meals recommended by Yum-me and accepted by users. Averageamountofnutrients perserving(normalized) Averageamountofnutrients perserving(normalized) Averageamountofnutrients perserving(normalized)

58. Creative Content Recommendation Bringing unstructured command traces into the recommendation of art Yang, Longqi, et al. "Personalizing Software and Web Services by Integrating Unstructured Application Usage Traces." Proceedings of the 26th International Conference on World Wide Web Companion. International World Wide Web Conferences Steering Committee, 2017.

59. Cold-start creative content recommendation Day to day work activities (Commands performed in professional design software)

60. … FC FC FC FC Two-step recommendation algorithm

61. utilization-to-vector (util2vec) action action action action action Representation Learning Paradigm

62. utilization-to-vector (util2vec) - sliding window sliding window

63. utilization-to-vector (util2vec) - sliding window sliding window

64. utilization-to-vector (util2vec) - inside each window prediction target 2n+1 actions (n=4) predictor inputs

65. utilization-to-vector (util2vec) - inside each window prediction target 2n+1 actions (n=4) predictor inputs

66. Concatenation/Average Softmax Predictor utilization-to-vector (util2vec) - predictor

67. Evaluation

68. OpenRec An open source and modular framework for extensible recommendation algorithms

69. Recap (C/R/Res/Adversarial/Rei nforcement) NN User Item Interaction Optimization (C/R/Res/Adversarial/Rei nforcement) NN (C/R/Res/Adversarial/Rei nforcement) NN

70. Real World Systems can only be more complex User Item Interaction Optimization … … … … … … User Item Interaction User Item Interaction

71. A Traditional (Monolithic) View of Recommender Systems … RS 1 RS 2 RS 3 RS n

72. OpenRec: Experimentation and innovation through Extension rating item text item textitem image rating user text item image item text rating rating user text user demogr item text item image integrator module extractor module interaction module R1 R2 R4R3

73. OpenRec: Architecture Module Extractor IntegratorInteraction BPR WARP PMF CML NeuMF … LF ResNet MLP LSTM FoodDist Concatenation Average Weighted sum …… Recommender News recommender system with users’ click history, Twitter posts and news topic modeling. Music recommender system with users’ listen history, lyrics and audio analysis … Utility Sampler Pairwise sampler Triplet sampler … Evaluator … AUC Recall@K

74. OpenRec: Examples input buff input holders user extractors item extractors interactions optimizerVisualCML Modules VBPR input buff input holders user extractors item extractors interactions optimizer reusable modules reusable functions build from scratch CML Interactionmodule … input buff input holders user extractors item extractors interactions optimizerVanillaBPR Modules reusable modules build from scratch LatentFactor BPR Extractor module Interactionmodule … …

75. Stay tuned, Late Fall: https://github.com/ylongqi/OpenRec OpenRec

76. Thank you! Longqi Yang Ph.D. Student, Computer Science, Cornell Tech, Cornell University Email: ylongqi@cs.cornell.edu Web: bit.ly/longqi Twitter: @ylongqi Connected Experiences Lab: http://cx.jacobs.cornell.edu/ Small Data Lab: http://smalldata.io/