Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Recommending and Searching (Research @ Spotify)

1.808 visualizaciones

Publicado el

These are the slides of a talk about some of our research at Spotify, as part of the celebration kickoff of Chalmers AI Research Centre in Gothenburg. I always like to make a story in my talk, and this time I wanted to reflect on the "push" (think recommender system) and "pull" (think search) paradigms. I am using this quote from Nicholas Belkin and Bruce Croft from their Communications of the ACM article published in 1992 to frame my story: "We conclude that information retrieval and information filtering are indeed two sides of the same coin. They work together to help people get the information needed to perform their tasks."

Publicado en: Tecnología
  • Sé el primero en comentar

Recommending and Searching (Research @ Spotify)

  1. 1. Recommending and Searching Research @ Spotify Mounia Lalmas Chalmers University of Technology, 4-5 March 2019
  2. 2. Making AI works at Spotify Qualitativeresearch Businessmetrics Algorithm(s) Training & Datasets Metric(s) Evaluation offline and online Interaction & feedbacks data Features (item) Features (user) Features (context)
  3. 3. What we do at Spotify
  4. 4. Spotify’s mission is to unlock the potential of human creativity — by giving a million creative artists the opportunity to live off their art and billions of fans the opportunity to enjoy and be inspired by it.
  5. 5. http://everynoise.com/
  6. 6. Our team mission: Match fans and artists in a personal and relevant way. ARTISTS FANS
  7. 7. songs playlists podcasts ... catalog search browse talk users What does it mean to match fans and artists in a personal and relevant way?Artists Fans
  8. 8. “We conclude that information retrieval and information filtering are indeed two sides of the same coin. They work together to help people get the information needed to perform their tasks.” Information filtering and information retrieval: Two sides of the same coin? NJ Belkin & WB Croft, Communications of the ACM, 1992.
  9. 9. “We can conclude that recommender systems and search are also two sides of the same coin at Spotify. They work together to help fans get the music they will enjoy listening”. PULL PARADIGM PUSH PARADIGM is this the case?
  10. 10. Home … the push paradigm
  11. 11. Home Home is the default screen of the mobile app for all Spotify users worldwide. It surfaces the best of what Spotify has to offer, for every situation, personalized playlists, new releases, old favorites, and undiscovered gems. Help users find something they are going to enjoy listening to, quickly.
  12. 12. Streaming UserBaRT Explore, Exploit, Explain: Personalizing Explainable Recommendations with Bandits. J McInerney, B Lacker, S Hansen, K Higley, H.Bouchard, A Gruson & R Mehrotra, RecSys 2018. BaRT: Machine learning algorithm for Spotify Home
  13. 13. BaRT (Bandits for Recommendations as Treatments) How to rank playlists (cards) in each shelf first, and then how to rank the shelves?
  14. 14. https://hackernoon.com/reinforcement-learning-part-2-152fb510cc54 Explore vs Exploit Flip a coin with given probability of tail If head, pick best card in M according to predicted reward r → EXPLOIT If tail, pick card from M at random → EXPLORE BaRT: Multi-armed bandit algorithm for Spotify Home
  15. 15. Success is captured by the reward function Reward Binarised Streaming Time Success is when user streams the playlist for at least 30s. BaRT UserStreaming
  16. 16. Is success the same for all playlists? Consumption time of a sleep playlist is longer than average playlist consumption time. Jazz listeners consume Jazz and other playlists for longer period than average users.
  17. 17. one reward function for all users and all playlists success independent of user and playlist one reward function per user x playlist success depends on user and playlist too granular, sparse, noisy, costly to generate & maintain one reward function per group of users x playlists success depends on group of users listening to group of playlists Personalizing the reward function for BaRT
  18. 18. Co-clustering Co-clustering Dhillon, Mallela & Modha, "Information-theoretic co-clustering”, KDD 2003. Caveat no theoretical foundation for selecting the number of co-clusters apriori group = cluster group of user x playlist = co-cluster
  19. 19. Co-clustering for Spotify Home Users Playlists User groups Playlist groups Any (interaction) signal can be used to generate the co-clusters.
  20. 20. Reward function per co-cluster using distribution of streaming time continuousadditivemean
  21. 21. Counterfactual methodology, which works like offline A/B Thresholding methods: mean, additive, continuous & random Baseline (one threshold), playlists only, users only, both One week of random sample of 800K+ users, 900K+ playlists, 8M user-playlist interactions Expected stream rate Experiments Deriving User- and Content-specific Rewards for Contextual Bandits. P Dragone, R Mehrotra & M Lalmas, WWW 2019.
  22. 22. Conclusions Accounting for user experience and playlist consumption matters. Co-clustering users and playlists surfaces patterns of user experience x playlist consumption. Using one interaction signal and a simple thresholding method can already provide effective personalised success metrics.
  23. 23. Metric 1 Metric 2 Metric 3 Multiple objective functionsRecommendation in a 2-sided Marketplace
  24. 24. ● Policy I: Optimizing Relevance ● Policy II: Optimizing Fairness ● Policy III: Probabilistic Policy ● Policy IV: Trade-off Relevance & Fairness ● Policy V: Guaranteed Relevance ● Policy VI: Adaptive Policy I ● Policy VI: Adaptive Policy II “Fairness” Relevance Optimising for fairness and satisfaction at the same time Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems. R Mehrotra, J McInerney, H Bouchard, M Lalmas & F Diaz, CIKM 2018. Recommendation in a 2-sided Marketplace
  25. 25. ML Lab An offline evaluation framework to launch, evaluate and archive machine learning studies, ensuring reproducibility and allowing sharing across teams. Offline Evaluation to Make Decisions About Playlist Recommendation Algorithms. A Gruson, P Chandar, C Charbuillet, J McInerney, S Hansen, D Tardieu & B Carterette, WSDM 2019. Offline evaluation for Home
  26. 26. Search … pull & push paradigms
  27. 27. Searching for music
  28. 28. Overview of the user journey in search TYPE/TALK User communicates with us CONSIDER User evaluates what we show them DECIDE User ends the search session INTENT What the user wants to do MINDSET How the user thinks about results
  29. 29. FOCUSED One specific thing in mind OPEN A seed of an idea in mind EXPLORATORY A path to explore ● Find it or not ● Quickest/easiest path to results is important ● From nothing good enough, good enough to better than good enough ● Willing to try things out ● But still want to fulfil their intent ● Difficult for users to assess how it went ● May be able to answer in relative terms ● Users expect to be active when in an exploratory mindset ● Effort is expected How the user thinks about results Just Give Me What I Want: How People Use and Evaluate Music Search. C Hosey, L Vujović, B St. Thomas, J Garcia-Gathright & J Thom, CHI 2019.
  30. 30. Focused mindset Search Mindsets: Understanding Focused and Non-Focused Information Seeking in Music Search. A Li, J Thom, P Ravichandran, C Hosey, B St. Thomas & J Garcia-Gathright, WWW 2019. 65% of searches were focused. When users search with a Focused Mindset Put MORE effort in search. Scroll down and click on lower rank results. Click MORE on album/track/artist and LESS on playlist. MORE likely to save/add but LESS likely to stream directly. Understanding mindset helps us understand search satisfaction. When users know what they want to find. The pull paradigm and how it translates to the music context. Findings from large-scale in-app survey + behavioral analysis.
  31. 31. Search by voice A type of push paradigm and how it translates to the music context. Findings from qualitative research. Users ask for Spotify to play music, without saying what they would like to hear (open mindset)
  32. 32. Search as recommendation Delivering for the open mindset. Conversational search. Non-specific querying is a way for a user to effortlessly start a listening session via voice. Non-specific querying is a way to remove the burden of choice when a user is open to lean-back listening. User education matters as users will not engage in a use-case they do not know about. Trust and control are central to a positive experience. Users need to trust the system enough to try it out.
  33. 33. Conclusions Focused mindset is a typical and common case of pull paradigm. Understanding the focus mindset can inform measures of search satisfaction. Open mindset is important for discovery and lean-back experiences. Conversational search (Voice) allows for pull & push paradigms if done right.
  34. 34. Some final words
  35. 35. Making AI works at Spotify Qualitativeresearch Businessmetrics Algorithm(s) Training & Datasets Metric(s) Evaluation offline and online Interaction & feedbacks data Features (item) Features (user) Features (context)
  36. 36. Making AI works at Spotify … in this talk Qualitativeresearch Businessmetrics Algorithm(s) Training & Datasets Metric(s) Evaluation offline and online Interaction & feedbacks data Features (item) Features (user) Features (context) BaRT ML-Lab Rewardfunction forBaRT Focused mindset in search 2-side marketplace Conversational search (Voice)
  37. 37. Thank you!

×