SlideShare una empresa de Scribd logo
1 de 31
Descargar para leer sin conexión
Multiple Objectives in Collaborative Filtering
Tamas Jambor and Jun Wang
University College London
Structure of the talk
• Motivation
• Multiple objectives
• User perspective
– Promoting less popular items
• System perspective
– Stock management
Motivation
• In the RecSys community, many research efforts
are focused on recommendation accuracy
• And yet accuracy is not a only concern
• Practical recommender systems might have
multiple goals
Improved Accuracy != Improved User experience
Algorithm
Additional
factors
Available resources
Cost of delivery
User interface
Diverse choices
Profitability per item
Advertisement
Improved user experience
Available resources
Cost of delivery
User interface
Diverse choices
Profitability per item
Advertisement
Additional
factors
Accuracy
Improved
user
experience
Handling Multiple objectives
• Accuracy is the main objective
– Defined in the baseline algorithm
• User perspective
– Define and consider user satisfaction as priority
• System perspective
– Consider additional system related objectives
• Objectives of the system might contradict
Where to optimize?
• In the objective function or as a post-filter?
• Post-filters have the advantage to
– Add to any baseline algorithm
– Extend easily
– Add multiple goals
The proposed optimization framework
(for each user)
• Add additional constraints of w
0
11:tosubject
ˆmax
w
w
rw
T
T
w

Properties of the framework
• Linear optimization problem
• Recommendation as a ranking problem
• Constraints provide the means of biasing the
ranking
User case – Promoting the Long Tail
Current systems are biased towards popular items
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
1 2 3 4 5 6 7 8 9 10
Probabilityofanyof100mostpopularitembeingat
rankingposition
Ranking Position
SVD
User-based
Item-based
Random Sample
Promoting the Long Tail
• Does that reflect real user needs?
• Popular items might not be interesting for the user
• Discovering unknown item could be more valuable
• The aim is to reduce recommending popular items
– if the user is likely to be an interested in alternative
choices
– keep recommending popular items otherwise
Promoting the Long Tail
• Extending the optimization framework
0
11
:tosubject
ˆmax
w
w
mwm
rw
T
u
T
T
w

 
Promoting the Long Tail and Diversification
• Diversifying the results
0
11
:tosubject
ˆmax
w
w
mwm
wwrw
T
u
T
TT
w





Diversification
• Increase the covariance between recommended
items
– Reduce the risk of expanding the system
– Provide a wider range of choice
Experimental setup
• MovieLens 1m dataset
• 3900 movies, 6040 users
• Five-fold cross validation
Evaluation metrics
• Recommendation as a ranking problem
• IR measures
– Normalized discounted cumulative gain (NDCG)
– Precision
– Mean reciprocal rank (MRR)
• Constraint specific measures
Results: Promoting the Long Tail
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
0.5
1 2 3 4 5 6 7 8 9 10
Probabilityofanyof100mostpopularitembeingat
rankingposition
Ranking Position
Baseline (SVD)
Long Tail Constraint
Long Tail Constraint and
Diversification (λ=6)
Random Sample
Results: Promoting the Long Tail
Baseline (SVD) LTC LTC and Div (λ=6)
NDCG@10 0.8808 0.8780 (-0.3%) 0.8715 (-1.0%)
P@10 0.8204 0.8207 (+0.2%) 0.8177 (-0.3%)
MRR 0.9518 0.9453 (-0.6%) 0.9349 (-1.7%)
System case – Resource Constraint
• Introducing external factors to the system
• Stock availability of recommended items
• The aim is to rank items lower, if less of them are
available
• Minimizing performance loss
Simulation
• Online DVD-Rental company
– Operates a warehouse
– Only a limited number of items are available
• Recommend items that are in stock higher in the
ranking list
Simulation
• User choice is based purely on recommendation
• Simulating the stock level for 50 days
– Present a list of items to a random number of users
– The probability that the item is taken depends on the
rank
– Cumulative probability depends on how many times the
item was shown and at which rank position
Cut-off point
• Threshold c controls the cut-off point from which
the system starts re-ranking items


 


cp
cps
s ti
titi
ti
,
,,
,
if0
if
Resource Constraint
• Extending the optimization framework
0
11
:tosubject
ˆmax


w
w
sws
rw
T
u
T
w


Evaluation: Monitoring the waiting list size
• Waiting list
– If item is not in stock, user puts it on their waiting list
– When item returns, it goes out to the next user
• Waiting list size represents how long a user has to
wait to get their favourite items
Results: Resource Constraint
0
20
40
60
80
100
120
140
160
180
0 5 10 15 20
Numberofitemsonthewaitinglist
Time (days)
baseline c=1.6 c=1.2 c=0.0
Results: Resource Constraint
• Trade-off between low waiting list size and good
performance
0.75
0.77
0.79
0.81
0.83
0.85
0.87
0.89
0.91
0 5 10 15 20
NDCG@3
Time (days)
baseline c=1.6 c=1.2 c=0.0
Results: Resource Constraint
c=0 c=0.4 c=0.8 c=1.2 c=1.6
NDCG@3(mean) -12.3% -4.32% -1.03% -0.43% -0.13%
NDCG@3(max) -14.7% -5.12% -1.34% -0.56% -0.50%
P@10(mean) -6.42% -3.37% -0.86% -0.06% -0.03%
P@10(max) -8.42% -3.91% -1.11% -0.24% -0.18%
Performance loss over 50 days
Conclusion
• Recommender systems have multiple objectives
• Multiple optimization framework
– Expand the system with minor performance loss
– It is designed to add objectives flexibly
– It can be added to any recommender system
• Two scenarios that offer practical solutions
– Long-tail items
– Stock simulation
Future plan
• Personalized digital content delivery
– Reduce delivery cost
• Diversification and the long tail
– Does recommendation kill diversity?
• Evaluate improved user experience
– User studies
Thank you.
References
• Deshpande, M., Karypis, G.: Item-based top-N recommendation algorithms.
ACM Trans. Inf. Syst. 22(1) (2004)
• Herlocker, J.L., Konstan, J.A., Borchers, A., Riedl, J.: An algorithmic
framework for performing collaborative filtering. In: SIGIR '99. (1999)
• Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for
recommender systems. Computer 42(8) (2009)
• Wang, J., de Vries, A.P., Reinders, M.J.T.: Unifying user-based and item-
based collaborative filtering approaches by similarity fusion. In: SIGIR '06:
Proceedings of the 29th annual international ACM SIGIR conference on
Research and development in information retrieval, New York, NY, ACM Press

Más contenido relacionado

Similar a Multiple objectives in Collaborative Filtering (RecSys 2010)

Quality andc apability hand out 091123200010 Phpapp01
Quality andc apability hand out 091123200010 Phpapp01Quality andc apability hand out 091123200010 Phpapp01
Quality andc apability hand out 091123200010 Phpapp01jasonhian
 
Bridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
Bridging the Gap: Machine Learning for Ubiquitous Computing -- EvaluationBridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
Bridging the Gap: Machine Learning for Ubiquitous Computing -- EvaluationThomas Ploetz
 
Customer Churn Analytics using Microsoft R Open
Customer Churn Analytics using Microsoft R OpenCustomer Churn Analytics using Microsoft R Open
Customer Churn Analytics using Microsoft R OpenPoo Kuan Hoong
 
From testing to quality governance and problem resolution.pdf
From testing to quality governance and problem resolution.pdfFrom testing to quality governance and problem resolution.pdf
From testing to quality governance and problem resolution.pdfXavier Escudero Sabadell
 
AWS re:Invent 2016: Getting to Ground Truth with Amazon Mechanical Turk (MAC201)
AWS re:Invent 2016: Getting to Ground Truth with Amazon Mechanical Turk (MAC201)AWS re:Invent 2016: Getting to Ground Truth with Amazon Mechanical Turk (MAC201)
AWS re:Invent 2016: Getting to Ground Truth with Amazon Mechanical Turk (MAC201)Amazon Web Services
 
Web Performance Bootcamp 2014
Web Performance Bootcamp 2014Web Performance Bootcamp 2014
Web Performance Bootcamp 2014Daniel Austin
 
Automated Parameterization of Performance Models from Measurements
Automated Parameterization of Performance Models from MeasurementsAutomated Parameterization of Performance Models from Measurements
Automated Parameterization of Performance Models from MeasurementsWeikun Wang
 
A Top-N Recommender System Evaluation Protocol Inspired by Deployed Systems
A Top-N Recommender System Evaluation Protocol Inspired by Deployed SystemsA Top-N Recommender System Evaluation Protocol Inspired by Deployed Systems
A Top-N Recommender System Evaluation Protocol Inspired by Deployed SystemsAlan Said
 
Web Performance BootCamp 2013
Web Performance BootCamp 2013Web Performance BootCamp 2013
Web Performance BootCamp 2013Daniel Austin
 
When a FILTER makes the di fference in continuously answering SPARQL queries ...
When a FILTER makes the difference in continuously answering SPARQL queries ...When a FILTER makes the difference in continuously answering SPARQL queries ...
When a FILTER makes the di fference in continuously answering SPARQL queries ...Shima Zahmatkesh
 
Big Data Day LA 2016/ Use Case Driven track - Shaping the Role of Data Scienc...
Big Data Day LA 2016/ Use Case Driven track - Shaping the Role of Data Scienc...Big Data Day LA 2016/ Use Case Driven track - Shaping the Role of Data Scienc...
Big Data Day LA 2016/ Use Case Driven track - Shaping the Role of Data Scienc...Data Con LA
 
Cassandra Applications Benchmarking
Cassandra Applications BenchmarkingCassandra Applications Benchmarking
Cassandra Applications Benchmarkingniallmilton
 
Argumentation in Artificial Intelligence: From Theory to Practice (Practice)
Argumentation in Artificial Intelligence: From Theory to Practice (Practice)Argumentation in Artificial Intelligence: From Theory to Practice (Practice)
Argumentation in Artificial Intelligence: From Theory to Practice (Practice)Mauro Vallati
 
Virtual segment brief
Virtual segment briefVirtual segment brief
Virtual segment briefAhmed Shaaban
 
CBM Requirements by Carl Byington - PHM Design, LLC
CBM Requirements by Carl Byington - PHM Design, LLCCBM Requirements by Carl Byington - PHM Design, LLC
CBM Requirements by Carl Byington - PHM Design, LLCCarl Byington
 
Machine learning algorithm for classification of activity of daily life’s
Machine learning algorithm for classification of activity of daily life’sMachine learning algorithm for classification of activity of daily life’s
Machine learning algorithm for classification of activity of daily life’sSiddharth Chakravarty
 

Similar a Multiple objectives in Collaborative Filtering (RecSys 2010) (20)

Quality andc apability hand out 091123200010 Phpapp01
Quality andc apability hand out 091123200010 Phpapp01Quality andc apability hand out 091123200010 Phpapp01
Quality andc apability hand out 091123200010 Phpapp01
 
Bridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
Bridging the Gap: Machine Learning for Ubiquitous Computing -- EvaluationBridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
Bridging the Gap: Machine Learning for Ubiquitous Computing -- Evaluation
 
Six sigma11
Six sigma11Six sigma11
Six sigma11
 
Customer Churn Analytics using Microsoft R Open
Customer Churn Analytics using Microsoft R OpenCustomer Churn Analytics using Microsoft R Open
Customer Churn Analytics using Microsoft R Open
 
From testing to quality governance and problem resolution.pdf
From testing to quality governance and problem resolution.pdfFrom testing to quality governance and problem resolution.pdf
From testing to quality governance and problem resolution.pdf
 
AWS re:Invent 2016: Getting to Ground Truth with Amazon Mechanical Turk (MAC201)
AWS re:Invent 2016: Getting to Ground Truth with Amazon Mechanical Turk (MAC201)AWS re:Invent 2016: Getting to Ground Truth with Amazon Mechanical Turk (MAC201)
AWS re:Invent 2016: Getting to Ground Truth with Amazon Mechanical Turk (MAC201)
 
Training - What is Performance ?
Training  - What is Performance ?Training  - What is Performance ?
Training - What is Performance ?
 
Web Performance Bootcamp 2014
Web Performance Bootcamp 2014Web Performance Bootcamp 2014
Web Performance Bootcamp 2014
 
Automated Parameterization of Performance Models from Measurements
Automated Parameterization of Performance Models from MeasurementsAutomated Parameterization of Performance Models from Measurements
Automated Parameterization of Performance Models from Measurements
 
A Top-N Recommender System Evaluation Protocol Inspired by Deployed Systems
A Top-N Recommender System Evaluation Protocol Inspired by Deployed SystemsA Top-N Recommender System Evaluation Protocol Inspired by Deployed Systems
A Top-N Recommender System Evaluation Protocol Inspired by Deployed Systems
 
Mini datathon
Mini datathonMini datathon
Mini datathon
 
Web Performance BootCamp 2013
Web Performance BootCamp 2013Web Performance BootCamp 2013
Web Performance BootCamp 2013
 
When a FILTER makes the di fference in continuously answering SPARQL queries ...
When a FILTER makes the difference in continuously answering SPARQL queries ...When a FILTER makes the difference in continuously answering SPARQL queries ...
When a FILTER makes the di fference in continuously answering SPARQL queries ...
 
Big Data Day LA 2016/ Use Case Driven track - Shaping the Role of Data Scienc...
Big Data Day LA 2016/ Use Case Driven track - Shaping the Role of Data Scienc...Big Data Day LA 2016/ Use Case Driven track - Shaping the Role of Data Scienc...
Big Data Day LA 2016/ Use Case Driven track - Shaping the Role of Data Scienc...
 
Cassandra Applications Benchmarking
Cassandra Applications BenchmarkingCassandra Applications Benchmarking
Cassandra Applications Benchmarking
 
Argumentation in Artificial Intelligence: From Theory to Practice (Practice)
Argumentation in Artificial Intelligence: From Theory to Practice (Practice)Argumentation in Artificial Intelligence: From Theory to Practice (Practice)
Argumentation in Artificial Intelligence: From Theory to Practice (Practice)
 
rerngvit_phd_seminar
rerngvit_phd_seminarrerngvit_phd_seminar
rerngvit_phd_seminar
 
Virtual segment brief
Virtual segment briefVirtual segment brief
Virtual segment brief
 
CBM Requirements by Carl Byington - PHM Design, LLC
CBM Requirements by Carl Byington - PHM Design, LLCCBM Requirements by Carl Byington - PHM Design, LLC
CBM Requirements by Carl Byington - PHM Design, LLC
 
Machine learning algorithm for classification of activity of daily life’s
Machine learning algorithm for classification of activity of daily life’sMachine learning algorithm for classification of activity of daily life’s
Machine learning algorithm for classification of activity of daily life’s
 

Último

Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Adtran
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1DianaGray10
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarPrecisely
 

Último (20)

Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™Meet the new FSP 3000 M-Flex800™
Meet the new FSP 3000 M-Flex800™
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
20150722 - AGV
20150722 - AGV20150722 - AGV
20150722 - AGV
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1UiPath Platform: The Backend Engine Powering Your Automation - Session 1
UiPath Platform: The Backend Engine Powering Your Automation - Session 1
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
AI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity WebinarAI You Can Trust - Ensuring Success with Data Integrity Webinar
AI You Can Trust - Ensuring Success with Data Integrity Webinar
 

Multiple objectives in Collaborative Filtering (RecSys 2010)

  • 1. Multiple Objectives in Collaborative Filtering Tamas Jambor and Jun Wang University College London
  • 2. Structure of the talk • Motivation • Multiple objectives • User perspective – Promoting less popular items • System perspective – Stock management
  • 3. Motivation • In the RecSys community, many research efforts are focused on recommendation accuracy • And yet accuracy is not a only concern • Practical recommender systems might have multiple goals
  • 4. Improved Accuracy != Improved User experience Algorithm Additional factors Available resources Cost of delivery User interface Diverse choices Profitability per item Advertisement
  • 5. Improved user experience Available resources Cost of delivery User interface Diverse choices Profitability per item Advertisement Additional factors Accuracy Improved user experience
  • 6. Handling Multiple objectives • Accuracy is the main objective – Defined in the baseline algorithm • User perspective – Define and consider user satisfaction as priority • System perspective – Consider additional system related objectives • Objectives of the system might contradict
  • 7. Where to optimize? • In the objective function or as a post-filter? • Post-filters have the advantage to – Add to any baseline algorithm – Extend easily – Add multiple goals
  • 8. The proposed optimization framework (for each user) • Add additional constraints of w 0 11:tosubject ˆmax w w rw T T w 
  • 9. Properties of the framework • Linear optimization problem • Recommendation as a ranking problem • Constraints provide the means of biasing the ranking
  • 10. User case – Promoting the Long Tail Current systems are biased towards popular items 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 1 2 3 4 5 6 7 8 9 10 Probabilityofanyof100mostpopularitembeingat rankingposition Ranking Position SVD User-based Item-based Random Sample
  • 11. Promoting the Long Tail • Does that reflect real user needs? • Popular items might not be interesting for the user • Discovering unknown item could be more valuable • The aim is to reduce recommending popular items – if the user is likely to be an interested in alternative choices – keep recommending popular items otherwise
  • 12. Promoting the Long Tail • Extending the optimization framework 0 11 :tosubject ˆmax w w mwm rw T u T T w   
  • 13. Promoting the Long Tail and Diversification • Diversifying the results 0 11 :tosubject ˆmax w w mwm wwrw T u T TT w     
  • 14. Diversification • Increase the covariance between recommended items – Reduce the risk of expanding the system – Provide a wider range of choice
  • 15. Experimental setup • MovieLens 1m dataset • 3900 movies, 6040 users • Five-fold cross validation
  • 16. Evaluation metrics • Recommendation as a ranking problem • IR measures – Normalized discounted cumulative gain (NDCG) – Precision – Mean reciprocal rank (MRR) • Constraint specific measures
  • 17. Results: Promoting the Long Tail 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 1 2 3 4 5 6 7 8 9 10 Probabilityofanyof100mostpopularitembeingat rankingposition Ranking Position Baseline (SVD) Long Tail Constraint Long Tail Constraint and Diversification (λ=6) Random Sample
  • 18. Results: Promoting the Long Tail Baseline (SVD) LTC LTC and Div (λ=6) NDCG@10 0.8808 0.8780 (-0.3%) 0.8715 (-1.0%) P@10 0.8204 0.8207 (+0.2%) 0.8177 (-0.3%) MRR 0.9518 0.9453 (-0.6%) 0.9349 (-1.7%)
  • 19. System case – Resource Constraint • Introducing external factors to the system • Stock availability of recommended items • The aim is to rank items lower, if less of them are available • Minimizing performance loss
  • 20. Simulation • Online DVD-Rental company – Operates a warehouse – Only a limited number of items are available • Recommend items that are in stock higher in the ranking list
  • 21. Simulation • User choice is based purely on recommendation • Simulating the stock level for 50 days – Present a list of items to a random number of users – The probability that the item is taken depends on the rank – Cumulative probability depends on how many times the item was shown and at which rank position
  • 22. Cut-off point • Threshold c controls the cut-off point from which the system starts re-ranking items       cp cps s ti titi ti , ,, , if0 if
  • 23. Resource Constraint • Extending the optimization framework 0 11 :tosubject ˆmax   w w sws rw T u T w  
  • 24. Evaluation: Monitoring the waiting list size • Waiting list – If item is not in stock, user puts it on their waiting list – When item returns, it goes out to the next user • Waiting list size represents how long a user has to wait to get their favourite items
  • 25. Results: Resource Constraint 0 20 40 60 80 100 120 140 160 180 0 5 10 15 20 Numberofitemsonthewaitinglist Time (days) baseline c=1.6 c=1.2 c=0.0
  • 26. Results: Resource Constraint • Trade-off between low waiting list size and good performance 0.75 0.77 0.79 0.81 0.83 0.85 0.87 0.89 0.91 0 5 10 15 20 NDCG@3 Time (days) baseline c=1.6 c=1.2 c=0.0
  • 27. Results: Resource Constraint c=0 c=0.4 c=0.8 c=1.2 c=1.6 NDCG@3(mean) -12.3% -4.32% -1.03% -0.43% -0.13% NDCG@3(max) -14.7% -5.12% -1.34% -0.56% -0.50% P@10(mean) -6.42% -3.37% -0.86% -0.06% -0.03% P@10(max) -8.42% -3.91% -1.11% -0.24% -0.18% Performance loss over 50 days
  • 28. Conclusion • Recommender systems have multiple objectives • Multiple optimization framework – Expand the system with minor performance loss – It is designed to add objectives flexibly – It can be added to any recommender system • Two scenarios that offer practical solutions – Long-tail items – Stock simulation
  • 29. Future plan • Personalized digital content delivery – Reduce delivery cost • Diversification and the long tail – Does recommendation kill diversity? • Evaluate improved user experience – User studies
  • 31. References • Deshpande, M., Karypis, G.: Item-based top-N recommendation algorithms. ACM Trans. Inf. Syst. 22(1) (2004) • Herlocker, J.L., Konstan, J.A., Borchers, A., Riedl, J.: An algorithmic framework for performing collaborative filtering. In: SIGIR '99. (1999) • Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender systems. Computer 42(8) (2009) • Wang, J., de Vries, A.P., Reinders, M.J.T.: Unifying user-based and item- based collaborative filtering approaches by similarity fusion. In: SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, New York, NY, ACM Press

Notas del editor

  1. Background statement Explain what we mean by multiple objectives in this context Give two examples Motivated from the user point of view Motivated from the system point of view User point of view would improve the user experience System point of view would take into account other external factors
  2. Background statement Explain what we mean by multiple objectives in this context Give two examples Motivated from the user point of view Motivated from the system point of view User point of view would improve the user experience System point of view would take into account other external factors
  3. Improving the accuracy of the system does not necessarily equal to improving user experience Defined the recommender algorithm how good it performs on a metrics and how fast it can do its job But we need additional factors that would define the whole system For example if we take a VOD service User interface affects how user reads, understands find information including the recommendation How interesting the movies are that the user gets System related factors include The underlying hardware of the system How much, how fast an item can be delivered External factors include How much the company earns on an item, for example some items have a higher profit margin Personalised advertisement, recommending preferred items
  4. User related and system related factors can directly improve user experience And we suggest that the combination of accuracy and other factors might improve user experience And the rest can help to maximize profit 
  5. Consider accuracy as the main objective
  6. Combine these into a single optimization framework. Take the prediction values of any baseline algorithm - r-hat. And add additional constraints that are equally important. Baseline algorithm to minimize errors. Higher weights represent higher importance Without constraints it returns the same as the original order
  7. Recommendation is considered to be a ranking problem Top-N list It is a linear optimization problem, so global solution can be found. Biasing the original order by using constraints
  8. Introducing two case studies to illustrate the use of the framework The first one is aim to promote items from the long tail. Assumption Current systems are biased towards popular items Picked the first 100 items that have received the highest ratings Where SVD is likely to place them Plot the distribution Figure show the probability that some popular are placed higher on the recommendation list by all the widely used recommender algorithms
  9. Does that reflect user needs? We assume that discovering unknown items are more valuable We aim to identify users who would like alternative choices And recommend from the long tail for them Keep recommending popular items if the user has a more mainstream taste
  10. We add this as an inequality constraint to the framework m is a vector that contains the mainstreamness value of each items in the recommended list m_u measures the mainstreamness of the user How these values are calculated are in the paper if you are interested
  11. We added another extra bit to the framework to diversify items
  12. Experimenting with diversification Promote item from the long tail that differ from each other Higher covariance That would reduce the risk of such an extension
  13. Movielens 1m dataset Around 4000 movies, 6000 users Five-fold cross validation
  14. Since we approach recommendation as a ranking problem We used the following IR measures
  15. The probability that popular items ranked higher is significantly reduced Only 32% of the users have popular items in the top position The baseline is 45% We get reduction until position four Then it is slightly worse than the baseline This is the case when user studies would provide a better way to evaluate performance
  16. Long tail constraint alone Long tail constraint with diversification Slight performance loss for all measures, except one.
  17. The other case – system case Adding other non-user related factors to the system the availably of certain items The aim is to rank items lower if we are about to run out of stock But also minimize performance loss
  18. The second scenario was evaluated by simulating the stock level Of an imaginary company for 50 days We presented a recommendation list to a random number of users each day The probability that a user took an item depended on the rank The cumulative probability up to the present point was based on a, How many times an item was show in the past B, And at which ranking position To evaluate the system we monitored the waiting list size
  19. We introduce a cut-off point from which the system will start to reorganize the ranking list.
  20. Adding it as a constraint to the system, in a same fashion s is a vector that represents the probability that the item is available at the given time For each item in the list s_u controls how cautious the system should be with stock level. e.g if s_u is higher the system starts penalizing items later As they are getting less available
  21. The experiment is designed to run out of stock This graph shows the waiting list size for the first 20 days With respect to parameter c It controls when the system should start penalizing items as they are running out of stock e.g if c is set to 0.8 then after it is 80% likely that an item is taken the system starts penalizing
  22. This table show the average and the maximum performance loss per day As you can see from c=0.8 and above there is only a slight performance loss of the system.
  23. Two papers Optimizing algorithm from the user point of view One way is to identify different errors A general framework to handle multiple goals Two scenarios to illustrate that
  24. Improving user experience might be validated using user studies