The document summarizes research on recommender systems in the media industry. It discusses how FD Mediagroup uses recommender systems for their SMART Radio and SMART Journalism products. Key aspects of building a recommender system that FD focuses on include relevance, usefulness, and trust. Relevance is evaluated using metrics like NDCG, MAP, and R-Precision. Usefulness considers both algorithmic goals like diversity and business goals. Trust is evaluated based on whether users engage with the recommender system.
Chintamani Call Girls: 🍓 7737669865 🍓 High Profile Model Escorts | Bangalore ...
RecSys in the Media Industry: Relevance, Recency, Popularity, and Diversity.
1. Recommender Systems in the Media Industry:
Relevance, Recency, Popularity, and Diversity
! David Graus
✉ david.graus@fdmediagroep.nl
🐦 @dvdgrs
2. David Graus / Recsys Summer School / 11/09/2019
whoami !
• 🎓 Academia
• BA Media Studies @ UvA (2008)
• MSc Media Technology @ Universiteit Leiden (2012)
• PhD Information Retrieval @ UvA (2017)
• 🏢 Industry
• Editor radio/online public broadcaster NTR (between BA & MSc)
• Research Intern @ Microsoft Research, US
• Data Scientist @ FD Mediagroep
• Lead Data Scientist @ FD SMART Journalism / BNR SMART Radio
2
3. David Graus / Recsys Summer School / 11/09/2019
In what is to follow…
• An introduction of FD Mediagroep
• Personalization & RecSys at FD Mediagroep:
• SMART Radio
• SMART Journalism
• Lessons learnt building/productizing a RecSys;
• Relevance
• Usefulness
• Trust
3
15. David Graus / Recsys Summer School / 11/09/2019
SMART Radio
• (Transcribe)
• Segment
• Tag
• Serve
15
16. David Graus / Recsys Summer School / 11/09/2019
Transcribe
16
17. David Graus / Recsys Summer School / 11/09/2019
Segment
• Based on metadata,
text, and audio.
17
18. David Graus / Recsys Summer School / 11/09/2019
Tag
• Simple multilabel text
classifier
• Trained on transcripts of
segments + associated tags
from website
18
19. David Graus / Recsys Summer School / 11/09/2019
Serve
• iOS/Android
app
19
31. David Graus / Recsys Summer School / 11/09/2019
Goal
• Help users to discover content [1]
31Meyer, F. Recommender systems in industrial contexts (2012)
34. David Graus / Recsys Summer School / 11/09/2019
Building a RecSys for FD
• Definition of Done:
Our RecSys needs to provide relevant and useful results,
and be trustworthy.
• Relevance: IR evaluation metrics
• Usefulness: What we want our RecSys to achieve
• Trust: Whether we see that people like/use our RecSys
34
35. David Graus / Recsys Summer School / 11/09/2019
1. Relevance
• Using different (ranking) metrics, e.g.;
• NDCG
• MAP
• RPrec
• Evaluating models offline
• Comparing offline vs. online performance
35
36. David Graus / Recsys Summer School / 11/09/2019
Relevance: NDCG
• Item’s score: Gain
• Sum scores: Cumulative Gain
• Penalize scores at lower ranks: Discounted Cumulative Gain
• (e.g., divide CG by log of item position)
• Normalized by “ideal” ranking: NDCG
36http://fastml.com/evaluating-recommender-systems/
37. David Graus / Recsys Summer School / 11/09/2019
MAP
• Precision: TP / TP+FP
• Average Precision: Precision at each rank
• Mean Average Precision: average AP across
queries
37
https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html
38. David Graus / Recsys Summer School / 11/09/2019
R-Precision
• Precision@k
• where k = R (= number of relevant documents)
38https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html
39. David Graus / Recsys Summer School / 11/09/2019 https://github.com/cvangysel/pytrec_eval
But…
• It’s all we can do…
39Garcin et al., Offline and Online Evaluation of News Recommender Systems at swissinfo.ch (RecSys 2014)
40. David Graus / Recsys Summer School / 11/09/2019
2. Usefulness
• 🤖 Part algorithmic
• What can we build?
• 💰 Part business/values
• What do we want our RecSys to do?
40
42. David Graus / Recsys Summer School / 11/09/2019
Popularity
• “It is generally not useful to recommend very popular items as they
are generally already known by the user” [1]
• Presentation bias
• Skewed clicks
• Issue in historic data and online evaluation
42[1] Meyer, F. Recommender systems in industrial contexts (2012)
43. David Graus / Recsys Summer School / 11/09/2019
Coverage
• Unleash the long tail?
43
Most read:
Same 6 articles for everyone
Personalized:
178 articles
44. David Graus / Recsys Summer School / 11/09/2019
💰 Business: What do we want/expect?
• ⚠ WIP Study where journalists, data scientists, product,
developers were interviewed to identify shared
perceptions, expectations, attitudes towards
algorithms.
44
45. David Graus / Recsys Summer School / 11/09/2019
Business: perception of algorithms
• Distinguish between audience & and journalism-related values.
• Results TBD, but think of aspects such as:
• broadness vs. personalizedness (audience)
• usability (audience)
• objectivity/neutrality (journalism)
• speed/immediacy (journalism)
• etc.
45
48. David Graus / Recsys Summer School / 11/09/2019
Short story
• Tailor-made RecSys: we are in control
• RecSys as “yet another ranking”
• The slightly longer story…
48
50. David Graus / Recsys Summer School / 11/09/2019
1. Should we worry?
[We] focus on empirical evidence of the spread of personalised
news services and its likely effects on political polarisation and
political information.
50[Zuiderveen Borgesius et al., 2016]
51. David Graus / Recsys Summer School / 11/09/2019
1. Should we worry?
• It’s difficult to bubble yourself
• Both offline:
• “Those who use a lot of partisan information also use an above-average
amount of mainstream news.”
• “[M]ost people by far still get their news via traditional sources, most
notably public-service television.”
• And online:
• “People who choose personalisation are more likely to use an above-average
amount of general-interest news as well.”
• “A recent study suggests that the influence of [the Facebook] algorithm is
lower than the influence of the user’s choices.”
51[Zuiderveen Borgesius et al., 2016]
52. David Graus / Recsys Summer School / 11/09/2019
1. ⚠ Take homes
• “[T]here is no empirical evidence that warrants any strong
worries about filter bubbles.”
• “One lesson we should have learned from the past is that panic
does not lead to sane policies. More evidence is needed on the
process and effects of personalisation, so we can shift the basis of
policy discussions from fear to insight.”
52[Zuiderveen Borgesius et al., 2016]
53. David Graus / Recsys Summer School / 11/09/2019
2. Some empirical evidence…
• On average, 11.7% of results show differences due to
personalization on Google.
• Varies widely by search query and by result ranking.
• Only found measurable personalization as a result of searching
with a logged in account and the IP address of the searching user.
53
54. David Graus / Recsys Summer School / 11/09/2019
2. Method
1. 👤
1. Get 200 volunteers with Google accounts
2. Have them issue the same set of queries
3. Compare results
2. 🤖
1. Construct Google bot accounts
• Vary aspects such as location, demographics, click behavior, browsing + search
history, etc.
2. Have them issue the same set of queries
3. Compare results
54[Hannák et al., 2013]
55. David Graus / Recsys Summer School / 11/09/2019
2. Findings 👤
• On average, 11.7% of results show differences due to
personalization on Google.
• Top ranks tend to be less personalized than bottom ranks.
55[Hannák et al., 2013]
56. David Graus / Recsys Summer School / 11/09/2019
2. Findings 👤
• ✅ A great deal of personalization based on location
(especially for company names, where users received different store locations).
• ❌ The least personalized results tend to be factual and health related queries.
56[Hannák et al., 2013]
57. David Graus / Recsys Summer School / 11/09/2019
2. Findings 🤖
✅ Logged in vs. “cleared cookies” account
✅ Geolocation
❌ Gender
❌ Age
❌ Search history
❌ Click history
❌ Browsing history
57[Hannák et al., 2013]
58. David Graus / Recsys Summer School / 11/09/2019
3. Bursting bubbles
58
59. David Graus / Recsys Summer School / 11/09/2019
3. Aim
Increase exposure to varied political opinions
with a goal of improving civil discourse
59[Yom-Tov et al. 2014]
60. David Graus / Recsys Summer School / 11/09/2019
3. Method
• Classify searchers into political leaning (using geo data)
60[Yom-Tov et al. 2014]
61. David Graus / Recsys Summer School / 11/09/2019
3. Method
• Infer political leaning of news sources from user behavior.
• Identify polarized search queries (with strong political leanings —
in both directions).
61[Yom-Tov et al. 2014]
62. David Graus / Recsys Summer School / 11/09/2019
3. Method
• Treatment group: Insert red results for blue users, and blue
results for red users
• Control group: Do not adjust results
62[Yom-Tov et al. 2014]
63. David Graus / Recsys Summer School / 11/09/2019
3. Methode
1. Short term: Compare clicks/behavior between control &
treatment.
2. Long term: Measure during two weeks, per user;
1. Polarization: Difference of user’s leaning-score compared to
average leaning across all sources.
2. Engagement: Average number of queries + average read
articles.
63
64. David Graus / Recsys Summer School / 11/09/2019
3. Findings 1
• Less clicks on inserted opposing sources.
• But:
“Results pages of the opposing viewpoint which had a similarity
higher than the average tended to be clicked 38% more than those
below the average.”
64[Yom-Tov et al. 2014]
65. David Graus / Recsys Summer School / 11/09/2019
3. Findings 2
• Polarization:
• Treatment: Average leaning ‘moves’ ~25% to centre
• Control: Negligible difference (~1%)
• Engagement:
• Treatment: Number of queries: +9% / articles read: +4%
• Control: Small reduction in both (~2.5%)
65[Yom-Tov et al. 2014]
66. David Graus / Recsys Summer School / 11/09/2019
4. How do algorithmic recommendations 🤖 compare
to human 👤?
66
67. David Graus / Recsys Summer School / 11/09/2019
4. Method
• 🤖 Generate article recommendations for news articles using
different (off the shelf) recommender systems (CF & CB).
• 👤 Compare to hand-picked article recommendations.
• Measure “diversity” of recommended articles:
• At content level
• At tag level
• At category level
• At sentiment/subjectivity level
67[Möller et al. 2018]
68. David Graus / Recsys Summer School / 11/09/2019
4. Findings
“Conventional recommendation algorithms at least preserve the
topic/sentiment diversity of the article supply.”
68[Möller et al. 2018]
69. David Graus / Recsys Summer School / 11/09/2019
4. ⚠ Take home
• Diversity is preserved with conventional recommender systems.
69[Möller et al. 2018]
71. David Graus / Recsys Summer School / 11/09/2019
⚠ Besides
• It is trivial to model and incorporate aspects such as diversity (as
you’ll (hopefully) learn during our hands-on 🙌…)
71
74. David Graus / Recsys Summer School / 11/09/2019
Why fair and transparent?
• User studies have shown:
• Our users want personalized content
• Our users care for transparency
• FD
• Verifiability is one of the core values for FD Mediagroup
• Transparency enables verifiability
74
75. David Graus / Recsys Summer School / 11/09/2019
The Context
75
User-data
Rec engine
Inferred
data
Recomme
ndations
76. David Graus / Recsys Summer School / 11/09/2019
The Context
76
79. David Graus / Recsys Summer School / 11/09/2019
Discourse
• Who is the target audience of the explanations?
• What is the goal?
• What purpose do the explanations serve?
79
81. David Graus / Recsys Summer School / 11/09/2019
Framework
81
82. David Graus / Recsys Summer School / 11/09/2019
Framework
82
I want to
be an expert
I want to
stay informed
I want to
broaden my
horizon
I want to
discover the
unexplored
Values
Broadness, diversity, autonomy, objectivity,
match with the user needs, controllability
89. David Graus / Recsys Summer School / 11/09/2019
User study
89
System
Evaluation
Your goal is to Broaden your Horizons.
There may be topics you do not normally read about,
but you may actually find interesting. Exploring this
helps to build a broad perspective on the issues that
matter to you.
Your goal is to Discover the Unexplored.
There may be topics that you haven’t explored
before that may actually become new interests.
Exploring new topics can promote creativity
and objectivity.
90. David Graus / Recsys Summer School / 11/09/2019
User study
90
Aim:
Study whether being offered an explanation of
reading behavior and a particular goal, would
influence the user’s intended reading
behaviorSystem
Evaluation
91. David Graus / Recsys Summer School / 11/09/2019
User study
91
System
Evaluation
Objective
Pick a persona from four
data-driven profiles
Random assign
goal-order
Explain the goals: Broaden Horizons,
Discover the unexplored
Goal A
Show
visualization
Questionnaire
Persona 1
Goal B
Persona 4
92. David Graus / Recsys Summer School / 11/09/2019
Reading Behavior Explanation
92
Similarity
Familiarity
93. David Graus / Recsys Summer School / 11/09/2019
Hypotheses
93
System
Evaluation
2. Broaden horizon:
Users choose topics that have high similarity and
high familiarity compared to the non-selected topics.
3. Discover the unexplored:
Users choose topics that have low similarity and
low familiarity compared to the non-selected topics.
1. Goal Framework:
Broaden Horizon chosen topics are more similar and
more familiar than Discover the unexplored
95. David Graus / Recsys Summer School / 11/09/2019
Results
95
System
Evaluation
2. Broaden horizon:
Users choose topics that have high similarity and
high familiarity compared to the non-selected topics.
3. Discover the unexplored:
Users choose topics that have low similarity and
low familiarity compared to the non-selected topics.
1. Goal Framework:
Broaden Horizon chosen topics are more similar and
more familiar than Discover the unexplored
96. David Graus / Recsys Summer School / 11/09/2019
Results
96
System
Evaluation
2. Broaden horizon:
Users choose topics that have high similarity and
high familiarity compared to the non-selected topics.
3. Discover the unexplored:
Users choose topics that have low similarity and
low familiarity compared to the non-selected topics.
1. Goal Framework:
Broaden Horizon chosen topics are more similar and
more familiar than Discover the unexplored
97. David Graus / Recsys Summer School / 11/09/2019
Results
97
System
Evaluation
2. Broaden horizon:
Users choose topics that have high similarity and
high familiarity compared to the non-selected topics.
3. Discover the unexplored:
Users choose topics that have low similarity and
low familiarity compared to the non-selected topics.
1. Goal Framework:
Broaden Horizon chosen topics are more similar and
more familiar than Discover the unexplored
98. David Graus / Recsys Summer School / 11/09/2019
Results
98
System
Evaluation
2. Broaden horizon:
Users choose topics that have high similarity and
high familiarity compared to the non-selected topics.
3. Discover the unexplored:
Users choose topics that have low similarity and
low familiarity compared to the non-selected topics.
1. Goal Framework:
Broaden Horizon chosen topics are more similar and
more familiar than Discover the unexplored
99. David Graus / Recsys Summer School / 11/09/2019
Results
99
System
Evaluation
2. Broaden horizon:
Users choose topics that have high similarity and
high familiarity compared to the non-selected topics.
3. Discover the unexplored:
Users choose topics that have low similarity and
low familiarity compared to the non-selected topics.
1. Goal Framework:
Broaden Horizon chosen topics are more similar and
more familiar than Discover the unexplored
100. David Graus / Recsys Summer School / 11/09/2019
Results
100
System
Evaluation
2. Broaden horizon:
Users choose topics that have high similarity and
high familiarity compared to the non-selected topics.
3. Discover the unexplored:
Users choose topics that have low similarity and
low familiarity compared to the non-selected topics.
1. Goal Framework:
Broaden Horizon chosen topics are more similar and
more familiar than Discover the unexplored
101. David Graus / Recsys Summer School / 11/09/2019
Results
101
System
Evaluation
2. Broaden horizon:
Users choose topics that have high similarity and
high familiarity compared to the non-selected topics.
3. Discover the unexplored:
Users choose topics that have low similarity and
low familiarity compared to the non-selected topics.
1. Goal Framework:
Broaden Horizon chosen topics are more similar and
more familiar than Discover the unexplored
Partial
support
102. David Graus / Recsys Summer School / 11/09/2019
What have we done
• Novel, scalable and generalizable framework of user-profile
explanations
• Exploration of reader data; domain-knowledge and
data-driven ontology
• Interface mockup
• User-study to evaluate the value-driven explanations
• “Explainability” as a product
102
103. David Graus / Recsys Summer School / 11/09/2019
Future
• Further designing
implementation of our
framework
• Formalize the
epistemic goals
together with editors
• Extend user studies/
focus groups to all
value-driven goals
!103
104. David Graus / Recsys Summer School / 11/09/2019
In summary
• Shown you where/how we personalize @ FD Mediagroep
• Shown you how we evaluate and can determine usefulness of
RecSys using IR metrics
• Told you that these metrics are not all there is to it
• Told you it is not so difficult to incorporate diversity, discount for
popularity, etc.
• What we’ll work on after the break
104