Poster describing the ERCIM-funded project on IR- and user-centric recommender system evaluation currently being undertaken in the Information Access group at CWI. Presented at UMAP 2013.

1. ERCIM
research
project
at
CWI
Focused
on
novel
aspects
of
recommender
systems
evalua3on
from
an
informa3on
retrieval
and
a
user-­‐
centric
perspec3ve.
Dura3on
and
personnel
§ January
2013
–
March
2014
§ 24
person
months
§ 2
postdocs
Centrum
Wiskunde
&
Informa:ca
www.cwi.nl
Alan
Said,
Alejandro
Bellogín,
Arjen
De
Vries,
Benjamin
Kille
{alan.said,
alejandro.bellogin,
arjen.de.vries}@cwi.nl,
kille@dai-­‐lab.de
Informa3on
Retrieval
and
User-­‐Centric
Recommender
System
Evalua3on
UMAP
2013
–
Rome,
Italy
Acknowledgement
This
work
is
carried
out
during
the
tenure
of
an
ERCIM
“Alain
Bensoussan”
Fellowship
Programme.
The
research
leading
to
these
results
has
received
funding
from
the
European
Union
Seventh
Framework
Programme
(FP7/2007-­‐2013)
under
grant
agreement
no.246016.
User-­‐centric
Evalua:on
A
personalized
offline
evalua:on
protocol
mimicking
real-­‐life
deployed
recommender
systems.
• Each
algorithm
is
trained
once
per
user
based
on
a
personalized
training
and
test
split.
• Candidate
test
items
are
selected
based
on
a
personal
ra:ng
value
threshold
(e.g.
mean).
Diversity-­‐oriented
evalua:on
methods
focusing
on:
• non-­‐accuracy-­‐based
evalua3on
metrics
• item
feature-­‐centric
evalua3on
• novelty
IR-­‐based
Evalua:on
A
coherence-­‐based
func:on
that
is
able
to
predict
the
user
performance
• Correlated
with
the
“magic
barrier”
• Improves
the
total
performance
of
the
system
when
used
to
group
users
into
difficult
and
easy
ones
Explore
correla:ons
between
metrics
typically
used
in
offline
experiments
(precision)
and
in
real
applica3ons
(e.g.,
CTR).
Perform
an
analysis
of
how
evalua3on
in
recommenda3on
should
be
performed
to
obtain
interpretable
and
comparable
results.
Related
Events
Workshop
on
Reproducibility
and
Replica1on
in
Recommender
Systems
Evalua1on.
In
conjunc3on
with
RecSys
2013
Workshop
on
Benchmarking
Adap1ve
Retrieval
and
Recommender
Systems.
In
conjunc3on
with
SIGIR
2013
ACM
TIST
Special
Issue
on
Recommender
System
Benchmarking
Further
Reading
Ar1st
popularity:
do
web
and
social
music
services
agree?
A.
Bellogín
et
al.
In
ICWSM
2013
User-­‐Centric
Evalua1on
of
a
K-­‐Furthest
Neighbor
Collabora1ve
Filtering
Recommender
Algorithm
A.
Said
et
al.
in
CSCW
2013
Poster
Abstract
Informa1on
Retrieval
and
User-­‐Centric
Recommender
System
Evalua1on
A.
Said,
A.
Bellogín
et
al.
in
UMAP
2013