1. The CLEF Initiative
I N I T I AT I V E Conference and Labs of the Evaluation Forum
http:/
/www.clef-initiative.eu/
The CLEF Initiative
From 2010 to 2012 and Onwards
Nicola Ferro
Information Management Systems (IMS) Research Group
Department of Information Engineering
University of Padua, Italy
MediaEval Benchmarking Initiative for Multimedia Evaluation
MediaEval 2012 Workshop, 5 October 2012, Santa Croce in Fossabanda, Pisa, Italy
2. I N I T I AT I V E
Outline
The CLEF Initiative
From CLEF 2010 to CLEF 2012
CLEF 2013 and Beyond
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 2
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
4. I N I T I AT I V E
The CLEF Initiative: Mission
The CLEF Initiative is a self-organized body
whose main mission is to promote
research, innovation, and development of
information access systems with an
emphasis on multilingual and multimodal
information with various levels of structure.
multilingual and multimodal system testing, tuning and evaluation;
investigation of the use of unstructured, semi-structured, highly-structured, and
semantically enriched data in information access;
creation of reusable test collections for benchmarking;
exploration of new evaluation methodologies and innovative ways of using
experimental data;
discussion of results, comparison of approaches, exchange of ideas, and
transfer of knowledge.
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 4
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
5. I N I T I AT I V E
The CLEF Initiative: Structure
The CLEF Initiative is structured in two main parts:
a series of Evaluation Labs, i.e. laboratories to
conduct evaluation of information access systems
and workshops to discuss and pilot innovative
evaluation activities;
a peer-reviewed Conference on a broad range of
issues, including
investigation continuing the activities of the Evaluation Labs;
experiments using multilingual and multimodal data; in
particular, but not only, data resulting from CLEF activities;
research in evaluation methodologies and challenges.
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 5
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
6. I N I T I AT I V E
The CLEF Initiative
promise
http://www.clef-initiative.eu/
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 6
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
7. I N I T I AT I V E
The CLEF Initiative: Steering Committee
Steering Committee Chair
Nicola Ferro, University of Padua, Italy
Deputy Steering Committee Chair for the Conference
Julio Gonzalo, National Distance Education University (UNED), Spain
Deputy Steering Committee Chair for the Evaluation Labs
Carol Peters, ISTI, National Council of Research (CNR), Italy
Members
Martin Braschler, Zurich University of Applied Sciences, Switzerland
Khalid Choukri, Evaluations and Language resources Distribution Agency (ELDA), France
Paul Clough, University of Sheffield, United Kingdom
Donna Harman, National Institute for Standards and Technology (NIST), USA
Djoerd Hiemstra, University of Twente, The Netherlands
Jaana Kekäläinen, University of Tampere, Finland
Emanuele Pianta, Centre for the Evaluation of Language and Communication Technologies (CELCT), Italy
Maarten de Rijke, University of Amsterdam UvA, The Netherlands
Giuseppe Santucci, Sapienza University of Rome, Italy
Alan Smeaton, Dublin City University, Ireland
Christa Womser-Hacker, University of Hildesheim, Germany
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 7
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
8. I N I T I AT I V E
CLEF: the Name
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 8
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
9. I N I T I AT I V E
CLEF: the Name
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 8
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
11. I N I T I AT I V E
Approach
Scientific and Technological Advancement in
Multilingual and Multimedia Information Systems
Workshops
Workshops
Workshops
Workshops
Workshops
Workshops
Labs and
Labs and
Labs and
Labs and
Labs and
Labs and
Conference
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 10
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
12. I N I T I AT I V E CLEF 2010: 20-23 September 2010 in Padua
Ov er 140
es and
attende onsors
exte rnal sp
http://www.clef2010.org/
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 11
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
13. I N I T I AT I V E
CLEF 2010 Organization
Honorary Chair
Carol Peters, ISTI-CNR, Italy
General Chairs
Maristella Agosti, University of Padua, Italy
Maarten de Rijke, University of Amsterdam, The Netherlands
Program Chairs
Nicola Ferro, University of Padua, Italy
Alan Smeaton, Dublin City University, Ireland
Lab Chairs
Martin Braschler, Zurich University of Applied Sciences, Switzerland
Donna Harman, NIST, USA
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 12
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
14. I N I T I AT I V E
CLEF 2010 Conference
12 papers (8 full papers and 4 short papers)
out of 21 submissions (17 full papers and 4 short papers)
Two keynote talks
Norbert Fuhr, “IR Between Science and Engineering, and the Role of
Experimentation”
Ricardo Baeza-Yates, “Retrieval Evaluation in Practice”
Other Evaluation Initiatives
Ellen Voorhees for TREC; Noriko Kando for NTCIR; Prasenjit Majumder
for FIRE; Jaap Kamps for INEX; Pavel Braslavski for ROMIP
Two panels
Donna Harman, Noriko Kando, Mounia Lalmas, Carol Peters, “The Four
Ladies of Experimental Evaluation”
“A PROMISE for Experimental Evaluation” by the partners of the project
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 13
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
15. I N I T I AT I V E
CLEF 2010 Labs and Workshops
5 Benchmarking activities
CLEF-IP: a benchmarking activity on intellectual property
ImageCLEF: a benchmarking activity on image retrieval
PAN: a benchmarking activity on plagiarism detection
RespubliQA: A benchmarking activity on question answering
using multilingual political data
WePS: A benchmarking activity on web people search
2 Workshops
CriES: a workshop aimed at exploring the evaluation of
searching for expertise in social media
LogCLEF: a workshop aimed at exploring methodologies for
studying search engine log files
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 14
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
17. I N I T I AT I V E CLEF 2011 Approach: Interleaved Sessions
16
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
18. I N I T I AT I V E CLEF 2011: 19-22 September 2011 in Amsterdam
Over 170
tende es and
at nsors
e xtern al spo
http://www.clef2011.org/
http://www.clef2011.org/
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 17
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
19. I N I T I AT I V E
CLEF 2011 Organization
General chairs
Julio Gonzalo, National Distance Education University (UNED),
Spain
Maarten de Rijke, University of Amsterdam, The Netherlands
Program chairs
Jaana Kekäläinen, University of Tampere, Finland
Mounia Lalmas, Yahoo! Research Barcelona, Spain
Lab chairs
Paul Clough, University of Sheffield, United Kingdom
Vivien Petras, Humboldt University, Germany
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 18
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
20. I N I T I AT I V E
CLEF 2011 Conference
14 papers (10 full papers and 4 short papers)
out of 23 submissions (19 full papers and 4 short papers)
Two keynote talks
Elaine Toms , “Would you trust your IR system to choose your date? Re-thinking
IR Evaluation in the 21st Century”
Omar Alonso, “Crowdsourcing for Information Retrieval Experimentation and
Evaluation”
Community Sessions
Other Evaluation Initiatives: Ellen Voorhees for TREC; Hideo Joho for NTCIR;
Jaap Kamps for INEX; Gareth Jones for MediaEval
CHORUS+ Session: Information Retrieval from Scientific Multimedia Data
Infrastructure Session: the Promise evaluation infrastructure, where Visual
Analytics meets Information Retrieval
Funding Opportunity Session: Intelligent Information Management - FP7 ICT
Workprogramme 2011 - 2012, Strategic Objective 4.4. - General information and
conditions of Call 8
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 19
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
21. I N I T I AT I V E
CLEF 2011 Labs and Workshops
5 Benchmarking activities
CLEF-IP: a benchmarking activity on intellectual property
ImageCLEF: a benchmarking activity on image retrieval
LogCLEF: a benchmarking activity on Multilingual Log File Analysis:
Language identification, query classification, success of a query
PAN: a benchmarking activity on plagiarism detection
QA4MRE: a benchmarking activity on the evaluation of Machine Reading
systems through Question Answering and Reading Comprehension Test
2 Workshops/Pilots
CHiC: a workshop aimed at moving towards a systematic and large-scale
evaluation of cultural heritage digital libraries and information access systems
and helping to shape a possible roadmap for it
MusicCLEF: a pilot benchmarking activity on the evaluation of music search
engines that are based both on audio content and on multilingual textual
descriptions
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 20
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
23. I N I T I AT I V E CLEF 2012: 17-20 September 2012, Rome
Ov er 190
es and
attende onsors
exte rnal sp
http://www.clef2012.org/
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 22
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
24. I N I T I AT I V E
CLEF 2012 Organization
General chairs
Tiziana Catarci, Sapienza University of Rome, Italy
Djoerd Hiemstra, University of Twente, The Netherlands
Program chairs
Anselmo Peñas, National Distance Learning University, Spain
Giuseppe Santucci, Sapienza University of Rome, Italy
Lab chairs
Jussi Karlgren, Swedish Institute of Computer Science,
Sweden
Christa Womser-Hacker, University of Hildesheim, Germany
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 23
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
25. I N I T I AT I V E
CLEF 2012 Conference
17 papers (9 full papers, 5 short papers, and 3 posters)
out of 33 submissions
Two keynote talks
Peter Clark, “From Information Retrieval to Knowledgeable Machines”
Tobias Schreck, “Visual Search and Analysis in Textual and Non-Textual
Document Repositories - Approaches, Applications, and Research
Challenges”
Community Sessions
Other Evaluation Initiatives: Donna Harman for TREC and TAC; Noriko
Kando for NTCIR; Pavel Braslavski for ROMIP; Prasenjit Majumder for
FIRE; Gareth Jones for MediaEval (remote)
Evaluation Clinic: participants had the possibility of meeting evaluation
experts and discussing with them their own evaluation problems
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 24
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
26. I N I T I AT I V E
CLEF 2012 Labs and Workshops
7 Benchmarking activities
CHiC: a last-year workshop turning into a benchmarking activity for the
cultural heritage domain based on Europeana collections
CLEF-IP: a benchmarking activity on intellectual property
ImageCLEF: a benchmarking activity on image retrieval
INEX: the well-known Initiative for Evaluation of XML retrieval joins efforts
with CLEF to target new synergies between multilingual, multimodal and
semi-structured information access
PAN: a benchmarking activity on plagiarism detection
QA4MRE: a benchmarking activity on the evaluation of Machine Reading
systems through Question Answering and Reading Comprehension Test
RepLab: a benchmarking activity on microblog data for online reputation
management
1 Workshop
eHealth: a workshop on new evaluation issues in the health domain,
related to the Louhi series of workshops on NLP in Health Informatics
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 25
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
28. I N I T I AT I V E
Next CLEFs
CLEF 2013
23 - 26 September 2013
Technical University of Valencia, Spain
More information soon on the CLEF 2013 Web site
http://www.clef2013.org/
CLEF 2014
15 - 19 September 2014
University of Sheffield, United Kingdom
CLEF 2015
Bids are open
Send a proposal at chair@clef-initiative.eu by 5 April 2013
Template available at: http://www.clef-initiative.eu/documents/71612/87713/
CLEF-Initiative-Template_for_bids.docx
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 27
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
29. I N I T I AT I V E
CLEF Publication
We are now missing a single publication that acts as an hub for citation of
all the CLEF activities (conference and labs) that happen during the year
Therefore, the Springer proceedings delivered at the conference will contain
the conference papers selected by the review process carried out by the program
committee
the lab overview papers, after a review process carried out by the lab chairs
To meet the deadlines to have this publication ready by September, we need
to reschedule the overall CLEF evaluation campaign period
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 28
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
30. I N I T I AT I V E
Conference - CLEF 2013
Call for Papers coming out soon, late
November 2012. Don’t miss it!!!
Paper submission deadline around late
April 2013
More emphasis on evaluation
Focused range of hot topics
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 29
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
31. I N I T I AT I V E
Lab Coordination Deadlines - CLEF 2013
1 October 2012: Call for Lab proposals
15 October 2012: Deadline for presentation of proposals
29 October 2012: Notification of acceptance/rejection
5 November 2012: Release of first lab flyer and call for participation
1 December 2012: Labs registration opens
15 December 2012: Labs begin to release data
15 December 2012 - 1 May 2013: Evaluation Campaign
1 May 2013: Result submission
1 June 2013: Results release
15 June 2013: Working notes papers submitted
30 June 2013: Lab overview papers submitted in Springer
format
30 June 2013 - 7 July 2013: Lab Chairs review lab overviews
15 July 2013: Springer proceedings done
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 30
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
32. I N I T I AT I V E ELIAS Lab Organizers Meeting at ECIR 2013
ELIAS can support a meeting at ECIR
2013 (24-27 March 2013) in Moscow
between Lab Organizers
to share ideas, plans, and strategies
to coordinate
http://www.elias-network.eu/
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 31
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
33. I N I T I AT I V E
Lab Coordination Deadlines - CLEF 2014
15 June 2013: Call for Lab Proposals
1 July 2013: Deadline for presentation of proposals
20 July 2013: Notification of acceptance/rejection
CLEF 2014: Release of first lab flyer and call for participation
1 November 2013: Labs registration opens
15 November 2013: Labs begin to release data
15 November 2013 - 1 May 2014: Evaluation Campaign
1 May 2014: Result submission
1 June 2014: Results release
15 June 2014: Working notes papers submitted
30 June 2014: Lab overview papers submitted in Springer
format
30 June 2014 - 7 July 2014: Lab Chairs review lab overviews
15 July 2014: Springer proceedings done
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 32
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy
34. I N I T I AT I V E
PROMISE: a supporter of CLEF promise
PROMISE
Participative Research labOratory for Multimedia
and Multilingual Information Systems Evaluation
FP7 ICT 2009.4.3, Intelligent Information Management
PROMISE Retreat Report
Prospects and Opportunities for
Information Access Evaluation
Editors
Nicola Ferro, University of Padua, Italy
http://www.promise-noe.eu/
Richard Berendsen, University of Amsterdam, The Netherlands
Allan Hanbury, Vienna University of Technology, Austria
Mihai Lupu, Vienna University of Technology, Austria
Vivien Petras, Humboldt University Berlin, Germany
Maarten de Rijke, University of Amsterdam, The Netherlands
Gianmaria Silvello, University of Padua, Italy
Network of Excellence co-funded by the 7th Framework Program of the European Commission, grant agreement no. 258191
The CLEF Initiative: From 2010 to 2012 and Onwards
Nicola Ferro 33
MediaEval 2012 Workshop, 5 October 2012, Pisa, Italy