Design and implementation_of_relevance_assessm

Design and Implementation of Relevance Assessments Using Crowdsourcing Omar Alonso 1 and Ricardo Baeza-Yates 2 1 Microsoft Corp., Mountain View, California, US 2 Yahoo! Research, Barcelona, Spain European Conference on Information Retrieval, 2011

Outline of presentation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Crowdsourcing ,[object Object],"Crowdsourcing is the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call." - Jeff Howe

Mechanical Turk ,[object Object]

Design and build HITs Put HITs on Amazon Mechanical Turk Collect evaluation results from MTurk Workers Approve evaluation results according to designed approval rules Pay Workers whose inputs have been approved Collected binary relevance Assessments of TREC 8 Amazon Mechanical Turk Workflow

Objective of the paper ,[object Object]

Methodology ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Methodology ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Experimental Setup ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Effect of Highlighting ,[object Object],[object Object],[object Object],[object Object]

Experiment with comments ,[object Object],[object Object],[object Object]

Recommendations ,[object Object],[object Object],[object Object],[object Object],[object Object]

Design and implementation_of_relevance_assessm

Recomendados

Recomendados

Más contenido relacionado

Similar a Design and implementation_of_relevance_assessm

Similar a Design and implementation_of_relevance_assessm (20)

Último

Último (20)

Design and implementation_of_relevance_assessm