Our presentation for the MediaEval 2014 Crowdsourcing Workshop. We showed that looking at the amount of HITs per worker can be a interesting thing to do.

1. Work Like a Bee
Taking Advantage of
Diligent Crowdsourcing Workers
Presentation: Michael Riegler, Simula Research Laboratory, Norway
Preben N. Oslen, Simula Research Laboratory
Pål Halvorsen, Simula Research Laboratory
michael@simula.no

2. The Idea
• Use “good” workers
• Like the task or good with it
• Will do it much better
• So to speak diligent workers
• Do most of the work
• Extra: additional crowd
workers

3. The Runs
• 4 different runs
• Run 1: MJV with additional crowdsourcing
• Run 2: Diligent Workers vote only
• Run3: MJV with weighted diligent workers
• Run4: Run3 with additional crowdsourcing

4. The Results
Run WF1 score True Labels
Predicted
Labels
R1 0.7207 183, 63, 291 192, 68, 279
R2 0.6919 183, 63, 291 208, 95, 234
R3 0.6912 183, 63, 291 208, 87, 242
R4 0.6919 183, 63, 291 208, 95, 234
Baseline 0.3809 183, 63, 291 0, 0, 537

5. The Analysis (i)
• Every run is better than the
baseline
• R1 is the best performing run
• Small difference between R1
and R2 shows that diligent
workers perform very good
WF1 score
0,73
0,718
0,705
0,693
0,68
R1 R2 R3 R4

6. The Analysis (ii)
• This can for example be used
to save costs
• R2, R3 and R4 perform similar
• Additional Crowdsourcing
does not improve results
WF1 score
0,73
0,718
0,705
0,693
0,68
R1 R2 R3 R4

7. The Conclusion
• Very simple approach
• Showed that it can work
• Quality of workers is more important than
the total number of workers
• No gain trough additional crowdsourcing
• Extend work on larger datasets

8. The End
Tusen takk og
ha det bra!