This document summarizes a game designed to combine human and computer vision to refine metadata for images. The game asks players to identify which leaf in a set is different. It was designed to be easy enough to not require expertise, but still provide useful data to researchers. A trial using Mechanical Turk found that players found the game enjoyable and were able to correctly identify leaves. The game aims to identify errors, test if color helps identification, and get feedback to improve the game experience.

1. Odd Leaf Out
Combining Human and Computer Vision
Arijit Biswas, Computer Science and Darcy Lewis, iSchool
Derek Hansen, Jenny Preece, Dana Rotman-University of Maryland’s iSchool
David Jacobs, Eric Stevens-University of Maryland Computer Science
Jen Hammock, Cynthia Parr-The Smithsonian Institution

3. Refining Metadata Associated with Images

4. Existing Image Crowdsourcing Games

5. How our game is different
• Anyone can play and can provide us with
useful information.
• No expertise necessary
• Capitalizes on strengths of humans and
algorithms
– Humans are better than algorithms at identifying
similarity of images

6. Game Mechanics

7. Game Mechanics

8. How Leaf Sets Are Constructed
• Designed to bring in useful data
• Not too easy or too hard
• Curvature based histograms used to get
features from leaf shapes.
– These features are used to find distance between
all possible pairs of leaves.

9. What’s in it for us if people play this game?
• Identify errors in the dataset
• Discover if color helps humans identify leaves
• Feedback on how enjoyable or difficult the
game is

10. Game Variations
Before Leaf is Chosen
Multiple Guesses Skip
After Leaf is Chosen
Contest after Game is Finished Contest Previous Round
Feedback Mechanism
When Feedback Occurs

11. Mechanical Turk Trial
1 2 3 4 5
0
5
10
15
20
25
30
Enjoyment
NumberCorrect

12. Mechanical Turk Trial
1 2 3 4 5
1
2
3
4
5
Difficulty
Enjoyment

13. Summary
• Anyone can help in Computer Vision research
work.
• Games can be fun for players and useful for
researchers.
• Humans are better than machines in judging
the similarity of two images.

14. Funding
This work is made possible by National Science Foundation
grant number 0968546