Presentation at the Human Computation for Science and Computational Sustainability workshop at NIPS 2012 in Lake Tahoe, NV.

1. Crowdsourcing Citizen Science
Data Quality with a
Human-Computer Learning Network
Wiggins, Gerbracht, Lagoze,Yu, Wong, & Kelling
7 December, 2012 ~ Lake Tahoe, NV
Workshop on Human Computation for Science and Computational Sustainability

2. Crowdsourcing Scientific Work

3. eBird
• Online checklist
program for bird
abundance &
distribution
• Data (mostly)
from recreational
birders; used
widely
• Over 100 million
records & growing eBird observations per month

4. Data Quality
Dogbird Catbird

5. Data Quality
X
Dogbird Catbird

6. The eBird HCLN
S Kelling, C Lagoze, W-K Wong, J Yu, T Damoulas, J Gerbracht, D Fink, C Gomes. 2012. eBird: A Human/Computer Learning
Network to Improve Biodiversity Conservation and Research. Artificial Intelligence.

7. Emergent Filters
Kelling, S., J.Yu, J. Gerbracht, and W. K. Wong. 2011. Emergent Filters: Automated Data Verification in a Large-scale Citizen Science
Project. Proceedings of the IEEE eScience Conference.

8. Modeling Expertise
Environmental Occupancy Detection
Covariates (Latent) Detection Covariates
oi dit
Xi$ Zi$ Yit$ Wit$
t=1,…,Ti$
i=1,…,N$

9. Occupancy-Detection-Expertise
Expertise Expertise
Covariates vj
Uj Ej
j=1,…,M
oi dit, fit
Xi Zi Yit Wit$
t=1,…,Ti
i=1,…,N

10. Average Detection Probabilities
0.20
0.15
0.10
0.05
0.00
h
o
w
n
er
al
y
h
tc
re
o
Ja
s
lo
in
h
ru
er
ha
Vi
as
-0.05
al
rd
ue
Th
H
r
ut
Sw
Ca
d
Th
Bl
e
N
de
d
lu
oo
n
ed
n
ea
d
B
er
ow
te
W
g
-h
at
th
as
in
Br
ue
re
or
re
w
G
Bl
N
h-
-b
ug
te
hi
Ro
W
n
er
th
or
N
Common birds Hard-to-detect birds
Yu, J., W. K. Wong, and R. A. Hutchinson. 2010. Modeling Experts and Novices in Citizen Science Data
for Species Distribution Modeling. IEEE 10th International Conference on Data Mining (ICDM),

11. Emergent Filters + Expertise
Spizella
passerina

12. Emergent Filters + Expertise
Spizella 50"
45"
passerina 40"
35"
30"
25"
20"
15"
10"
5"
0"
8)Jan" 8)Feb"8)Mar" 8)Apr" 8)May" 8)Jun" 8)Jul" 8)Aug" 8)Sep" 8)Oct" 8)Nov" 8)Dec"

13. Emergent Filters + Expertise
Spizella 50"
45"
passerina 40"
35"
30"
25"
20"
15"
10"
5"
0"
8)Jan" 8)Feb"8)Mar" 8)Apr" 8)May" 8)Jun" 8)Jul" 8)Aug" 8)Sep" 8)Oct" 8)Nov" 8)Dec"
50"
45"
40"
35"
30"
25"
20"
15"
10"
5"
0"
8)Jan" 8)Feb"8)Mar" 8)Apr"8)May" 8)Jun" 8)Jul" 8)Aug" 8)Sep" 8)Oct" 8)Nov" 8)Dec"

14. Emergent Filters + Expertise
Spizella 50"
45"
passerina 40"
35"
30"
25"
20"
15"
10"
5"
0"
8)Jan" 8)Feb"8)Mar" 8)Apr" 8)May" 8)Jun" 8)Jul" 8)Aug" 8)Sep" 8)Oct" 8)Nov" 8)Dec"
50" 50"
45" 45"
40" 40"
35" 35"
30" 30"
25" 25"
20" 20"
15" 15"
10" 10"
5" 5"
0" 0"
8)Jan" 8)Feb"8)Mar" 8)Apr"8)May" 8)Jun" 8)Jul" 8)Aug" 8)Sep" 8)Oct" 8)Nov" 8)Dec" 1(Jan" 1(Feb"1(Mar" 1(Apr"1(May" 1(Jun" 1(Jul" 1(Aug" 1(Sep" 1(Oct" 1(Nov"1(Dec"

15. Improving Spatial Coverage
Locations in NY with eBird submissions in 2009

16. Improving Spatial Coverage
Areas with enough data for emergent filters

17. Future Work
• Preliminary studies integrated into eBird for better
data quality on multiple levels
• Resulting human-computer learning network will
use eBird data in new ways
• Evaluation of motivation, learning, and skills related
to expertise ranking & birding routes

18. Thanks!
www.ebird.org
@AndreaWiggins
andrea.wiggins@cornell.edu
www.andreawiggins.com
Acknowledgements
• Leon Levy Foundation
• Wolf Creek Foundation
• National Science Foundation Grants OCI-0830944, CCF-0832782, ITR-0427914, DBI-1049363,
DBI-0542868, DUE-0734857, IIS-0748626, IIS-0844546, IIS-0612031, IIS-1050422,
IIS-0905385, IIS-0746500, IIS-1209589, AGS-0835821, CNS-0751152, CNS-0855167.