1. The document discusses strategies for increasing engagement in crowdsourcing projects through targeted interventions. It presents a machine learning approach to predict when users are at risk of disengaging and then intervenes with personalized messages. 2. The interventions are tested in different conditions on the Galaxy Zoo crowdsourcing platform, varying the message content and timing. Results show that prediction-based interventions can significantly increase user contributions compared to no intervention or random interventions. 3. The authors conclude that increasing engagement requires an intelligent intervention design combined with predictive timing of the interventions. They also discuss the importance of engagement for citizen science projects and make their code for the intervention platform openly available.

1. 1
Intervention Strategies for Increasing
Engagement in Crowdsourcing: Platform,
Predictions and Experiments
Avi Segal, Ben-Gurion University, Israel
Ya’akov (Kobi) Gal, Ben-Gurion University, Israel
Ece Kamar, Microsoft Research, Redmond WA, USA
Eric Horvitz, Microsoft Research, Redmond WA, USA
Alex Bowyer, University of Oxford, UK
Grant Miller, University of Oxford, UK

2. Citizen Science (Zooniverse.org)
Wikipedia: scientific research conducted, in whole
or in part, by amateur or nonprofessional
scientists.
2

3. Galaxy Zoo
Conclusions
3

4. The Disengagement Problem
4
Eveleigh, Alexandra, et al. "Designing for dabblers and deterring drop-outs in citizen science.” CHI 2014
Sauermann, Henry, and Chiara Franzoni. "Crowd science user contribution patterns and their implications." PNAS 2015

5. The Approach: Intervention
5
Dimension Description Examples
Target Target population for intervention all, newcomers, heavy users
Time Intervention timing periodical; as soon as risk factor is
identified; one day after disengagement
Duration Intervention (interruption) duration for x minutes; until intervention is
acknowledged; while user is on webpage
Media Type Method of presentation text; graphics; audio; video; mixed
Channel Medium of delivery email; web page; modal message
Mechanism The method used for this
intervention
Messages; help system; task routing;
competition; achievements; …
Content Content of the intervention
communication
“...If GalaxyZoo didn't suit you, then…”;
checkmark image;

6. Disengagement
Prediction
Intervention Evaluation
Dimensions: When & What
6

7. The Intervention Server Platform
7

8. Predicting Disengagement
8
Predict
Disengagement
Intervene
Evaluate
Supervised
Learning
Supervised
Learning
User will disengage within given window
Session & History
1M records
13.5K users
Andrew Mao, Ece Kamar, and Eric Horvitz. "Why stop now? predicting
worker engagement in online crowdsourcing.” HCOPM 2013

9. Intervention Message: Helpful
9
Predict
Disengagement
Intervene
Evaluate
Please don’t stop just yet. You’ve been
extremely helpful so far. Your votes are
really helping us to understand deep
mysteries about galaxies.
Ryan, Richard M., and Edward L. Deci. "Self-determination theory
and the facilitation of intrinsic motivation, social development, and
well-being." APA(2000)

10. Intervention Message: Community
10
Predict
Disengagement
Intervene
Evaluate
Thousands of people are taking part in
the project every month. Visit Talk at
talk.galaxyzoo.org to discuss the
images you see with them.
Kraut, Robert E., and Paul Resnick. "Encouraging contribution to online
communities." Building successful online communities: Evidence-based
social design 2011

11. Intervention Message: Anxiety
11
Predict
Disengagement
Intervene
Evaluate
We use statistical techniques to get the
most from every answer; So, you don’t
need to worry about being “right”.
Just tell us what you see.
Segal, Avi, et al. "Improving productivity in citizen science
through controlled intervention." WWW 2015

12. Intervention GUI
12
Predict
Disengagement
Intervene
Evaluate

13. Conditions
13
Predict
Disengagement
Intervene
Evaluate
# Type
1. No Intervention
2. Random Timing, Helpful Message
3. Random Timing, Community Message
4. Random Timing, Anxiety Message
5. Prediction Based Timing, Helpful Message
6. Prediction Based Timing, Community Message
7. Prediction Based Timing, Anxiety Message

14. Impact on Contributions
14
Predict
Disengagement
Intervene
Evaluate

15. Impact of History
15
Predict
Disengagement
Intervene
Evaluate

16. Conclusion
Increasing engagement = intervention design +
intelligent timing
Code is open sourced
Sequential decision making with uncertainty
Off policy policy evaluation
Engagement Importance
16

17. Thank You & Questions
avise@post.bgu.ac.il

18. Human Aware AI?
30
Learn
Interven
e
Evaluate
“Intervention” title 1
Papers this morning ~100
Learning ~80
? 10
Intervention? ~10
Large scale int. ~2
% 2%
Is this what we want?Is this what we want?