Invited Talk at the 2012 Frontiers of Information Science and Technology (FIST) meeting in Shanghai, China (December 12, 2012).

1. The Rise of Crowd Computing
Matt Lease
School of Information @mattlease
University of Texas at Austin ml@ischool.utexas.edu

2. What is Information Science?
2

3. Information Schools
www.ischools.org
– At 30 universities across: N. America, Europe, Asia
– China: Nanjing University, School of Info. Management
Wobbrock et al. (2009). Reflections on the Future of
iSchools from Inspired Junior Faculty. 3

4. Crowdsourcing
@mattlease 4

5. Crowdsourcing
• Jeff Howe. Wired, June 2006.
• Take a job traditionally
performed by a known agent
(often an employee)
• Outsource it to an undefined,
generally large group of
people via an open call
• New application of principles
from open source movement
5

6. A Couple of Examples from China
6

7. 7

8. Amazon Mechanical Turk (MTurk)
• Marketplace for crowd labor (microtasks)
• Created in 2005 (still in “beta”)
• On-demand, scalable, 24/7 global workforce
8

9. Collecting Data (“Gold Rush”)
@mattlease 9

10. MTurk for Computer Vision (UIUC)
• Li: ImageNet’07 – Sorokin & Forsythe: CVPR08
• 4K labels for US $60
10

11. Snow et al. (EMNLP 2008)
• MTurk annotation for 5 Tasks
– Affect recognition
– Word similarity
– Recognizing textual entailment
– Event temporal ordering
– Word sense disambiguation
• 22K labels for US $26
• High agreement between
consensus labels and
gold-standard labels
11

12. Alonso et al. (SIGIR Forum 2008)
• MTurk for Information Retrieval (IR)
– Judge relevance of search engine results
12

13. Kittur, Chi, & Suh (CHI 2008)
• MTurk for User Studies
• “…make creating believable invalid responses as
effortful as completing the task in good faith.”
13

14. Beyond MTurk
@mattlease 14

15. ESP Game (Games With a Purpose)
von Ahn & Dabbish (2004)
15

16. reCaptcha
von Ahn et al. (2008). In Science.
16

17. Human Sensing: Passive & Active
August 12, 2012 17

18. Crowd Computing
@mattlease 18

19. What is a Computer?
19

20. • What was old is new
• Crowdsourcing: A New
Branch of Computer Science
– D.A. Grier, IEEE President
• Tabulating the heavens:
computing the Nautical
Almanac in 18th-century
England
– M. Croarken (2003) Princeton University Press, 2005
20

21. The Mechanical Turk
The original, constructed and
unveiled in 1770 by Wolfgang
von Kempelen (1734–1804)
J. Pontin. Artificial Intelligence, With Help From
the Humans. New York Times (March 25, 2007)
21

22. The Human Processing Unit (HPU)
Davis et al. (2010)
HPU
22

23. A New Design Space for
Intelligent Systems
@mattlease 23

24. Ethics Checking: The Next Frontier?
• Mark Johnson’s address at ACL 2003
– Transcript in Conduit 12(2) 2003
• Think how useful a little “ethics checker and
corrector” program integrated into a word
processor could be!
24

25. Soylent: A Word Processor with a Crowd Inside
• Bernstein et al., UIST 2010
25

26. fold.it
S. Cooper et al. (2010)
Alice G. Walton. Online Gamers Help Solve Mystery of
Critical AIDS Virus Enzyme. The Atlantic, October 8, 2011.
26

27. VizWiz aaaaaaaa
Bingham et al. (UIST 2010)
Matt Lease - ml@ischool.utexas.edu 27

28. PlateMate (Noronha et al., UIST’10)
28

29. @mattlease 29

30. What about data quality?
• Research on statistical quality control methods
– Online vs. offline, feature-based vs. content-agnostic
– Worker calibration, noise vs. bias, weighted voting
– See research in my lab by Jung, Kumar, Ryu, & Tang
• Human factors matter too!
– Instructions, design, interface, interaction
– Names, relationship, reputation (Klinger & Lease’11)
– Fair pay, hourly vs. per-task, recognition, advancement
– For contrast with MTurk, see Kochhar (2010)
30

31. Real vs. Synthetic Data
Modeling crowd behavior & testing model
Tang & Lease (CIR’11)
31

32. What about benchmarking?
• How do alternative methods perform?
– Common tasks & datasets provide evaluation foundation
– Shared tasks measure progress & spur innovation
• Across Tasks, .e.g.
– Translation & Transcription
– Search Evaluation
– Verification & Correction
• Across Data Scales
• NIST TREC Crowdsourcing Track (shared task)
– 3rd annual evaluation in 2013 – Please join us! 32

33. How to Design Optimal Workflows?
33

34. What about context?
By minimizing context, greater task efficiency
can often be achieved
– e.g. “Can you name who is in this photo?”
34

35. Context & Informed Consent
• What is the larger task I’m contributing to?
• Who will benefit from it and how?
Jonathan Zittrain, Harvard 35

36. What about sensitive data?
• Not all data can be publicly disclosed
– User data
– Intellectual property
– Legal confidentiality
• Need to restrict who is in your crowd
– Critical for industry adoption
– Distinguish workforce vs. technology
• Provide software platform or “internal” crowd
36

37. 37

38. What about worker fraud?
• Bots (AI) emulating humans (emulating AI…)
– Artificial Artificial Artificial Intelligence
• Also sybil attacks (use of multiple identities)
38

39. What about requestor fraud?
• Refusing to pay for work, installing malware
• Using crowds to defeat online metrics/security
39

40. Robert Sim, MSR Summit’12
40

41. What about regulation?
• Wolfson & Lease (ASIS&T’11)
• As usual, technology is ahead of the law
– employment law
– patent inventorship
– data security and the Federal Trade Commission
– copyright ownership
– securities regulation of crowdfunding
• Take-away: don’t panic, but be mindful
– Understand risks of “just in-time compliance”
41

42. What about ethics?
• Silberman, Irani, and Ross (2010)
– “How should we… conceptualize the role of these
people who we ask to power our computing?”
– Power dynamics
– “Abstraction hides detail”
• Fort, Adda, and Cohen (2011)
– “…opportunities for our community to deliberately
value ethics above cost savings.”
42

43. Who are
the workers?
• A. Baio, November 2008. The Faces of Mechanical Turk.
• P. Ipeirotis. March 2010. The New Demographics of
Mechanical Turk
• J. Ross, et al. Who are the Crowdworkers? CHI 2010.
43

44. How much to pay?
Psychology, economics, and ethics
• What motivates strong performance?
• Primary or supplemental income?
• Effect on local economies?
• Better to pay something (if low)
or pay nothing at all (e.g., games)?
44

45. Digital Dirty Jobs
• The Googler who Looked at the Worst of the Internet
• Policing the Web’s Lurid Precincts
• Facebook content moderation
• The dirty job of keeping Facebook clean
• Even linguistic annotators report stress &
nightmares from reading news articles!
45

46. What about freedom?
• Crowdsourcing vision: empowering freedom
– work whenever you want for whomever you want
• Risk: people being compelled to perform work
– Digital sweat shops? Digital slaves?
– We really don’t know (and need to learn more…)
– Traction? Human Trafficking at MSR Summit’12
46

47. The Future of Crowd Work
Paper to appear at CSCW 2013
by: Kittur, Nickerson, Bernstein, Gerber,
Shaw, Zimmerman, Lease, and Horton 47

48. Conclusion
• Crowdsourcing is quickly transforming practice
in industry and academia via greater efficiency
• Crowd computing enables a new design space
for applications, augmenting state-of-the-art AI
with human computation to offer
new capabilities and user experiences
• With people at the center of this new computing
paradigm, important research questions span
both technological and social/societal challenges
48

49. Thank You!
Students: Past & Present
– Catherine Grady (iSchool)
– Hyunjoon Jung (iSchool)
– Jorn Klinger (Linguistics)
– Adriana Kovashka (CS)
– Abhimanu Kumar (CS)
ir.ischool.utexas.edu/crowd
– Hohyon Ryu (iSchool)
– Wei Tang (CS)
– Stephen Wolfson (iSchool)
Support
– John P. Commons Fellowship
– Temple Fellowship
Matt Lease - ml@ischool.utexas.edu - @mattlease 49

50. REFERENCES & RESOURCES
August 12, 2012 50

51. 2012 Conferences & Workshops
• AAAI: Human Computation (HComp) (July 22-23)
• AAAI Spring Symposium: Wisdom of the Crowd (March 26-28)
• ACL: 3rd Workshop of the People's Web meets NLP (July 12-13)
• AMCIS: Crowdsourcing Innovation, Knowledge, and Creativity in Virtual Communities (August 9-12)
• CHI: CrowdCamp (May 5-6)
• CIKM: Multimodal Crowd Sensing (CrowdSens) (Oct. or Nov.)
• Collective Intelligence (April 18-20)
• CrowdConf 2012 (October 23)
• CrowdNet - 2nd Workshop on Cloud Labor and Human Computation (Jan 26-27)
• EC: Social Computing and User Generated Content Workshop (June 7)
• ICDIM: Emerging Problem- specific Crowdsourcing Technologies (August 23)
• ICEC: Harnessing Collective Intelligence with Games (September)
• ICML: Machine Learning in Human Computation & Crowdsourcing (June 30)
• ICWE: 1st International Workshop on Crowdsourced Web Engineering (CroWE) (July 27)
• KDD: Workshop on Crowdsourcing and Data Mining (August 12)
• Multimedia: Crowdsourcing for Multimedia (Nov 2)
• SocialCom: Social Media for Human Computation (September 6)
• TREC-Crowd: 2nd TREC Crowdsourcing Track (Nov. 14-16)
• WWW: CrowdSearch: Crowdsourcing Web search (April 17)
51

52. Surveys
• Ipeirotis, Panagiotis G., R. Chandrasekar, and P. Bennett. (2009).
“A report on the human computation workshop (HComp).” ACM
SIGKDD Explorations Newsletter 11(2).
• Alex Quinn and Ben Bederson. Human Computation: A Survey
and Taxonomy of a Growing Field. In Proceedings of CHI 2011.
• Law and von Ahn (2011). Human Computation
August 12, 2012 52

53. 2013 Events Planned
Research events
• 1st year of HComp as AAAI conference
• 2nd annual Collective Intelligence?
Industrial Events
• 4th CrowdConf (San Francisco, Fall)
• 1st Crowdsourcing Week (Singapore, April)
August 12, 2012 53

54. Journal Special Issues 2012
– Springer’s Information Retrieval (articles now online):
Crowdsourcing for Information Retrieval
– IEEE Internet Computing (articles now online):
Crowdsourcing (Sept./Oct. 2012)
– Hindawi’s Advances in Multimedia Journal: Multimedia
Semantics Analysis via Crowdsourcing Geocontext
August 12, 2012 54

55. 2011 Tutorials and Keynotes
• By Omar Alonso and/or Matthew Lease
– CLEF: Crowdsourcing for Information Retrieval Experimentation and Evaluation (Sep. 20, Omar only)
– CrowdConf (Nov. 1, this is it!)
– IJCNLP: Crowd Computing: Opportunities and Challenges (Nov. 10, Matt only)
– WSDM: Crowdsourcing 101: Putting the WSDM of Crowds to Work for You (Feb. 9)
– SIGIR: Crowdsourcing for Information Retrieval: Principles, Methods, and Applications (July 24)
• AAAI: Human Computation: Core Research Questions and State of the Art
– Edith Law and Luis von Ahn, August 7
• ASIS&T: How to Identify Ducks In Flight: A Crowdsourcing Approach to Biodiversity Research and
Conservation
– Steve Kelling, October 10, ebird
• EC: Conducting Behavioral Research Using Amazon's Mechanical Turk
– Winter Mason and Siddharth Suri, June 5
• HCIC: Quality Crowdsourcing for Human Computer Interaction Research
– Ed Chi, June 14-18, about HCIC)
– Also see his: Crowdsourcing for HCI Research with Amazon Mechanical Turk
• Multimedia: Frontiers in Multimedia Search
– Alan Hanjalic and Martha Larson, Nov 28
• VLDB: Crowdsourcing Applications and Platforms
– Anhai Doan, Michael Franklin, Donald Kossmann, and Tim Kraska)
• WWW: Managing Crowdsourced Human Computation
– Panos Ipeirotis and Praveen Paritosh
55

56. 2011 Workshops & Conferences
• AAAI-HCOMP: 3rd Human Computation Workshop (Aug. 8)
• ACIS: Crowdsourcing, Value Co-Creation, & Digital Economy Innovation (Nov. 30 – Dec. 2)
• Crowdsourcing Technologies for Language and Cognition Studies (July 27)
• CHI-CHC: Crowdsourcing and Human Computation (May 8)
• CIKM: BooksOnline (Oct. 24, “crowdsourcing … online books”)
• CrowdConf 2011 -- 2nd Conf. on the Future of Distributed Work (Nov. 1-2)
• Crowdsourcing: Improving … Scientific Data Through Social Networking (June 13)
• EC: Workshop on Social Computing and User Generated Content (June 5)
• ICWE: 2nd International Workshop on Enterprise Crowdsourcing (June 20)
• Interspeech: Crowdsourcing for speech processing (August)
• NIPS: Second Workshop on Computational Social Science and the Wisdom of Crowds (Dec. TBD)
• SIGIR-CIR: Workshop on Crowdsourcing for Information Retrieval (July 28)
• TREC-Crowd: Year 1 of TREC Crowdsourcing Track (Nov. 16-18)
• UbiComp: 2nd Workshop on Ubiquitous Crowdsourcing (Sep. 18)
• WSDM-CSDM: Crowdsourcing for Search and Data Mining (Feb. 9)
56

57. More Books
July 2010, kindle-only: “This book introduces you to the
top crowdsourcing sites and outlines step by step with
photos the exact process to get started as a requester on
Amazon Mechanical Turk.“
57

58. Bibliography
 J. Barr and L. Cabrera. “AI gets a Brain”, ACM Queue, May 2006.
 Bernstein, M. et al. Soylent: A Word Processor with a Crowd Inside. UIST 2010. Best Student Paper award.
 Bederson, B.B., Hu, C., & Resnik, P. Translation by Interactive Collaboration between Monolingual Users, Proceedings of Graphics
Interface (GI 2010), 39-46.
 N. Bradburn, S. Sudman, and B. Wansink. Asking Questions: The Definitive Guide to Questionnaire Design, Jossey-Bass, 2004.
 C. Callison-Burch. “Fast, Cheap, and Creative: Evaluating Translation Quality Using Amazon’s Mechanical Turk”, EMNLP 2009.
 P. Dai, Mausam, and D. Weld. “Decision-Theoretic of Crowd-Sourced Workflows”, AAAI, 2010.
 J. Davis et al. “The HPU”, IEEE Computer Vision and Pattern Recognition Workshop on Advancing Computer Vision with Human
in the Loop (ACVHL), June 2010.
 M. Gashler, C. Giraud-Carrier, T. Martinez. Decision Tree Ensemble: Small Heterogeneous Is Better Than Large Homogeneous, ICMLA 2008.
 D. A. Grier. When Computers Were Human. Princeton University Press, 2005. ISBN 0691091579
 JS. Hacker and L. von Ahn. “Matchin: Eliciting User Preferences with an Online Game”, CHI 2009.
 J. Heer, M. Bobstock. “Crowdsourcing Graphical Perception: Using Mechanical Turk to Assess Visualization Design”, CHI 2010.
 P. Heymann and H. Garcia-Molina. “Human Processing”, Technical Report, Stanford Info Lab, 2010.
 J. Howe. “Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business”. Crown Business, New York, 2008.
 P. Hsueh, P. Melville, V. Sindhwami. “Data Quality from Crowdsourcing: A Study of Annotation Selection Criteria”. NAACL HLT
Workshop on Active Learning and NLP, 2009.
 B. Huberman, D. Romero, and F. Wu. “Crowdsourcing, attention and productivity”. Journal of Information Science, 2009.
 P.G. Ipeirotis. The New Demographics of Mechanical Turk. March 9, 2010. PDF and Spreadsheet.
 P.G. Ipeirotis, R. Chandrasekar and P. Bennett. Report on the human computation workshop. SIGKDD Explorations v11 no 2 pp. 80-83, 2010.
 P.G. Ipeirotis. Analyzing the Amazon Mechanical Turk Marketplace. CeDER-10-04 (Sept. 11, 2010)
58

59. Bibliography (2)
 A. Kittur, E. Chi, and B. Suh. “Crowdsourcing user studies with Mechanical Turk”, SIGCHI 2008.
 Aniket Kittur, Boris Smus, Robert E. Kraut. CrowdForge: Crowdsourcing Complex Work. CHI 2011
 Adriana Kovashka and Matthew Lease. “Human and Machine Detection of … Similarity in Art”. CrowdConf 2010.
 K. Krippendorff. "Content Analysis", Sage Publications, 2003
 G. Little, L. Chilton, M. Goldman, and R. Miller. “TurKit: Tools for Iterative Tasks on Mechanical Turk”, HCOMP 2009.
 T. Malone, R. Laubacher, and C. Dellarocas. Harnessing Crowds: Mapping the Genome of Collective Intelligence.
2009.
 W. Mason and D. Watts. “Financial Incentives and the ’Performance of Crowds’”, HCOMP Workshop at KDD 2009.
 J. Nielsen. “Usability Engineering”, Morgan-Kaufman, 1994.
 A. Quinn and B. Bederson. “A Taxonomy of Distributed Human Computation”, Technical Report HCIL-2009-23, 2009
 J. Ross, L. Irani, M. Six Silberman, A. Zaldivar, and B. Tomlinson. “Who are the Crowdworkers?: Shifting
Demographics in Amazon Mechanical Turk”. CHI 2010.
 F. Scheuren. “What is a Survey” (http://www.whatisasurvey.info) 2004.
 R. Snow, B. O’Connor, D. Jurafsky, and A. Y. Ng. “Cheap and Fast But is it Good? Evaluating Non-Expert Annotations
for Natural Language Tasks”. EMNLP-2008.
 V. Sheng, F. Provost, P. Ipeirotis. “Get Another Label? Improving Data Quality … Using Multiple, Noisy Labelers”
KDD 2008.
 S. Weber. “The Success of Open Source”, Harvard University Press, 2004.
 L. von Ahn. Games with a purpose. Computer, 39 (6), 92–94, 2006.
 L. von Ahn and L. Dabbish. “Designing Games with a purpose”. CACM, Vol. 51, No. 8, 2008.
59

60. Bibliography (3)
 Shuo Chen et al. What if the Irresponsible Teachers Are Dominating? A Method of Training on Samples and
Clustering on Teachers. AAAI 2010.
 Paul Heymann, Hector Garcia-Molina: Turkalytics: analytics for human computation. WWW 2011.
 Florian Laws, Christian Scheible and Hinrich Schütze. Active Learning with Amazon Mechanical Turk.
EMNLP 2011.
 C.Y. Lin. Rouge: A package for automatic evaluation of summaries. Proceedings of the workshop on text
summarization branches out (WAS), 2004.
 C. Marshall and F. Shipman “The Ownership and Reuse of Visual Media”, JCDL, 2011.
 Hohyon Ryu and Matthew Lease. Crowdworker Filtering with Support Vector Machine. ASIS&T 2011.
 Wei Tang and Matthew Lease. Semi-Supervised Consensus Labeling for Crowdsourcing. ACM SIGIR
Workshop on Crowdsourcing for Information Retrieval (CIR), 2011.
 S. Vijayanarasimhan and K. Grauman. Large-Scale Live Active Learning: Training Object Detectors with
Crawled Data and Crowds. CVPR 2011.
 Stephen Wolfson and Matthew Lease. Look Before You Leap: Legal Pitfalls of Crowdsourcing. ASIS&T 2011.
60