Crowd computing utilizes both crowdsourcing and human computation to solve problems. Crowdsourcing enables more efficient and scalable data collection and processing by outsourcing tasks to a large, undefined group of people. Human computation allows software developers to incorporate human intelligence and judgment into applications to provide capabilities beyond current artificial intelligence. Examples discussed include Amazon Mechanical Turk, various crowd-powered applications, and how crowdsourcing has helped label large datasets to train machine learning models.
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
The Rise of Crowd Computing - 2016
1. The Rise of Crowd Computing
Matt Lease
School of Information @mattlease
University of Texas at Austin ml@utexas.edu
Slides:
slideshare.net/mattlease
2. “The place where people & technology meet”
~ Wobbrock et al., 2009
“iSchools” now exist at 65 universities around the world
www.ischools.org
What’s an Information School?
2
3. • Crowd Computing =
Crowdsourcing + Human Computation
• Crowdsourcing enables new levels of efficiency
& scalability in data collection & processing
• Human Computation lets us build next-
generation applications today, providing
capabilities beyond state-of-the-art AI
Roadmap
8. Crowdsourcing
• Jeff Howe. Wired, June 2006.
• Take a job traditionally
performed by a known agent
(often an employee)
• Outsource it to an undefined,
generally large group of
people via an open call
8
9. • Marketplace for paid crowd work (“micro-tasks”)
– Created in 2005 (remains in “beta” today)
• On-demand, scalable, 24/7 global workforce
• API lets human labor be integrated into software
– “You’ve heard of software-as-a-service. Now this is human-as-a-service.”
Amazon Mechanical Turk (MTurk)
10. Beyond Mechanical Turk: An Analysis of
Paid Crowd Work Platforms
Vakharia and Lease, iConference 2015
Qualitative assessment of 7 crowd work platforms
11. Collecting Data from Crowds
MTurk sparks 2008 “gold rush” for ML training data
• Information Retrieval: Alonso et al., SIGIR Forum
• Human-Computer Interaction: Kittur et al., CHI
• Computer Vision: Sorokin & Forsythe, CVPR
• NLP: Snow et al, EMNLP
– Annotating human language
– 22,000 labels for only US $26
– Crowd’s consensus labels can
replace traditional expert labels
16. ACM Queue, May 2006
16
“Software developers with innovative ideas for
businesses and technologies are constrained by the
limits of artificial intelligence… If software developers
could programmatically access and incorporate human
intelligence into their applications, a whole new class
of innovative businesses and applications would be
possible. This is the goal of Amazon Mechanical Turk…
people are freer to innovate because they can now
imbue software with real human intelligence.”
20. Ethics Checking: The Next Frontier?
• Mark Johnson’s address at ACL 2003
– Transcript in Conduit 12(2) 2003
• Think how useful a little “ethics checker and
corrector” program integrated into a word
processor could be!
20
21. Soylent: A Word Processor with a Crowd Inside
• Bernstein et al., UIST 2010
21
27. AAAI Human Computation (HCOMP) Conference
www.humancomputation.com
October 30-November 3, 2016 in Austin
• Give a Talk @ Industry Track
• Attend (learn & network)
• Become a Sponsor!
28. The Future of Crowd Work
Paper @ CSCW 2013 by
Kittur, Nickerson, Bernstein, Gerber,
Shaw, Zimmerman, Lease, and Horton 28
29. Summary
• Crowd Computing =
Crowdsourcing + Human Computation
• Crowdsourcing transforms data collection &
processing via greater efficiency & scalability
• Human Computation lets us build next-
generation applications today, providing
capabilities beyond state-of-the-art AI