The Rise of Crowd Computing
Matt Lease
School of Information @mattlease
University of Texas at Austin ml@utexas.edu
Slides...
“The place where people & technology meet”
~ Wobbrock et al., 2009
“iSchools” now exist at 65 universities around the worl...
• Crowd Computing =
Crowdsourcing + Human Computation
• Crowdsourcing enables new levels of efficiency
& scalability in da...
Motivation
@mattlease
AI effectiveness is often limited by training data size
Problem: creating labeled data is expensive!
Banko and Brill (2001)
What do we do when even
state-of-the-art AI isn’t good enough?
Crowdsourcing
@mattlease
Crowdsourcing
• Jeff Howe. Wired, June 2006.
• Take a job traditionally
performed by a known agent
(often an employee)
• O...
• Marketplace for paid crowd work (“micro-tasks”)
– Created in 2005 (remains in “beta” today)
• On-demand, scalable, 24/7 ...
Beyond Mechanical Turk: An Analysis of
Paid Crowd Work Platforms
Vakharia and Lease, iConference 2015
Qualitative assessme...
Collecting Data from Crowds
MTurk sparks 2008 “gold rush” for ML training data
• Information Retrieval: Alonso et al., SIG...
12
SQUARE:
A Benchmark
for Crowd
Consensus
@HCOMP’13 ir.ischool.utexas.edu/square
(open source)
Human Computation
@mattlease
14
Zensors
Laput et al., CSCW 2015
15
ACM Queue, May 2006
16
“Software developers with innovative ideas for
businesses and technologies are constrained by the
l...
PlateMate (Noronha et al., UIST’10)
17
VizWiz aaaaaaaa
Bigham et al. (UIST 2010)
18Matt Lease - ml@ischool.utexas.edu
“Amazon Remembers”
19
Ethics Checking: The Next Frontier?
• Mark Johnson’s address at ACL 2003
– Transcript in Conduit 12(2) 2003
• Think how us...
Soylent: A Word Processor with a Crowd Inside
• Bernstein et al., UIST 2010
21
MonoTrans:
Translation by monolingual speakers
22
• Bederson et al.,
2010
• See also: Morita & Ishidi, ACM IUI 2009
Counting by Hybrid Divide-&-Conquer
JellyBean
Sarma et al.,
HCOMP 2015
23
Scribe (Lasecki et al., 2012)
Real-time Captioning by Non-professionals
24
fold.it
S. Cooper et al. (2010)
Alice G. Walton. Online Gamers Help Solve Mystery of
Critical AIDS Virus Enzyme. The Atlan...
@mattlease
AAAI Human Computation (HCOMP) Conference
www.humancomputation.com
October 30-November 3, 2016 in Austin
• Give a Talk @ I...
The Future of Crowd Work
Paper @ CSCW 2013 by
Kittur, Nickerson, Bernstein, Gerber,
Shaw, Zimmerman, Lease, and Horton 28
Summary
• Crowd Computing =
Crowdsourcing + Human Computation
• Crowdsourcing transforms data collection &
processing via ...
Matt Lease - ml@utexas.edu - @mattlease
Thank You!
ir.ischool.utexas.edu/crowd
Slides: slideshare.net/mattlease
Próxima SlideShare
Cargando en…5
×

The Rise of Crowd Computing - 2016

742 visualizaciones

Publicado el

Talk at South-by-southwest (SXSW) on crowdsourcing + human computation = crowd computing., March 11, 2016. See http://www.humancomputation.com.

Global growth in Internet connectivity and participation is driving a renaissance in human computation: use of people rather than machines to perform certain computations for which human competency continues to exceed that of state-of-the-art algorithms (e.g. “AI-hard” tasks such as interpreting text or images). While current AI limitations will certainly improve with time, using human computation lets us bulid applications which deliver superior results today. Just as cloud computing now enables us to harness vast Internet computing resources on demand, new crowdsourcing APIsenable us to build computing systems which integrate human computation at run-time, invoking crowd labor on-demand and at-scale. Moreover, we can achieve the best of both worlds by integrating automated AI with human computation, creating hybrid systems with capabilities greater than the sum of their parts. When AI falls short, not only can human computation meet the immediate end-user need, but the results can be fed back into the system to further improve the AI. As a consequence, AI limitations are no longer a bottleneck to delivering innovative, new applications. Such enhanced capabilities have begun to change how we design and implement intelligent systems. While early work in crowd computing focused only on collecting more data from crowds to better train AI, we are increasingly seeing hybrid, socio-computational system emerge which creatively blend human computation and AI at run-time to solve hard computing problems. As such, we find ourselves today in an exhilarating new design space in which intelligent system capabilities are seemingly limited only by our imagination and creativity in designing new algorithms to compute effectively using crowds as well as silicon.

Publicado en: Tecnología
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

The Rise of Crowd Computing - 2016

  1. 1. The Rise of Crowd Computing Matt Lease School of Information @mattlease University of Texas at Austin ml@utexas.edu Slides: slideshare.net/mattlease
  2. 2. “The place where people & technology meet” ~ Wobbrock et al., 2009 “iSchools” now exist at 65 universities around the world www.ischools.org What’s an Information School? 2
  3. 3. • Crowd Computing = Crowdsourcing + Human Computation • Crowdsourcing enables new levels of efficiency & scalability in data collection & processing • Human Computation lets us build next- generation applications today, providing capabilities beyond state-of-the-art AI Roadmap
  4. 4. Motivation @mattlease
  5. 5. AI effectiveness is often limited by training data size Problem: creating labeled data is expensive! Banko and Brill (2001)
  6. 6. What do we do when even state-of-the-art AI isn’t good enough?
  7. 7. Crowdsourcing @mattlease
  8. 8. Crowdsourcing • Jeff Howe. Wired, June 2006. • Take a job traditionally performed by a known agent (often an employee) • Outsource it to an undefined, generally large group of people via an open call 8
  9. 9. • Marketplace for paid crowd work (“micro-tasks”) – Created in 2005 (remains in “beta” today) • On-demand, scalable, 24/7 global workforce • API lets human labor be integrated into software – “You’ve heard of software-as-a-service. Now this is human-as-a-service.” Amazon Mechanical Turk (MTurk)
  10. 10. Beyond Mechanical Turk: An Analysis of Paid Crowd Work Platforms Vakharia and Lease, iConference 2015 Qualitative assessment of 7 crowd work platforms
  11. 11. Collecting Data from Crowds MTurk sparks 2008 “gold rush” for ML training data • Information Retrieval: Alonso et al., SIGIR Forum • Human-Computer Interaction: Kittur et al., CHI • Computer Vision: Sorokin & Forsythe, CVPR • NLP: Snow et al, EMNLP – Annotating human language – 22,000 labels for only US $26 – Crowd’s consensus labels can replace traditional expert labels
  12. 12. 12 SQUARE: A Benchmark for Crowd Consensus @HCOMP’13 ir.ischool.utexas.edu/square (open source)
  13. 13. Human Computation @mattlease
  14. 14. 14
  15. 15. Zensors Laput et al., CSCW 2015 15
  16. 16. ACM Queue, May 2006 16 “Software developers with innovative ideas for businesses and technologies are constrained by the limits of artificial intelligence… If software developers could programmatically access and incorporate human intelligence into their applications, a whole new class of innovative businesses and applications would be possible. This is the goal of Amazon Mechanical Turk… people are freer to innovate because they can now imbue software with real human intelligence.”
  17. 17. PlateMate (Noronha et al., UIST’10) 17
  18. 18. VizWiz aaaaaaaa Bigham et al. (UIST 2010) 18Matt Lease - ml@ischool.utexas.edu
  19. 19. “Amazon Remembers” 19
  20. 20. Ethics Checking: The Next Frontier? • Mark Johnson’s address at ACL 2003 – Transcript in Conduit 12(2) 2003 • Think how useful a little “ethics checker and corrector” program integrated into a word processor could be! 20
  21. 21. Soylent: A Word Processor with a Crowd Inside • Bernstein et al., UIST 2010 21
  22. 22. MonoTrans: Translation by monolingual speakers 22 • Bederson et al., 2010 • See also: Morita & Ishidi, ACM IUI 2009
  23. 23. Counting by Hybrid Divide-&-Conquer JellyBean Sarma et al., HCOMP 2015 23
  24. 24. Scribe (Lasecki et al., 2012) Real-time Captioning by Non-professionals 24
  25. 25. fold.it S. Cooper et al. (2010) Alice G. Walton. Online Gamers Help Solve Mystery of Critical AIDS Virus Enzyme. The Atlantic, October 8, 2011. 25
  26. 26. @mattlease
  27. 27. AAAI Human Computation (HCOMP) Conference www.humancomputation.com October 30-November 3, 2016 in Austin • Give a Talk @ Industry Track • Attend (learn & network) • Become a Sponsor!
  28. 28. The Future of Crowd Work Paper @ CSCW 2013 by Kittur, Nickerson, Bernstein, Gerber, Shaw, Zimmerman, Lease, and Horton 28
  29. 29. Summary • Crowd Computing = Crowdsourcing + Human Computation • Crowdsourcing transforms data collection & processing via greater efficiency & scalability • Human Computation lets us build next- generation applications today, providing capabilities beyond state-of-the-art AI
  30. 30. Matt Lease - ml@utexas.edu - @mattlease Thank You! ir.ischool.utexas.edu/crowd Slides: slideshare.net/mattlease

×