1. 1 TurKit: A Toolkit for Human Computation Algorithms Rob Miller & Greg Little User Interface Design Group MIT CSAIL Joint work with Lydia Chilton, Max Goldman, Jeff Bigham, Aubrey Tatarowicz, Rajeev Najak, Michael Bernstein June 10, 2010

2. Outline Iterative human computation TurKit: a toolkit for human computation algorithms Systems with human computation inside VizWiz: vision for blind users with camera phones Soylent: putting a crowd inside Microsoft Word June 10, 2010 2

3. What is TurKit?

4. 2 Ways What is TurKit?

5. Way 1

6. Way 1 ADD 1, 2

7. Way 1 ADD 1, 2 JMP somewhere

8. Way 1 ADD 1, 2 JMP somewhere …

9. Way 1 ADD 1, 2 JMP somewhere … TURK “How do you feel?”

10. Way 1 ADD 1, 2 JMP somewhere … TURK “How do you feel?”

11. Way 2

12. Way 2

13. Way 2

14. Way 2

15. Way 2

16. Way 2

17. Way 2

18. Way 2

19. Way 2

20. Way 2

21. Demo

22. Demo TurKit

23. What can you do with TurKit?

24. x50

25. 31 19

28. Improve

29. Improve

30. Vote Improve

31. Vote Improve

32. Handwriting Recognition

33. Image Description

35. better package

36. far more competitive offer

37. more competitive offer

39. Subjective Ratings x

40. Subjective Ratings

41. Subjective Ratings

42. Workflow Comparisons

43. Workflow Comparisons

44. Workflow Comparisons

45. Workflow Comparisons

46. Workflow Comparisons

47. Iterative

48. Parallel Iterative

49. Parallel Iterative

50. Parallel Iterative

51. Which is better?

52. Which is better? 30 images

53. Which is better? 30 images 6 iterations

54. Which is better? 30 images 6 iterations

55. Which is better?

56. Which is better?

57. Which is better?

58. Larger Algorithms

59. Larger Algorithms

60. Larger Algorithms

61. Outline Iterative human computation TurKit: a toolkit for human computation algorithms Systems with human computation inside VizWiz: vision for blind users with camera phones Soylent: putting a crowd inside Microsoft Word June 10, 2010 58

62. VizWiz: Helping the Blind See 59 joint work with Jeff Bigham (University of Rochester) Which door is the women’s restroom? June 10, 2010

63. Helping the Blind See 60 MechanicalTurk the left one LEFT on the left June 10, 2010

64. Field Study Field deployment with 11 blind iPhone users Answers received within a minute for ~5 cents a question Latency can be reduced to less than 30 seconds by keeping workers warmed up (at $4 per hour) June 10, 2010 61

65. All Mobile Users Are Situationally Disabled 62 joint work with Rajeev Najak MechanicalTurk When is Barack Obama speaking at MIT? June 10, 2010

66. Soylent: Putting a Crowd inside MS Word June 10, 2010 joint work with Michael Bernstein 63

67. Crowd-Driven Proofreading June 10, 2010 64

68. Crowd-Driven Shortening June 10, 2010 65

69. Bigger Idea: REAL Wizard of Oz Wizard of Oz is a tried-and-true prototyping technique in AI and HCI Putting a human behind the curtain until we figure out how to put software there Crowd computing enables Wizard of Oz systems that are useful and deployable So we can start collecting data about how the system is really used in practice Adding the AI backend becomes a performance or cost optimization 66 June 10, 2010

70. Outline Iterative human computation TurKit: a toolkit for human computation algorithms Systems with human computation inside VizWiz: vision for blind users with camera phones Soylent: putting a crowd inside Microsoft Word Funded in part by June 10, 2010 67