1. Introducing
Amazon
Mechanical
Tu r k
Presented by:
John Hoskins
Sr. Manager
June 10, 2011

2. Agenda
 Introduction
A brief overview of Mechanical Turk
 Use Cases & Case Studies
Practical applications of Mechanical Turk and examples of
how others have used it.
 Getting Started
Options for implementing with Mechanical Turk
© 2011 Amazon.com, Inc. or its Affiliates.

3. Business Challenges
 Content Moderation
Our community is generating 10,000 comments per day –
how can we eliminate SPAM?
 Product Feedback
We just released a new product or service – how do we
judge how it’s being received?
 Discoverability
We have thousands of legacy media assets – how do we
make them searchable?
© 2011 Amazon.com, Inc. or its Affiliates.

4. When to use Mechanical Turk?
 Tasks that require human judgment:
 Recognition
 Reasoning
 Context
 Prior Knowledge
 Algorithmic solutions are inefficient or impractical
 Requires flexibility due to unpredictable fluctuations in
volume
© 2011 Amazon.com, Inc. or its Affiliates.

5. What is
Mechanical Turk?
© 2011 Amazon.com, Inc. or its Affiliates.

6. Mechanical Turk is a
marketplace for work.
Mechanical Turk gives businesses and developers access to
an on-demand, scalable workforce.
 Flexibility: Scale your workforce up and down quickly
 Accuracy: Get high quality, efficient and cost effective
results.
 Price: Pay only when you are satisfied with the results.
 Speed: Start receiving results in minutes
© 2011 Amazon.com, Inc. or its Affiliates.

7. Workforce
500,000Workers
190+ Countries
 Who are the Workers?
 Workers are global, available 24x7
 Workers choose the work they want to do and work from
anywhere, anytime
 Workers are dedicated to your tasks and can work in
parallel
 Valuable source of feedback on how to improve your HITS
© 2011 Amazon.com, Inc. or its Affiliates.

8. Your Workers
 Identify and manage your best Workers
• All available Workers
All Workers • 500,000 +
• 24x7, 365
• Any Worker who
Your completes a HIT
Workers • Evaluate performance
on your HITS
Your Trusted • Your best Workers
with proven quality
Workers and availability
© 2011 Amazon.com, Inc. or its Affiliates.

9. How are businesses using it?
 Data Management  Categorization
 Data Verification  Classification
 Data Entry & Collection  Tagging
 Data De-duplication  Sentiment Analysis
 Algorithm Training
 Content & Media  Business Services
 Moderate Photos & Content  Search Relevancy
 Content Creation & Editing  Product Usability Testing
 Transcription  Research
© 2011 Amazon.com, Inc. or its Affiliates.

10. Search Enhancement / Relevance
 Search discovery requires categorization and expansion
or classification of attributes – keywords that users can
use to find content that relates to them.
 Adding meta data – such as detailed descriptions,
attribute tags and categories can improve discoverability.
 How it works:
Item description is Content is more
Workers add Tags are added as
sent to Mechanical easily discovered
creative & attributes to the
Turk for review by through Web and
descriptive tags item
Workers Site search
© 2011 Amazon.com, Inc. or its Affiliates.

11. Moderation of User Generated
Content
 The volume and speed at which user generated content
is being generated – and maintaining site guidelines is a
real problem faced by site publishers.
 Filtering content for appropriateness and usefulness
requires review..
 How it works:
Content is sent Web site
Workers review Inappropriate
to Mechanical maintains
according to content is
Turk for review community
rules deleted
by Workers expectation
© 2011 Amazon.com, Inc. or its Affiliates.

12. Data Handling
 Resolve Data conflicts directly in your business process
 Remove duplicates
 Normalize data entries
 Resolve conflicts and collisions
 Enhance entries
 How it works:
Items flagged
Business Workers review Business
for conflict are
Process Ingests item and Process
sent to
Data Elements resolve conflict continues
Mechanical Turk
© 2011 Amazon.com, Inc. or its Affiliates.

13. Advantages of Mechanical Turk
Gives you access to an on-demand workforce
Eliminates
100% Efficiency
Staffing Issues
Maintain the flexibility your business demands
Use only the capacity you need, when you need it
Converts a fixed cost into a variable cost
Lowers
Significantly reduces ongoing headcount expenses
Costs
Offers a pay-as-you-go staffing model
Eliminates lead time of interviewing and staffing
Reduces work
Work gets done faster, by working in parallel
turnaround time
Work continues around the globe, around the clock
© 2011 Amazon.com, Inc. or its Affiliates.

14. Getting Started
Easy, scalable options for organizations of all sizes
© 2011 Amazon.com, Inc. or its Affiliates.

15. How it works:.
“Validate,
“Design & Publish”
Pay & Go”
© 2011 Amazon.com, Inc. or its Affiliates.

16. Your Project  Tasks
 “Working Backwards”
 What are the business results that you want?
 How will you use those results?
 What format do the results need to be in?
 Which tasks benefits from human intelligence?
 Break your overall project into smaller “tasks”
 Consolidate like items and processes
 Define instructions for each task
© 2011 Amazon.com, Inc. or its Affiliates.

17. Example Workflow
 Content Moderation with API integration
© 2011 Amazon.com, Inc. or its Affiliates.

18. The Key components of your HIT
 Instruction
 Guidelines for how to
do a HIT
 Data
 “batch” concept
What kind of dress is this?
 Question(s) Cocktail
Sun dress
Bridal dress
© 2011 Amazon.com, Inc. or its Affiliates.

19. HIT Design
 Instructions
 The guidelines you provide on how to do your HIT
 Define what an acceptable answer looks like
 Account for ambiguity
 Ergonomics – get work done faster
 Design for Worker and reviewer efficiency
 Make sure all Workers do equal work for equal $
 Defensive design – make sure your questions can be answered as
expected
 Workflow
 Don‟t try to make one HIT do all of the work
 Split the process into logical stages
 Maintain „Like Reward for Like Work‟
© 2011 Amazon.com, Inc. or its Affiliates.

20. Work Catalog
 Workers
discover
available HITs
through the
listings catalog
 Name of
Requester,
Name of HIT,
Price and
Qualifications
are displayed
© 2011 Amazon.com, Inc. or its Affiliates.

21. Adjudication Strategies
 Leverage different methods of determining the correct
answer by designing for accuracy in your HIT:
 Multiple Choice
 Plurality
 Known Answers (“Gold Standards”)
 Free-form
 Multiple HIT Workflow‟s where a new Worker “edits” or
“grades” previous Worker submissions
 Creative Tasks
 Bonus Payments
© 2011 Amazon.com, Inc. or its Affiliates.

22. Balances & Levers
 Leverage market dynamics to optimize speed, price and
accuracy to meet your business needs.
Speed
Price
Accuracy
© 2011 Amazon.com, Inc. or its Affiliates.

23. Requester access:
Apps & Solution
Web UI API & CLT
Providers
• Easy Access • Programmatic • Experts at
• Create HITS in Access Workflow
HTML • Multiple SDKS design
• Upload and • Massive scale • They do the
manage data training and
via CSV management
of Workers for
specific kinds
of tasks
© 2011 Amazon.com, Inc. or its Affiliates.

24. Applications & Solution Providers
 Mechanical Turk supports a robust ecosystem of providers
that have built or can build solutions to meet your needs.
© 2011 Amazon.com, Inc. or its Affiliates.

25. Thank you.
John Hoskins
hoskins@amazon.com
206.266.8105
@amazonmturk
Facebook.com/amazonmturk
mechanicalturk.typepad.com
© 2011 Amazon.com, Inc. or its Affiliates.