The document outlines the schedule and content for a CrowdFlower University training. The day includes sessions on workforce management and the CrowdFlower platform, process deconstruction and job creation, job monitoring and reports, case studies, and the product roadmap. Breaks are scheduled between sessions.

1. CrowdFlower University
October 21, 2013

2. Schedule for the day
1:00 - 1:05 // Arrive, Plug-in
1:05 - 1:30 // Intro to Workforce Management and the
CrowdFlower Platform
1:30 - 2:30 // Process Deconstruction & Job Creation
2:30 - 2:40 // Break
2:40 - 3:30 // Job Monitoring and Reports
3:30 - 4:00 // From the Field: Case Studies
4:00 - 4:10 // Break
4:10 - 4:30 // What’s to come: Product Roadmap
4:30 - 5:00 // Q&A
2
CrowdFlower, Inc. – Proprietary and Confidential

3. Purpose
CrowdFlower University will teach you the CrowdFlower
Platform, empowering you to manage workforces online.
3
CrowdFlower, Inc. – Proprietary and Confidential

4. Success Center
4
CrowdFlower, Inc. – Proprietary and Confidential

5. Success Center
success.crowdflower.com
5
CrowdFlower, Inc. – Proprietary and Confidential

6. Workforce Management
6
CrowdFlower, Inc. – Proprietary and Confidential

7. Workforce Management
Geography
Performance Levels and Skills
Workflows
Test Questions
Channels
7
CrowdFlower, Inc. – Proprietary and Confidential

8. Geography
8
CrowdFlower, Inc. – Proprietary and Confidential

9. Geography
9
CrowdFlower, Inc. – Proprietary and Confidential

10. Performance Levels and Skills
10
CrowdFlower, Inc. – Proprietary and Confidential

11. Skills and Performance Levels
The CrowdFlower Platform enables you to add
Skill requirements and Performance Levels to
your job. Enabling the Skilled Crowds will likely
increase the quality of your results and may
decrease the throughput of your judgments.
11
CrowdFlower, Inc. – Proprietary and Confidential

12. Workflows
12
CrowdFlower, Inc. – Proprietary and Confidential

13. Workflows
13
CrowdFlower, Inc. – Proprietary and Confidential

14. Test Questions
14
CrowdFlower, Inc. – Proprietary and Confidential

15. Test Questions
Test Questions are a powerful method for
measuring the quality of the contributors in your
job. By providing the answers to a small
percentage of units, you can tell how accurate
someone is, and automatically remove anyone
who doesn't meet your requirements.
15
CrowdFlower, Inc. – Proprietary and Confidential

16. Channels
16
CrowdFlower, Inc. – Proprietary and Confidential

17. Channels
17
CrowdFlower, Inc. – Proprietary and Confidential

18. CrowdFlower Terminology
18
CrowdFlower, Inc. – Proprietary and Confidential

19. CrowdFlower Terminology
•
•
Unit - One row on a spreadsheet (One company, one
officer)
•
Dataset - The entire set of data for a project
•
Task - A subset of data from the whole dataset
•
Workflow - Multiple jobs that strung together
•
Channel - A website in which a contributor can access
a task
•
Test Questions - Training data created in the platform
•
19
Judgment - One (of many) answers given by an
individual contributor
Contributor - Person completing assignments online
CrowdFlower, Inc. – Proprietary and Confidential

20. Process Deconstruction and Job
Creation
20
CrowdFlower, Inc. – Proprietary and Confidential

21. Process Deconstruction & Job Creation
Let’s learn how to build a job!
Today we’re going to build and run a sentiment analysis
job about CrowdFlower.
21
CrowdFlower, Inc. – Proprietary and Confidential

22. Best Practices – Job Construction
22
CrowdFlower, Inc. – Proprietary and Confidential

23. Best Practices – Job Construction
•
•
Leverage the Template Library
•
Use the Instructions Template
•
Help contributors find the data (provide links, images,
etc)
•
Provide examples in the instructions
•
Leverage CML logic wherever possible
•
Use validations to control text inputs
•
23
Break down problem into smallest components
Allow an "escape" option for multiple choice
questions regarding information in the job or a link: "I
don't know", "I couldn't find it", "the link is broken",
etc.
CrowdFlower, Inc. – Proprietary and Confidential

24. Best Practices – Test Questions
24
CrowdFlower, Inc. – Proprietary and Confidential

25. Best Practices – Test Questions
•
•
Enable random data for Test Questions
•
Copy units to source
•
Be as thorough in your Test Question reasons as
possible – these will help train the crowd throughout
the task
•
Create Test Questions out of edge case (subjective
or tricky) units to train the crowd
•
You don’t need to answer every question
•
25
Try to create an even distribution of answers
For subjective questions you can allow for multiple
correct answers
CrowdFlower, Inc. – Proprietary and Confidential

26. Best Practices – Job Calibration
26
CrowdFlower, Inc. – Proprietary and Confidential

27. Best Practices – Job Calibration
•
•
Set variable judgments on trickier CML
elements
•
27
Set the Max Judgments Per IP at the number
of units in an assignment multiplied by the
amount of gold in the task
• This ensures that a contributor will not
see the same gold unit twice
Set threshold for trust of contributors
CrowdFlower, Inc. – Proprietary and Confidential

28. Launch!
28
CrowdFlower, Inc. – Proprietary and Confidential

29. Task Monitoring and Reports
29
CrowdFlower, Inc. – Proprietary and Confidential

30. Task Monitoring and Reports
Task Monitoring
Interpreting Reports
Interpreting Results
*This session will teach you how to understand your results.
30
CrowdFlower, Inc. – Proprietary and Confidential

31. Task Monitoring
31
CrowdFlower, Inc. – Proprietary and Confidential

32. Task Monitoring
• Review Test Questions
• Monitor Contributors
32
CrowdFlower, Inc. – Proprietary and Confidential

33. Interpreting Reports
33
CrowdFlower, Inc. – Proprietary and Confidential

34. Interpreting Reports
34
CrowdFlower, Inc. – Proprietary and Confidential

35. Interpreting Results
35
CrowdFlower, Inc. – Proprietary and Confidential

36. Interpreting Results
• Confidence Scores
• Aggregation Methods
• Auditing for Accuracy
36
CrowdFlower, Inc. – Proprietary and Confidential

37. Case Studies
37
CrowdFlower, Inc. – Proprietary and Confidential

38. Case Studies
In this session we will cover several practical applications
of the CrowdFlower Platform.
38
CrowdFlower, Inc. – Proprietary and Confidential

39. Search Relevance Testing
39
CrowdFlower, Inc. – Proprietary and Confidential

40. Example 1
40
CrowdFlower, Inc. – Proprietary and Confidential

41. Example 2
41
CrowdFlower, Inc. – Proprietary and Confidential

42. Master Data Management
42
CrowdFlower, Inc. – Proprietary and Confidential

43. Find URL
43
CrowdFlower, Inc. – Proprietary and Confidential

44. Find About Us Page
44
CrowdFlower, Inc. – Proprietary and Confidential

45. Categorize Businesses
45
CrowdFlower, Inc. – Proprietary and Confidential

46. De-Duplication
46
CrowdFlower, Inc. – Proprietary and Confidential

47. Content Categorization
47
CrowdFlower, Inc. – Proprietary and Confidential

48. Photo Categorization
48
CrowdFlower, Inc. – Proprietary and Confidential

49. Product Categorization
49
CrowdFlower, Inc. – Proprietary and Confidential

50. Content Moderation
50
CrowdFlower, Inc. – Proprietary and Confidential

51. Photo Moderation
51
CrowdFlower, Inc. – Proprietary and Confidential

52. Text Moderation
52
CrowdFlower, Inc. – Proprietary and Confidential

53. Image and Text Interpretation
53
CrowdFlower, Inc. – Proprietary and Confidential

54. Image Transcription
54
CrowdFlower, Inc. – Proprietary and Confidential

55. OCR Verification
55
CrowdFlower, Inc. – Proprietary and Confidential

56. Product Roadmap
56
CrowdFlower, Inc. – Proprietary and Confidential

57. Q4 2013
Automatic Test Question access for Enterprise customers
only
A wizard for first-time users so that your team needs less
training
A reputation score for our contributors and improvements
to our performance levels
57
CrowdFlower, Inc. – Proprietary and Confidential

58. Q1 2014
In-platform templates for you to get started quickly
Improvements to our GUI
Team collaboration on jobs
In-platform auditing tool so that your internal team and
our strongest contributors can easily evaluate the output
of your job
58
CrowdFlower, Inc. – Proprietary and Confidential

59. And beyond…
New task analytics for improved slicing and dicing of your
data
Workflow building tool
Minimize human input (e.g. variable pay where you set
the budget, we set the pay for contributors)
59
CrowdFlower, Inc. – Proprietary and Confidential

60. Questions?
60
CrowdFlower, Inc. – Proprietary and Confidential

61. Contact Me
Nathan Zukoff
Platform Program Manager
nathan@crowdflower.com
973-997-1921
61
CrowdFlower, Inc. – Proprietary and Confidential