1. Crowdsourcing for Watson
Lora Aroyo
Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo
Monday, December 10, 12

2. Observation
language is ambiguous & vague
people interpret it from different
hu points of view
am m
big an
uit dis
y o agr
r v ee
ag me
ue nt
ne m
ss ay
of sig
lan na
gu l
ag
e
Flickr: elkabong
Monday, December 10, 12

3. our goal is ...
not to enforce agreement
to capture different view points
adapt to different annotation tasks
Flickr: elkabong
Monday, December 10, 12

4. so we need to ...
understand the range of disagreements by creating a
space of possibilities with frequencies & similarities
teach machines to reason in the disagreement space
score their output based on where it falls in this space
Flickr: elkabong
Monday, December 10, 12

5. Harnessing the Crowd
not just to save money
Flickr: AmyJanelle
Monday, December 10, 12

6. Monday, December 10, 12

7. Monday, December 10, 12

8. Worker
#1
MICRO-­‐TASK ?
Answer
?
?
Worker
#2
Answer
?
Worker
#3
Answer
Worker
#N
Answer
Monday, December 10, 12

9. The Task Answers
Lora Aroyo
Monday, December 10, 12

10. The Task Answers
Lora Aroyo
Monday, December 10, 12

11. The Task Answers
Lora Aroyo
Monday, December 10, 12

12. The Task Answers
Lora Aroyo
Monday, December 10, 12

13. The Task Answers
Lora Aroyo
Monday, December 10, 12

14. OLANZAPINE is an atypical antipsychotic, approved by the U.S.Food
and Drug Administration (FDA) for the treatment of SCHIZOPHRENIA
and bipolar disorder.
Is SCHIZOPHRENIA related to OLANZAPINE ?
treated_by treats may_treat
RESPIRATORY ALKALOSIS is a medical condition in which increased
respiration (HYPERVENTILATION) elevates the blood pH (a condition
generally called alkalosis).
RESPIRATORY
Is HYPERVENTILATION related to
ALKALOSIS
?
diagnosed_by cause_of cause symptom_of
Lora Aroyo
Monday, December 10, 12

15. OLANZAPINE is an atypical antipsychotic, approved by the U.S.Food
and Drug Administration (FDA) for the treatment of SCHIZOPHRENIA
and bipolar disorder.
Is SCHIZOPHRENIA related to OLANZAPINE ?
treated_by treats may_treat
RESPIRATORY ALKALOSIS is a medical condition in which increased
respiration (HYPERVENTILATION) elevates the blood pH (a condition
generally called alkalosis).
RESPIRATORY
Is HYPERVENTILATION related to
ALKALOSIS
?
diagnosed_by cause_of cause may_cause symptom_of
Lora Aroyo
Monday, December 10, 12

16. He was the first physician to identify the relationship between
HEMOPHILIA and HEMOPHILIC ARTHROPATHY.
HEMOPHILIC
Is HEMOPHILIA related to ?
ARTHROPATHY
other
treated_by cause_of cause may_cause symptom_of
has_manifestation
It just says there is a relation between the two
but gives no specifics about what the relation is
There is a relationship between the two disorders but the
sentence does not indicate what that relationship is.
identify the relationship between
Lora Aroyo
Monday, December 10, 12

17. Sentence-Relation Distribution
Lora Aroyo
Monday, December 10, 12

18. Expert Annotators
Lora Aroyo
Monday, December 10, 12

19. Crowd vs. Experts
Lora Aroyo
Monday, December 10, 12

20. Crowd vs. Experts
x
Lora Aroyo
Monday, December 10, 12

21. The Dark Side of Crowdsourcing
Disagreement
• disagreement is beautiful, except when it results from spamming
• crowdsourcing has to account for people that want to get paid for
not doing any work
• spammers generate disagreement for the wrong reasons
• most spam detection requires gold standard
Monday, December 10, 12

22. Position
Artificially
restricting
humans d
Machines oes not h
will learn elp mach
from dive ines to le
arn.
rsity
Flickr: elkabong
Monday, December 10, 12

23. JOIN our team
• Dr. Watson choose
Medical Synonyms Game Your
• Medical Detective Thesis
Gamification of Medical project
Diagnosis
• To CATCH a Thief
SPAM detection in
call now:
lora.aroyo
crowdsourcing with no Gold
Standard @vu.nl
Monday, December 10, 12