3. http://lora-aroyo.org @laroyo 3
E-LEARNING & AI
To understand the value of semantic technologies for e-learning
we need to understand the people, specifically how they
interact and consume information
8. http://lora-aroyo.org @laroyo
EMPIRE OF THE EXPERTS
8
80’s
Advances in hardware and SDEs
PCs, workstations, Symbolics, Sun
New architectures like Hypercube
LISP, Prolog, OPS
AI can now BUILD SYSTEMS
Primary focus on experts and rules
What is the knowledge of experts
Graphs, logic, rules, frames
How do experts reason?
Deduction, induction
Work on form & process academic
inside the system, to make the
reasoning inside the system proper
and as good as possible
Industry forged ahead with ad-hoc &
proprietary systems and actually tried
to build expert systems
Originals of uncertain KR
Fuzzy, probabilistic
9. http://lora-aroyo.org @laroyo
EMPIRE OF THE EXPERTS
9
80’s
Piero Bonissone and
the DELTA/CATS expert
system for locomotive repair
with David Smith, a
locomotive repair expert
10. http://lora-aroyo.org @laroyo
EMPIRE OF THE EXPERTS
10
80’s
Buchanan and Shortliff’s
MYCIN project at Stanford
built a huge rule base
for medical diagnosis
working with an extensive
team of medical experts.
11. http://lora-aroyo.org @laroyo
KNOWLEDGE
ACQUISITION FROM
EXPERTS
11
90’s
Common KADS
founded by Bob Wielinga as
a methodology for expert
knowledge acquisition. It
was deeply psychology
based - it was about people,
about their knowledge and
especially about their
expertise. How do people
know what they know, and
how can you acquire that
knowledge?
19. http://lora-aroyo.org @laroyo
ONE TRUTH
19
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
7 MYTHS ABOUT HUMAN ANNOTATION
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
20. http://lora-aroyo.org @laroyo 20
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
7 MYTHS ABOUT HUMAN ANNOTATION
ALL EXAMPLES EQUAL
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
21. http://lora-aroyo.org @laroyo 21
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
7 MYTHS ABOUT HUMAN ANNOTATION
DISAGREEMENT BAD
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
22. http://lora-aroyo.org @laroyo 22
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
7 MYTHS ABOUT HUMAN ANNOTATION
EXPERTS RULE
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
23. http://lora-aroyo.org @laroyo 23
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
7 MYTHS ABOUT HUMAN ANNOTATION
ONE IS ENOUGH
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
24. http://lora-aroyo.org @laroyo 24
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if
examples cause disagreement - add
instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with
old7 MYTHS ABOUT HUMAN ANNOTATION
BINARY WORLD
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
25. http://lora-aroyo.org @laroyo 25
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
DOES THIS SENTENCE EXPRESS TREATS RELATION?
26. http://lora-aroyo.org @laroyo 26
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
DOES THIS SENTENCE EXPRESS TREATS RELATION?
27. http://lora-aroyo.org @laroyo 27
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
DOES THIS SENTENCE EXPRESS TREATS RELATION?
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
28. http://lora-aroyo.org @laroyo 28
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
WHAT DO EXPERTS SAY?
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
✓
✓
✘
29. http://lora-aroyo.org @laroyo 29
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
WHAT DOES A LAY ANNOTATOR SAY?
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
✓
✓
✘
30. http://lora-aroyo.org @laroyo 30
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
WHAT DOES ANOTHER LAY ANNOTATOR SAY?
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
✓
✘
✘
31. http://lora-aroyo.org @laroyo 31
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
WHAT DOES A THIRD LAY ANNOTATOR SAY?
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
✓
✓
✓
32. http://lora-aroyo.org @laroyo 32
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
WHAT DOES THE CROWD SAY?
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
Intuition: This is better
95%
75%
50%
33. http://lora-aroyo.org @laroyo 33
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
95%
75%
50%
There’s a difference between these two
This one isn’t utterly wrong
BETTER
WORSE
WHAT DOES THE CROWD SAY?
34. http://lora-aroyo.org @laroyo 34
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if
examples cause disagreement - add
instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with
old
COMFORT ZONE
Disrupted
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
35. http://lora-aroyo.org @laroyo 35
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if
examples cause disagreement - add
instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with
old
COMFORT ZONE
Disrupted
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
36. http://lora-aroyo.org @laroyo 36
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if
examples cause disagreement - add
instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with
old
COMFORT ZONE
Disrupted
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
37. http://lora-aroyo.org @laroyo 37
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if
examples cause disagreement - add
instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with
old
COMFORT ZONE
Disrupted
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
38. http://lora-aroyo.org @laroyo 38
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if
examples cause disagreement - add
instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with
old
COMFORT ZONE
Disrupted
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
39. http://lora-aroyo.org @laroyo 39
For prevention of malaria, use only in individuals traveling to
malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
ENCOURAGING DISAGREEMENT
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
Intuition: This is better
95%
75%
50%
41. http://lora-aroyo.org @laroyo
WORKER VECTOR FOR A SENTENCE
treats associated _with othersymptom
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
42. http://lora-aroyo.org @laroyo
MANY WORKERS FOR THE SAME SENTENCE
treats otherassociated _withsymptom
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
43. http://lora-aroyo.org @laroyo
ALL WORKER VECTORS
AGGREGATED IN A SENTENCE VECTOR
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
treats othernoneassociated _withsymptommanifestation
side
effect
44. http://lora-aroyo.org @laroyo
SENTENCE VECTORS FOR THE 3 SENTENCES
treats othernoneassociated _withsymptommanifestation
side
effect
treats othernoneassociated _withcontraindicatesmanifestation
treats other
45. http://lora-aroyo.org @laroyo 45
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if
examples cause disagreement - add
instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with
old
TIME TO DISRUPT THE
COMFORT ZONE
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
71. http://lora-aroyo.org @laroyo
treats othernoneassociated _withsymptommanifestation
side
effect
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
Sentence ambiguity
QUALITY ASSESSMENT
74. http://lora-aroyo.org @laroyo
Agreement as percentage
CROWDTRUTH METRICS
For prevention of malaria, use only in individuals traveling to
malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
25% 25% 75% 12% 12% 50%
treats othernoneassociated _withcontraindicatesmanifestation
75. http://lora-aroyo.org @laroyo
For prevention of malaria, use only in individuals traveling to
malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
CROWDTRUTH METRICS
Applying all sides of the triangle
treats other
76. http://lora-aroyo.org @laroyo
CROWDTRUTH METRICS
For prevention of malaria, use only in individuals traveling to
malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
treats other
Applying all sides of the triangle
99%
83. http://lora-aroyo.org @laroyo
On the role of user-generated metadata in audio visual collections (2011).
R. Gligorov, M. Hildebrand, J. van Ossenbruggen, G. Schreiber, L. Aroyo K-CAP2011
VIDEO METADATA ENRICHMENT
The Netherlands Institute for Sound and Vision
1
90. http://lora-aroyo.org @laroyo
H2020 ReTV
Trans-Vector Platform (TVP)
Lora Aroyo, (coordinator) VU
Amsterdam, Computer Science
Lyndon Nixon, MODUL, AT
Vasileios Mezaris, CERTH, GR
Arno Scharl, Weblyzard, AT
Bea Knecht, Zattoo, DE
Johan Oomen, Sound and Vision, NL
Nicolas Patz, Rundfunk
Berlin-brandenburg, DE
91. http://lora-aroyo.org @laroyo
CAPTURING BIAS
Startimpuls for the Dutch National Science Agenda
Lora Aroyo, (coordinator) VU
Amsterdam, Computer Science
Alessandro Bozzon, TU Delft CS &
Delft Data Science
Alec Badenoch, Utrecht University,
Media & Culture Studies
Antoaneta Dimitrova, Leiden
University, Institute of Public
Administration
Johan Oomen, Netherlands
Institute for Sound and Vision
92. http://lora-aroyo.org @laroyo
CROWDTRUTH ROCKS!
Disagreement is signal
CrowdTruth is a spatial representation
of meaning that harnesses disagreement
Crowds bring natural diversity
CrowdTruth defines hyper-dimensional
space to represent ambiguity
Crowds help gathering
real human semantics
93. http://lora-aroyo.org @laroyo
The world is full
of shades of grey
Experts and crowds
are complimentary
Capturing and understanding opinions,
perspectives and contexts in the center
of understanding people
TIME TO BREAK FREE
CrowdTruth defines multi-dimensional
space to measure quality