Human Computation

HUMAN COMPUTATION
Irene Celino – irene.celino@cefriel.com
Cefriel, Viale Sarca 226, 20126 Milano
Seminar @ Data Semantics course – April 11th, 2018

1. Introduction
2. Linked Data and Knowledge Graph Refinement
3. Human Computation and Games with a Purpose
4. Examples of GWAP for Data Linking
5. Truth Inference and Open Science
6. Guidelines
7. Indirect People Involvement
2copyright © 2018 Cefriel – All rights reserved

from ideation to business value
3
1. INTRODUCTION
Is the Web a pure technological artefact?
What role can people play on the Web?
copyright © 2018 Cefriel – All rights reserved

WEB AS A SOCIAL ARTEFACT
“The Web isn’t about what you can do with computers.
It’s people and, yes, they are connected by computers.
But computer science, as the study of
what happens in a computer, doesn’t tell
you about what happens on the Web”
– sir Tim Berners-Lee

Open Source Software
“Given enough eyeballs, all bugs are
shallow.” Eric S. Raymond
(The Cathedral and the Bazaar)
OPEN EVERYTHING
Open Content
“It is easy when you skip the
intermediaries”
original motto of Creative Commons
(EN video) (IT video)
Open Data
“Raw. Data. Now.” Tim Berners-Lee
(The year open data went worldwide –
TED Talk)

COOPERATION ON THE WEB TO PRODUCE OPEN KNOWLEDGE

WISDOM OF CROWDS
• “Why the Many Are Smarter Than the Few and
How Collective Wisdom Shapes Business, Economies, Societies and Nations”
• Criteria for a wise crowd
• Diversity of opinion (importance of interpretation)
• Independence (not a “single mind”)
• Decentralization (importance of local knowledge)
• Aggregation (aim to get a collective decision)
• The are also failures/risks in crowd decisions:
• Homogeneity, centralization, division, imitation, emotionality
James Surowiecki
The wisdom of crowds
Anchor, 2005

8
2. LINKED DATA & KNOWLEDGE
GRAPH REFINEMENT
Do we need to involve people in Semantic Web systems?
What semantic data management tasks can we effectively “outsource” to humans?

HUMANS IN THE SEMANTIC WEB
• Knowledge-intensive and/or context-speciﬁc character of Semantic Web tasks:
• e.g., conceptual modelling, multi-language resource labelling, content annotation with
ontologies, concept/entity similarity recognition, …
• Need to engage users and involve them in executing tasks:
• e.g., wikis for semantic content authoring, folksonomies to bootstrap formal
ontologies, instance creation by data entry, …

SEMANTIC WEB TASKS (ALSO) FOR HUMANS
Fact level
Schema level
Collection Creation CorrectionValidation Filtering Ranking Linking
Conceptual
modelling
Ontology
population
Quality
assessment
Ontology re-
engineering
Ontology
pruning
Ontology
elicitation
Knowledge
acquisition
Ontology
repair
Knowledge
base update
Data search/
selection Link
generation
Ontology
alignment
Ontology
matching

AUTOMATIC METHODS IN THE SEMANTIC WEB?
• Knowledge Graph Refinement (and, in general, linked dataset refinement) is an
emerging and hot topic to (1) identify and correct errors and (2) add missing knowledge
• e.g., completing type assertions via classification, predicting relations from textual
sources, finding erroneous type assertions, identifying erroneous literal values
through anomaly/outlier detection, …
• Statistical and machine learning approaches require some partial gold standard,
i.e. a “ground truth” dataset to train automatic models
• Ground truth is usually put together manually by expert
• Sourcing gold standard from humans is expensive!
Heiko Paulheim. Knowledge graph refinement: A survey of
approaches and evaluation methods. Semantic Web Journal, 2017

DATA LINKING
• Creation of links in the form of RDF triples (subject, predicate, object)
• Within the same dataset (i.e. generating new connections between resources of the
same dataset or knowledge graph)
• Across different datasets (i.e. creating RDF links, as named in the Linked Data world)
• Note:
• In literature, data linking often means finding equivalent resources (similarly to record
linkage in database research), i.e. triples with correspondence/match predicate (e.g.
owl:sameAs)  in the following, data linking is intended in its broader meaning (i.e. links
with any predicate)

DATA LINKING: SOME DEFINITIONS
• Resources R is the set of all resources (and literals), whenever possible also described by the
respective types. More specifically: R = Rs ∪ Ro, where Rs is the set of resources that can take the
role of subject in a triple and Ro is the set of resources that can take the role of object in a triple; as said
above the two sets are not necessarily disjoint, i.e. it can happen that Rs ∩ Ro ≠ ∅.
• Predicates P is the set of all predicates, whenever possible also described by the respective domain
and range.
• Links L is the set of all links; since links are triples created between resources and predicates it is:
L ⊂ Rs × P × Ro; each link is defined as l = (rs,p,ro) ∈ L with rs ∈ Rs, p ∈ P, ro ∈ Ro.
L is usually smaller than the full Cartesian product of Rs, P, Ro, because in each link (rs,p,ro) it
must be true that rs ∈ domain(p) and ro ∈ range(p).
• Link scores σ is the score of a link, i.e. a value indicating the confidence on the truth value of the link;
usually σ ∈ [0,1]; each link l ∈ L can have an associated score.

CASES OF DATA LINKING
• Link creation: a link l is created: given R = Rs ∪ Ro and P, the link l = (rs,p,ro), with rs ∈ Rs,
p ∈ P, ro ∈ Ro is created and added to L
• e.g., music classification: assign one or more music styles to audio tracks by creating the link
(track,genre,style)
• Link ranking: given the set of links L, a score σ ∈ [0,1] is assigned to each link l. The score
represents the probability of the link to be recognized as true. Links can be ordered on the basis of their
score σ, thus obtaining a ranking
• e.g., ranking photos depicting a specific person (an actor, a singer, a politician) to identify the
pictures in which the person is more recognizable or more clearly depicted
• Link validation: given the set of links L, a score σ ∈ [0,1] is assigned to each link l. The score
represents the actual truth value of the link. A threshold t ∈ [0,1] is set so that all links with score
σ ≥ t are considered true
• e.g., assessing the correct music style identification in audio tracks (music classification)

15
3. HUMAN COMPUTATION &
GAMES WITH A PURPOSE
What goals can humans help machines to achieve? How to involve a crowd of persons?
What extrinsic rewards (money, prizes, etc.) or intrinsic incentives can we adopt to
motivate people?

HUMAN COMPUTATION
• Human Computation is a computer science technique in which a computational process
is performed by outsourcing certain steps to humans. Unlike traditional computation,
in which a human delegates a task to a computer, in Human Computation the computer
asks a person or a large group of people to solve a problem; then it collects, interprets
and integrates their solutions
• The original concept of Human Computation by its inventor Luis von Ahn derived from the
common sense observation that people are intrinsically very good at solving some
kinds of tasks which are, on the other hand, very hard to address for a computer;
this is the case of a number of targets of Artiﬁcial Intelligence (like image recognition or
natural language understanding) for which research is still open
Edith Law and Luis von Ahn. Human computation.
Synthesis Lectures on Artiﬁcial Intelligence and Machine Learning, 2011

HUMAN COMPUTATION
Problem: an Artificial Intelligence
algorithm is unable to achieve an
adequate result with a satisfactory
level of confidence
Solution: ask people to intervene
when the AI system fails, “masking”
the task within another human
process
Example: https://www.google.com/recaptcha/

CROWDSOURCING
• Crowdsourcing is the process to outsource tasks to a “crowd” of distributed people.
The possibility to exploit the Internet as vehicle to recruit contributors and to assign
tasks led to the rise of micro-work platforms, thus often (but not always) implying a
monetary reward. The term Crowdsourcing, although quite recent, is used to indicate a
wide range of practices; however, the most common meaning of Crowdsourcing implies
that the “crowd” of workers involved in the solution of tasks is different from the traditional
or intended groups of task solvers
Jeff Howe. Crowdsourcing: How the power of the crowd
is driving the future of business. Random House, 2008

CROWDSOURCING
Problem: a company needs to
execute a lot of simple tasks,
but cannot afford hiring a
person to do that job
Solution: pack tasks in
bunches (human intelligence
tasks or HITs) and outsource
them to a very cheap workforce
through an online platform
Example: https://www.mturk.com/

CITIZEN SCIENCE
• Citizen Science is the involvement of volunteers to collect or process data as part of
a scientific or research experiment; those volunteers can be the scientists and
researchers themselves, but more often the name of this discipline “implies a form of
science developed and enacted by citizens” including those “outside of formal scientific
institutions”, thus representing a form of public participation to science. Formally, Citizen
Science has been defined as “the systematic collection and analysis of data; development
of technology; testing of natural phenomena; and the dissemination of these activities by
researchers on a primarily avocational basis”.
Alan Irwin. Citizen science: A study of people, expertise
and sustainable development. Psychology Press, 1995

CITIZEN SCIENCE
Example: https://www.zooniverse.org/
Problem: a scientific
experiment requires the
execution of a lot of simple
tasks, but researchers are busy
Solution: engage the general
audience in solving those tasks,
explaining that they are
contributing to science,
research and the public good

SPOT THE DIFFERENCE…
• Similarities:
• Involvement of people
• No automatic replacement
• Variations:
• Motivation
• Reward (glory, money, passion/need)
• Hybrids or parallel!
Citizen Science
Crowdsourcing
Human
Computation

GAMES WITH A PURPOSE
• A GWAP lets to outsource to humans some steps of a computational process in an
entertaining way
• The application has a “collateral effect”, because players’ actions are exploited to
solve a hidden task
• The application *IS* a fully-fledged game (opposed to gamification, which is the use
of game-like features in non-gaming environments)
• The players are (usually) unaware of the hidden purpose, they simply meet game
challenges
Luis Von Ahn. Games with a purpose. Computer, 39(6):92–94, 2006
Luis Von Ahn and Laura Dabbish. Designing games with a purpose.
Communications of the ACM, 51(8):58–67, 2008

GAMES WITH A PURPOSE (GWAP)
Problem: it’s the same of
Human Computation (ask
humans when AI fails)
Solution: Solution: hide the
task within a game, so that
users are motivated by game
challenges, often remaining
unaware of the hidden purpose,
task solution comes from
agreement between players

25
4. GWAPS FOR DATA LINKING
Can we embed data linking tasks within Games with a Purpose?

26
• Input: set of all links
<asset>
foaf:depiction
<photo>
• Goal: assign score 𝜎 to
rank links on their
recognisability/representa-
tiveness
• The score 𝜎 is a function of
𝑋 𝑁 where 𝑋 is the no. of
successes (=recognitions)
and 𝑁 the no. of trials of
the Bernoulli process
(guess or not guess)
realized by the game
• Cultural heritage assets in Milano and their pictures
LINK RANKING
http://bit.ly/indomilando
Pure GWAP with
hidden purpose
Points, badges,
leaderboard as
intrinsic reward
Link ranking is a result
of the “agreement”
between players
But also an
educational
“collateral effect”
Irene Celino, Andrea Fiano, Riccardo Fino. Analysis of a Cultural Heritage Game with a Purpose
with an Educational Incentive. 16th International Conference on Web Engineering, 2016

27
• Input: set of links
<land-area>
clc:hasLandCover
<land-cover>
• Goal: assign score 𝜎 to
each link to discover the
“right” land cover class
• Score 𝜎 of each link is
updated on the basis of
players’ choices
(incremented if link
selected, decremented if
link not selected)
• When the score of a link
overcomes the threshold
𝜎 ≥ 𝑡 , the link is considered
“true” (and removed from
the game)
• Two automatic classifications in disagreement:
<land-cover-assigned-by-DUSAF> ≠ <land-cover-assigned-by-GL30>
LINK VALIDATION
https://youtu.be/Q0ru1hhDM9Q
http://bit.ly/foss4game
Pure GWAP with
not-so-hidden purpose
(played by “experts”)
Points, badges,
leaderboard as
intrinsic reward
A player scores if he/she
guess one of the two
disagreeing classifications
Link validation is a result
of the “agreement”
between players
Maria Antonia Brovelli, Irene Celino, Andrea Fiano, Monia Elisa Molinari, Vijaycharan Venkatachalam.
A crowdsourcing-based game for land cover validation. Applied Geomatics, 2017

28
• Input: set of subject
resources (pictures) and
object resources
(classification categories)
• Goal: create links
<picture> hasCategory
<category> and assign
score 𝜎 to each link
• Score 𝜎 of each link is
updated on the basis of
players’ choices
(incremented if link
selected)
• When the score of a link
overcomes the threshold
𝜎 ≥ 𝑡 , the link is considered
“true” (and the picture is
removed from the game)
• Identify pictures of cities from above between those taken on board of the ISS (the pictures are
used then in a scientific process in light pollution research)
LINK COLLECTION & VALIDATION
http://nightknights.eu
Pure GWAP with
not-so-hidden purpose
(but played by anybody)
Points, badges,
leaderboard as
intrinsic reward
A player scores if he/she
agrees with another player
“Bonus” intrinsic reward
with NASA pictures!
Gloria Re Calegari, Gioele Nasi, Irene Celino. Human Computation vs. Machine Learning:
an Experimental Comparison for Image Classification. Human Computation Journal, 2018.

29
5. TRUTH INFERENCE &
OPEN SCIENCE
How do we aggregate the contributions from the crowd?
Are individual contribution of any value?

AGGREGATION OF CONTRIBUTIONS
• The same task is usually given to multiple human contributors (named workers in crowdsourcing)
• Results on the same task are then aggregated across different contributors (“wisdom of crowds”)
• How to perform the truth inference process?
• Simplistic solution: majority voting across all contributors
• But… are all contributors “created equal”? No! Less simplistic solutions:
• Majority voting across “quality” contributors (filtering out “spammers”)
• Weighted majority voting with estimation of contributors “reliability”
• Expectation maximization
• Message passing… and a lot more!
• How to compute contributor reliability?
• Assessment tasks (gold standard) with known solution to measure reliability
• History of contributions/past behaviours to compute a “reputation” value

TRUTH INFERENCE GENERIC ALGORITHM
Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, Reynold Cheng.
Truth Inference in Crowdsourcing: Is the Problem Solved? VLDB 2017
Input: contributions
Output: truth and reliability
Step 2: compute an estimation
of contributor reliability (e.g.
precision on truth estimation)
Step 1: compute an
estimation of the truth
(e.g. majority voting)
Iterate until convergence (e.g.
until some difference w.r.t.
previous step is really small)

OPEN SCIENCE: ENABLING COMPARE & CONTRAST
• Open Science has the aim to make scientific research and data accessible to all levels of society
• Repeatability and reproducibility are among the foundational principles of open science
• Human Computation aims at involving people in some step of the scientific process
• Human contributors generate data to solve assigned tasks
• Algorithms aggregate contributions in the truth inference process
• Can we compare different truth inference algorithms?
• Yes, if we make available the data of the Human Computation process!
• What can we share, e.g. in the case of data linking tasks?
• “True” and “false” links
• Confidence scores of the links
• Individual contributions and aggregation process

PROV-O AND HUMAN COMPUTATION ONTOLOGY
• Provenance is information about entities, activities, and people involved in producing
a piece of data or thing (used to assess its quality, reliability or trustworthiness)
• W3C defined the PROV-O ontology to capture provenance information
https://www.w3.org/TR/prov-o/
• The Human Computation ontology extends PROV-O
to describe the data shared within a Human Computation Process
http://swa.cefriel.it/ontologies/hc
• Data linking process information can be published
according to linked data principles described with the HC ontology
(e.g. data from the Urbanopoly GWAP at http://swa.cefriel.it/linkeddata/)
aggregatedFrom
Contributor
Contribution
Human
Computation Task
provo:Agent
provo:Entity
provo:Activity
Consolidated
Information
solvedBy
enabledBy
contributionFrom
solutionTo
aggregatedBy
Human
Computation
Algorithm
Irene Celino. Human Computation VGI Provenance: Semantic Web-based Representation and Publishing.
IEEE Transactions on Geoscience and Remote Sensing, 2013

34
6. GUIDELINES
Is it that easy to involve people on the Web?
What should we care of when designing a human computation system?

MICE AND MEN (OR: KEEP IT SIMPLE)
• Crowdsourcing workers behave like mice
• Mice prefer to use their motor skills (biologically cheap, e.g. pressing a lever to get food) rather
than their cognitive skills (biologically expensive, e.g. going through a labyrinth to get food)
• Workers prefer/are better at simple tasks (e.g. those that can be solved at first sight) and
discard/are worse at more complex tasks (e.g. those that require logics)
• Crowdsourcing tasks should be carefully designed
• Tasks as simple as possible for the workers to solve
• Complex tasks together with other incentives (e.g. variety/novelty)
Panos Ipeirotis. On Mice and Men: The Role of Biology in Crowdsourcing,
Keynote talk at Collective Intelligence, 2012.

DIVIDE ET IMPERA (OR: FIND-FIX-VERIFY)
• Find-Fix-Verify crowd programming pattern
• A long and “expensive” task…
• Summarize a text to shorten its total length
• …is decomposed in more atomic tasks…
1. find sentences that need to be shortened
2. fix a sentence by shortening it
3. verify which summarized sentence maintains original meaning
• …and the complex task is turned into a workflow of simple
tasks, and each step is outsourced to a crowd
M. Bernstein, G. Little, R. Miller, B. Hartmann, M. Ackerman, D. Karger, D. Crowell, K. Panovich.
Soylent: A Word Processor with a Crowd Inside, UIST Proceedings, 2010.

COMPARE AND CONTRAST
• A sort of “wisdom of the crowd(sourcing methods)”:
(1) apply different approaches to solve the same problem
and (2) compare results
• Which is the best approach
for a specific use case?
• Which is the most suitable crowd?
• Is human computation better/faster/cheaper
than machine computation?
• Knowledge Graph Refinement: use Human Computation
to “crowdsource” a gold standard and then use it to train
some statistical/machine learning algorithm
input
task
output
solution
Human
Computation
Machine
Computation
input
task
output
solution
Human Computation
Machine Computation
input
task
output
solution
Machine
Computation
Human
Computation
input
task
output
solution
Machine
Computation
Human
Computation
Human
Computation
Gloria Re Calegari, Gioele Nasi, Irene Celino. Human Computation vs. Machine Learning:
an Experimental Comparison for Image Classification. Human Computation Journal, 2018.

FINAL NOTE ON DISAGREEMENT
• Is there always a “right answer”? Or is there a “crowd truth”?
• Not always true/false, because of human subjectivity,
ambiguity and uncertainty
• Disagreement across contributors is not necessarily bad,
but a sign of: different opinions, interpretations, contexts,
perspectives, …
• Remember the long tail theory…
• …and ask yourself who are your users
and who you want to involve
Lora Aroyo, Chris Welty. Truth is a Lie: 7 Myths about Human Annotation. AI Magazine 2014.

39
7. INDIRECT PEOPLE INVOLVEMENT
Are there indirect ways to involve humans in data processing?

HUMANS AS A SOURCE OF INFORMATION
• People are not only task executors, they are also information providers!
• Opportunistic sensing
• Voluntary or involuntary digital traces of human-related activities
• e.g., phone call logs, GPS traces, social media activities
• Open content and cooperative knowledge
• Data explicitly provided by people can “hide” further information
• e.g., logs of wiki editing, statistical distribution of contributes

FROM POI INFORMATION AND PHONE CALL LOGS TO LAND USE
• General topic: exploit “low-cost” information about a geographic area as features to
train a predictive model that outputs “expensive” information about the same area
• “Inexpensive” input information:
• Geo-information about points of interests
• Mobile traffic data processed using different time series techniques –
smoothing, decomposition, ﬁltering, time-windowing
• “Expensive” output information:
• Land use characterization (usually collected through long and expensive
workflows that mix machine processing and costly human labour)
Gloria Re Calegari, Emanuela Carlino, Diego Peroni, Irene Celino. Extracting Urban Land Use from Linked Open Geospatial Data. IJGI, 2015
Gloria Re Calegari, Emanuela Carlino, Diego Peroni, Irene Celino. Filtering and Windowing Mobile Traffic Time Series for Territorial Land Use Classification. COMCOM, 2016

FROM SPATIAL ANALYTICS TO GEO-ONTOLOGY ENGINEERING
• OpenStreetMap collects information about points of interest (POI)
• Spatial distribution and conglomeration of specific POIs can give hints
about the geographical space
• Re-engineering of spatial features through comparison between areas:
same POI type shows different distribution  evidence for different
semantics (e.g. what is a pub in Milano vs. London)
• Semantic specification of spatial neighbourhoods:
• Emerging neighbourhoods from spatial clustering of POIs (opposed
to administrative divisions)
• Spatial version of tf-idf to compare between different areas (e.g.
central or peripheral areas in different cities) and to characterise
neighbourhoods (e.g. shopping district)
Gloria Re Calegari, Emanuela Carlino, Irene Celino, Diego Peroni. Supporting Geo-Ontology
Engineering through Spatial Data Analytics. 13th Extended Semantic Web Conference, 2016

MILANO
viale Sarca 226,
20126,
Milano - Italy
LONDON
4th floor
57 Rathbone Place
London W1T 1JU – UK
NEW YORK
One Liberty Plaza,
165 Broadway, 23rd Floor,
New York City, New York, 10006 USA
Cefriel.com
Thanks for your attention!
Any question?
Irene Celino
Knowledge Technologies
Digital Interaction Division
irene.celino@cefriel.com

Human Computation

Recommended

Recommended

More Related Content

What's hot

What's hot (15)

Similar to Human Computation

Similar to Human Computation (20)

More from Irene Celino

More from Irene Celino (20)

Recently uploaded

Recently uploaded (20)

Human Computation