Slides of the INSEMTIVES talk at Semtech 2010, 2010-06-22

1. Reputation, community, Web
2.0, and games: incentives for
semantics – about getting people
involved
www.insemtives.eu 1
Katharina Siorpaes and Elena Simperl
SemTech 2010

2. Executive summary
• Many tasks related to semantic content authoring cannot be undertaken
without human contribution.
• User motivation is essential for semantic applications to achieve critical
mass and ensure sustainable growth.
• How to encourage user participation
– Incentives
– Technology design
– Usability engineering
– Games with a purpose
www.insemtives.eu 2

3. About the speakers
www.insemtives.eu 3
Katharina Siorpaes Elena Simperl

4. Agenda
• Human-driven semantic content authoring
• Incentives and participatory design
• Example: Casual games
• Example: Virtual worlds
• Conclusion
www.insemtives.eu 4

5. HUMAN-DRIVEN SEMANTIC
CONTENT AUTHORING
Human-driven semantic content authoring
Incentives and participatory design
Example: Casual games
Example: Virtual worlds
Conclusions
www.insemtives.eu 5

6. Human vs computational intelligence
• Ordered sequence of tasks into which the
authoring exercise can be divided
• Required skills and expertise
www.insemtives.eu 6

7. Example: semantic annotation
www.insemtives.eu 7

8. Example: ontology evaluation
www.insemtives.eu 8

9. Example: ontology alignment
www.insemtives.eu 9

10. MILLION DOLLAR QUESTION
HOW CAN WE MOTIVATE PEOPLE
TO DEDICATE THEIR VALUABLE TIME
TO ANNOTATE THEIR/OUR DATA?
www.insemtives.eu 10

11. INCENTIVES AND PARTICIPATORY
DESIGN
Human-driven semantic content authoring
Incentives and participatory design
Example: Casual games
Example: Virtual worlds
Conclusions
www.insemtives.eu 11

12. www.insemtives.eu 12
Motivation vs incentives
• Incentives are ‘rewards’ assigned by an external
‘judge’ to a performer for undertaking a specific task.
• Common belief (among economists): incentives can
be translated into a sum of money for all practical
purposes.
• Incentives can be related to both extrinsic and
intrinsic motivations.

13. www.insemtives.eu 13
Extrinsic vs intrinsic motivation
• Extrinsic motivation if task is considered
– Boring, dangerous, useless, socially undesirable,
dislikable by the performer.
• Intrinsic motivation if
– The performer likes what he/she is doing
– The act is satisfying in itself (it can happen for
many different reasons).

14. Web 2.0 is hot, Semantic Web is
not?
www.insemtives.eu 14

15. Web 2.0 motivators
www.insemtives.eu 15

16. User-empowered Web
• Comprehensive incentives studies are
available (Kuznetsov, 2004; Marlow et
al.,2006; Wikipedia, 2009).
www.insemtives.eu 1616

17. Example: Wikipedians
• Reciprocity: Altruistic contributors receive a
benefit in return.
• Community: “Wikipedians […] feel needed”, there
is “a sense of common purpose and belonging “.
• Reputation: Contributors “develop identities in
order to be respected, trusted, and appreciated
by peers”.
• Autonomy: Contributors enjoy “the freedom of
independent decision”.
Kuznetsov, 2004
www.insemtives.eu 17

18. Factors influencing incentives design
• Tasks or an ordered collection of tasks into
which the authoring exercise can be divided.
• Skills required to undertake the task.
• Goal of the authoring exercise.
• Social structure, i.e. social relationships
among the subjects participating in the
exercise.
• Nature of the good.www.insemtives.eu 18

19. Example: image sharing on a social
platform
• Task: attaching concepts and
relationships to images
(iPhoto).
• Skills: identifying objects in
images and choosing the
right description.
• Goal: sharing and retrieval.
• Social structure: Friends (in
social network).
• Nature of the good:
annotations accessible to me
and my friends.
www.insemtives.eu 19

20. Example: UNSPSC and eCl@ss
alignment
• Task: ontology alignment
• Skills: domain knowledge,
modeling skills
• Goal: interoperability
• Social structure:
enterprise, team
• Nature of good: for
benefit of enterprise
www.insemtives.eu 20

21. Technology and application design
• Involve users in the design process.
– Design workshops, interviews etc.
• Take usability serious.
– Design for efficiency, effectiveness and user
satisfaction.
• Design for sociability.
– Support sharing, contributing, collaborating,
identity-building.
www.insemtives.eu 21

22. Technology and application design
(ii)
• Respect trust/security and safety/privacy
issues.
– Support visibility and awareness.
• Design for fun.
• Follow an user experience design approach.
• Support a user’s individual identity in a (and
contribution to a) community with collaborative
activities.
www.insemtives.eu 22

23. Example: image annotation in
OKEnterprise (i)
• Involvement of users in design
process
– Interviews and user
workshops
• Usability
– Interviews, usability
evaluations
• Design for sociability
– User interaction, visibility of
contributions, registered
users, collaborative working
• Trust and privacy
– Trust in enterprise portal,
rating
– Privacy: users have control
over their data
www.insemtives.eu 23

24. Example: image annotation in
OKEnterprise (ii)
• Design for fun
– Annotator of the week,
rankings
– Casual games
• User-experience design
approach
– Evaluate and revise
depending on user
experience
• Support user’s individual
identity
– Registration of users,
individual user pages
www.insemtives.eu 24

25. NOW: FUN AND COMPETITION:
GAMES FOR SEMANTICS!
Human-driven semantic content authoring
Incentives and participatory design
Example: Casual games
Example: Virtual worlds
Conclusions
www.insemtives.eu 25

26. 1/28/2015
ESWC 2008 26
OntoPronto: Step 1

27. OntoPronto: Step 2

28. 1/28/2015
ESWC 2008 28
OntoTube

29. OntoTube

30. Results
n=2905 rounds
n=2234
12.9%
10.2%
76.9%
Either skipped or no
consensus found in
the first task
Challenges in which
only the first task was
completed
consensually
Challenges in which
both tasks were
completed
consensually
97.6%
0.5%
1.9%
Challenges in which
at least the first task
was completed
consensually
Wrong judgements of
ontological nature
Wrong abstractions

31. MASSACRE – MASive Semantic
Annotation Creation Game
www.insemtives.eu 31

32. Phratris – Phrase Analysis TETRIS
www.insemtives.eu 32

33. playence’ Uhani (i)
www.insemtives.eu 33
Check
www.playence.com!

34. playence’ Uhani (ii)
www.insemtives.eu 34

35. Casual games
1. Steep learning curve
2. Fast game play: little time effort required
3. Simple implementation (simple interface and
graphics)
4. Low hardware efforts (usually browser or
mobile app)
5. Low bandwidth requirements
6. Mass audience
www.insemtives.eu 35

36. How to design your own game
1. Specify output
2. Identify input
3. Choose type of game and define game play
4. Based on previous decisions, define game
play and adapt underlying game ontology
5. Adapt or define export algorithm
6. Evaluate output
www.insemtives.eu 36

37. Guidelines
1. Timed response
2. Score keeping
3. Player skill level
4. High score lists
5. Randomness
5. Random player pairing
6. Player testing
7. Repetition
8. Taboo outputs
www.insemtives.eu 37
Luis von Ahn. Games With A Purpose.
IEEE Computer Magazine, June 2006. pp
96-98.

38. Challenges
• Identifying suitable tasks in semantic content creation.
• Designing games.
• Designing a usable, attractive interface.
• Identifying suitable knowledge corpora.
• Preventing cheating.
• Defusing typical pitfalls of conceptual modeling.
• Distribution of labor.
• Fostering user participation.
• Deriving formal representations.
• Scalability and performance.
www.insemtives.eu 38

39. ANNOTATION IN VIRTUAL WORLDS
Human-driven semantic content authoring
Incentives and participatory design
Example: Casual games
Example: Virtual worlds
Conclusions
www.insemtives.eu 39

40. Tiny Planets
• Kids TV property licensed in over
100 countries worldwide
• Education
• Diverse broadcaster base – from
Al Jazeera to American Armed
Forces network
• Unusual age profile – kids 4 -11
• Brand extensions into web and
virtual world
www.insemtives.eu 40

41. Tiny Planets (ii)
• Tiny Planets website – 40,000
unique visitors per month
• 100,000 page impressions
• Average stay 8 minutes
• MTP virtual world – active
47,000 user accounts
• Books, Fun, TV, Learning, Labs
sites as well as virtual world
• Real and virtual currencies
• Cross-site account creation
www.insemtives.eu 41

42. • Engage kids with annotation
games as a means of expression
• Combination of ‘rating’, ‘tagging’,
and pre-scripted comments
• Locate content within virtual
world to inform friends, give
clues in games, give an
‘emotional’ context to items by
expressing opinions. Search for
clues and comments left by
friends
• Reward annotations with virtual
currency
www.insemtives.eu 42
Virtual world

43. Real world
• Real-world science experiment
based on Galaxy Zoo format
• Crater marking using data from
NASA LRO
• Use My Tiny Planets account
for sign in and performance
tracking
• Reward annotations with
achievements and virtual
currency
• And kudos; kids get to know
they’re doing science for real,
and its going to be used.
www.insemtives.eu 43

45. Incentives
• Earn Stars to spend on Virtual
Goods, games or videos
• Earn awards that can be seen by
other players in your passport
• Rise in Rank as you perform
targeted activities
• Join the Pro circuit so that your
top scores can be seen by other
players
• Buy Keys, which unlock new
areas and new activities in
which to spend your Stars.
www.insemtives.eu 45

50. CONCLUSIONS
Human-driven semantic content authoring
Incentives and participatory design
Example: Casual games
Example: Virtual worlds
Conclusions
www.insemtives.eu 50

51. Summary
• Human contribution is needed for selected tasks in
semantic content authoring.
• Web 2.0 has impressively demonstrated how strong
incentives can foster user participation and create
successful applications.
• Turning semantic content creation tasks into games is
an especially intriguing idea because of the potentially
large amounts of human-produced data that can be
created.
• Challenges are manifold: knowledge corpora,
interesting game design, ensuring uptake, difficulty of
tasks, etc.
www.insemtives.eu 51

52. 1/28/2015 www.insemtives.eu 52
Realizing the Semantic Web by
encouraging millions of end-users
to create semantic content.