OpenShift Commons Paris - Choose Your Own Observability Adventure
Melt Barcelona Variazioni Vuorikari Final
1. Collaborative Content Annotation
in
http://lre.eun.org
Multilingual Europe
Riina Vuorikari, Frans Van Assche
European Schoolnet
1st VARIAZIONI Workshop
Barcelona Nov 29 2007
2. Collaborative Content Annotation
in
http://lre.eun.org
Multilingual Europe
* motivation and context
* federated architecture
* co-existence of LOM and
unstructured metadata
* envisaged enrichment services
3. Collaborative Content Annotation
in
http://lre.eun.org
Multilingual Europe
* what does multi-linguality
mean for tags and how do
users deal with it?
4. Motivation of this work
http://lre.eun.org
European education,
especially that of K-12 education,
is inherently
multi-lingual and multi-cultural.
10. New challenges for repositories
European teachers have access to multiple
repositories of digital learning resources by
http://lre.eun.org
– Educational Authorities,
– publishers,
– other teachers,.
Users become more demanding and expect
services that are seen elsewhere (rss,
personalised feed-selections, bookmarks,
community rankings ..)
12. Since 1999 EUN's goal:
http://lre.eun.org
to facilitate the access to
multi-lingual
learning resources repositories
13. EUN + Federation of LORs
http://lre.eun.org
=
Learning Resources Exchange
(LRE)
14. Semantic interoperability for K-12
Long process in semantic interoperability
since 1999
http://lre.eun.org
– First DC based Application Profile in 2001
15. Semantic interoperability for K-12
LOM based Application Profile v 3.0
http://insight.eun.org/intern/shared/data/insight/lre/AppProfilev3p0.pdf
http://lre.eun.org
18. Challenge for users
End-users (e.g. teachers) have difficulties to
discover and find resources from educational
repositories
http://lre.eun.org
– Metadata does not always match search terms
Locating content across linguistic and
national borders within Europe has proven
hard
– Despite the use of a multilingual Thesaurus and
controlled vocabularies
19. New Mission Critical
Metadata
Egology for
http://lre.eun.org
Learning
Technologies
=> co-existence of structured and
unstructured metadata
=> by expert indexers and
end-users
22. Enrichment: by experts
By expert indexers:
http://lre.eun.org
Original
Metadata
Edited by expert
Merged
indexer
Metadata
Samgi
generated Metadata
instance
“toolbox will be extended with tools for the
development of taxonomies, and with tools for
automatic translation and vocabulary
management, as well as for collecting
attention metadata to track what users
actually do with the MELT infrastructure”
24. Collaborative content enrichment
in multilingual Europe
Addition to the traditional LOM
http://lre.eun.org
By users interacting with the portal, resources
and other users
Four main tools:
– social bookmarking and tags in multiple
languages
– rating of usefulness
– pedagogical annotations (used in learning events)
– levels of user engagement when interacting with
the system (what is viewed, how many times,..)
26. Social bookmarking
To store, organise, share and search
bookmarks of web pages. Keep found things
http://lre.eun.org
found!
Bookmarks are usually public and shared
Users organise bookmarks with informal tags
(instead of the traditional folders)
Find like-minded users with similar interest.
27. What is a tag?
Metadata externally applied to an item
http://lre.eun.org
Can be used for sorting or managing
A hook for aggregating
Provides identifier and/or description
Personal marker
by Thomas Vander Wal
37. .. allows new ways to discover both
resources and people!
http://lre.eun.org
38. How do users bookmark?
January 1 to October 31 2007
http://lre.eun.org
A post : a triple of (user, item, {tag})
1022 posts to favourites
142 users
682 different learning resources (items)
1029 multilingual tags recorded to the system,
some of which were reused by users
39. How do users bookmark?
Distribution of posts and inverse Power Law
http://lre.eun.org
130
120
110
100
Number of Bookmarks
90
80
Column J
70
Column K
60
50
40
30
20
10
0
0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150
40. How do users tag?
Most posts consisted Percent of tags/posting
of only one tag Onetag 79%
2 tags 15%
http://lre.eun.org
3 tags 4%
4 tags 1%
5 tags 0.39%
More than half of 6 tags 0.23%
tags were used just 7 tags 0.08%
once (54%)
Only about 10% of
tags were reused
more than twise
No guided tagging
in this pilot! This
may change.
41. Are tags found useful?
LO Type Lang Keyword % of votes
http://lre.eun.org
1 Tag En healthy meal 100%
1 Thesaurus En health education 77%
2 Tag En EU 77%
2 Tag Pl E uropa 62%
3 Thesaurus En physics 85 %
3 Thesaurus En mathematics 92 %
4 Thesaurus En mathematics 92 %
4 Tag En G eoG ebra-program 62%
5 Thesaurus En G eography 92 %
5 Thesaurus En Island 6 9%
Vuorikari, R., Ochoa, X., Duval, E. Analysis of User Behavior on MultilingualTagging of
Learning Resources. In Workshop proceedings of the EC-TEL conference:
SIRTEL07 (EC-TEL ’07) (Crete, Greece, September 17-20, 2007)
42. Are tags found useful?
LO Type Lang Keyword % of votes
http://lre.eun.org
1 Tag En healthy meal 100%
1 Thesaurus En health education 77%
Tags, producedE Uwith no outlay,
2 Tag
En 77%
2 Tag
Pl E uropa
show an E n physics
encouraging and
62%
3 Thesaurus 85 %
potentialn gain in overall92 %
3 Thesaurus
E mathematics
4 Thesaurus
En mathematics
usefulness!
92 %
4 Tag
En G eoG ebra-program 62%
5 Thesaurus En G eography 92 %
5 Thesaurus En Island 6 9%
Vuorikari, R., Ochoa, X., Duval, E. Analysis of User Behavior on MultilingualTagging of
Learning Resources. In Workshop proceedings of the EC-TEL conference:
SIRTEL07 (EC-TEL ’07) (Crete, Greece, September 17-20, 2007)
43. Does the language matter?
http://lre.eun.org
Need for better ways to identify the language
– Give rules (if the user first preferred languages is..,
then..)
– Automate the recognition of languages
– Out-source it to users
44. “Travel well” tags
About 15% of tags contain a general term, a
http://lre.eun.org
name, place, etc. that is easily understood
without translation
e.g. AIDS, software, EU, Euroopa, Europa,
europe, Evropa, geograafia, Pythagoras, etc.
45. What's the point of travel well tags?
If those tags need no translation or language
http://lre.eun.org
filtering to be understood, and
..if they can be identified:
We can be sure to show at least some tags to
users
– whose language preferences we don't know, and
– in whose language there are no tags or keywords
available.
46. User engaging with resources
Steps taken:
- views page
http://lre.eun.org
- views metadata
- bookmarks and
tags
- rates
- what about the
actual use?
47. Semantic analysis of tags
Factual tags 63%
(Golder: item topics, kinds of
item, category refinements)
http://lre.eun.org
Subjective tags 29%
( Golder: item qualities)
Personal tags 3%
(Golder: item ownership,
self-reference, tasks
organisation)
5% other
Sen et al. (2006).
48. Why tag categories?
In Sen et al. (2006) it
was found that tags
of different categories
can be useful for
http://lre.eun.org
different tasks
In our case it is too
early to say anything,
but ...we'll have an
eye on it!
49. Metadata LOM tags
social bookmarks
folksonomy social tagging
multi-linguality social classification
http://lre.eun.org
thanks! for your attention
learning resources user communities
discover resources and items
questions?
teachers social navigation
social traces
paths, trails
http://www.cs.kuleuven.ac.be/~hmdb/infovis/calibrate/calibrate.html
flock