In the last few years, the Information Engineering research unit @ University of Alcala has developed a productive research line on the use of these data to identify underlying knowledge such as social relations, user preferences or hints on the quality of the resources to create applications for recommendation, filtering an item or learner clustering, to name just a few. Successes and pitfalls of this research are illustrated from the past experiences of one member of the group.
ICT role in 21st century education and it's challenges.
Ratings, tags, bookmarks and other species: some examples of quantitative research on information filtering in TEL
1. Ratings, tags, bookmarks and
other species: some examples of
quantitative research on
information filtering in TEL
Salvador Sánchez-Alonso
2. About this talk
Some context
about me, my group and my research
Research coordinates
Revision of successful cases
Conclussions and open research directions
Practical exercise
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
3. About me
Remember Pecha-kucha?
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
4. About IE
Research lines
Projects
Doctorate studies
Publications
Journals and conferences
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
5. About our PhD program
Not officially online but definitely not face-to-face!
High performance-oriented
No more “read this” or “have a look at…”
Lots of autonomous work but with REAL help/guidance
Procedure:
Presentation (including CV)
Finding a few ideas PhD candidate likes
Writing objectives
Usually avoiding conferences unless veeeeeery junior
Paper accepted in an impact factor journal PhD finished.
2-3 years usually enough
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
6. Objectives of this talk
Find research opportunities in quantitative TEL
research
Learn from our experience
See how TEL research can target high impact factor
journals
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
7. Assumptions
You are familiar with
The concept of Learning object
The concept of metadata
Learning objects repositories
… and of course with IEEE LOM standard
You have (ideally) visited one or more learning object
repositories (e.g. MERLOT, CNX, Organic.Edunet…)
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
9. The process
Develop a research question related to some
functionality and state a hypothesis (not formally yet)
Identify the data source
Build a software engine to collect the data
Find the more apropriate technique(s) to analyse the
data and apply it on the dataset
Use statistics to assess if the hypothesis holds
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
14. Social Network Analysis (SNA)
Social network: a graph made up of nodes (individuals,
organizations...) and edges representing relationships
between nodes (friendship, work, membership...)
Social Network Analysis: a set of techniques to discover
features of a network by means of its numerical or visual
representation.
Find network measures such as betweenness and centrality
Most SNA software uses a plain text ASCII data format to
represent the network
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
15. Collaborative filtering
Informally: a form of automating the process of "word-of-mouth"
But you would rather like to hear the opinions of those who have
interests similar to your own!
Basic mechanism:
A large group of people's preferences are registered
Using a similarity metric, a subgroup of people is selected whose
preferences are similar to the preferences of the person who seeks
advice;
A (possibly weighted) average of the preferences for that subgroup is
calculated;
The resulting preference function is used to recommend options on
which the advice-seeker has expressed no personal opinion as yet.
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
16. Ontologies
Formal representation of knowledge
Concepts, relations and properties are represented in
an ontology language (eg OWL)
Ontologies can be used to
Enhance information retrieval
Power advanced services such as more accurate web
search
Communicate between systems
Evaluate the correctness of a conceptual model
…
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
17. Statistical profiling
A set of techniques that allow to discover patterns or
correlations in large quantities of data
Helps in dealing with the increasing data overload,
helping to discriminate information from noise
Metrics:
Precision: the fraction of correct instances among those
that the algorithm believes to belong to the relevant
subset
Recall: the fraction of correct instances among all
instances that actually belong to the relevant subset
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
19. Assesing LO
reusability from
their metadata
PhD. Javier Sanz
Timeline: August 2008 -April 2010
Status: Final
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
20. Hypothesis
• It is possible to find an aprioristic reusability
evaluation based on LO metadata
• This metric would span all the factors affecting the
reusability of a learning object
• It is possible to compute reusability in an
automated way
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
21. Functionalities
• Estimating reusability would provide useful information
when it comes to selecting reusable objects
• This measure of reusability might constitute an indicator of
quality which would allow for search results to be ordered,
with those which have greater possibilities of being reused
taking priority.
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
22. The process in a nutshell
“Polish” the hypothesis
Gather data from 2 repositories: MERLOT and eLERA
Find the metadata elements having an impact on
reusability
Create the metrics
Adjust them with empirical data
Assess their effectiveness
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
23. Method
Cohesion
Learning object
Reusability
Size
+
Expert
Metadata Technological Portability Agregation
Educational Portability
Repository
> Aggregation methods: Weighted mean, Choquet’s integral, Multiple linear regression
> Evaluation of the efficiency of the model: Average absolute error, Average relative
error, Correlation between the real and the estimated value, Quality of the prediction
> Expert questionnaire + LORI
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
24. Factors, metrics and metadata
elements
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
25. Results
– Statistically significant
correlation between
estimated reusability and
content quality evaluated
by the experts.
– Statistically significant
correlation between
estimated reusability and Correlation K endall’s Spearman’s Rho
Tau
effectiveness as a
Content quality 0.228 0.307
learning tool and ease of Effectiveness 0.153 0.217
use. Ease of use 0.190 0.250
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
26. Publications
Sanz-Rodriguez, J., Dodero, J. and Sanchez-Alonso, S.
(2010). Metrics-based evaluation of learning object JCR
reusability. Software Quality Journal 19(1), pp. 121-140.
Sanz-Rodríguez, Dodero, Sánchez-Alonso. (2010) Ranking
Learning Objects through Integration of Different Quality
Indicators, IEEE Transactions on Learning Technologies, 2008
3(4), pp. 358-363.
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
27. Expanding queries
in LO repositories
with ontologies
PhD. Alejandra Segura
Timeline: June 2009 - December 2010
Status: Final
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
28. Hypothesis
Evaluación y resultados
Query expansion can help repository users to find
additional relevant resources not retrieved using the
regular built-in search
Scenario of application: A teacher searches for
educational resources to design a new course or to
create a new resource
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
29. Procedure
Query Query Search
other
Synonym
Original
Part of
Is a
extraction
Expanded concepts
Query
is a
Expansion
part of
• Exists
• Aproximate Remove
• Doesnt exist other duplicates
synonyms
List of LO Filter results
A
Contrast B +/- Relevance
results C evaluation
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
30. Procedure
24 Official courses in genetics
In Universities and Higher education institutions
Syllabi published in the web
Academic period 2009
711 different concepts identified (lists of contents)
91 test queries (concept retrieval frequency >1)
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
31. Method
Experts in the field evaluated the results (3
experts each query).
Topical relevance
Expert profile: medical practitioners and genetics
specialists, 5 years experience in teaching and practice
extrictly necessary.
Expert correlation analysis using rater agreement
metrics
Precission and recall where used to state relevance
and novelty
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
32. Results
Rater agreement moderate according to
Cohen’s Kappa and Spearman correlation
COVERAGE: More than half the results (54%)
retrieved from non-expanded queries are also
retrieved when expanding the query.
NOVELTY: 53% of the relevant LOs retrieved by
the expanded queries are new (e.g. different
from those retrieved without expansion).
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
33. Results
0,90
0,80
0,70
0,60
0,50
0,40
0,30
0,20
0,10
0,00
isa hermanos isa hijos isa padres par todo par partes syn exacto
Novedad Cobertura
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
34. Conclusions
Conclusiones
Results of expanded queries are affected by:
The quality of the ontology
Built-in retrieval mechanism
Intrinsic characteristics of the learning objects collection
Best novelty results when:
Polysemic queries
Results from all types of expansions are merged in a
unique list
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
35. Publications
Segura, A., Sánchez-Alonso, S., Garcia-Barriocanal, E.
and Prieto, M. (2011). An empirical analysis of ontology-
JCR
based query expansion for learning resource searches
using MERLOT and the Gene ontology. Knowledge Based
Systems, 24(1), pp. 119-133.
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
36. Exploring affiliation
network models as a
collaborative filtering
mechanism in e-learning
Not linked to any PhD
Status: ready for anyone interested
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
37. Hypothesis
Social network analysis of relations in learning
environments will make it possible to re-configure…
A) the learning contents and/or activities
E.g. including new activities, changing the future course
structure or taking other kind of actions.
B) the learning environment
E.g. group formation, rearranging groups once the course
is being delivered.
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
38. Method
Modeling and analysing learners participation in
activities organized around communication forums,
(very common in e-learning environments)
Forum participation as an affiliation network (a kind of
social network with different types of nodes)
One of the possible applications: identifying common
interests of groups of learners.
Technique: Blockmodeling (aimed at transforming an
apparently non-coherent network into a more easily
comprehensible arrangement)
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
39. Results
We identified two
different groups, one
interested in the
learning tools used
during the course and
the other group more
interested in the
theoretical aspects of
the course.
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
40. Results
33-slice cluster with
introductory topics (most
people are interested)
As long as course
progresses interest is
less cohesive
16-slice cluster for topics
on general LO definitions
and concepts
8-slice for highly
technical issues
(SCORM + LD)
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
41. Publications
Rodríguez, D., Sicilia, MA., Sánchez-Alonso, S., Lezcano,
L. and García-Barriocanal, E. (2009) Exploring affiliation
JCR
network models as a collaborative filtering mechanism in e-
learning, Interactive Learning Environments. DOI:
10.1080/10494820903148610
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
42. Automated quality
assessment of
Learning Objects
PhD. Cristian Cechinel
Timeline: from July 2009
Status: Almost final
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
43. Hypothesis
Some intrinsic features of learning objects stored in
existing repositories may present significant
difference between highly rated (good) learning
objects and poorly rated (not-good) learning objects.
These quantitative measures could serve as the
basis for the development of models for quality
prediction.
44. Functionality
Developing models for automatically assessing
quality of learning objects inside repositories
based on the intrinsic features of the resources
45. Method
Identification of intrinsic metrics of learning resources
that could serve as potential indicators of quality
Database gathered from the MERLOT repository by
using a web crawler.
In total, 35 metrics were extracted from 6,740
learning objects. From these resources, only 1,765
(27.27%) had at least one peer review or one user
rating and were used in the analysis, the rest were
discarded
46. Metrics
Class of Measure Metric
Link Measures Number of Links, Number of Unique Links, Number of Internal Links,
Number of Unique Internal Links, Number of External Links, Number of
Unique External Links
Text Measures Number of Words, Number of words that are links
Graphic, Interactive and Number of Images, Total Size of the Images (in bytes), Number of
Multimedia Measures Scripts, Number of Applets, Number of Audio Files, Number of Video
Files, Number of Multimedia Files
Site Architecture Size of the Page (in bytes), Number of Files for downloading, Total
Measures Number of Pages
Evaluation Metadata Number of Personal Collections
(contrast metric)
47. Method
Learning objects were classified into three groups
(good, average, poor) according to their ratings.
A Mann-Whitney (Wilcoxon) test was performed to
evaluate whether the selected metrics presented
significant difference in their medians between the
groups of good and not-good (average + poor)
resources
Kolmogorov-Smirnov test was performed to evaluate
differences regarding the distributions.
48. Preliminary Results
The two groups of ratings available on MERLOT
(i.e. peer-reviewers and user comments ratings)
differ substantially regarding the intrinsic
characteristics of the resources.
The tested metrics present different profiles and
tendencies between good and not-good materials
depending on the category of discipline and the
type of material to which the resource belong
49. Preliminary Results
We developed a Linear Discriminant Analysis (LDA) to
build models in order to distinguish
1. good from not-good resources,
2. good from average resources, and
3. good from poor resources
For the Science and Technology discipline intersected
with the Simulation material type in the context of peer-
reviews thresholds.
The third model achieved 91.49% of overall accuracy, with
a squared canonical correlation of 0.81130 (significant at
the 99% level)
50. Preliminary Results
The pursuit for an automated model for the quality
evaluation of learning objects must consider the
development of profiles taking into account the
intersection of the categories of disciplines and material
types, as well as the distinct groups of raters.
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
51. Publications
Cechinel, C., Sánchez-Alonso, S. and García-Barriocanal,
E. (2011). Statistical profiles of highly-rated learning
JCR
objects. Computers & Education, 57(1), pp. 1255-1269.
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
52. Can 3D platforms
improve training of
trainers' programs?
PhD. Carlos M. Lorenzo
Timeline: from April 2010
Status: in progress
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
53. Hypothesis
Using Massively Multiuser Online Learning
environments (MMOL) platforms in virtual courses can
improve training-of-trainers program
The aim is to explore how a specific MMOL Platform
facilitates online tutor’s tasks in a rich virtual learning
environment with a pedagogical framework, and to
identify essential issues of interactivity in this context
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
54. Method
MMOL session using the
collaborative LORI approach
A group of users contribute their
individual evaluations on a learning
object and try to reach a consensus
after hearing everyone else’s opinion.
2D session in LCMS
Comparing results in terms of
satisfaction and efectiveness
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
55. The experiments
A prototype of 3D MMOL platform was created in a
realXtend server with an interactive space called
“MadriPolis”.
Case A: 11 master
students Both cases consist in training-
of-trainers experiences about
• LCMS on-line tutor experiment
collaborative Learning Object
• MMOL on-line tutor experiment evaluation based on Learning
Object Review Instrument
Case B: 10 graduated (LORI) with the Convergent
students Participation Model (CPM)
(Vargo et al., 2003) to
• LCMS on-line tutor experiment determinate the quality of e-
• MMOL on-line tutor experiment learning resources
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
56. Results
•Case «A». Data
•LCMS Experim. • Data Analisys
Collection and SNA:
•MMOL Experim.
•Case «B» •Log events. • Density
•MMOL Experim. •On-line surveys • Centrality
•LCMS Experim. •Direct
Observations
•Triangulation
Case studies Evaluation
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
57. Publications
Manuscript submitted to Computer & Education (April
2011). Still waiting… JCR
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
58. Recommenders
inside learning object
repositories:
requirements for
meaningful datasets
Not linked to any PhD
Status: ready for anyone interested
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
59. Hypothesis
Implicit communities found via SNA
blockmodeling & component analysis
have a potential for recommending
learning objects to repository users
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
60. Application
Implicit communities found via SNA
blockmodeling & component analysis
have a potential for recommending
learning objects to repository users
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
61. Method
Evaluate parameters for Collaborative Filtering
Algorithms for two datasets from MERLOT
1. Resources including ratings given by users
2. Resources present in the users’ Personal
Collections
Generating recommendations for the datasets
using optimized parameters
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
62. Method
Compare the results generated by the different
algorithms for the two datasets
Contrast the results of the recommendations
generated by the algorithms with existing
endorsement mechanisms of the repository
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
63. Results
Very high Precision values (varying from 20% to
100%) and not so high Recall percentages (with a
maximum of 18%).
Recommendations generated are related to other
endorsement mechanisms in MERLOT
Big differences between the recommendations
generated using the two distinct datasets
Reinforcement of the initial idea that these two datasets
represent very distinct information
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
64. Future work
Involving users’ opinions in the process
Contrasting if recommendations for a given user
fall in the disciplinary area of that user or are
crossing disciplines
Evaluating if users are already familiar with the
recommended resources, and if they would
recommend such resources to their fellows
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
65. Publications
Sánchez Alonso, S., Sicilia, MA., García, E., Pagés, C. and
Lezcano, L. (2011) Social models in open learning object
repositories: A simulation approach for sustainable JCR
collections. Simulation Modelling Practice and Theory
19(1): 110-120
Sicilia, M.-Á., García-Barriocanal, E., Sánchez-Alonso, S.,
& Cechinel, C. (2010). Exploring user-based recommender
results in large learning object repositories: the case of
MERLOT. Procedia Computer Science, 1(2), 2859-2864.
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
66. Conclussions and
open reseach
directions
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
67. Lessons learned
Quantitative research is usually well received by
impact factor journals editorial boards
It is feasible to have a PhD ready in about 2 years
Respect repositories’ policies on acceptable use
Collecting data, either manually or through automated
processes may be not permitted
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
68. Lessons learned
LOR data should be shared!
Evangelize repository owners to share data for
research and to include that in their conditions for use.
A common dataset sharing format for LOR needed
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
69. Open research opportunities
More in-depth study of social interaction in LCMS
(software ready for SNA in Moodle)
Open courseware research studies similar to those
presented (OCW Finder crawler ready)
Several project-related research (e.g. Assessment
of automated translation mechanisms in
Organic.Edunet)
Any other research extending previous cases...
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
70. Your turn!
5 next minutes [individually]: think of an experiment
similar to those reported in my talk
15 minutes [all]: share your ideas with us
[In pairs] Write down a summary with at least:
Hypothesis
Functionality
Techniques
Assessment metod
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species
71. To be written down
Salvador: salvador.sanchez@uah.es
Want to know more about our distance PhD program
@ IE-UAH?
Talk to me today or email me at your wish
Fancy to work with us in a EU project?
Contact me or prof. Sicilia: msicilia@uah.es
JTEL Summer school 2011 - Ratings, tags, bookmarks and other species