• Divergent opinions about the usefulness and understandability of the
course report: activity of students by step and responses in the quizzes
and tests.
• Students´ satisfaction and comments by step were among the most
useful information for the presented monitor goals as in [2].
How educators value data analytics about their
MOOCs
Introduction Results
Research focus
Methods Conclusions and future work
Konstantinos Michos, Davinia Hernández-Leo & Manel Jiménez
{kostas.michos, davinia.hernandez-leo, manel.jimenez}@upf.edu
ICT Department, Communication Department, Universitat Pompeu Fabra, Barcelona
References
1. Dyckhoff, A. L., Lukarov, V., Muslim, A., Chatti, M. A., & Schroeder, U.: Supporting action research with learning analytics. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (pp.
220-229). ACM (2013).
2. Stephens-Martinez, K., Hearst, M. A., & Fox, A.: Monitoring MOOCs: Which information sources do instructors value?. In Proceedings of the first ACM conference on Learning@ scale conference (pp. 79-88). ACM (2014).
• Data analytics provided to educators in MOOCs: e.g. students´ profiles,
students´ online behavior, students´ satisfaction with the course.
• Supporting the role of educators, their decisions and their actions
based on the data analytics [1].
• Learning design & Data anaytics
.
• Survey study with closed and open-ended questions.
• 4 Educators who developed two editions of three different MOOCs in
UPF.
RQ1: Which information sources and visualizations (from FutureLearn
reports) are most useful for MOOC educators?
RQ2: What information sources (from FutureLearn reports) help MOOC
educators to identify problems and potential improvements in a redesign
of the course?
• Problem: How do educators value the different sources of data and
how do they connect them with their course re-design?
• FutureLearn case – Summary Reports about: a) pre-course survey
b) course report c) satisfaction surveys.
• Questions about the usefulness and understandability of the different
provided data analytics from their own MOOCs. Connection with
monitoring goals as in [2]:
1. Problems with the activities-tasks.
2. Struggling students and what they are struggling with.
3. Difficulty of the grading activities.
4. Appropriateness of course difficulty level for students.
5. Engaging content for students.
6. Most difficult part of the course.
7. Improving the presentation of a topic.
8. Least interesting content for students.
22
22
28
22
28
22
25
19
41
28
41
47
59
44
34
38
34
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
a. Previous experiences of students
b. Enrollement cummulative growth
c. Activity by step
d. Comments by step
e. Quizzes and tests
f. Students' satisfaction
Percent of educators’ responses (n = 32)
Informationsources
Usefulness of the information sources for all the monitoring goals
Neutral Agree Strongly agree Not applicable
29
29
25
29
29
25
25
13
4
13
8
4
17
33
25
38
29
33
21
13
42
29
58
29
29
33
38
63
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
1. Problems with the activities-tasks
2. With what students struggle
3. Difficulty of the grading activities
4. Appropriatness of course difficulty
5. Engaging content for students
6. Most difficult part of the course
7. Improving the presentation of a topic
8. Least interesting content
Percent of educators’ responses (n= 24)
Monitoringgoals
Usefulness of the information sources for each of the
monitoring goals
Neutral Agree Strongly agree Not applicable
• Εducators redesign actions based on their availble data: simplification
of the content. Interventions in specific parts of the course.
Redesigned quizzes and tests based on the levels of shown difficulty.
• Final decissions were determined after discussion between the groups
of educators and the FutureLearn platform.
• Proposal for interaction analysis and words in sentiment analysis.
• Preliminary results indicated that educators perceived specific data
analytics particularly useful, with some divergences in their opinions.
• Difficulty to connect the information sources with the monitoring goals.
• Extend the study with more educators.
• How topics, learning design and audience
characteristics influence the educators.
• Monitoring goals defined by educators
before their course.
• Study in detail the educators’ actions
based on the presented data analytics.
• Take in to account educators’ divergent
needs in the design of MOOC dashboards.
 During the course: monitoring, interventions, dynamic changes
in the course design, prompts in the forums.
 After the course: reflection on what happened to improve re-
runs of the course. Decisions in design-time based on past
cohorts of MOOC students