Meetup Data Science Northeast NL, October 13, 2016.

1. Claudia Hauff
Web Information Systems, TU Delft
Large-scale Learning
Analytics

2. It’s data that is on the Web … Web data … lets find the
Web Information Systems people!
✤ 40+ MOOCs
✤ 1+ Million enrollments
✤ From primary school to PhD level
✤ Lots of user data (click logs)

3. Our goals
Data
Knowledge
Application
to learning
Gain actionable insights into learner
behaviours at scale.
a. Data Science
b. Big data processing
Increase our knowledge about learners by
looking beyond the learning platform
a. Web data analytics
Design technology interventions that
enable adaptive learning at scale.
a. Web data analytics
b. Human-centered design
c. Learning technologies

4. Learner profiling beyond
the MOOC platform
ACM WebScience 2016
Guanliang Chen, Dan Davis, Jun Lin, Claudia Hauff, and Geert-Jan Houben. Beyond the MOOC platform:
Gaining Insights about Learners from the Social Web, ACM WebScience, pp. 15-24, 2016.

5. Whythis research?
Learner
Before the MOOC
NOTHING
Engagement, retention, …
During the MOOC
NOTHING
After the MOOC

6. Howto solve the problem?
We propose:
a deeper understanding about learners
can be gained by exploring their traces
on the Social Web.

7. Whatresearch questions?
1
On what Social Web platforms can a significant fraction of
MOOC learners be identified?
Are learners who demonstrate specific traits
on the Social Web drawn to certain types of MOOCs? 2
To what extent do Social Web platforms enable us to
observe (specific) user attributes
that are relevant to the online learning experience?
3

8. Learner identification
across Social Web platforms
edX learners
Email Login name Full name+ +
1. Explicit Matching
Profile images & links
Identification via
emails
2. Direct Matching
Identification via profile
links from Step 1
3. Fuzzy Matching
Search learners by their
login & full names
Compare:
1. profile link
2. profile image
3. login & full names

9. Social Web platforms
involved in our work

11. Matching results
for 18 DelftX MOOCs
Lowest Highest Overall
Gravatar 4.37% 23.49% 7.81%
Twitter 4.99% 17.58% 7.78%
Linkedin 3.90% 11.05% 5.89%
StackExchange 1.23% 21.91% 4.58%
GitHub 3.43% 41.93% 10.92%

12. Matching results
for 18 DelftX MOOCs
Lowest Highest Overall
Gravatar 4.37% 23.49% 7.81%
Twitter 4.99% 17.58% 7.78%
Linkedin 3.90% 11.05% 5.89%
StackExchange 1.23% 21.91% 4.58%
GitHub 3.43% 41.93% 10.92%
On average, 5% of learners can be identified on globally
popular Social Web platforms.

13. Learners on
Linkedin
- Using job titles & skills to characterise learners
Spreadsheet MOOC
- Software Engineer
- Business Analyst
- …
Design Approach MOOC
- Co founder
- UX designer
- …

14. Learners on
StackExchange
- Functional
Programming
learners in
StackOverflow
- To what extent do
learners change their
question/answering
behaviour during
and after a MOOC?

15. Take-home
Messages
On average, 5% of learners from 18 DelftX MOOCs
can be identified on 5 globally popular Social Web platforms. 1
Learners with specific traits prefer different types of MOOCs.2
Learners’ post-course behaviour can be investigated
by using their external Social Web traces.3

16. Learning Transfer:
does it take place?
Best Paper Nominee at
ACM Learning At Scale 2016
An Investigation into the Uptake of Functional
Programming in Practice
Guanliang Chen, Dan Davis, Claudia Hauff and Geert-Jan Houben, Learning Transfer: does it take place in
MOOCs?, ACM Learning At Scale, pp. 409-418, 2016.

17. Whatis learning transfer?
Learning transfer is the application
of knowledge or skills gained in a
learning environment to another
context.

18. Whydo we care?
Learning transfer is a more important
measure of learning in MOOCs than
retention, success or engagement.

19. FP101x
@flickr:christiaan_008
Course programming language: Haskell
Run as a typical video-lecture based MOOC
Assessment: 288 Multiple Choice questions
Introduction to Functional Programming
37,485 learners registered.
41% engaged with the course.
5% completed the course.
33% were active on GitHub (1.1M events).

20. Whatdid we do?
FP101x
logs
surveys
coding
activities
3 months 2.5 years + 0.5 years
+ +
email address
Are changes made in a
functional language?

21. GitHub
10+ million registered users
hosting, collaboration and organisation
the most popular social coding platform
founded in 2007
long-term
large-scale
detailed

22. detailed logs
code changes
project meta-data

23. A sanity check

24. Are “GitHub learners” different?
GitHub
learners
Non-GitHub
learners
#Learners 12,415 25,070
Completion rate 7.71% 4.03%
Avg. time watching
videos
49.1 min 27.7 min
Avg. #questions
attempted
31.3 17.5
Avg. accuracy of
learners’ answers
23.4% 12.9%
GitHub learners are more engaged than non-GitHub learners
and exhibit higher levels of knowledge.

25. Are “Expert learners” different?
Expert GitHub
learners
Novice GitHub
learners
#Learners 1,721 10,694
Completion rate 15% 6.5%
Avg. time watching
videos
78.6 min 44.4 min
Avg. #questions
attempted
57.9 27.0
Avg. accuracy of
learners’ answers
38.0% 21.1%
Expert learners are more engaged than Novice learners
and exhibit higher levels of knowledge.

26. To what extent do engaged learners
exhibit learning transfer?
5-10% >30%10-30%<5%

27. To what extent do engaged learners
exhibit learning transfer?
5-10%

28. Which type of learner is more likely
to display learning transfer?
flickr@ConalGallagher
Intrinsically motivated Extrinsically motivated

29. Which type of learner is more likely
to display learning transfer?
flickr@ConalGallagher
Intrinsically motivated

30. Which type of learner is more likely
to display learning transfer?
Experienced Inexperienced

31. Which type of learner is more likely
to display learning transfer?
Experienced

32. Which type of learner is more likely
to display learning transfer?
High-spacing
learning routine
Low-spacing
learning routine

33. Which type of learner is more likely
to display learning transfer?
High-spacing
learning routine

34. Learners who transfer
quickly move to Scala
FP101x

35. Conclusions
Most transfer learning findings from the classroom hold.
The observed transfer rate is low: 8.5%.
Learners quickly moved on after the course to
industrially-relevant functional languages.
@flickr:torsten-reuschling

36. From Learners to Earners:
Enabling MOOC Learners to Apply
their Skills and Earn Money in an
Online Market Place
IEEE Transactions on Learning
Technologies
Guanliang Chen, Dan Davis, Markus Krause, Efthimia Aivaloglou, Claudia Hauff and Geert-Jan Houben. Can
Learners be Earners? Investigating a Design to Enable MOOC Learners to Apply their Skills and Earn Money in
an Online Market Place, IEEE Transactions on Learning Technologies.

37. What
MOOCs aim to educate the world.
Most successful learners are already
highly educated.
Learners from developing countries
are underrepresented.
is the problem?

38. Whatis the problem?
EX101x: Data Analysis to the MAX()

39. How
Pay learner at scale: recommend tasks
from online market places to learners
that are relevant to the course material.
can we tackle it?

40. Howcan we tackle it?

41. What
1) To what extent do online market
places contain relevant tasks?
2) Are learners able to solve
real-world tasks with high quality?
do we need to look at?

42. Setup
1) Weekly spreadsheet “bonus
exercises” drawn from Upwork
(manually checked) in EX101x
2) Accuracy check
3) Quality check (code smells)

43. Howare learners doing?
Good accuracy & quality.

44. Built a working
recommender.
Deployed in
a MOOC by the
end of October.

45. Our goals one more time…
Data
Knowledge
Application
to learning
Gain actionable insights into learner
behaviours at scale.
Increase our knowledge about learners by
looking beyond the learning platform
Design technology interventions that
enable adaptive learning at scale.

46. MOOCs are vital to bring higher
education to the world.
Lots of unexplored potential.
Plenty of data.
Many users.
http://bit.ly/lambda-lab
Overall …