You can find the full recorded webinar here: https://www.elearninglearning.com/frs/10314249/enhancing-the-learning-experience-by-leveraging-learner-analytics
The importance of learner analytics is well understood among the eLearning community – but practical knowledge of how to capture and use those numbers is much harder to find. The effective application of learner analytics requires a deep understanding of everything from delivering and tracking learning events, to interpreting and leveraging learning data to improve your future courses. Either of these areas would be difficult for anyone to pick up quickly, and with the growth of analytics, many are finding themselves in a position where they’re expected to do both.
Fortunately, we’re here to help. In this webinar, Seth Puri, EVP of Global Sales and Marketing for Gutenberg Technology, will deliver a guide to collecting and assessing learning outcomes – and demonstrate how doing so can help you deliver a differentiated eLearning experience.
You’ll walk away with a wealth of knowledge on learner analytics, including:
- The must-have components of effective student analytics tracking
- Which analytics work best to determine course effectiveness
- How to interpret learning data: student performance vs. course performance
- What’s next: the future of adaptive learning
5. Enhancing the Learning Experience by
Leveraging Learner Analytics
A guide to how collecting & assessing learning outcomes
helps you deliver a differentiated elearning experience.
6. Agenda
1. EdTech Solutions that Achieve Learning Objectives
○ A review of the must-have solution components
○ All-in-one vs. modular solutions
○ How to deliver & track learning events: xAPI/TinCan vs. IMS Caliper
2. Translating Learning Events into Learning Analytics
○ What analytics work best to determine course effectiveness?
○ How to interpret learning data: learner performance vs. course performance
○ How instructors can leverage data for course improvements
○ What’s next: Adaptive learning
8. WebReader, iOS, Android AppsLMS
Learning Management System
User
Mgmt
Authoring Solution / LCMS
Project ManagementImport
(XML, ePub, PDF)
Authoring
styling, content, activities, media, ...User
Mgmt
Content Distribution
API
LTI
Analytics
Teaching Content
Content Distribution
Content Creation
Analytics
Read, Interact
& Learn
Learning Library
Content Producers / Authors Instructors Learners
Analytics
User
Mgmt
build, manage & distribute
teaching & learning content
manage, assemble & distribute
learning content
learn on learning content
Learning Event Server
1. EdTech Solutions that Achieve Learning Objectives
Global Solution Architecture
9. User Management
● User invitation,
registration, login
● SSO, access control,
module permissions
● Project teams & project
permissions
➢ Users: Education Publishers, Authors, Business Learning Centers, Communication Dpt, ...
➢ Solution Role: Build, Manage & Distribute teaching, learning or communication content
Project Management
● New project creation
● Project import via various
formats (ePub, PDF, XML, …)
● Project list, sort, search, delete
● Project information & metadata
● Project settings & accessibility
Analytics
Insights for content creators based on end-user interactions to improve content
Project Authoring
● Project theme/layout: colors,
fonts, styles, layout CSS
● Project enrichment: media,
activities, widgets, …
● Print/digital or learner/instructor
specific content
Project distribution
● Project versioning
● Project team-review
● Distribution to learners:
LMS, LTI direct linking,
custom API, direct to
readers
1. EdTech Solutions that Achieve Learning Objectives
The Authoring Solution
10. User Management
● User invitation,
registration, login, SSO
● Instructor management
● Classrooms / Groups
Content Library
● Content preview, bookmarks
● Content assembling from
learning objects
● External content import
Analytics
Insights for Instructors on content end-user interactions to improve learning
Content Authoring
● Simplified authoring
● Insertion of media, activities
Content Distribution
● Distribution to group of
learners / users
● Assignation, expiration
date, specific settings
➢ Users: Teachers, Instructors, Communication Managers
➢ Solution Role: Manage & Distribute learning or communication content
1. EdTech Solutions that Achieve Learning Objectives
The Learning Management Solution
11. User Management
● User registration,
login, SSO
● User profile, settings
Learning Library
● List, sort content, detailed info
● Content download for offline reading
● Search content
● Additional content purchase
● Social
Analytics
Summary of interactions with content, progress reports
Content Readers
● Content display with layout
● Media rendering
● Activities & assessments
● Search in content
● Accessibility
● User annotations
● Interaction events
➢ Users: Students, Learners, Employees, Customers, ...
➢ Solution Role: Discover & Interact with the distributed content
Android Apps
Chromebooks
MacOS
iOS apps
Windows
1. EdTech Solutions that Achieve Learning Objectives
The Readers for End-Users
12. To build & deliver digital content, we see companies follow a similar development path:
To integrate with the legacy ecosystem already in place, specialized solutions need to:
● Be modular with open API available
● Support industry standards: IMS LTI, Caliper in Education/Publishing, Scorm / Tincan for Business Learning
● Integrate with market-leading solutions such as Moodle or Blackboard LMS
Start with an all-in-one
solution, generally a LMS
that can author
& deliver to the web.
Addition & integration of more
specialized solutions.
1
Expansion of digital distribution
=> Richer needs,
workflow industrialization
2
Authoring Solution
Mobile & Web
Distribution Solutions
Analytics Solution?
1. EdTech Solutions that Achieve Learning Objectives
Integrated or Modular Solution?
13. Analytics are about building a quantitative analysis of the end-users’ behavior with your content.
● Recipients: end-user, instructors, content producers, …
● Types of analytics: aggregate figures, ratio, charts & graphs (dataviz)
Track meaningful end-user
interactions
with content (= events)
Flow events into different user
interfaces
Conclude
most meaningful analytics for
recipient based on events
Build interfaces that provide
easy access to desired
info / analytics
1. EdTech Solutions that Achieve Learning Objectives
Getting Started with Analytics
14. xAPI (Project TinCan)
Initiative led by the &
Open-source, Apache v2 license
Successor of SCORM since 2013
API Goals:
● Allow applications to share data about human
performance on instructional content & context information
● Store information in a Learning Record Store (LRS)
➔ JSON format over HTTP + OAuth
➔ Service and device agnostic = versatile, transferable
from LMS to mobile devices, wearables, search
engine, gamification, blog, webinar, CRM, ...
24 conformant Learning Record Stores, 39 official adopters
➔ Versatile, but mostly adopted in business learning
Best practices for collecting end-user interaction events led to 2 different analytics standards:
Caliper v1.1
Initiative led by the , started 2015
Closed consortium which controls the Caliper IP
Goals:
● Approach to describing, collecting & exchanging learning
activity data
● Way of measuring learning activities & effectiveness
● Enables new services to measure, report & visualize
➔ JSON format over HTTPS/TLS with API key
➔ Focus around the LMS only => adopted or being
adopted by the main LMS vendors
Only 11 solutions certified by IMS, ex: Blackboard, Canvas,
VitalSource + others having “adopted” ex: Moodle, Elsevier
➔ Focused on learning, mostly in e-education
1. EdTech Solutions that Achieve Learning Objectives
Analytics Standards
16. xAPI (Project TinCan)
The Activity Provider delivers the learning experiences and
communicates the learning info to the Learning Record Store.
Learning experiences are described through statements:
● Actor: individual/group performing an action within an Activity.
● Verb: action performed during the learning experience.
The xAPI only specifies how to create meaningful Verbs.
● Object: something that the Actor interacted with,
ex: a book, an elearning course, a unit of instruction, an activity
● Result (optional): measured outcome of the statement
ex: score, success, completion, response, activity duration
● Context (optional): additional information ex: the instructor,
inclusion in a broader activity or team activity
● Attachment (optional): evidence of a learning experience
ex: a record of communication, an essay, a video, or a
certificate
● TimeStamp, Authority: device/service delivering the statement
Caliper v1.1
An activity within an Application is tracked through a Sensor
Endpoint that sends events to a Record Store via the Sensor
API.
A Metric Profile describes a type of learning activity with a
controlled & constrained vocabulary.
ex: Annotation, Assessment, Forum, Media, Reading, Outcome, …
● Actor
● Action
● Activity
● Optional: EventTime, target, generated, referrer, edApp, group,
membership, session, federatedSession (LTI), extensions
1. EdTech Solutions that Achieve Learning Objectives
Analytics Standards Cont.
18. Standard Analytics*
1. Content Popularity > Volume & Audience
> Check time spent, # of pages read & # of unique users
> Compare between content subparts & similar content blocks
● Beware that time spent can be rigged; check how pages read are
counted (ie. scroll in the page, assess minimum time spent)
● Check the distribution scenarios; volume can come from content
popularity either among instructors or learners
2. Learner Engagement > Interactions with Content
Digital learning brings the unvalued benefit of engaging learners through
interactivities, making the learning more active and thus efficient.
> Check # of interactions & time spent with videos, audio, quizzes or
widgets within a content block
3. Learner Success > Scores
> Check learner score across different activities to assess their
understanding of a course
> Compare between content blocks to assess their relative difficulty
* Standard learning stats:
- Time spent
- Average result
- Last activity date
- Sessions, ...
2. Translating Learning Events into Learning Analytics
What analytics best determine course effectiveness?
19. Advanced Analytics
1. Learner Cohort Analysis
> Check % of pages read & volume of interactions by learner over time
If % is plateauing at a low level vs other content, this might be due to not
having enough engaging content or having irrelevant courses.
2. Time-Efficient Learning
> How much time is needed on average for a learner to master a topic
having learned with your content?
Compare average score on the same assessment between readers having
interacted with different content pieces with similar learning objectives.
3. Learner Progression
> Identify correlations between learner result progression over time and
their interaction with specific content or piece of subcontent.
Example of Cohort Analysis
2. Translating Learning Events into Learning Analytics
What analytics best determine course effectiveness?
20. Analyze Learner Performance Over Time, Across Courses
● Check how learner average performance progressed after a course:
○ Assess before & after the course
○ Compare course performance vs difficulty
● Check correlation between performance with time spent & volume of
engagement:
○ Check performance of most engaged learners vs the others
○ Take note that more engaged learners are often low-level learners
Analyze Performance Distribution Across Learners
● Check how course is performing among low-level, standard &
high-level learners:
○ Assess before & after the course
○ High-level students usually need less teaching content => course
performance is mostly reflected on the performance increase on
low-level & standard learners
How do you clearly distinguish the respective impact of
learning content, instructor methodology & learner work
on learner performance ?
2. Translating Learning Events into Learning Analytics
Interpreting Learning Data: Learner Performance vs.
Course Performance
21. Learning Content Improvements
● Course Clarity & Efficiency
○ How are learners performing on the content?
Is the score distribution binary or more evenly distributed?
○ Are learners spending too much time on the content in
relation to the performance increase?
➢ Review the course layout and pedagogy
● Course Engagement
○ Is the level of learner engagement matching the level of
other content?
➢ Add interactive elements & activities to engage learner
● Course Level / Difficulty
○ Compare the average score across learners for the
different course sections and across courses vs
expectations
Teaching Improvements
Teaching efficiency is highly correlated with learner engagement / level of
activity and teaching personalization.
● Engage Learners
○ Ensure courses & assignments have a good mix of activities
○ Encourage learners to interact more with the content
● Personalize Teaching
○ Start with a quick test to break down the classroom into ability groups
○ Differentiate assignments for each ability group via digital learning
○ Identify the sticking points & root-causes for low-level learners:
■ Assess pre-requisites
■ Slow down learning pace / try a more progressive learning method,
(ex: step-by-step > application > more complex assessments)
■ Try other pedagogies & learning content types (videos, schema, …)
Which data & actions can improve your teaching ?
2. Translating Learning Events into Learning Analytics
Leveraging Data for Course Improvements
22. What is Adaptive Learning?
Adaptive learning is about automatically personalizing the learning path of
each learner to optimize the efficiency of their learning experience for
self-defined learning objectives or by assignment from an instructor.
Analyzing learner grades helps you to continually determine:
● 🔽Is a learner struggling? → teach prerequisite knowledge
● = Is a learner progressing as expected? → no action required
● 🔼Is a learner already mastering the knowledge? → save time & push them
learning content for more advanced or other supplementary knowledge
What’s Next: Adaptive Learning
➔ It can also try to identify per learner the most efficient types of learning
content, learning pace, optimal hours of learning, …
➔ A learning profile is built for each learner over time based on all their
interactions with the content & assessment results.
➔ Memory anchoring features, is available, can ensure knowledge is correctly
memorized over time by regularly reviewing it (recurrence of testing is
adapted to each learner).
2. Translating Learning Events into Learning Analytics
23. Prerequisites & Adaptive Learning Engine
1. Add a Semantic Layer to Your Content
○ Atomic learning objects (LOs) provide more granularity for
personalizing each learner’s learning path
○ LOs to be metattagged with role & target knowledge
ex: “LO 1 teach addition with 2 digits”
○ Build a knowledge tree that links the different knowledge of the
content together to detail which ones are prerequisites to others
2. Track Learner Behavior: Assessment scores, time spent, interactions
with media and interactivities, …
Based on the LO semantic & learner interactions, adaptive learning
engines leverage bayesian or neural network algorithms to dynamically
identify which next LO should be shown to each learner.
Limits & Perspective
● Requires additional work to add semantic layer to learning content
● Fully automated engines (ex: blackbox) work better for self-paced
learning, but are harder to incorporate with an instructor pedagogy
● Works best as an instructor aid with learning recommendations &
insights on learners
2. Translating Learning Events into Learning Analytics
What’s Next: Adaptive Learning
24. Summary
1. EdTech Solutions that Achieve Learning Objectives
○ The must-have components:
■ A global architecture
■ A combination of authoring, learning management and end-user solutions
○ All-in-one vs. modular: we recommend all-in-one, pending your unique needs
○ xAPI/TinCan vs. IMS Caliper: either, as long as you can build a quantitative analysis of user behavior
2. Translating Learning Events into Learning Analytics
○ Standard analytics: Time spent, # pages read, # unique users, learner scores
○ Advanced analytics: Learner cohort analysis, time efficiency, learner progression
■ Both kinds of data help educators pivot course improvement and student engagement
○ Adaptive learning: the future of personalizing the elearning experience for all learners
