1.
The RISE Framework: Using learning analytics
for the continuous improvement of open
educational resources
Robert Bodily, Rob Nyland, and David Wiley

2.
Project Background
◈ Bob and Rob were members of John Hilton’s OER Research
Class at BYU
◈ In the class we were using the COUP framework for OER
Research
◆ Cost
◆ Outcomes
◆ Use
◆ Perceptions
◈ We wanted to understand what learning analytics could bring to
the conversation around OER

3.
The 5 R’s of OER
Retain
Reuse
Revise
Remix
Redistribute

4.
“Because the Open Curriculum is licensed in such a way that we can
revise materials directly, OHSU is able to engage in a highly
data-driven curriculum improvement process. As students go
through the online courses, their clicks and learning paths can be
tracked and recorded. This information can be combined with item
response theory and learning outcome analysis data to set priorities
for curriculum or assessment revision empirically. This data-driven
process of curriculum improvement should allow the Open
Curriculum to reach a very high level of quality very quickly.”
- David Wiley, 2009

5.
Continuous Improvement
Learning
Analytics
OER
Data-Driven
Continuous
Improvement

6.
Continuous Improvement at OLI
“A key attribute of the OLI
environment is that while students
are working through the course, we
are collecting analytics data and
using those data to drive multiple
feedback loops”
-Strader & Thille (2012)

7.
Resource Inspection, Selection, and Enhancement
(RISE)
Inspection: Analyze course resources using a variety of automated
methods
Selection: Identify resources that need additional evaluation
Enhancement: Facilitate enhancement through an automated
inspection and selection process (enhancement is usually done
manually by an instructional designer).

8.
Four Quadrant Evaluation Structure
High
Grades
High student prior knowledge,
inherently easy learning outcome,
highly effective content, poorly written
assessment
Effective resources, effective
assessment, strong outcome
alignment
Low
Grades
Low motivation or high life distraction of
students, too much material, technical
or other difficulties accessing resources
Poorly designed resources,
poorly written assessments,
poor outcome alignment,
difficult learning outcome
Low Use High Use

9.
Our Approach: Case Study
◈ Fall 2015 Introduction to Business Course (N = 1002)
◈ OER courseware platform
◆ OER online text
◆ Videos
◆ Simulations
◆ Formative assessments
◆ Practice assessments
◆ Summative assessments
◈ Assessments and resources are all linked to course outcomes

10.
Data Tracking
Resource Use Data
◈ Tracked with Google Analytics
◈ Defined as pageviews (other options include time spent and ratings)
Assessment Data
◈ Tracked with Open Assessments
◈ Defined as grade on summative assessments

11.
Types of Analyses
1. Outer quadrant analysis
2. Page type analysis
3. Extreme outlier analysis

12.
Outer Quadrant Analysis
◈ Blue dotted line is the
additive z-score
threshold.
◈ Black lines are the
median values for
pageviews and grades

13.
Outer Quadrant Analysis
Page Title Grade PV/student A.Z-Score Quadrant
Reading: The Four Ps of Marketing 0.81 1.64 3.65 1
Outcome: Recruiting Employees 0.88 0.53 3.4 2
Video: Legal Rights Under Implied
Warranties
0.53 0.57 3.6 3
Reading: The Organization Chart and
Reporting Structure
0.64 1.89 4.56 4

14.
Page Type Analysis
Reading Pages Self-Check Pages Outcome Pages
Video Pages
Simulation
Pages

15.
◈ External search traffic: 1500 pageviews
◈ Course search traffic: 1626 pageviews
◈ Once external search traffic was removed, it
was no longer an outlier.
Extreme Outlier Analysis
Page Title Grade PV/student A.Z-Score
Reading: Stages of an
Economy
0.83 2.82 7.79

16.
Enhancement
◈ Where the continuous improvement “rubber meets the road”
◈ Start with a theory of the domain
◈ Examine assessments
◆ Evaluate student response patterns / IRT / CFA
◆ Review assessments for design problems (negative language)
◆ Redesign or replace assessments
◈ Move on to content
◆ Examine content for assessment alignment (outcome, words)
◆ Examine content for micro-design problems (mismatch)
◆ Redesign or replace content

17.
Questions?