Case study presentation for the Learning and Knowledge Analytics 2013 MOOC.

1. Using Learner Analytics to
Understand Student Achievement in
a Large Enrollment Hybrid Course
slides posted:
John Whitmer, Ed.D.
Associate Director, Academic Technology Services
California State University, Office of the Chancellor
Society for Learning Analytics Research | LAK 2013 Case Study
February 19, 2013

2. Outline
1. Context
2. Methods & Tools
3. Findings
4. Conclusions & Next Steps

3. 1. CONTEXT

4.  Founded in 1887
 15,257 FTES, 95% from
California, serves 12 counties
 Primarily residential,
undergraduate teaching
college
 Campus in California State
University system
(23 colleges, 44,000 faculty
and staff, 437,000 students)

5. CSU Budget Proposed Increase!
Source: CSU Chancellor’s Office
http://bit.ly/X7LYeK

6. Case Study: Intro to Religious Studies
• Undergraduate, introductory, high demand
• Redesigned to hybrid delivery format
54 F’s
through “academy eLearning program”
• Enrollment: 373 students
(54% increase on largest section)
• Highest LMS (Vista) usage
entire campus Fall 2010
(>250k hits)
• Bimodal outcomes:
• 10% increase on final exam
• 7% & 11% increase in DWF
• Why? Can’t tell with aggregated data

7. Driving Conceptual Questions
1. How is student LMS use related to academic
achievement in a single course section?
2. How does that finding compare to the relationship of
achievement with traditional student characteristic
variables?
3. How are these relationships different for
“at-risk” students (URM & Pell-eligible)?
4. What data sources, variables and methods are most
useful to answer these questions?

8. University
Gender Freq. Percent Average Difference
Female 231 62% 51% 11%
Male 142 38% 48% -10%
Age 0%
17 22 6%
18-21 302 81%
22-30 22 6%
31+ 1 0%
Under-represented
Minority
No 264 71% 73% -2%
Yes 109 29% 27% 2%
Pell-eligible Freq. Percent
No 210 56%
Yes 163 44%
First Attend College Freq.
No 268 72%
Yes 105 28%
Enrollment Status Freq.
Continuing Student 217 58%
Transfer 17 5%
First-Time Student 139 37%

9. 2. METHODS & TOOLS

10. Methods at a Glance
 Data Sources: 1) LMS logfiles, 2) SIS data,
3) Course data
 Process
1. Clean/filter/transform/reduce data (70% effort)
2. Descriptive / exploratory analysis (20% effort)
3. Statistical analysis (10% effort)
 Factor analysis
 Correlation single variables
 Regression multiple variables; partial & complete

11. Tools Used
App Function
Excel Early data exploration; simple sorting; tables
for print/publication
Tableau Complex data summaries and explorations;
complex charts; presentation charts
Final/formal descriptive data; statistical
analysis; some charts (scatterplots)
Statistical analysis (factor analysis)

12. Variables
Student Characteristic Independent Variables
Gender
Under Represented Minority (URM)
Pell-Eligible
High School GPA
First in Family to Attend College
Student Major (Discipline)
Enrollment Status
Interaction URM & Gender
Interaction URM & Pell-Eligibility
Learning Management System Usage Variables
Total LMS course website hits
Total LMS course dwell time
Administrative tool website hits
Assessment tool website hits
Content tool website hits
Engagement tool website hits
Dependent Variable: Final Course Grade

13. Missing Data On Critical Indicators

14. Final data set: 72,000 records (-73%)

15. LMS Use Consistent across Categories
Factor Analysis of LMS Use Categories

16. 3. FINDINGS

17. Clear Trend: Grade w/Mean LMS Hits

18. Question 1 Results:
Correlation LMS Use w/Final Grade
Scatterplot of
Assessment Activity
Hits vs. Course
Grade

19. Question 2 Results:
Correlation: Student Char. w/Final Grade
Scatterplot of
HS GPA vs. Course
Grade

20. Conclusion: LMS Use Variables better
Predictors than Student Characteristics
LMS Student
Use Characteristic
Variables Variables
18% Average
(r = 0.35–0.48)
> 4% Average
(r = -0.11–0.31)
Explanation of change Explanation of change
in final grade in final grade

21. Smallest Largest
LMS Use Variable Student
>
Characteristic
(Administrative
Activities) (HS GPA)
r = 0.35 r = 0.31

22. Combined Variables Regression Final Grade by
LMS Use & Student Characteristic Variables
LMS Student
Use Characteristic
Variables Variables
25%
(r2=0.25)
> +10%
(r2=0.35)
Explanation of change Explanation of change
in final grade in final grade

23. Question 3 Results:
Regression by “At Risk” Population Subsamples

24. At-Risk Students: “Over-Working Gap”
24

25. Activities by Pell and Grade
Extra effort
in content-
related
activities

26. 4. CONCLUSIONS & NEXT STEPS

27. Conclusions
1. At the course level, LMS use better predictor of
academic achievement than student demographics
(what do, not who are).
2. Small strength magnitude of complete model
demonstrates relevance of data, but suggests that
better methods could produce stronger results.
3. LMS data requires extensive filtering to be useful;
student variables need pre-screening for missing
data.

28. More Conclusions
4. LMS use frequency is a proxy for effort. Not a
very complex indicator.
5. Student demographic measures need revision
for utility in Postmodern era (importance to
student, more frequent sampling, etc.).
6. LMS effectiveness for at-risk students may be
caused by non-technical barriers. Need
additional research!

29. Ideas & Feedback
Potential for improved LMS analysis methods:
 social learning
 activity patterns
 discourse content analysis
 time series analysis
Group students by broader identity, with unique
variables:
 Continuing student (Current college GPA, URM, etc.
 First-time freshman (HS GPA, SAT/Act, etc)

30. Feedback? Questions?
John Whitmer
jwhitmer@calstate.edu
Slides
Complete monograph
http://bit.ly/15ijySP
Twitter: johncwhitmer