Presentation delivered at WASC ARC conference on April 11, 2013 on the CSU Data Dashboard and Chico State Learning Analytics case study.
Chico State Case Study: Academic technologies collect highly detailed student usage data. How can this data be used to understand and predict student performance, especially of at-risk students? This presentation will discuss research on a high-enrollment undergraduate course exploring the relationship between LMS activity, student background characteristics, current enrollment information, and student achievement.
CSU Data Dashboard: By monitoring on-track indicators institutional leaders can better understand not only which milestones students are failing to reach, but why they are not reaching them. It can also help campuses to design interventions or policy changes to increase student success and to gauge the impact of interventions.
TỔNG ÔN TẬP THI VÀO LỚP 10 MÔN TIẾNG ANH NĂM HỌC 2023 - 2024 CÓ ĐÁP ÁN (NGỮ Â...
Improving Student Achievement with New Approaches to Data
1. John Whitmer, Ed.D.
Academic Technology Services
California State University, Office of the Chancellor
WASC ARC Conference
April 11, 2013
Improving Student Achievement with
New Approaches to Data:
Learning Analytics &
the CSU Data Dashboard
slides @ slideshare.net/JohnWhitmer/
2. slides @ slideshare.net/JohnWhitmer/
Outline
1. Context: California State University &
Graduation Initiative
2. Chico State Learning Analytics Case Study
3. CSU Data Dashboard Project
4. Next Steps
5. Discussion
4. slides @ slideshare.net/JohnWhitmer/
California State University
http://calstate.edu
23 campuses
437,000 FTE students
44,000 faculty and staff
Largest, most diverse, &
one of the most
affordable university
systems in the country
Play a vital role in the
growth & development of
California's communities
and economy
7. slides @ slideshare.net/JohnWhitmer/
New Approaches to Using Data
Enable data-driven decision making for
interventions earlier in the student experience by
1. Integrate new data sources & variables
2. Disseminate findings to a broader audience
3. Provide ability to interact with data analysis,
conduct ad-hoc and custom reporting
9. slides @ slideshare.net/JohnWhitmer/
Economist. (2010, 11/4/2010). Augmented business: Smart systems will disrupt lots of industries, and perhaps the entire economy.
The Economist.
200MB of data emissions annually!
10. slides @ slideshare.net/JohnWhitmer/
Source: jisc_infonet @ Flickr.com
Source: jisc_infonet @ Flickr.com
Logged into course within 24
hours
Interacts frequently in
discussion boards
Failed first exam
Hasn’t taken college-level
math
No declared major
11. slides @ slideshare.net/JohnWhitmer/
Case Study: Intro to Religious Studies
• Undergraduate, introductory, high
demand
• Redesigned to hybrid delivery format
through “academy eLearning program”
• Enrollment: 373 students
(54% increase on largest section)
• Highest LMS (Vista) usage
entire campus Fall 2010
(>250k hits)
• Bimodal outcomes compared to
traditional course
• 10% increase on final exam
• 7% & 11% increase in DWF
• Why? Can’t tell with aggregated data
54 F’s
13. slides @ slideshare.net/JohnWhitmer/
Learner Analytics
“ ... measurement, collection, analysis and
reporting of data about learners and their
contexts, for purposes of understanding and
optimizing learning and the environments in
which it occurs.” (Siemens, 2011)
14. slides @ slideshare.net/JohnWhitmer/
Pervasive Adoption of Learning Management Systems
Institution-Supported IT
Resources and Tools. Reprinted
from “The ECAR Study of
Undergraduate Students and
Information Technology,” Eden
Dahlstrom, 2012 by the
EDUCAUSE Center for Applied
Research.
15. slides @ slideshare.net/JohnWhitmer/
Guiding Questions
1. How is student LMS use related to academic
achievement in a single course section?
2. How does that finding compare to the relationship of
achievement with traditional student characteristic
variables?
3. How are these relationships different for
“at-risk” students (URM & Pell-eligible)?
4. What data sources, variables and methods are most
useful to answer these questions?
16. slides @ slideshare.net/JohnWhitmer/
LMS Use Variables
1. Administrative Activities
(calendar, announcements)
2. Assessment Activities
(quiz, homework, assignments,
grade center)
3. Content Activities
(web hits, PDF, content pages)
4. Engagement Activities
(discussion, mail)
Student Char. Variables
1. Enrollment Status
2. First in Family to Attend
College
3. Gender
4. HS GPA
5. Major-College
6. Pell Eligible
7. URM and Pell-Eligibility
Interaction
8. Under-Represented
Minority
9. URM and Gender
Interaction
17. slides @ slideshare.net/JohnWhitmer/
Tools Used
App Function
Excel Early data exploration; simple sorting; tables
for print/publication
Tableau Complex data summaries and explorations;
complex charts; presentation charts
Final/formal descriptive data; statistical
analysis; some charts (scatterplots)
Statistical analysis (factor analysis)
Statistical analysis (charts)
19. slides @ slideshare.net/JohnWhitmer/
Predict the trend
LMS use and final grade is _______ compared to
student characteristics and final grade:
a) 50% smaller
b) 25% smaller
c) the same
d) 200% larger
e) 400% larger
20. slides @ slideshare.net/JohnWhitmer/
Predict the trend
LMS use and final grade is _______ compared to
student characteristics and final grade:
a) 50% smaller
b) 25% smaller
c) the same
d) 200% larger
e) 400% larger
23. slides @ slideshare.net/JohnWhitmer/
Combined Variables Regression Final Grade by
LMS Use & Student Characteristic Variables
LMS
Use
Variables
25%
(r2=0.25)
Explanation of change
in final grade
Student
Characteristic
Variables
+10%
(r2=0.35)
Explanation of change
in final grade
>
24. slides @ slideshare.net/JohnWhitmer/
Predict the trend
LMS use and final grade is ______ for “at-risk”*
students compared to not at-risk students?
a) 50% smaller
b) 20% smaller
c) No difference
d) 20% larger
e) 100% larger
Relationship indicates how strongly LMS use is correlated
with final grade; lower value equals less impact
*at-risk = BOTH under-represented minority and Pell-eligible
25. slides @ slideshare.net/JohnWhitmer/
Predict the trend
LMS use and final grade is ______ for “at-risk”*
students compared to not at-risk students?
a) 50% smaller
b) 20% smaller
c) No difference
d) 20% larger
e) 100% larger
*at-risk = BOTH under-represented minority and Pell-eligible
29. slides @ slideshare.net/JohnWhitmer/
Conclusions
1. LMS use is a better predictor of academic
achievement than student characteristics.
– LMS use frequency is a proxy for effort.
2. LMS data requires extensive filtering to be useful;
student variables need pre-screening for missing
data.
3. LMS effectiveness for at-risk students may be
caused by non-technical barriers.
4. Small strength magnitude suggests that better
methods could produce stronger results.
31. slides @ slideshare.net/JohnWhitmer/
Next Steps
Potential for improved LMS analysis methods:
time series analysis
social learning
activity patterns
discourse content analysis
Group students by broader identity, with unique
variables:
Continuing student (Current college GPA, URM, etc.)
First-time freshman (HS GPA, SAT/Act, etc)
33. slides @ slideshare.net/JohnWhitmer/
THE FRAMEWORK
Advancing by Degrees: A
Framework for Increasing
College Completion by
Offenstein, Moore &
Schulock
Institute for Higher
Education Leadership and
Policy and The Education
Trust (http://bit.ly/10QtMXC)
34. slides @ slideshare.net/JohnWhitmer/
This research describes
academic patterns (or leading
indicators) that occur early in
the pipeline that can be tracked
and monitored in real time
against milestones on the
graduation route.
37. slides @ slideshare.net/JohnWhitmer/
Milestones Leading Indicators
Year-to-year Retention
Transition to college level coursework
(English and Math)
Earn one year of college level credits
Complete General Education
Complete degree
Remediation
Begin remedial coursework in the first term, if
needed.
Complete needed remediation
Gateway Courses
Complete college-level math and/or English in
the first or second year
Complete a college-success course or other
first-year experience program
Credit Accumulation and Related Academic
Behaviors
Complete high percentage of courses
attempted (low rate of course dropping and/or
failure)
Complete 20-30 credits in the first year
Earn summer credits
Enroll full time
Enroll continuously, without stop-outs
Register on-time for courses
Maintain adequate academic progress
38. slides @ slideshare.net/JohnWhitmer/
Driving Questions for Dashboard
1. What percentage of students reach each of the
leading indicators?
2. What is the impact of reaching each of the
leading indicators on success rate?
3. Does meeting any of the indicators reduce or
eliminate gaps between student demographic
groups?
40. slides @ slideshare.net/JohnWhitmer/
Purpose
Demonstrate potential value of combined
reporting and statistics
Evaluate availability and integration of data
Pilot potential tools in real-world scenario
NOTE: production system may be dramatically
different from POC, given lessons learned and
scalability
56. slides @ slideshare.net/JohnWhitmer/
What’s Now … And Next
Conducting 3 multi-campus pilots
1. mCURL: Moodle Common Usage Reporting & Learning
Analytics: (8 CSU & 2 UC campuses)
2. Blackboard Analytics for Learn (3 campuses)
3. LMS-agnostic campus surveys
Investigating additional pilot with LMS-agnostic tool to
move beyond “clickometry” into social network analysis,
discourse analysis, etc.
Raises question for MOOC research: relationship between
student intent/motivation, student characteristics/leading
indicators, MOOC use, and achievement
Context: California State University & Graduation Initiative (5)Chico State Learning Analytics Case Study (20)CSU Data Dashboard Project (20)Next Steps (5)Q & A (10)
Kathy
John
John
John
Opportunity: If you have large number of students not meeting a particular indicator, gives you an opportunity
Overall graduation rates and goalsAchievement gapShows that we’re “green” for retention rates, but yellow for rates by achievement gap
Overall graduation rates and goalsAchievement gapShows that we’re “green” for retention rates, but yellow for rates by achievement gap
Overall graduation rates and goalsAchievement gapShows that we’re “green” for retention rates, but yellow for rates by achievement gap
Overall graduation rates and goalsAchievement gapShows that we’re “green” for retention rates, but yellow for rates by achievement gap
Drill into system – select multiple ethnicities. See the variation by overall ethnicity
Select Bakersfield campus – problem in second-year retention – but no problem by achievement gap
Bakersfield by gender – big problem for male, especially URM male students.