Presentation about the E2Coach computer tailored communication for physics education at the University of Michigan

1. What to do with actionable intelligence:
E2Coach as an intervention engine
Mysteries of teaching and learning
What do students know when they come? What
do they do while in class? What do they learn
and remember?
Can we predict outcomes? When? For whom?
Can we change outcomes? What should we do?
When? How? For whom?
How learning analytics and
computer tailored
communications enable us to
provide individualized
feedback, encouragement, and
advice to thousands of
students
5/8/2012 LAK 12: Vancouver
Tim McKay, University of Michigan Departments of Physics and Astronomy, LSA Honors Program

2. Learning Analytics and Knowledge
• Education systems generate • Learning Analytics: the
a rich stream of increasingly Better-Than-Expected
accessible data which can project for introductory
inform teaching and learning physics
• I’m from big data cosmology, – Generates actionable
how 100s of millions of intelligence
galaxies get to be the way • An intervention: adaptation
they are. We’re exploring of PH computer tailored
what data tells us about how communication in E2Coach
students get to be the way – Acts on the intelligence
they are provided by LA
Our E2Coach application – grown from analytics, aware of general
information like identity and goals, reaching into real time, content
5/8/2012 specific student performance data
LAK 12: Vancouver

3. Better-than-expected in Physics
• LA investigation of two • Admissions information
year-long intro physics – High school GPA
sequences – SAT and ACT
– State and Country of origin
• 48,579 students over 14 – First generation and SES
years – Gender
• Institutional data => • Internal UM information
construct predictions of – Cumulative GPA
student outcomes – Number of credits: UM and
transfer
• Identify those who do – Exam scores
better (and worse) than – Homework grades
expected: Find out why – Final grade in this course
Much more data is available…
5/8/2012 LAK 12: Vancouver

4. Essential findings from BTE
• Student grades can be • Significant performance
predicted with half disparities are apparent
letter grade accuracy – Gender: especially
– Incoming UM GPA the strong in courses where
most powerful predictor female students are
– Weak additional seriously
information in SAT/ACT underrepresented
Math – First generation college
students
• There is real dispersion:
– Students from low socio-
students do better (and economic status
worse) than expected households
5/8/2012 LAK 12: Vancouver

5. One-to-one line…
One sigma dispersion
around the mean for
each bin
Mean and error
on the mean for
each bin
5/8/2012 LAK 12: Vancouver

6. Exploring BTE/WTE
Learning analytics make all of
Male students Non first-gen students
these explorations possible, even
the qualitative ones. They
Female students First-gen students
provide actionable intelligence.
Gendered performance
Performance disparity seen for
first-generation college students,
disparity seen in intro also for low SES students…
physics courses nationwide
5/8/2012 LAK 12: Vancouver

7. What to do with actionable
intelligence?
• We now have John • Options for response:
Campbell’s ‘obligation – Tell someone and let
of knowing’: how we them act: student,
expect them to instructor, advisors: scale
remains a challenge
perform, and what
leads to success – Develop tools which act
directly in response to
• How can we tailor our student state:
approaches and • Intelligent tutors etc.
interactions to optimize • Computer tailored
communication systems
the success of all?
5/8/2012 LAK 12: Vancouver

8. Tailoring is well established and tested in public health, and has seen major
commercial application. An extensive body of peer-reviewed research reports on
tests of efficacy in design across interventions ranging from smoking cessation and
diabetes control to cancer treatment decision making and depression. This research
provides a strong base for the design of new computer tailored interventions
5/8/2012 LAK 12: Vancouver

9. MTS was built by the University of Michigan’s Center for
Health Communications Research, an established leader in
computer tailored public health interventions. MTS is a
mature, fully open-source software system for computer
tailored communication.
5/8/2012 LAK 12: Vancouver

10. E2Coach:
tailored support for • Three groups of players:
physics students – Department of Physics
– CHCR leadership and staff
• Used LA and MTS to – Consultants from across
construct “E2Coach”: an the campus
Electronic Expert • Project goals:
coaching system for intro – Improved performance and
physics courses affect for all students
• You can find a basic – Reduced disparities
information about the The E2Coach team:
project online: Tim McKay, Kate Miller, Jared Tritz, Gus
Evrard, Dave Gerdes in Physics
http://sitemaker.umich.edu/ecoach Vic Strecher, Ed Saunders, Holly Derry,
5/8/2012 LAK 12: Vancouver Mike Nowak at CHCR

11. How does E2Coach work?
Where the real Expertise of hundreds of
effort lies students, dozens of instructors
and behavior change experts Individually
Detailed personalized
information messages:
about what we all
thousands
of students
and their
MTS agree we
would say to
each
current student, if
The Michigan Tailoring System: a mature
status open-source software system for only we
creating content designed specifically could…
for an individual based on data about
that individual
5/8/2012 LAK 12: Vancouver

12. Expertise and Information
• Structured interviews • Knowledge of each
with faculty course and its structure
• Survey of 70+ student • Real-time input from
study group leaders the course gradebook
• Better-than-expected • Input from the student
interviews – Background, goals and
• Input from students interests, planned effort,
desired and expected
with different grades, self-efficacy,
backgrounds has confidence in physics
extreme relevance!
• Opt-In: 54% (953 total)
5/8/2012 LAK 12: Vancouver

13. What we provide
• Tailored advice on all aspects of the course,
including testimonials from relevant peers
5/8/2012

14. Performance
feedback
5/8/2012 LAK 12: Vancouver

15. First measures of impact
• First term ended four • Testing in fall using
days ago: final scores for fractional factorial design
enrolled students 2.3%
(4 ) higher
• Currently examining Score differences Actual
effects vs. usage observed in 104 measured
random samples score
• Disparities on gender, difference
SES, first-gen status
• This is a complex
intervention, with many
parts: which are key?
5/8/2012 LAK 12: Vancouver

16. E2Coach in the LA landscape…
Computer
tailored
communication
SOLAR: Open
Learning
Analytics: an BTE project
integrated & and other
modularized analytics
platform
Siemans, G., et al. July 2011
http://solaresearch.org
5/8/2012 LAK 12: Vancouver

17. Learning Analytics at Michigan
• Redesigning for Physics • UM Provost has
in the fall empowered Learning
– Full enrollment Analytics Task Force
– Much tighter approach • Charge:
• Expanding to situations – Improve information
w/diverse student environment for LA
bodies – Support LA projects with
– Intro Stats: 1800 three funding cycles
– Epidemiology: Masters – Revise institutional
metrics used for
– Freshman orientation teaching and learning
5/8/2012 LAK 12: Vancouver