Presentation given at the November 2011 NERCOMP Workshop on Seeking Evidence of Impact

1. Assessment:
Designs and Approaches
Gail Matthews-DeNatale
Northeastern University
Dave Consiglio
Bryn Mawr College

2. The Cycle
Identify
Scope
Interpret
Data
Share &
Act Upon
Findings
Form
Research
Question
Design &
Implement
Research
You Are
Here

3. On a Dime
http://www.flickr.com/photos/jdhancock/3544855402

4. Gold Standard
http://www.flickr.com/photos/bullionvault/3591745555

5. Three Scenarios
Activity

6. Online Enrollment
Your Dean wants to raise the cap
on enrollment in online courses.
The cap limit will be revisited on an
annual basis. What research can
you perform with your students
and your faculty at your institution
to inform these decisions?

7. Clickers
Your institution has been using
classroom response systems
(a.k.a. “clickers”) for several years.
Usage is robust. You’re leading a
task force to assess the impact of
clickers on learning.

8. Blended Learning
A pilot initiative is underway at
your school to integrate blended
courses across the curriculum.
What will be the research
component for the pilot, how will
the research be performed, and
how will it be used?

9. North Star Questions
What do you want to know/learn?
What’s the purpose?
What’s the population?
Who will receive the results?
Who could/should be involved?

10. Scenario Planning
20
Minutes
What do you want to know?
What’s the population?
What’s the purpose?
Who will be involved?
Who will receive the results?

11. Gold Standard
http://www.flickr.com/photos/bullionvault/3591745555

12. Gold Standard

13. Reliability
and Validity
Gold Standard

14. Controls
Gold Standard

15. Example: Try Science

16. Qualitative Methods (Mostly)
Background information
(years teaching, education, prior OL)
Activity and work products
(discussion texts, projects, reports)
Formative feedback responses
Teacher follow-up
interviews
Program retention rates
Pre-post thought experiments
Self-reported increase in confidence
Change in classroom practice
(pre- post -observation/video)
Tool
NVIVO

17. Interviews
I thought I knew everything
about density. When I did my
physics unit in class, I thought I
was all set. As long as you know
the formula, you know density --
that's the way I taught it before.
Try Science blew my mind. At first I was a little bit angry
because I found out that for the three years I taught
density, in some cases, I know truly believe that I was
teaching it incorrectly. The assumptions that I made from
reading it instead of doing it were wrong.

18. Turning Qual into Quant
96% Program Retention Rate

19. Cost: $600,000
Report: 121 pages
Some Numbers

20. Quantitative Example: CATS
Goal: Study effect of interactive instruction on student learning
Tools: Standardized concept inventory, demographic and
attitudinal survey of students, teaching practice survey of instructors
Population: Undergraduate students taking Astronomy 101
Study group: Thousands of students of dozens of higher
education institutions from across the country
Process: Pre and post tests using concept inventory, student
survey demographic and attitudinal survey, measure classroom and
institutional variables
Results: Statistical analysis of individual and classroom
influences on learning gains

21. Your Task Over Lunch
What would these “gold standard”
methodologies would look like if
they were done “on a dime”?
How might you adapt them for use
in situations that are under-
resourced and pressed for time?