A powerpoint summarizing a scholarly paper presented at the Association for the Study of Higher Education (ASHE) on nonresponse to online course evaluations. Uploaded 16 Nov 2010. Indianapolis, IN.

1. WHO DOESN’T RESPOND AND WHY? AN
ANALYSIS OF NONRESPONSE TO ONLINE
STUDENT EVALUATIONS OF TEACHING
Meredith J.D. Adams & Paul D. Umbach

2. CONTEXT
• Participation decline while survey requests are
increasing. (Nonresponse increases the risk for
error.)
• Ongoing change from paper to web-based
course evaluations.
• Results from course evaluations are used for
research and decision-making processes about
personnel.

3. FRAMEWORK
• Survey Theories
– Salience (Dillman et al., 2002; Groves et al., 2004,
2009)
– Survey Fatigue (Groves et al., 2004; Porter &
Whitcomb, 2005)
• Academic Disciplines framework (Holland,
1966; Rosen et al., 1997; Smart et al., 2000)
• Previous survey and course evaluation research
at postsecondary institutions

4. PURPOSE OF STUDY AND RQs
• Purpose of study is to investigate factors of
participation in course evaluations
• Research questions
– To what extent did course salience increase the
likelihood of response? To what extent did between-
level interactions influence these results?
– To what extent did survey fatigue influence
response?

5. METHODS OF ANALYSIS
• Study measured likelihood of response
• Hierarchical modeling techniques
• 30 variables
– Students (demographics, housing, number of surveys
students completed)
– Course (if department of course and student’s major
matched; grade earned)

6. RESULTS
Statistically significant
factors associated with
nonrepsonse:
– Gender (Male)
– Ethnicity (African-American,
Asian)
– Housing (off-
campus/commuters)
– Athletes
– Grades (Ds, Fs,
ungraded
students/courses)
– Age (traditional age)
– Nontransfers
– Class rank
(sophomores & juniors)
– Students with more
than 10 SETs to
complete

7. 48.9%
response rate
(overall)

8. RESULTS (continued)
• Using Holland’s six major academic types
– Realistic majors were more likely to respond than
social, artistic, conventional, enterprising, and
investigative disciplines.
– Social majors were less likely to respond than all
other students if the course was in the same
department as the student’s major.
• Most variables were no longer statistically
significant when the course was in the same
department as their major.

10. DISCUSSION
• Mostly aligned with previous research and
theories of survey participation*
– Exceptions = Environment of major/course, class
rank
• Introduced new potential influences on
participation
– Campus housing, athletes, transfer status, etc.
* Avery et al., 2006; Cohen, 1981; Clarksberg, et al., 2008; Dey, 1997; Dillman et al., 2002, 2009; Fidelman, 2007; Groves et
al., 2004, 2009; Johnson et al., 2002; Jones, 2009; Kaplowitz et al., 2004; Lepkowski & Couper, 2002; Marsh, 2007; Moore &
Tarnai, 2002; Porter et al., 2004; Porter & Umbach, 2006a; Porter & Whitcomb, 2005; Sax et al., 2003, 2008

11. DISCUSSION
• Nonresponse Bias
• Salience
– Course Location
– Grades
– Environment of Student’s Major
• Survey Fatigue

12. LIMITATIONS
• Only one institution
• Only undergraduates
• Limited information about the course, the
faculty, class size, etc.

13. IMPLICATIONS
• Evaluate the major’s environment/culture
– How to increase salience?
• Encourage students unlikely to respond
• Examine how we utilize course evaluations

14. FOR MORE INFORMATION
meredith_adams@ncsu.edu
This paper can be found at:
1. https://cedupload.ncsu.edu/dropbox/uploads/Adams%20Um
bach%20Course%20Evaluation%20Nonresponse%20ASHE
%202010.pdf
2. http://bit.ly/9zjMAY
3. http://goo.gl/ByiZo
4. This PowerPoint Presentation is available at
http://www.slideshare.net/meredithNCSU/ashe-2010-course-
eval-nonresponse