CARMA Summer Workshop

1. Encouraging Responding
CARMA Internet Research Module
Jeff Stanton

2. Dillman Vs. Groves
Tailored Design - Dillman Leverage-Saliency – Groves
Use an advance email contact Follow-up procedures do not
or pre-screening procedure work in a vacuum; their
Second contact: Email a link to effectiveness depends
the questionnaire with a upon the survey context
carefully constructed and characteristics of the
recruitment message;
include pre-study incentive respondent
Third contact: Reminder Topic interest moderates the
message targeted to non- effectiveness of response-
respondents enhancing techniques
Subsequent contacts: Thank Community involvement
you messages, or second moderates the
reminders
Provide any post-study effectiveness of response-
incentives for participation enhancing techniques
May 15-17, 2008 Internet Data Collection Methods (Day 2-2)

3. Elias Veris: Survey Gamification

4. Strategies for Encouraging Resp.
Use multiple response enhancing
strategies to increase the proportion of 1. Pre-notification
the sample that responds. The single 2. Publicity
most powerful technique for increasing 3. Instrument Design
response is having an interesting topic. 4. Incentives
The second most powerful technique is 5. Survey length
having interesting questions. Each 6. Reminders
technique may only increment 7. Opportunity
response rate by a few percentage 8. Response Monitoring
point. Each increment decreases the 9. Establish Importance
potential for bias, although response 10. Provide Feedback
rates under 95% do not generally
eliminate it.
May 15-17, 2008 Internet Data Collection Methods (Day 2-4)

5. Response Enhancing Techniques and Pct
Increases in Mail Surveys
Monetary incentives: 12% (Yammarino et al.,
1991) to 19% (Hopkins & Gullickson, 1993)
Advance notice: 13% (Roth and BeVier, 1998)
Follow-up/reminder: 6% (Yu and Cooper , 1983)
Personalization: 9% (Martin, Duncan, Powers &
Sawyer, 1989)
Each strategy has been tested and found
effective in Internet surveys, although
different improvement %s

6. Anseel: Org Surveys 1995-2008
Advance notice, personalization, topic salience, identification numbers (to
improve anonymity), and sponsorship all have generally positive
effects
Combining several response enhancing techniques is advisable
Results consistent with leverage-saliency
Consider how respondent characteristics relate to each response enhancing
technique
Surveys of managers: Mention sponsorship
Surveys of executives: Mention business relevance
Consumers: Don’t mention sponsorship
Non-managerial employees: Incentives ineffective
Non-working respondents: Incentives are effective
Personal distribution of surveys
Advance notice and follow-up have become less important over the years

7. Thorpe et al. 2009 (Family Practice)
Improving response rates when surveying
physicians:
Used Dillman’s Total Design Method including four
contacts by mail, RR = 48%
With the addition of registered mail with return
receipt plus $25 gift cards (unconditional
presentation), response rates for subsequent
survey studies rose to over 75%

8. Personally notify your potential participants that
they will be receiving a survey in the near
future.
Actively publicize the survey.
Be sensitive to the actual physical design of your
survey. For example, how questions are
ordered may impact respondent
participation.
– A study by Roberson and Sundstrom (1990)
suggests placing the more interesting and easy
questions first and demographic questions last.

9. Provide incentives, if appropriate. Inexpensive
items such as pens, key chains, or certificates
for free food/drink can increase responses.
Keep the survey to a reasonable length. A theory-
driven approach to survey design helps
determine what is absolutely necessary to
include in the survey instrument.
Send reminder notes. Response rates may bump up
3-7% with each reminder note, but keep in mind
that there's a point of diminishing returns when
you irritate people who have chosen not to
participate.

10. Give everyone the opportunity to participate
(e.g., paper surveys where required,
scheduling time off the phone in the call
centers, etc.). At one company for example,
most surveys run for 10 business days and
span across three work weeks.
Track response rates so that HR generalists
and/or the survey coordinators can identify
units with low response rates and contact the
responsible manager to increase responses.

11. Foster commitment to the survey effort. For
example, you can involve a wide range of
employees (across many levels) in the survey
development process. Link the content of the
survey to important business outcomes.
Provide respondents with survey feedback after the
project is completed. Be careful not to abandon
your participants once getting the data you
wanted from them. You are paving the way (or
failing to do so!) for future survey efforts.
Personalization of the survey invitation. Personal
signature as part of cover letter.
Topic salience

12. Even when controlling for the presence of other
techniques, advance notice, personalization,
identification numbers, and salience, are
associated with higher response rates.
Need to do more to just get the same results as the
past.
Target facilitation strategy to who you are
surveying.
– For top executives, salience of the survey topic was
most key. Incentives were counterproductive
– Incentives worked for unemployed individuals

13. Pre-screening w/Active Response
Dear sampled individual:
In one week, we will begin a study on this topic
where you will have the opportunity to earn $X
for completing a survey that is Y minutes long. Click
here if you expect to participate:
http://mysrv.com/screen.htm?participate=yes
Please click here if you plan to decline participation or
do not wish to receive the recruitment message:
http://mysrv.com/screen.htm?participate=no
Internet Data Collection Methods (Day 2-
May 15-17, 2008
13)

14. Dealing with Non-Contacts
Cull
bad
Advance Recruiting addrs
Screening Message Ethics question:
Who gets rewards?
1st Rem
How much missing
Message
data allowed?
2nd Rem
Message
Cull Cull
good
refusals Distribute
resps
Rewards
Internet Data Collection Methods (Day 2-
May 15-17, 2008
14)