Washington Evaluators Brown Bag by Stephen Axelrad

1. Washington Evaluators
Evening Brown Bag Session
Moving Beyond the Monkey
An overview of the latest online survey software
products and their implications for evaluators
Stephen Axelrad
Senior Consultant
Organization & Strategy
Survey & Assessment Services
Washington, DC
May 27, 2009

2. Getting started
 Your name
 Your organization
 Your typical evaluands
 Online survey software
– What products do you use?
– What evaluation projects do you use online surveys?
– What have been the benefits?
– What have been the challenges?
Booz Allen Hamilton Proprietary
2

3. Agenda
 Online survey concepts and issues
 Choosing an online survey software solution
 Examples
 Concluding points and guidance
Booz Allen Hamilton Proprietary
3

4. Moving Beyond the Monkey
 Online survey concepts and issues
 Choosing an online survey software solution
 Examples
 Concluding points and guidance
Booz Allen Hamilton Proprietary
4

5. In the Online Surveys issue of New Directions in Evaluation, Ritter
and Sue (2007) identify the respondent and survey characteristics
that are necessary for online surveys to work
 Evaluators can use online surveys when the following conditions are true for potential
respondents
– Respondents have regular access to the Internet (e.g., home or office)
– Respondents have provided their e-mail addresses to an organization who is authorized to
forward such list for research purposes
– Respondents in the sampling frame are geographical dispersed
– Respondents cannot be accessed easily through mail or interviewing modes of
administration (e.g., highly mobile or no land lines)
 Evaluators can use online surveys when the following characteristics are present in the survey
– Survey is self-administered and contains skip logic
– Survey includes open-ended questions which may be easier to type than handwrite
– Survey questions solicit responses to an image or audio/video clip
Booz Allen Hamilton Proprietary
5

6. Ritter and Sue (2007) illustrate how online surveys solve some old
challenges associated with other survey modes but also introduce
some new ones
 Online surveys overcome several obstacles associated with other survey methods
– Online surveys are less expensive than mail, telephone, or in-person methods
– Online surveys collect data that can be more easily recovered and stored than paper-based
surveys
– Online surveys allow for confidential and sensitive information to be collected without the
need for an interviewer and stored in electronically secure data warehouses
– Online surveys can include complex screening and branching options without confusing the
respondent and avoid interviewer error
 Online surveys may introduce new challenges not associated with other methods
– Online surveys may not work with populations that have less access or familiarity with the
Internet
– Online surveys need to have designs that are compatible with diverse browser setups
– Online surveys posted directly to the Web make it very difficult to obtain a sampling frame
Booz Allen Hamilton Proprietary
6

7. Discussion Questions
 What types of evaluations are more suited to online surveys?
 Just because evaluators can use online surveys in an evaluation, should evaluators use online
surveys?
 Are evaluators over-using online surveys?
 Is it possible to have a credible and rigorous evaluation with just online surveys? Should
evaluators aim to integrate multiple modes (e.g., mail, in-person, telephone)?
 How can online surveys enhance the value of evaluations to organizations and the
constituents they reach?
Booz Allen Hamilton Proprietary
7

8. Moving Beyond the Monkey
 Online survey concepts and issues
 Choosing an online survey software solution
 Examples
 Concluding points and guidance
Booz Allen Hamilton Proprietary
8

9. Evaluators have limited available resources for choosing an online
survey software solution
 A Google search for “online*survey*software” yielded 180,000 results
 Many software product review sites are not helpful to evaluators
– Some of these “product review” sites are created by software vendors and reviews are
written deceptively to favor their products
– Other legitimate sites evaluate the technical qualities but are written from an informationtechnology perspective and not an applied research or evaluation perspective
 The American Evaluation Association site lists more than 60 different online survey software
solutions
– The AEA site does not offer guidance on the quality or suitability of the different products or
vendors
– The AEA site includes products that range dramatically in cost and features
 Evaluators cannot rely upon their organization’s IT departments because these individuals
have little if any expertise in survey research or evaluation
Booz Allen Hamilton Proprietary
9

10. Evaluators must choose a software solution that includes features,
which parallel the steps in the survey research process
 Steps in the survey research process
Identify potential Create
Specify response
Invite respondents
Administer survey Analyze resultsReport results
respondents
survey questions
options
 Important features in an online survey software solution (i.e., functional requirements)
 Address
books
 Order
flexibility
 Restricting
ability
 E-mail
invitations
 Changing
format
 Reminder
scheduling
 Prepopulation
 Completion
status
 Translation
 Visibility &
Rotation
 Quota &
conditional
sampling
 Question
library
 Response
library
 Unique IDs  Skip logic
 Import lists
 Conditional
branching
 Duplication
 Audio/video
detection
 Number of
responses
stored
Booz Allen Hamilton Proprietary
 Panel setup
 Log in
 Browser
restrictions
 Survey
navigation
 Saving
 Chart &
table
formatting
 Missing data
 Look and
feel
 Partially
completed
responses
 Reporting
over time
 Data
validation
 Built-in
statistics
 Exporting to
SPSS, SAS
 Setting
templates
 Exporting to
Word, PDF
or
PowerPoint
10

11. Evaluators must consider the range of costs and user
requirements in selecting a software solution
 Evaluators need to develop a budget that can cover the range of costs associated with
different software solutions
– Product costs: survey publishing, web hosting, subscription fees, upgrades
– User costs: number of licenses, training, technical support
– Capacity costs: number of surveys, number of responses
 Evaluators need to develop user requirements that establish a hierarchy of users with varying
degrees of privileges that match their responsibilities in the survey research process
– Group administrators
– Survey editors/approvers
– Survey authors
– Survey analysts
– Survey translators
– Report viewers
Booz Allen Hamilton Proprietary
11

12. Evaluators must weigh the benefits and drawbacks of choosing
internal or off-the-shelf solutions
Internal Solutions
Commercial Off-The-Shelf (COTS) Solutions
 Benefits
 Benefits
– You can design a product that matches your
unique research and evaluation needs
– You are not as dependent on your
organization’s IT department
– You can develop surveys that use data from
your organization’s database
– You can consult with colleagues from other
organizations about the quality of the product
 Drawbacks
– Your organization must have an IT department
that has qualified and available application
developers
– Your organization must have networks and
servers capable of hosting the survey and
handling large amounts of data
– Your organization must be willing to wait several
months before a solution is available to
implement
– Your organization must be willing to invest
several thousand dollars up front
Booz Allen Hamilton Proprietary
– You can purchase an advanced software
solution that can satisfy most if not all of your
research and evaluation needs
 Drawbacks
– You may have to develop workarounds for some
research and evaluation projects
– You may encounter a software product that was
not designed with researchers and evaluators in
mind
– You may have to pay extensive fees in product
upgrades, technical support packages and
training
12

13. Discussion Questions
 Should AEA and other professional associations with researchers and evaluators as members
have online survey product guides?
 Given some of the advanced features in online survey software packages and the need to
interact with IT professionals, what technological capabilities do evaluators need to have in
order to decide on and implement online survey software solutions?
 How can evaluators determine if COTS products were designed with input from researchers
and evaluators?
 What factors do evaluators need to consider when determining the return on investment they
could get from internally developed solutions versus COTS packages?
Booz Allen Hamilton Proprietary
13

14. Moving Beyond the Monkey
 Online survey concepts and issues
 Choosing an online survey software solution
 Examples
 Concluding points and guidance
Booz Allen Hamilton Proprietary
14

15. Examples
 Internally developed solutions
– Needs assessment survey
– Implementation evaluation survey
 COTS packages
– Gravic Remark Web Survey
– Prezza Tech Checkbox Survey
– Snap Professional Edition
Booz Allen Hamilton Proprietary
15

16. COTS Example – Gravic Remark Web Survey
Booz Allen Hamilton Proprietary
16

17. COTS Example – Prezza Tech Checkbox Survey
Booz Allen Hamilton Proprietary
17

18. COTS Example – Prezza Tech Checkbox Survey
Booz Allen Hamilton Proprietary
18

19. COTS Example – Snap Survey Professional Edition
Booz Allen Hamilton Proprietary
19

20. COTS Example – Snap Survey Professional Edition
Booz Allen Hamilton Proprietary
20

21. Moving Beyond the Monkey
 Online survey concepts and issues
 Choosing an online survey software solution
 Examples
 Concluding points and guidance
Booz Allen Hamilton Proprietary
21

22. Evaluators should develop an online survey software strategy and
examine different solutions based on their research and evaluation
needs
 Evaluators should develop an online survey software strategy
– Identify the aspects of your evaluation practice that involve online surveys
– Understand the implications of using online surveys
– Develop a budget
– Develop user and functional requirements based on the survey research process
 Evaluators should examine the range of online survey software solutions
– Consider the benefits and drawbacks for using internally developed versus COTS solutions
– Compare different software features to your user and functional requirements
– Be open to technological innovations only if they help answer your evaluation question
Booz Allen Hamilton Proprietary
22

23. References and additional reading to learn more about research
and practice related to online surveys
 E-mail me at axelrad_stephen@bah.com for an electronic copy of this presentation or give me
your business card at the end of the session
 American Evaluation Association (n.d.). Surveys. Retrieved April 11, 2009, from
http://www.eval.org/Resources/surveylinks.asp.
 Ritter, L.A., & Sue, V.M. (2007). Using online surveys in evaluation. New Directions in
Evaluation, 115, 1-64.
 Witte, J.C. (Ed.; 2009). Special issue: Web surveys. Sociological Methods & Research, 283455.
Booz Allen Hamilton Proprietary
23

24. Feedback
 Questions
 Comments
 Implications for Your Current Work
Booz Allen Hamilton Proprietary
24