The document provides guidance on designing effective surveys. It discusses types of survey questions such as open-ended, closed-ended, Likert scale, and multiple choice questions. Specific tips are provided, such as keeping questions short and simple, avoiding double-barreled or biased questions, and pre-testing surveys. Different data collection techniques like interviews, observations, and online surveys are also overviewed, including their pros and cons. Resources for writing good survey questions and online survey tools are listed.
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Survey
1. Designing a SurveyDesigning a Survey
EDUU600EDUU600
Adapted from McMillan & SchumacherAdapted from McMillan & Schumacher
2. Make items clearMake items clear
Avoid double-barreled questions (avoidAvoid double-barreled questions (avoid
and)and)
Respondents must be competent toRespondents must be competent to
answer (and provide reliable information)answer (and provide reliable information)
Questions should be relevantQuestions should be relevant
Short, simple items are bestShort, simple items are best
Avoid negative itemsAvoid negative items
Avoid biased items or termsAvoid biased items or terms
Writing Questions andWriting Questions and
StatementsStatements
3. Types of ItemsTypes of Items
Closed Form – Structured response whereClosed Form – Structured response where
subject chooses between predeterminedsubject chooses between predetermined
responsesresponses
Open Form – subject writes in any responseOpen Form – subject writes in any response
Scaled itemsScaled items
– Gradations, levels, or valuesGradations, levels, or values
– Likert scaleLikert scale
– Semantic differential scaleSemantic differential scale
– Ranked itemsRanked items
– Checklist itemsChecklist items
4. Data Collection TechniquesData Collection Techniques
Pros and ConsPros and Cons
Paper/pencilPaper/pencil
– Economical/standardEconomical/standard
– Norms inappropriateNorms inappropriate
– Must be able to readMust be able to read
Alternative AssessmentAlternative Assessment
– Holistic/authenticHolistic/authentic
– Subjective ratingSubjective rating
– Costly/time-consumingCostly/time-consuming
Questionnaire or SurveyQuestionnaire or Survey
– Economical/easy to scoreEconomical/easy to score
– Response rate/inability to probe and clarifyResponse rate/inability to probe and clarify
– Biased/ambiguous itemsBiased/ambiguous items
5. Interview and ObservationInterview and Observation
InterviewInterview
– Flexible/include nonverbal responsesFlexible/include nonverbal responses
– Costly/time consumingCostly/time consuming
– Can be anonymousCan be anonymous
– Effect of interviewer and interviewer biasEffect of interviewer and interviewer bias
ObservationObservation
– Captures natural behaviorCaptures natural behavior
– Costly/time consumingCostly/time consuming
– Observer biasObserver bias
– Not anonymousNot anonymous
6. Designing a QuestionnaireDesigning a Questionnaire
StronglyStrongly
AgreeAgree
AgreeAgree NeutralNeutral DisagreeDisagree StronglyStrongly
DisagreeDisagree
AlwaysAlways Most of theMost of the
TimeTime
SometimesSometimes RarelyRarely NeverNever
Very happyVery happy SomewhatSomewhat
happyhappy
Neither sadNeither sad
or happyor happy
SomewhatSomewhat
sadsad
Very SadVery Sad
Use a Likert Scale, (or combination of Likert, ranking and differential)
Like Dislike
Important Unimportant
7. Smileys for KidsSmileys for Kids
Questions must be readQuestions must be read
Child can respond by circling correctChild can respond by circling correct
responseresponse
May respond verbally or by pointingMay respond verbally or by pointing
8. Open-ended QuestionsOpen-ended Questions
Questions to which there is not one definiteQuestions to which there is not one definite
answer.answer.
May be a good way to break the ice with a surveyMay be a good way to break the ice with a survey
Gives respondents an opportunity to answer inGives respondents an opportunity to answer in
their own words.their own words.
Example: "Are there any other comments aboutExample: "Are there any other comments about
the course you would like to add?"the course you would like to add?"
Responses to open-ended questions can be veryResponses to open-ended questions can be very
useful, often yielding quotable material.useful, often yielding quotable material.
Drawback is that the responses are more difficultDrawback is that the responses are more difficult
to catalogue and interpretto catalogue and interpret
From types of survey questions -
http://coe.sdsu.edu/eet/Articles/surveyquest/index.htm
9. Closed-ended QuestionsClosed-ended Questions
Have a finite set of answers from which theHave a finite set of answers from which the
respondent chooses.respondent chooses.
One of the choices may be "Other" but need toOne of the choices may be "Other" but need to
provide a write-in responseprovide a write-in response
Easy to standardizeEasy to standardize
Data gathered from closed-ended questions lendData gathered from closed-ended questions lend
themselves to statistical analysisthemselves to statistical analysis
More difficult to write than open-ended questionsMore difficult to write than open-ended questions
Design choices must include allDesign choices must include all the possiblethe possible
answers a respondent could give for eachanswers a respondent could give for each
questionquestion
From types of survey questions -
http://coe.sdsu.edu/eet/Articles/surveyquest/index.htm
10. Likert ScaleLikert Scale
When you want to know respondents'When you want to know respondents'
feelings or attitudes about somethingfeelings or attitudes about something
Respondents must indicate how closelyRespondents must indicate how closely
their feelings match the question ortheir feelings match the question or
statement on a rating scale.statement on a rating scale.
Number at one end of the scale representsNumber at one end of the scale represents
least agreement, or "Strongly Disagree”least agreement, or "Strongly Disagree”
Number at the other end of the scaleNumber at the other end of the scale
represents most agreement, or "Stronglyrepresents most agreement, or "Strongly
Agree"Agree"
From types of survey questions -
http://coe.sdsu.edu/eet/Articles/surveyquest/index.htm
11. Multiple ChoiceMultiple Choice
When you want respondents to pickWhen you want respondents to pick
the best answer or answers fromthe best answer or answers from
among all the possible options,among all the possible options,
Multiple-choice questions are easy toMultiple-choice questions are easy to
lay out on a written survey.lay out on a written survey.
Include specific directions about howInclude specific directions about how
many answers to select directly aftermany answers to select directly after
the question.the question.
From types of survey questions -
http://coe.sdsu.edu/eet/Articles/surveyquest/index.htm
12. Other Closed QuestionsOther Closed Questions
Ordinal: When you need all possibleOrdinal: When you need all possible
answers to be rank orderedanswers to be rank ordered
Categorical: When the possibleCategorical: When the possible
answers for a question are categoriesanswers for a question are categories
– Each respondent must "belong" inEach respondent must "belong" in
exactly one of themexactly one of them
Numerical: When the answer mustNumerical: When the answer must
be a real numberbe a real number
From types of survey questions -
http://coe.sdsu.edu/eet/Articles/surveyquest/index.htm
13. When to Use What Type Question
Type of question... Best Used for...
Open-ended
Breaking the ice in an interview; when respondents' own
words are important; when the surveyor doesn't
know all the possible answers.
Closed-ended
Collecting rank ordered data; when all response choices
are known; when quantitative statistical results are
desired.
Likert-scale To assess a person's feelings about something.
Multiple-choice
When there are a finite number of options (remember to
instruct respondents as to the number of answers to
select).
Ordinal To rate things in relation to other things.
Categorical
When the answers are categories, and each respondent
must fall into exactly one of them.
Numerical For real numbers, like age, number of months, etc.
http://coe.sdsu.edu/eet/Articles/surveyquest/index.htm
14. InfoPoll Survey TipsInfoPoll Survey Tips
Write a short questionnaireWrite a short questionnaire
Use simple wordsUse simple words
Relax your grammarRelax your grammar
Assure a common understandingAssure a common understanding
Start with interesting questionsStart with interesting questions
Don't write leading questionDon't write leading question
Avoid double negativesAvoid double negatives
Balance rating scalesBalance rating scales
http://www.accesscable.net/~infopoll/tips.htm
15. InfoPoll Survey TipsInfoPoll Survey Tips
Don't make the list of choices too longDon't make the list of choices too long
Avoid difficult conceptsAvoid difficult concepts
Avoid difficult recall questionsAvoid difficult recall questions
Use Closed-ended questions rather thanUse Closed-ended questions rather than
Open-ended onesOpen-ended ones
Put your questions in a logical orderPut your questions in a logical order
Pre-test your surveyPre-test your survey
Name your survey to pique interestName your survey to pique interest
Include cover memo or introductionInclude cover memo or introduction
http://www.accesscable.net/~infopoll/tips.htm