1. Topic 3
POLS2110
Jan 28 & Feb 4
Questionnaire design

2. Questionnaire design: Procedure
POLS2110-T3
1. Types of questions
2. Question wording
3. Formatting – Options
4. As a whole: Cover letter, layout
2

3. Types
POLS2110-T3
1. Personal:
 Sex, age-group,…
2. Attitudinal:
 attitudes or opinions
 “Agree” / “Support”
3. Behavioral:
 E.g., Did you cast a vote during the Legislative
Council 2008?
4. Closed- and open-ended questions
 Reasons given vs. you give reasons
 Will you offer “Others (reasons)”?
3

4. Types
POLS2110-T3
5. Sensitive
 Personal, social, cultural and political orientations:
sexual orientation, drug & alcohol use, mental
health issues, illegal activities, controversial public
issues, political belief, identities.
 E.g.,
 Do you agree that the June 4th Incident is a conspiracy?
 Do you take drug at school?
4

5. Constructing questionnaire
POLS2110-T3
Guides for asking questions
 The surveyor should ask questions only from
those likely to be able to answer them accurately
 Questions should be relevant: don't ask questions
on topics that respondents don't care about, haven't
thought about, or that are irrelevant to the research
problem
 Respondent must be competent to answer: don't
ask questions that the respondent won't accurately
be able to answer
5

6. Question wording
POLS2110-T3
1. Specific questions/statements for specific
answers
 Avoid double-barreled questions (make sure the
question asks only one clear thing)
 Each item should consist of one concept/term
 E.g., Do I care about my students and their
families?
 How many concepts/terms?
 How would you revive the question?
6

7. Question wording
POLS2110-T3
2. Simple language:
 Clear and understandable
 avoid slang, jargon, and technical terms
 Short items are best (so that they may be read,
understood, and answered quickly)
 E.g.,
Do you agree that “the Legislative Council’s split
voting system should remain in passing a bill”?
你是否同意立法會在通過議案時繼續使用分組點票制度？
 (If you don‟t study HK politics), which term you find it
difficult to understand?
 How to change the question so as to make it
understandable
7

8. Question wording
POLS2110-T3
3. Avoid ambiguous questions
 A word having at least two possible interpretations
 E.g.,
Do you agree that “the public should supervise
the council”?
你是否同意「在香港，議會應受到公眾的監督」？
 Which term seems to be ambiguous
 How to make changes?
8

9. Question wording
POLS2110-T3
4. Avoid vague words
 Make items clear: don't assume the person you
are questioning knows the terms you are using
 When unique and unusual terms need to be
defined in questionnaire items, use very clear
definitions
9

10. Question wording
POLS2110-T3
5. Avoid leading questions
 Avoid biased items and terms: be sensitive to the
effect of your wording on respondents.
 E.g.,
Do you agree that “Rational voters should not cast a vote
for those candidates who resist China and disturb Hong
Kong”?
你是否同意「理性的選民不應該在立法會選舉裡投反中亂
港的候選人一票？」
 What do you feel?
 Any possible implications?
 How to make changes?
10

11. Question wording
POLS2110-T3
 Avoid negative or double negative items
 E.g.,
- if you ask “whether librarians should not be paid more”,
it will confuse respondents;
- I am confident in teaching vs. I am not confident in
teaching;
- I like to teach well-behaved students vs. I do not like to
teach badly-behaved student
11

12. Question wording
POLS2110-T3
6. Avoid presuming questions
 E.g., Do you agree with the view that “as June 4
Turmoil is a conspiracy organized by the Western
countries, the running of the candlelight vigil
annually undermines the harmonious relationship
with mainland China”?
12

13. Question wording
POLS2110-T3
7. Try to avoid hypothetical questions
 Except such hypothetical questions can tape the
respondents‟ views from different perspectives
 Use and interpret cautiously
 E.g., If Hong Kong is no longer suitable for you,
would you seek means to leave HK?
1. Yes; 2. Would like to but can‟t; 3. NO; 4.
DK/Refuse to answer (HKTP, 1999)
13

14. Question wording
POLS2110-T3
8. Decide carefully whether a question should be
asked in a personalized form
 Use “I” as the subject
 “I support Henry Tang to be the next Chief
Executive” VS
“Henry Tang should be the next Chief Executive”
 Any differences?
 Which one would you prefer?
14

15. Question wording
POLS2110-T3
9. Avoid questions involving memory
 Factual accuracy is meaningless when knowing public
opinion
 (However,) In some cases, factual accuracy aims to
interpret the social/political meaning behind the
chosen option
 E.g.,
- For your understanding, July 1st is ___.
A. a rally B. A public holiday C. a day that HK
returned to China
- Can you name any three of the Legislative Council
members who are Civic Party‟s members?
[Do you think this question is meaningful to evaluate
the level of understanding local politics from
respondents?]
15

16. Question wording
POLS2110-T3
10. Deal with embarrassing, controversial and
sensitive questions carefully
 E.g., Sexual, political orientations; marital status;
age
 Possible solutions:
 Skip such questions?
 Put such questions at the end of the questionnaire?
16

17. Question wording
POLS2110-T3
Ask the embarrassing questions in non-
embarrassing ways. E.g.,
“Did you kill your wife?”
1. The Everybody approach:
 “As you know, many people have been killing their wives these
days. Do you happen to have killed yours?”
2. The numbered card: (face-to-face survey only)
 Would you please read off the number on this card which
corresponds to what became of your wife?” (HAND CARD TO
RESPONDENT)
1. Natural Death 2. I killed her 3. Other (What?)
(GET CARD BACK FROM RESPONDENT BEFORE
PROCEEDING!)
17

18. Asking questions which require
sensitive answers
POLS2110-T3
Respondents may distort their answers
 If the question poses an actual threat
 If they want to make themselves look good
 If they are maintaining their own self-image
18

19. Formatting
POLS2110-T3
1. Multi-Point Rating
 e.g. Strongly Agree to Strongly Disagree
 Choose from 2-point rating scale up to 10-point
rating scale
 Specify rating labels
 Include/Exclude "Unable to Rate" option?
 Low to high or high to low rating label direction
19

20. POLS2110-T320

21. POLS2110-T321

22. POLS2110-T322

23. Rating
POLS2110-T3
1. True scale: e.g.,
 completely true, mostly true, somewhat true, slightly true,
not at all true
 Completely true, generally true, generally untrue,
completely untrue
2. Always to never: e.g.,
 Always, usually, sometimes, rarely, never
3. Very good  very poor
4. Important  not important at all/unimportant
5. Likely  unlikely
6. Smiley faces [^_^], [o_o], [-_-], [>_<]
23

24. 2. Likert-scale/Semantic Differential
POLS2110-T3
 5 rating points to 10 rating points
 Specific rating anchors
 Include/Exclude "Unable to Rate"
24

25. POLS2110-T325

26. POLS2110-T326

27. POLS2110-T327

28. 3. Multiple choice
POLS2110-T3
 Select one or select multiple
 Limit number of selections ("Select up to X choices")
 Randomize choices
28

29. POLS2110-T329

30. 4. Ranking
POLS2110-T3
 Define the meaning of numbers (e.g., 1 is the most
favorite/important whereas 5 is the least)
 Rank each of the given items without overlapping
30

31. 5. open-ended questions
POLS2110-T3
 Comments, opinions, feelings, reasons, etc.
31

32. POLS2110-T332

33. 6. Skip the irrelevant question
POLS2110-T3
1. Do you watch TV news for local news?
Yes ___ (Go to Q.2) No ___ (Go to Q.3)
2. If yes, which channel do you watch for local news?
A. TVB Jade
B. TVB Pearl
C. ATV Local
D. ATV World
E. Others: ____________________
3. Do you watch for local news through other platforms?
A. Internet
B. Newspapers
C. Magazines
D. Wisenews
33

34. Recap: Common problems
POLS2110-T3
1. Unclear question because of vague words
2. Two or more questions
3. Wordy or lengthy questions
4. Question contains negatives
5. Question contains jargon
6. Response categories overlap
7. Unbalanced response options
8. Mismatch between the question & the responses
9. Respondent does not have understanding to answer
question
10. Not all respondents can answer the question –
branching needed
34

35. Sample
POLS2110-T335

36. Sample
POLS2110-T336

37. Sample
POLS2110-T337

38. Cover letter
POLS2110-T3
 What is the study is about, why it is important,
how the study results will be used
 Why the respondent is important to the study
 How the respondent was selected
 Promise of confidentiality
 a photo number to call if the respondent has
questions & who will be reached at that number
 Also, want good quality headed notepaper with
survey organization (and sponsor included) and
signature by responsible researcher
38

39. Recommendations
POLS2110-T3
 Keep it short. Think of 10p per word. How much
$ can you save?
 Make sure purpose is unbiased:
 Poor e.g.,
“It is important to keep big business from harming
the environment, so we are doing this survey to get
your honest opinions on the extent to which you feel
the environment should be protected”
 Better e.g.,
“It is unclear whether people want either more or
less to be done by state government to protect the
environment than is now being done…”
39

40. Cover Letter Sample
POLS2110-T340

41. Questionnaire as a Whole
POLS2110-T3
1. clear, specific instructions on the questionnaire
on how to mark answers
For postal,
 typically have „tick‟ boxes 
 need overall instructions: “Please tick the box that
best describes you or write in the information
requested.”
 Need prominent “question by question” instructions
(e.g., „Tick one box only”, “write number in box”)
 Both sets of instructions also needed to web
surveys
41

42. E.g,
POLS2110-T3
Q.1 How long have you/your household been a tenant
of this landlord?
Tick one box only 
Under 1 year 
1-5 years 
6-10 years 
11-15 years 
16+ years 
Don‟t know/can‟t remember 
Q.2 How many people usually live here in total?

Write number in box
42

43. Questionnaire as a Whole
POLS2110-T3
2. First questions should
 Should apply to everyone
 Be easy to answer
 Be interesting
 Be connected to what R understands as topic of
survey
 Be closed
 Not be sensitive!
 So where should you put your demographic
variables?
43

44. Questionnaire as a Whole
POLS2110-T3
3. Meaningful order
 It is useful for the respondent to sense the flow, or
natural progression, of the instrument
4. Definition (be sure these are part of the
question)
 In postal surveys, R rarely reads instructions
booklets
 In web surveys where underlined words provide
definitions, Rs rarely use
 Remember to put definitions first in interview
surveys
44

45. Questionnaire as a Whole
POLS2110-T3
5. Routing
 If paper self-completion, avoid complex routing 
confuse or be ignored
 Usually best to have everyone answer every
question (i.e. no routing)
 If routing, use arrows as well as prominent explicit
text
45

46. E.g.,
POLS2110-T3
Q.1 Normally, do you work out everyday?
 No (Skip to 4)
 Yes
Q.2 (If yes) How many minutes per day to you
work out?
_____ minutes per day
…
Q.4
46

47. Questionnaire as a Whole
POLS2110-T3
6. Open-ended questions
 In self-completion, be sure to provide the appropriate
amount of write-in-space (too much or too little can lead
to item nonresponse)
7. Sections
 In interview surveys, have the interview read out a
sentence (e.g., “The next few questions are about…”)
 In self-completion, use short one or two world labels that
are eye catching
8. Question numbers
 In self-completion, start numbering with 1 and number
consecutively. Avoid sub-parts even though you think
this will make the questionnaire look shorter.
47

48. Questionnaire as a Whole
POLS2110-T3
9. Do not cram lots of questions together with small
print to try and save space
 Better to be generous with space to make the task look
easy and to give R a sense of progress in filling in the
questionnaire.
10. Place sensitive questions toward the end (but not
the very last questions!)
11. “Thank you”
 Make sure interviewers thank the respondent
 In self-completion, write a large “thank you” at the
end.
 Provide written instructions (at the end) about how to
return the questionnaire
 If posted, supply a return envelope
48