The document outlines the objectives and content of a survey design workshop. It discusses key topics like questionnaire design, levels of measurement, sampling, and implementation issues. The workshop aims to help participants understand rigorous survey planning, common survey methods, questionnaire design best practices, and critically reviewing example surveys.

1. Survey Design Workshop Inter-University Research Workshop Program Dr. James Neill Centre for Applied Psychology University of Canberra 1 February, 2011

3. Introductions

4. Logins & Resources

5. Research methods

6. Questionnaire design

7. Levels of measurement

8. Sampling

9. Evaluation

10. Objective 1 Understand the importance of a rigorous , step-by-step process in planning, developing & implementing research questionnaires

12. Telephone survey

13. Mail survey

14. Internet/mobile survey

16. Response formats,

17. Layout, and

18. Pre- and pilot testing

20. Sample size

22. Student in-progress examples with a view towards planning, drafting, and/or revising of an initial draft (pilot) survey.

24. Readings

25. Books about surveys design and survey research (check library)

26. Introductions Introductions

28. Quasi-experimental

29. Non-experimental

31. Control over extraneous variables

33. Control over some extraneous variables

34. Groups are “naturally occuring”

36. Minimal control over extraneous variables

38. Often use interviews or questionnaires .

39. Involve real-world samples .

40. Often quantitative , but can be qualitative .

42. Since the 1980's, theories and principles evolved to create a unified perspective on the design, conduct, and evaluation of surveys.

44. Impartial : selects units of the population without prejudice or preference

45. Representative : includes units that together are representative of the problem under study and the population affected by it

47. Quantitative : assigns numerical values to nonnumerical character

48. Self-monitoring : procedures can be designed in ways that reveal any unplanned and unwanted distortions (biases) that may occur 8 Survey Research Characteristics Backstrom & Hursh-C é sar (1981, pp. 3-4)

50. Replicable : other people using the same methods in the same ways can get essentially the same results 8 Survey Research Characteristics Backstrom & Hursh-C é sar (1981, pp. 3-4)

52. Access to wide range of participants

53. Potentially large amounts of data

54. May be more ethical (than experiments)

56. Data may be 'superficial'

57. Can be costly to obtain representative data

58. Self-report data only

59. Potentially low compliance rates

60. Example Surveys Unit Satisfaction Survey Community Library Use

61. The research process 1. Establish need for info/ research 2. Problem definition/ Hypotheses 3. Research design 4. Sampling/ Data collection 5. Data analysis 6. Reporting

62. The research process

64. Types of questionnaires

65. Questionnaire development

66. Writing questions

67. Types of questions

68. Response formats  LOM

69. Survey formatting

71. A measuring instrument

72. A way of converting fuzzy psychological stuff into hard data for analysis What is a survey?

73. Types of surveys Types of surveys Self - administered Interview - administered Postal Delivered and collected Telephone Face to face structured interview Web-based

74. Questionnaire development 1. Formulate generic questionnaire 2. Expand the questionnaire Turn into separate sections based on study objectives. Draft qs & response formats 4. Finalise questionnaire & implement Question order & funnel qs 3. Pre-test, pilot test, & redraft

76. Ensure all questions relate to research objectives

77. For explanatory objectives or hypotheses ensure both dependent and independent variables exist

79. Few will read it without good prompting and being easy-to-read

81. Explain how to do the survey in a user-friendly manner

82. Example: Life Effectiveness Questionnaire

83. Instructions: Example

85. Purpose of survey?

86. What's involved?

87. Explain any risks/costs/rewards

88. How will results be used?

89. Human ethics approval #

90. How is consent given / not given?

91. Voluntary - can choose not to continue anytime

92. More info: Complaints, how to obtain results, contact details etc.

94. No. of pages

95. Margins

96. Double vs. single-siding

97. Colour, etc. Layout

99. only include relevant questions Layout

101. Indicate the end

102. Say thanks! Layout

104. Ask screening questions first, rather than later. Does the participant qualify for the survey? (esp. for internet surveys)

105. Use funneling/branching questions to move respondents through survey

106. Start off with easy to answer and engaging questions

107. More controversial questions in middle section

108. Expanding the Survey

110. Consider response format e.g, Income in categories or ranges

112. Show cards for responses may help for face to face interviews

115. Are redundant

116. Are misunderstood

118. Pilot test on a small sample from the target population, analyse, & revise

119. Types of Questions Be able to justify and defend your choices ...

121. Check related research & questionnaires

122. Draft items (for important, fuzzy constructs aim to have multiple indicators)

123. Pre-test & revise

125. Be clear

126. Be brief

127. Avoid big words

128. Use simple and correct grammar

129. Writing questions – Don'ts Inapplicable – must apply to all respondents Over-demanding – e.g., recall of detail or time-consuming, unnecessary questioning Ambiguous – meaning must be clear to all respondents Double negative – e.g., Do you not disapprove of tax reforms?

130. Writing questions – Don'ts Double-barrelled - e.g., “Do you think speed limits should be lowered for cars & trucks?” Leading - e.g., “don’t you see some danger in the new policy?” Loaded – e.g., “Do you advocate a lower speed limit to save human lives?” vs “Does traffic safety require a lower speed limit?”

133. Order effects

136. Information may be less rich

137. Types of Questions

138. Accuracy of recall decreases over time

140. The question could be accurately answered by an observer. e.g., How times during 2009 did you visit a G.P.? ______

142. There is no “true”, factual answer.

143. Many possible answers per unit.

144. Can't be accurately answered by an observer. e.g., Think about the visits you made to a G.P. during 2010. How well did you understand the medical advice you received? perfectly very well reasonably poorly not at all

146. One criticism of this distinction: There is no such thing as “objective” and that all responses are subjective.

148. Useful for descriptive, exploratory work

149. Difficult and subjective to analyse

150. Time consuming

152. Useful for descriptive, exploratory work

153. Difficult and subjective to analyse

154. Time consuming

155. Open-ended questions: Examples What are the main issues you are currently facing in your life? How many hours did you spend doing university study last week? _________

157. Useful for hypothesis testing

158. Easy and objective to analyse

159. Time efficient

160. Levels of Measurement = Type of Data Stevens (1946)

162. O rdinal

163. I nterval

164. R atio

165. Discrete vs. continuous Discrete - - - - - - - - - - Continuous ___________

166. Each level has the properties of the preceding levels, plus something more!

169. Categorical / nomimal example: Phrenological labels

171. Ordinal / ranked example: Ranked importance Rank the following aspects of the university according to what is most important to you (1 = most important through to 5 = least important)  Quality of the teaching and education  Quality of the social life  Quality of the campus  Quality of the administration  Quality of the university's reputation

174. Interval example: 8 point Likert scale

177. Ratio scale: Time

178. Why do levels of measurement matter? Different analytical procedures are used for different levels of data. More powerful statistics can be applied to higher levels

179. Levels of measurement: Revision question Fill in all cells Descriptive Statistics Ratio Interval Ordinal / Rank Nominal / Categorical Graphs Examples Prop-erties Level

181. Multichotomous

182. Verbal frquency scale

183. The list (multiple response)

184. Ranking

185. Likert scale

186. Graphical rating scale

187. Semantic differential

188. Non-verbal (idiographic)

189. Dichotomous 2 response options e.g., Excluding this trip, have you visited Canberra in the previous five years? __ Yes __ No Provides the simplest type of quantification

190. Multichotomous How many hours did you spend doing university study last week? __ less than 5 hours __ > 5 to 10 hours __ > 10 to 20 hours __ more than 20 hours

191. Multichotomous More than two possible answers e.g., What type of attractions in your current trip to Canberra most appeal to you? __ historic buildings __ museum/art galleries __ parks and gardens

192. Verbal frequency scale Over the past month, how often have you argued with your intimate partner? 1. All the time 2. Fairly often 3. Occasionally 4. Never 5. Doesn’t apply to me at the moment

193. The list (multiple response) Provides a list of answers for respondents to choose from e.g., Tick any words or phrases that describe your perception of Canberra as a travel destination: __ Exciting __ Important __ Boring __ Enjoyable __ Interesting __ Historical

194. The list (multiple-response) What are the main issues that you are currently facing in your life? (tick all that apply) __ financial __ physical / health __ academic __ employment / unemployment __ relationships __ other (please specify)

195. Ranking Helps to measure the relative importance of several items Rank the importance of these reasons for visiting Canberra (from 1 (most) to 4 (least)): __ to visit friends and relatives __ for business __ for educational purposes __ for holiday/ sightseeing

196. Likert scale Measures strength of feeling or perception. Indicate your degree of agreement with this statement: “ I am an adventurous person.” (circle the best response for you) 1 2 3 4 5 strongly disagree neutral agree strongly disagree agree 1 2 3 4 5 strongly agree neutral disagree strongly agree disagree

197. Graphical rating scale How would you rate your enjoyment of the movie you just saw? Mark with a cross (X) not enjoyable very enjoyable

198. What is your view of smoking? Tick to show your opinion. Bad ___:___:___:___:___:___:___ Good Strong ___:___:___:___:___:___:___ Weak Masculine ___:___:___:___:___:___:___ Feminine Unattractive ___:___:___:___:___:___:___ Attractive Passive ___:___:___:___:___:___:___ Active Semantic differential

199. Non-verbal scale Point to the face that shows how you feel about what happened to the toy. Also called an idiographic scale .

200. Verbal frequency scale Over the past month, how often have you argued with your intimate partner? 1. All the time 2. Fairly often 3. Occasionally 4. Never 5. Doesn’t apply to me at the moment

202. Watch out for too few or too many options.

204. Average = 3 to 9

205. Maximum = 10? Basic guide: 7 +/- 2 Number of response options?

206. AGREEMENT ABOUT SOMETHING 2-Categories DISAGREE AGREE 3-Categories DISAGREE NEUTRAL AGREE 4-Categories STRONGLY MILDLY MILDLY STRONGLY DISAGREE DISAGREE AGREE AGREE 5-Categories STRONGLY MILDLY MILDLY STRONGLY DISAGREE DISAGREE NEUTRAL AGREE AGREE Number of response options? Likert scale example

207. Watch out for too many or too few response options “ Capital punishment should be reintroduced for serious crimes ” 1 = Agree 2 = Disagree 1 = Very, Very Strongly Agree 7 = Slightly Disagree 2 = Very Strongly Agree 8 = Disagree 3 = Strongly Agree 9 = Strongly Disagree 4 = Agree 10 = V. Strongly Disagree 5 = Slightly Agree 11 = V, V Strongly Disagree 6 = Neutral

208. Example: How could this question be improved? How old are you? ___ 18-20 ___ 20-22 ___ 22-30 ___ 30 and over

209. Are you satisfied with your marriage and your job? __________________________ Example: How could this question be improved?

210. You didn’t think the food was very good, did you? _____ Yes _____ No Example: How could this question be improved?

211. Environmental issues have become increasingly important in choosing hotels. Are environmental considerations an important factor when deciding on your choice of hotel accommodation? ____ Yes ____ No Example: How could this question be improved?

212. What information sources did you use to locate your restaurant for today’s meal? (please tick appropriate spaces) ____ yellow pages ____ Internet ____ word of mouth Example: How could this question be improved?

213. Comparison of data collection methods Alreck & Settle (1995; 32) Alreck and Settle (1995:32)

215. Ensure categories do not overlap

217. Only ask questions that relate to objectives

218. Tricks? Font size/double sided photocopying/numbering sections

220. Respondent’s level of interest

221. Colour of paper

222. Accompanying letter / introduction

223. Mail surveys - self-addressed stamped return envelope

224. Rewards

225. Reminders or follow up calls

227. Pilot test – try out on a small sample from the target population & revise

228. Be assertive and interactive about seeking feedback – ask questions & observe

229. “The customer is always right.”

232. redundancy

233. skewed response items

234. misinterpreted items

236. LUNCH BREAK

238. are particularly good

239. you would like to ask about

241. Examine structure, design issues and question styles

242. Note cover page and details provided

243. Sampling

245. What is sampling?

246. Why sample?

247. Sampling methods

248. Example: Shere Hite’s survey

250. What is sampling? “Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen.” - Trochim (2006)

252. If the sample is representative, the sample data allows inferences to be drawn about the total population.

254. Sampling method

255. Adequacy of sample size

256. Response rate – both the % & representativeness of people in sample who actually complete survey It is better to have a small, representative sample than a large, unrepresentative sample.

258. Systematic

260. Convenience

261. Snowball

263. Selection occurs entirely by random chance

264. Also called representative sampling

266. Useful if clear study area or population is identified

268. Randomly select # through table of random #s or by computer

270. Respondents (units) selected from a list/file.

271. Useful when survey population is similar e.g. List of students

272. Select sample at regular intervals from the population e.g., every 5 th person on a list, starting at a random number between 1 and 5

274. Then random selection from within each stratum

275. Improves representativeness

276. e.g., Telephone interviews using post-code strata

278. Useful for exploratory research and case study research

279. Able to get large sample size quickly

280. Limitations include potential bias and non-representativeness

282. Due to time/financial constraints

283. e.g. surveying all those at a tourist attraction over one weekend

285. e.g., Frequent flyers to contact regarding service quality in an airline setting

287. Respondents recommend other respondents

288. e.g., in studying ecstasy users, gain trust of a few potential respondents and ask them to recommend the researcher to other potential respondents

290. Select sampling method

291. Calculate sample size for desired power.

292. Maximise return rate

294. Selection sampling method

295. Calculate required sample size

296. Maximise return rate

297. Sampling Example: Shere Hite ‘American Sexology’

299. Media furors & worldwide attention

300. 127-item questionnaire about marriage & relations between sexes

301. Sample: 4500 USA women, 14 to 85 years

302. Conclusion: Society and men need to change to improve lives of women

306. 98% wanted “basic changes” to love relationships

307. 84% were emotionally unsatisfied

308. 95% reported emotional & psychological harassment from their men

309. Some of the critical comments.... She goes in with prejudice & comes out with a statistic. The survey often seems merely to provide an occasion for the author’s own male-bashing diatribes. Hite uses statistics to bolster her opinion that American women are justifiably fed up with American men.

311. 4,500 replied (4.5% return rate)

312. “ We get pretty nervous if respondents in our survey go under 70%. Respondents to surveys differ from nonrespondents in one important way: they go to the trouble of filling out what in this case was a very long, complicated, and personal questionnaire.” - Regina Herzog, University of Michigan Institute for Social Research Hite's response rate & selection bias

314. Objectivity – watch out for manipulating the survey questions and results interpretation to suit your personal conjectures Lessons from Hite's male-female relations survey

315. Measurement Error

317. Sources of measurement error Non-sampling (e.g., unreliable or invalid tests) Sampling (e.g., non-rep. sample) Personal bias (e.g., researcher favours a hypothesis) Paradigm (e.g., Western focus on individualism)

319. Sensitive to target constructs

320. Clear wording on questions/instructions

322. Use standard protocol To minimise measurement error

324. Minimise length / time / hassle

325. Offer rewards / incentives

326. Coloured paper

327. Call backs / reminders

329. Check data To minimise measurement error

331. Selection sampling method

332. Calculate required sample size

333. Maximise return rate

335. How might you get hold of a sample frame?

336. What sampling technique would you use?

338. Gives the estimated range of values into which we expect other samples to fall say 95% of the time

339. Social sciences = at least 90% or above, preferably 95%

345. Identify the number of surveys to be done

346. Once you have completed that number, do not do any additional surveys HOWEVER….

349. Identify target population and confidence level

350. Calculate margin of error

353. A survey is a standardised stimulus designed to convert fuzzy psychological phenomenon into hard data.

355. Sampling - probability & non-prob.

356. Levels of measurement & parametric / non-parametric stats

357. Ethical considerations

358. Sources of measurement error

362. Free and open source software.

363. http://www.openoffice.org/product/impress.html