Many of us receive multiple requests to complete surveys every day. Some of us find that colleagues or clients think of ‘doing a survey’ as the same as ‘doing some research’ – which may explain why organizations send out so many survey requests.
In this webinar, you’ll meet the Survey Octopus, Caroline Jarrett’s friendly way of talking about the many issues that make surveys one of the most challenging research methods.
The Survey Octopus will help you to:
Explain to colleagues that a survey may not be the first research method to try
Help to justify a choice to work with a “non significant” number of responses
Think about the steps that go into delivering a survey that works
As a bonus, Caroline will also explain how her Survey Octopus maps into the Total Survey Error concept that underpins the work of many survey methodologists.
Giulio Michelon, Founder di @Belka – “Oltre le Stime: Sviluppare una Mentalit...
Surveys that work: a webinar for UXPA 2021
1. Surveys that work
An introduction to
the Survey Octopus
and Total Survey Error
Caroline Jarrett
@cjforms
#UXPA2021
2. What would you do for a dollar?
Caroline Jarrett @cjforms (CC) BY SA-4.0 2
3. Caroline Jarrett @cjforms (CC) BY SA-4.0 3
$1 in the envelope beats $10 guaranteed later
Caroline Jarrett @cjforms (CC) BY SA-4.0 3
4. People will only respond if they trust
you. After that, it's a balance between
the perceived reward from filling in the
survey compared to the perceived
effort that's required. Strangely
enough, if a reward seems 'too good to
be true' that can also reduce the
response.
Response
Diagram inspired by Dillman, D.A. (2000)
“Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 4
6. Caroline Jarrett @cjforms (CC) BY SA-4.0 6
I’m the forms specialist
Image credit: Flickr, taxrebate.org.uk 6
7. Caroline Jarrett @cjforms (CC) BY SA-4.0 7
Why do people answer questions?
Caroline Jarrett @cjforms (CC) BY SA-4.0 7
8. People ask me about surveys
“Please have a look at this survey”
“Tell me whether this is a good question”
“How many people do I need in my sample?”
“Will that be statistically significant?”
Caroline Jarrett @cjforms (CC) BY SA-4.0 8
9. I wrote a book
It seemed easier than
continuing to answer
lots of survey questions
https://rosenfeldmedia.com/books/surveys-that-work/
Caroline Jarrett @cjforms (CC) BY SA-4.0 9
10. Let’s think about these topics today
“Please have a look at this survey”
“Tell me whether this is a good question”
“How many people do I need in my sample?”
“Will that be statistically significant?”
Caroline Jarrett @cjforms (CC) BY SA-4.0 10
11. The survey is a
systematic method
for gathering information from
(a sample of) entities
for the purpose of
constructing quantitative descriptors
of the attributes of the larger population
of which the entities are members.
Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor &
Tourangeau, Roger (2004).Survey methodology. Hoboken, NJ: John Wiley & Sons.
Caroline Jarrett @cjforms (CC) BY SA-4.0
12. I change the definition a bit
systematic method becomes process
gathering information becomes ask questions
entities become people
quantitative descriptors become numbers
attributes of the
larger population become make decisions
Caroline Jarrett @cjforms (CC) BY SA-4.0
13. The survey is a process
for getting answers to questions
from (a sample of) people
for the purpose of
getting numbers
that you can use to
make decisions
Caroline Jarrett @cjforms (CC) BY SA-4.0
14. Caroline Jarrett @cjforms (CC) BY SA-4.0 14
Let’s rearrange that somewhat
The survey is a
process for getting
answers to questions
To make decisions From people
getting numbers
15. Caroline Jarrett @cjforms (CC) BY SA-4.0 15
Start with why and who; end with the number
The survey is a
process for getting
answers to questions
Why you want to ask Who you want to ask
The number
20. Caroline Jarrett @cjforms (CC) BY SA-4.0 20
Establish your goals for the survey
What do you want to know?
Why do you want to know?
What decision will you make
based on these answers?
What number do you
need for the decision?
21. Key Point 2 The aim of a survey is
to get a number that will
help with a decision
Caroline Jarrett @cjforms (CC) BY SA-4.0
22. Caroline Jarrett @cjforms (CC) BY SA-4.0 22
Let’s have a good look at that process
The survey is a
process for getting
answers to questions
Why you want to ask Who you want to ask
The number
23. Caroline Jarrett @cjforms (CC) BY SA-4.0 23
The Survey Octopus has things to think about
Why you want to ask Who you want to ask
The number
25. Caroline Jarrett @cjforms (CC) BY SA-4.0 25
I made a process from the things to think about
Goals
Questions
Questionnaire
Response
Sample
Fieldwork
Response
Reports
25
26. Here are the 7 steps as a linear process
Establish
your goals for
the survey
Decide who
to ask and
how many
Build the
questionnaire
Run the
survey from
invitation to
follow-up
Clean and
analyse the
data
Present the
results
Questions
you need
answers to
People you
will invite to
answer
Goals Sample Questionnaire Fieldwork
People who
actually
answer
Responses Reports
Answers Decisions
Test the
questions
Questions
Questions
people can
answer
Questions
people can
interact with
Caroline Jarrett @cjforms (CC) BY SA-4.0
27. People ask me about surveys
“Please have a look at this survey”
“Tell me whether this is a good question”
“How many people do I need in my sample?”
“Will that be statistically significant?”
Caroline Jarrett @cjforms (CC) BY SA-4.0 27
28. Caroline Jarrett @cjforms (CC) BY SA-4.0 28
Is this a good question?
Would you recommend us to
a friend or family member?
28
29. Caroline Jarrett @cjforms (CC) BY SA-4.0 29
“Recommend to friend/family” can be OK
A shop selling clothes
What do you want
to know?
Whether they will recommend
Why do you want to
know?
To compare numbers
over time
What number do
you need to make a
decision?
Some good reasons
(investigate and fix problems)
Some bad ones (punish staff)
29
30. Caroline Jarrett @cjforms (CC) BY SA-4.0 30
“Recommend to friend/family” can be very weird
A shop selling clothes A hospital ward
What do you want
to know?
Whether they will recommend ????
Why do you want to
know?
To compare numbers
over time
Because someone
said they had to ask
What number do
you need to make a
decision?
Some good reasons
(investigate and fix problems)
Some bad ones (punish staff)
????
What decision?
33. Key Point 3 To find out whether a
question is a good one,
test it with
people who will answer it
Caroline Jarrett @cjforms (CC) BY SA-4.0
34. People ask me about surveys
“Please have a look at this survey”
“Tell me whether this is a good question”
“How many people do I need in my sample?”
“Will that be statistically significant?”
Caroline Jarrett @cjforms (CC) BY SA-4.0 34
36. Caroline Jarrett @cjforms (CC) BY SA-4.0 36
Start with the number of answers you need
Fieldwork:
Who answers?
36
37. Caroline Jarrett @cjforms (CC) BY SA-4.0 37
Start with the number of answers you need
10,000
Fieldwork:
Who answers?
37
38. Caroline Jarrett @cjforms (CC) BY SA-4.0 38
Whether they’ll answer depends on effort
Questions:
What are you asking about?
How many questions?
38
39. Caroline Jarrett @cjforms (CC) BY SA-4.0 39
Whether they’ll answer depends on effort
20
Questions:
What are you asking about?
How many questions?
39
40. Caroline Jarrett @cjforms (CC) BY SA-4.0 40
And on the reward you’re offering
Goals:
Why are you asking?
Is helping you a reward in itself?
Are you offering any other incentive?
40
41. Caroline Jarrett @cjforms (CC) BY SA-4.0 41
And on the reward you’re offering
Goals:
Why are you asking?
Is helping you a reward in itself?
Are you offering any other incentive?
Make
improvements
41
42. Caroline Jarrett @cjforms (CC) BY SA-4.0 42
How do our users feel about this?
Sample:
the list you
sample from
42
43. Caroline Jarrett @cjforms (CC) BY SA-4.0 43
How do our users feel about this?
Sample:
the list you
sample from
Ask
anyone?
43
44. Caroline Jarrett @cjforms (CC) BY SA-4.0 44
How are you delivering the questionnaire?
Fieldwork:
How are you
asking?
44
45. Caroline Jarrett @cjforms (CC) BY SA-4.0 45
How are you delivering the questionnaire?
As a
banner on
the website
Fieldwork:
How are you
asking?
45
46. Response rates vary by method of asking
Method of asking Possible
response rate
Possible replies for
1000 invitations
Mailed paper survey Up to 70% 700
Email invitation to selected
group, specific one-off survey
Up to 20% 200
Routine email after every
transaction
Maybe 5% 50
Banner invitation on a website Less than 0.1% 1
Caroline Jarrett @cjforms (CC) BY SA-4.0 46
47. There are many influences on response rate
It depends on
• How you’re asking
• Why you’re asking
• What you’re asking and
the type of questionnaire you make
• Who you’re asking
Fieldwork
Goals
Questions
Questionnaire
Sample
Caroline Jarrett @cjforms (CC) BY SA-4.0
48. Caroline Jarrett @cjforms (CC) BY SA-4.0 48
And now it’s easy to work out how many to ask
Sample:
the number of
people to ask
48
49. Work through from response rate
Topic Answer Effect on response rate
How you’re asking Website banner Maybe 1 in 1000 (0.1%)
Why you’re asking Make improvements OK, no change
What you’re asking 20 questions Quite long, reduces rate
The type of
questionnaire
Pops up from banner OK, no change
Who you’re asking Any website visitor Not personal, reduces rate
Possible eventual rate Maybe 1 in 2500 (0.04%)
Caroline Jarrett @cjforms (CC) BY SA-4.0
50. Caroline Jarrett @cjforms (CC) BY SA-4.0 50
But wait, there’s more!
Response:
Dealing with the
answers
50
51. Caroline Jarrett @cjforms (CC) BY SA-4.0 51
Who will process those 200,000 data items?
Response:
Dealing with the
answers
10,000
20
51
52. Caroline Jarrett @cjforms (CC) BY SA-4.0 52
And do you really need all of them?
Report:
Creating the decision
52
53. Key Point 5 A sample size calculation
needs
the entire Survey Octopus
Caroline Jarrett @cjforms (CC) BY SA-4.0
54. People ask me about surveys
“Please have a look at this survey”
“Tell me whether this is a good question”
“How many people do I need in my sample?”
“Will that be statistically significant?”
Caroline Jarrett @cjforms (CC) BY SA-4.0 54
55. What type of significance do you need?
• A result that is statistically significant is one that is
mathematically unlikely to be the result of chance
• A result that is significant in practice is one that is
meaningful in the real world
Caroline Jarrett @cjforms (CC) BY SA-4.0 55
56. Caroline Jarrett @cjforms (CC) BY SA-4.0 56
If you ask the wrong questions, you’ll fail at validity
(Lack of)
validity
Sampling
error
56
57. Key Point 6 Asking one person
the right question
gets better results
than asking 10,000 people
the wrong question
Caroline Jarrett @cjforms (CC) BY SA-4.0
58. Caroline Jarrett @cjforms (CC) BY SA-4.0
Many statisticians aren’t keen, either
Scientists rise up against
statistical significance
https://www.nature.com/articles/d41586-019-00857-9
58
59. Caroline Jarrett @cjforms (CC) BY SA-4.0 59
Let’s look at another crucial error
Non-response
error
59
60. Non-response error happens when
the people who do not respond are different
to the people who do respond
in a way that affects your decision
Caroline Jarrett @cjforms (CC) BY SA-4.0 60
61. Jane Matthews told me a story
• 20 people attend a workshop; they all seem to enjoy it
• Only get 3 or 4 back from a web survey
“If we rely on those responses,
we might be at risk of making bad decisions”
• Now changing to phoning half the people
Credit: https://janematthews.com/
62. Caroline Jarrett @cjforms (CC) BY SA-4.0 62
There are errors all around the Survey Octopus
(Lack of)
validity
Measurement
error
Processing
error
Coverage
error
Sampling error
Non-response
error
Adjustment
error
62
63. Key Point 7 Significance in practice
comes from
making good decisions
throughout the survey
Caroline Jarrett @cjforms (CC) BY SA-4.0
64. Caroline Jarrett @cjforms (CC) BY SA-4.0 64
The aim is to get the best number you can,
within the resources you have
What you want to ask about
The reason you’re doing it
The questions you ask
The answers you get
The answers you use
Who you want to ask
The list that you sample from
The sample you ask
The ones who respond
The ones whose responses
you can use
The number
(Lack of)
validity
Measurement
error
Processing
error
Coverage error
Sampling error
Non-response
error
Adjustment
error
66. Total Survey Error diagram as presented in
Groves, R. M., F. J. Fowler, M. P. Couper, J. M.
Lepkowski, E. Singer and R. Tourangeau (2009).
Survey methodology. Hoboken, N.J., Wiley.
66
People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
This is a genuine invitation from local government, but the layout and images in the invitation make it look as if it's an approach from some sort of spammer or scammer.
The octopus again; we've looked at 6 of the 8 tentacles.
The octopus again; we've looked at 6 of the 8 tentacles.
If you're buying a baby carriage, then the 'recommend to a friend' question would probably be obvious, interesting and appropriate. If you're in hospital having a miscarriage, it would be cruelly inappropriate.
The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
The octopus again. This time we're looking at 'the questions we ask'.
The octopus again. This time we're looking at 'the questions we ask'.
The resources you have will help you to decide on the reward you’re offering
The resources you have will help you to decide on the reward you’re offering
The octopus, with focus on 'The list you sample from'
The octopus, with focus on 'The list you sample from'
The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
The survey octopus has 8 tentacles. We'll visit each one in the next few slides. We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
The octopus again; we've looked at 6 of the 8 tentacles.
The octopus again; we've looked at 6 of the 8 tentacles.
The octopus again; we've looked at 6 of the 8 tentacles.
This is a more conventional way of looking at the octopus tentacles
This slide translates the issues into the technical terms used by survey methodologists