The Survey Octopus - getting valid data from surveys, presentation for UX in the City 2016
•Descargar como PPTX, PDF•
3 recomendaciones•2,392 vistas
Getting valid results from surveys: meet the Survey Octopus. Surveys are a powerful research method, but not easy to get right. The Survey Octopus is a way of thinking through the issues that will ensure that you'll get solid results from your survey that you can use to make decisions. Presentation from the UX in the City conference, Oxford, March 2016
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (11)
Destacado
Destacado (14)
Similar a The Survey Octopus - getting valid data from surveys, presentation for UX in the City 2016
Similar a The Survey Octopus - getting valid data from surveys, presentation for UX in the City 2016 (20)
Más de Caroline Jarrett
Más de Caroline Jarrett (13)
Último
Último (20)
The Survey Octopus - getting valid data from surveys, presentation for UX in the City 2016
- 1. The Survey Octopus Getting valid data from surveys Caroline Jarrett @cjforms 2016 #uxinthecity
- 2. Caroline Jarrett @cjforms What would you do for a dollar? 2
- 3. Caroline Jarrett @cjforms (CC) BY SA-4.0 $1 in the envelope beats $10 (guaranteed) later Caroline Jarrett @cjforms (CC) BY SA-4.0 3
- 4. Caroline Jarrett @cjforms (CC) BY SA-4.0 4
- 5. Caroline Jarrett @cjforms (CC) BY SA-4.0 Response relies on effort, reward, and trust People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 5
- 6. Caroline Jarrett @cjforms (CC) BY SA-4.0 I’m a forms specialist Image credit: Flickr, taxrebate.org.uk 6
- 7. Caroline Jarrett @cjforms (CC) BY SA-4.0 Why do people answer questions? Caroline Jarrett @cjforms (CC) BY SA-4.0 7
- 8. Caroline Jarrett @cjforms “Please have a look at this survey” 8
- 9. Caroline Jarrett @cjforms The survey is a systematic method for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members. Groves, Robert M.; Fowler, Floyd J.; Couper, Mick P.; Lepkowski, James M.; Singer, Eleanor & Tourangeau, Roger (2004).Survey methodology. Hoboken, NJ: John Wiley & Sons.
- 10. Caroline Jarrett @cjforms The survey is a process for gathering information from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
- 11. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) entities for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
- 12. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
- 13. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of getting numbers of the attributes of the larger population of which the entities are members.
- 14. Caroline Jarrett @cjforms The survey is a process for getting answers to questions from (a sample of) people for the purpose of getting numbers that you can use to make decisions
- 15. Caroline Jarrett @cjforms To make decisions People The survey is a process for getting answers to questions getting numbers
- 16. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim of a survey is to get a number that helps you to make a decision 16
- 17. Caroline Jarrett @cjforms “The numbers” or “The number”? 17
- 18. Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork used to be expensive so a survey was a rare event. Image credit: http://www.census.gov/history/www/genealogy/decennial_census_records/ 18
- 19. Caroline Jarrett @cjforms (CC) BY SA-4.0 1950s mindset: “Ask Everything” Survey = Big Honkin’ Survey 19
- 20. Caroline Jarrett @cjforms (CC) BY SA-4.0 2016 mindset: the Light Touch survey • Choose ONE question • Find ONE person • Ask the question, face-to-face • See if you can make ONE decision • Improve, iterate, increase 20
- 21. Tip Interview first Caroline Jarrett @cjforms (CC) BY SA-4.0 21
- 22. Caroline Jarrett @cjforms One way to iterate, improve, increase Time for new question Caroline Jarrett @cjforms (CC) BY SA4.0 22
- 23. Caroline Jarrett @cjforms (CC) BY SA-4.0 The aim of a survey is to get a number that helps you to make a decision 23
- 25. Caroline Jarrett @cjforms Asking the right question Asking the right people 25
- 28. Caroline Jarrett @cjforms (CC) BY SA-4.0 Asking the right people is better than asking lots of people Sample: the list you sample from Caroline Jarrett @cjforms (CC) BY SA-4.0 28
- 30. Caroline Jarrett @cjforms (CC) BY SA-4.0 There’s often a ‘zone of indifference’ Hate it Love it 30
- 31. Caroline Jarrett @cjforms (CC) BY SA-4.0 Feeling special is rewarding People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 31
- 32. Caroline Jarrett @cjforms (CC) BY SA-4.0 Ask fewer people to get better response Sample: the number of people to ask Caroline Jarrett @cjforms (CC) BY SA-4.0 32
- 33. Caroline Jarrett @cjforms (CC) BY SA-4.0 Try mail, phone or face-to-face for better response rates Fieldwork: Who answers? Caroline Jarrett @cjforms (CC) BY SA-4.0 33
- 34. Did we get answers from the right people? Is this sample representative? Image credit: Caroline Jarrett / CorelDraw 34
- 35. Population of assorted birds Is this sample representative? Image credit: Caroline Jarrett / CorelDraw 35
- 36. Caroline Jarrett @cjforms (CC) BY SA-4.0 We don’t just want answers, we want representative answers Response Caroline Jarrett @cjforms (CC) BY SA-4.0 36
- 37. Caroline Jarrett @cjforms Asking the right question Asking the right people 37
- 38. Caroline Jarrett @cjforms Asking one person the right question is better than Asking 10,000 people the wrong question 38
- 40. Caroline Jarrett @cjforms In your last five days at work, what percentage of your work time do you estimate that you spent using publicly- available online services (not including email, instant messaging, and search) to do your work using a work computer or other device? % Caroline Jarrett @cjforms (CC) BY SA-4.0 40
- 41. Caroline Jarrett @cjforms (CC) BY SA-4.0 Helps a lot if you ask good questions Questions: What are you asking about? How many questions? Caroline Jarrett @cjforms (CC) BY SA-4.0 41
- 42. Caroline Jarrett @cjforms (CC) BY SA-4.0 Response relies on effort, reward and trust People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response. Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 42
- 43. Caroline Jarrett @cjforms (CC) BY SA-4.0 A good question works in three ways Obvious Interesting Appropriate 43
- 44. Caroline Jarrett @cjforms (CC) BY SA-4.0 Why did you visit our website today? Obvious Interesting Appropriate 44
- 45. Caroline Jarrett @cjforms (CC) BY SA-4.0 Would you recommend us to a friend or family member? In a shop, buying a baby buggy In a hospital, having a miscarriage Obvious Yes Interesting Yes Appropriate Yes Cruelly inappropriate 45
- 46. Caroline Jarrett @cjforms (CC) BY SA-4.0 Would you recommend us to a friend or family member? In a shop, buying a baby buggy In a hospital, having a miscarriage Obvious Yes Interesting Yes Appropriate Yes Cruelly inappropriate 46
- 47. Tip Test your questions by interviewing in context Caroline Jarrett @cjforms (CC) BY SA-4.0 47
- 48. Caroline Jarrett @cjforms (CC) BY SA-4.0 We don’t want any old answers. We want useful answers. Response Caroline Jarrett @cjforms (CC) BY SA-4.0 48
- 49. Caroline Jarrett @cjforms In your last five days at work, what percentage of your work time do you estimate that you spent using publicly- available online services (not including email, instant messaging, and search) to do your work using a work computer or other device? % Caroline Jarrett @cjforms (CC) BY SA-4.0 49
- 50. Caroline Jarrett @cjforms On your most recent working day, what percentage of time did you spend using the Internet? % Caroline Jarrett @cjforms (CC) BY SA-4.0 50
- 51. Caroline Jarrett @cjforms (CC) BY SA-4.0 The approximate curve of forgetting Major life event Noticeable, occasional Unremarkable, repetitive Perfect recall Day Gone Week Month YearHour 51
- 52. Caroline Jarrett @cjforms (CC) BY SA-4.0 Can’t remember = can’t answer Response Caroline Jarrett @cjforms (CC) BY SA-4.0 52
- 53. Caroline Jarrett @cjforms (CC) BY SA-4.0 Insights are the answers that you use to make the decisions Insights Caroline Jarrett @cjforms (CC) BY SA-4.0 53
- 54. Caroline Jarrett @cjformsCaroline Jarrett @cjforms (CC) BY SA-4.0 54
- 55. Caroline Jarrett @cjforms "Phone photography" by Petar Milošević - Own work. Licensed under CC BY-SA 3.0 via Commons - https://commons.wikimedia.org/wiki/File:Pho ne_photography.jpg#/media/File:Phone_phot ography.jpg Modified by Caroline Jarrett 55
- 56. Tip Always allow for ‘other’ Design by @RickyBuchanan; t-shirt from nopitycity.com or zazzle.co.uk 56
- 57. Caroline Jarrett @cjforms Can we make the decision yet? Can we make the decision yet? Can we make the decision yet? Caroline Jarrett @cjforms (CC) BY SA4.0 57
- 58. Tip Try writing your presentation first 58
- 59. Caroline Jarrett @cjforms (CC) BY SA-4.0 We’ve met the Survey Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0 59
- 60. What could possibly go wrong? Caroline Jarrett @cjforms (CC) BY SA-4.0 60
- 61. Caroline Jarrett @cjforms (CC) BY SA-4.0 Caroline Jarrett Twitter @cjforms http://www.slideshare.net/cjforms carolinej@effortmark.co.uk 61
Notas del editor
- This is a genuine invitation from local government, but the layout and images in the invitation make it look as if it's an approach from some sort of spammer or scammer.
- People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
- The survey sits between 'what you want to ask', 'who you want to ask' and 'the number'
- A process starting with one person face to face, continues through 10 people by phone, gets to 100 people by email or pop-up. It’s best to check that your question works with one person before you hassle 10 people with it. Then check it works with 10 people before you send it to 100. Once you’ve tried it on 100 people, you might be more interested in a new question than getting more answers on this question
- The survey sits between 'what you want to ask', 'who you want to ask' and 'the number'
- Prank leaves Justin Bieber facing tour of North Korea By Daniel Emery Technology reporter, BBC News 5 July 2010 Image caption It is highly unlikely Bieber would be given permission to enter North Korea Canadian singer Justin Bieber's has become the target of a viral campaign to send him to North Korea. A website polled users as to which country he should tour next, with no restrictions on the nations that could be voted on. There are now almost half a million votes to send the singer to the secretive communist nation. The contest, which ends at 0600 on 7 July, saw North Korea move from 24th to 1st place in less than two days. Many of the votes are thought to originate from imageboard website 4chan, which has built a reputation for triggering online viral campaigns.
- The octopus, with focus on 'The list you sample from'
- People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
- We’ll get our survey to the people who will answer in what the survey methodologists call ‘fieldwork’ – that might be a pop-up a website, a mail survey, or face-to-face interviews.
- The octopus, with focus on "the ones whose answers you use"
- The octopus again. This time we're looking at 'the questions we ask'.
- People will only respond if they trust you. After that, it's a balance between the perceived reward from filling in the survey compared to the perceived effort that's required. Strangely enough, if a reward seems 'too good to be true' that can also reduce the response.
- Obvious questions require no effort to answer. Interesting questions are rewarding to answer. Appropriate questions are ones that inspire trust in the respondent.
- People come to the web with their own questions, so they’re likely to know why they are on your website. It’s a relatively interesting question to answer, and it’s appropriate to ask visitors why they’re visiting.
- In your last five days at work, what percentage of your work time do you estimate that you spent using publicly-available online services (not including email, instant messaging, and search) to do your work using a work computer or other device?
- Two questions from a survey: '24: Do you use a Windows or Mac computer'? '25. What is your gender'?
- Photo of a Samsung (android) mobile with the same questions as previous slide. If you only have an Android mobile, how do you answer ‘do you use a Windows or Mac computer’ when the answer options are ‘Windows’, ‘Mac’ and ‘Both’?
- A model wears a t-shirt with Gender: 'Male' (crossed out), 'Female' (crossed out) and 'Other' (added and ticked'.
- A process starting with one person face to face, continues through 10 people by phone, gets to 100 people by email or pop-up. It’s best to check that your question works with one person before you hassle 10 people with it. Then check it works with 10 people before you send it to 100. Once you’ve tried it on 100 people, you might be more interested in a new question than getting more answers on this question
- The 8 tentacles of the Survey Octopus are: Left side: Goals: the resources you have Questions: the questions you ask Response: the answers you get Insights: the answers you use Right side: The list you sample from The sample you ask The ones who answer The ones whose answers you use
- The 8 tentacles of the Survey Octopus are: Left side: Goals: the resources you have Questions: the questions you ask Response: the answers you get Insights: the answers you use Right side: The list you sample from The sample you ask The ones who answer The ones whose answers you use