Internet User Experience Conference 4/24/2017 - Ann Arbor, MI
Presenters: Farris Khan and Chris Farnum
Like Jan Brady, online surveys are the middle child. They occupy the space between analytics platforms which are highly quantitative and user testing which can often be very qualitative. Many people turn their noses up at surveys including some business stakeholders and even UX-ers. Have you ever heard a stakeholder say, “I never fill out one of those. I always just close them.” Though they are often misunderstood and overlooked (“Marcia, Marcia, Marcia!”) surveys can be a powerful tool for measuring and baselining your user experience. There are different types of surveys, and how you choose and configure them makes all the difference. Farris and Chris will take you on a tour of several different approaches to survey programs. We’ll outline their potential benefits as well as pitfalls. And we promise that we won’t be wearing wigs.
4. User testing
and research
So easy to
connect with.
The darling.
A little goes a long way.
(i.e. qualitative)
@BobbleHeadGuru @crfarnum #iue17
5. Analytics and
big data
So popular
And sophisticated.
The attention getter.
You can count on a reaction.
(i.e. quantitative)
@BobbleHeadGuru @crfarnum #iue17
10. Types of
Surveys
Comment Card
Surveys
Longitudinal
Satisfaction Surveys
Detailed
Open Ended
Surveys
One question
Quick Feedback
NPS
Online Product, Service
Surveys
LeveragingWebsite
More QuantitativeMore Qualitative
@BobbleHeadGuru @crfarnum #surveyUX #iue17
Ad Hoc
Surveys
12. Detailed
Open Ended
Surveys
Comment
CardSurveys
ACanary in a
Coal Mine
More QuantitativeMore Qualitative
@BobbleHeadGuru @crfarnum #surveyUX #iue17
Comment Card
Surveys
Longitudinal
Satisfaction Surveys
One question
Quick Feedback
NPS
Online Product, Service
Surveys
LeveragingWebsite
Ad Hoc
Surveys
18. TheUX
Perspective on
Comment
Cards
Canary in a coal mine Tempest in a teapot
One comment card can cause a lot of chaos, especially
in the hands of a well meaning stakeholder.
Analyzing lots of comment cards can indicate trends
and help judge the severity of issues.
@BobbleHeadGuru @crfarnum #surveyUX #iue17
-OR-
20. Online Product, Service
Surveys
LeveragingWebsite
Ad Hoc
Feedback
Nimble
Optimization
More QuantitativeMore Qualitative
@BobbleHeadGuru @crfarnum #surveyUX #iue17
Comment Card
Surveys
Longitudinal
Satisfaction Surveys
Detailed
Open Ended
Surveys
Ad Hoc
Surveys
One question
Quick Feedback
NPS
21.
22. TheUX
Perspective on
Ad Hoc
Surveys
Overlap remote user testing.
Use for:
Design Input
Detailed feedback
Understand users
UXD “sweet spot”
Get involved!
@BobbleHeadGuru @crfarnum #surveyUX #iue17
27. Customer
Satisfaction
Model via
ForeSee
Questions include “How likely
are you to recommend Blue
Cross Blue Shield's website to a
family member, friend or
colleague?
Three weighted, independent
questions determine C-sat
Questions include “Please rate
how well the site layout helps
you find what you need.”
https://www.foresee.com/why-foresee/proven-approach/
@BobbleHeadGuru @crfarnum #surveyUX #iue17
32. TheUX
Perspective on
Longitudinal
Surveys
A word of caution…
Metrics like satisfaction are bellwether indicators,
not outcomes.
The real value comes from examining specific
outcomes for target user segments and/or tasks.
@BobbleHeadGuru @crfarnum #surveyUX #iue17
33. Satisfaction
Score is just
the tip of the
iceberg…
Satisfaction
score
Voice of
customer data
@BobbleHeadGuru @crfarnum #surveyUX #iue17
34. When you
analyze
results, the real
questions
are…
What dates?
Which personas?
What tasks?
Task success?
Narrow and segment to find insights.
@BobbleHeadGuru @crfarnum #surveyUX #iue17
39. No survey is an
island
@BobbleHeadGuru @crfarnum #surveyUX #iue17
Surveys work best when combined with other
techniques like user testing, user research, and analytics.
40. FinalThoughts
Build on previous surveys, iterate
• Surveys can be used iteratively, in a series.
• For example, the results of an initial ad hoc survey can be
used to inform the questions in a longitudinal survey.
Get to know the survey programs in your
organization.
• Check out existing reports
• Get access to the tools
• Work with theVOC and analytics team to improve the
surveys and interpret findings
@BobbleHeadGuru @crfarnum #surveyUX #iue17
Comment Cards
Used to get biased feedback
A canary in a coal mine
Like a card that you fill out at a restaurant.
Do NOT use for quantitative Satisfaction analysis
Meta analysis… clustering of issues, open ended comments
You will only get those that are happy
…. And likely more of those that are upset
Ad Hoc Surveys:
Add Survey Distribution (send via email, intercept)
Used to get quick feedback in areas that just launched.
Directional, not statistically significant
Do not use for too long.
Use for making a A/B Decisions,
Then move on to the next one.
Be Nimble
Was it easy to use.. Would you be likely to use it in the future.
Can be helpful in getting:
detailed input on designs
feedback on an existing site
fleshing out understanding of personas, tasks, and user needs
Lots of overlap with remote user testing.
UXD can (should) participate closely in designing the ad hoc surveys.
Health with Ad Hoc / Deep DiveAdd Survey distributionShow Foresee.
Used to get long term feedback.
Leverage Open Ended Comments
Random Intercept
Bell weather
Segment questions (Task completion, primary reason, frequency of use)
Proquest: Have an assignment in the UX team. Update or redesign: Arm yourself with information today and what their comments are.
Pinpoint functionality… with Primary Reason…
Use for long term monitoring.
Most complex
Most misunderstood
NPS… Members Scores.