One of the shortcomings of many user interviews is the vast gulf between what people think they do versus what they actually do, not to mention what they may have forgotten having done. Fortunately, new research tools are filling that gap by allowing users to quickly provide feedback from their phone right after they use a product, allowing for the capturing of rich, emotional details. These tools are breathing new life into a traditional research tool, the diary study. This talk share the best practices I’ve developed for designing a digital diary study that collects relevant and insightful data. It will be framed by examples from a recent diary study exploring how people use their fitness trackers (Fitbit, Jawbone Up, etc). Attendees will come away with not only an understanding of how much rich data can be collected this way, but with the basic knowledge needed to execute their own digital diary studies.

1. Learning from
Users in their
Natural Habitat
Sara Cambridge
sara@gotoresearch.com
@maybanks
2015 UXPA

2. What is the best way to learn
about the experience that people
are having with your product?

3. 3
Interviews don’t capture the range of ways people
do things or the rich, in-the-moment details that
people quickly forget.

4. Digital diary studies allow you to:
• Get instant feedback as people naturally use a
product or service
• Ask them follow-up questions throughout the study
• Access everything through a digital platform

5. 5
Are You Wearing One?

6. Fitness Tracker Case Study
We recruited 40 people who use fitness trackers to participate in
several digital diary studies over 8 months.
The first diary study had them report on 5 things they do with their
tracker. With every entry, they submitted a short video and answered
a few questions. Over 170 entries were collected.
We did 4 other diary studies with them over the course of 8 months to
better understand how their fitness tracker usage evolved and the
role it plays in their overall health goals.
6

7. 7
Erica, 33
“It’s 10:53pm, and I’m only at
7388 steps….we’re about to go
take the dog for a walk. Got to
make sure we get to our
10,000 steps before the end of
the night.”

8. What This Presentation Will Cover
What a Diary Study is
The Benefits of Using Diary Studies
How Diary Studies Compare to Other Research Methods
Traditional Diary Studies vs. Digital Diary Studies
What the Participant Experience Is Like
How to Create a Digital Diary Study
Findings From Our Fitness Tracker Diary Studies
8

9. What is a Diary
Study?

10. A Diary Study is a Cultural Probe
Cultural probes use a collection of tools, artifacts
and tasks intended to provoke the user to think
about their environment in new ways….With
minimal intrusion, researchers can glean insights
into participants’ environments that can help to
identify problem statements and uncover new
opportunities.
10
Adam Starkman & Jennifer Chow
http://designresearchtechniques.com/casestudies/cultural-probes/

11. Diary Studies Have 4 Key Elements
Theme-based: the subject and questions are suggested by the
researcher.
Self-reported: the actual reporting is done by the participant, using
whatever platform/prompts were provided.
In-the-moment: the participant is asked to do the reporting
immediately after engaging in the activity, in order to capture as much
detail as possible.
Longitudinal: the goal is to capture a range of uses, rather than a
singular experience. This can include how the user’s experience
evolves over time.
11

12. Traditional Diary Study
Mozilla used paper diary studies in 2012 to learn how participants
were using bookmarks in Firefox.
They asked 10 people to record their experiences with bookmarks
over 3 days. Each day had a different set of questions.
12
Brian Groudan, Mozilla
research, 2012 /
https://blog.mozilla.org/
ux/page/24/

13. Traditional Diary Study
People came in for an in-person interview afterwards, where they
discussed their entries and sorted them by theme.
Guiding design principles were culled from the analysis that were the
basis for design better bookmarking tools.
13

14. Why Use
Diary Studies?

15. Looking for the Details
The goal of research is to understand not just what people do (and
want and believe), but why.
This is complex and requires a deep understanding of people’s
behavior.
Knowing details about their behavior is necessary to get to that
understanding.
15

16. However people often have a hard time remembering those details
when not engaged in the activity.
This is especially true for habitual behaviors.
Habits are formed when our brains:
Recognize that we repeatedly do the same sequence of actions (like
checking email) after being triggered by a specific cue (such as
picking up our phones).
The brain “chunks” that sequence into one automatic routine.
This allows our brain to save mental energy for more complex tasks.
Barriers to Collecting Details
16The Power of Habit, Duhigg

17. A consequence of having a sequence of behaviors “chunked”
together is that details of the habit become accessible only after the
cue has been activated.
So outside the actual ‘doing’ of the task we have difficulty
remembering the details.
This is why gathering information in context collects such rich
details.
Barriers to Collecting Details
17The Power of Habit, Duhigg

18. 18
The best way to collect these rich details are either
observation (contextual interviews) or in-the-
moment reporting (diary studies).

19. How Diary Studies
Compare to
Other Methods

20. 20
Quantitative methods only explains what people do
while qualitative methods offer insight into why
people do them.
Focus Groups
Diary Studies
Usability Testing
Surveys
Big Data
Contextual Interviews
why people do it
what people do
Quantitative
(conscious & logical)
Qualitative
(subconscious &
emotional)

21. 21
Once you understand why, your insights can move
from evaluative (improvement) to generative (new
opportunities).
Focus Groups
Diary Studies
Usability Testing
Surveys
Big Data
Contextual Interviews
evaluative
Data
Stories
generative

22. 22
Yesterday’s session about Storytelling by Mary
Wharmby discussed this.

23. 23
Karen, 26,
grad student
“I got drunk with my friend and
convinced her to download
Snapchat on the contingency that
I send her a [Snapchat] everyday.
I use my [Jawbone Up] to remind
me at 7pm to send her a
picture….it vibrates on my wrist.”

24. How to Use Diary
Studies with Other
Research Methods

25. Insight Study Process
An Insight Study is a series of 2-3 focused research cycles built
around a specific question. Multiple research methods are used in
order to capture different types of insights.
Each cycle concludes with an Insight Review that shares the findings
with the client team.
The benefit for the client is that they:
• Deliver real-time findings that can be acted upon right away
• Offer an opportunity to refine the next phase of research based on
the learnings.
25

26. 26
Insight Study
Methods:
Contextual Interviews
allow us the deepest
insights by observing
people’s usage firsthand.
Diary Studies collect in-the-
moment behaviors around
a variety of issues
throughout the project.
Remote Interviews fill in
the gaps by probing
deeper into the diary study
entries and related areas.

27. The Benefits of Using Mixed Methods
Using mixed methods allow us to validate our findings and take
advantage of the unique strengths that the various research methods
offer.
• Diary studies let us collect actual behaviors around a range of
issues or track ongoing usage patterns.
• Contextual interviews give us the deepest insights by letting us
observe their natural environment directly.
• Remote interviews give us access to people who are
geographically distributed and allow more frequent opportunities
to engage with them.
27

28. What is a Digital
Diary Study?

29. 29
Getting people to actually do the reporting is a
major obstacle for traditional diary studies.
http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731

30. 30
Smartphones are ideal for digital diary studies
since people use them in short spurts throughout
the day.
http://kotaku.com/hong-kong-residents-supposedly-cant-live-without-their-476758731

31. 31
Digital diary studies can be as simple as text or
email prompts that people reply to.

32. Digital diary study platforms
32
Name Platform Cost
24Tru Web/app diary study high
CrowdTap Web/app diary study high
UX 360 Web diary study high
Dscout App diary study medium
Dedoose Web diary study low
Tumblr private blog low
Typeform online survey low

33. 33
$595 for recruiting
(or you can bring in your own participants)
$295 for each study

34. 34
Dscout has great guides and online support to help
get you started.

35. The Participant
Experience

36. 36
This view is only if recruiting through dscout.
Participant reviews
opportunities on app
Apply for a study Fill out screener

37. 37
Once invited to a study, this is what all participants
see.
Email invite to mission Q1: Begin with media prompt
(photo, video or text)
Inside the app

38. 38
Studies can have up to 10 questions each, and can
be any mix of multiple choice, rating, or open-
ended questions.

39. How to Conduct a
Digital Diary Study

40. The 3 Phases of a Diary Studies
40
The goal:
a concise, focused
study with the right
participants.
The goal:
high participation and
high quality data.
The goal:
to turn data into
relevant insights.

41. • Clarify your research question
• Refine and pretest your study
• Minimize the work for your participants
• Recruit the right participants

43. Clarify Your Research Question
Our Initial Questions:
How are activity trackers used by people attempting to make
concrete improvements to their fitness habits?
Can we discern patterns around when activity trackers are more or
less effective at helping people make those changes?
43

44. 44
Make a list of possible questions. You will have too
many. Edit and sort them by theme.

45. 45
Define the goals for each study. This will help
maintain focus as you refine the questions.

46. 46
Diary Studies are basically surveys with a twist.

47. 47
That means answer options, which require a lot of
refining. The more time spent building the study,
the better the data will be.

48. Make it Easy for your Participant to
Succeed
A guiding principle as you build your study should be to minimize the
work your participant has to do.
48

49. 49
Consider how often you ask them to report. It
should be enough to collect useful data, but not
enough to cause fatigue.

50. 50
Pairing themed multiple-choice questions with
open-ended ones creates a logical flow for people
and answers both “what” and “why”.

51. 51
Another benefit of multiple-choice is you can
quickly see trends as the data comes in.

52. 52
Open-ended questions provide the richest data but
are more work for both for you and the participant.
Use them sparingly.
I think it should send reminders on when to be more
active when I'm lagging in my goal such as vibrating.
I do wish that other activities
could be tracked with the
Fitbit without having to input it
myself.
Maybe if I could get
some sort of notification
on my phone if a specific
friend or group of friends
had passed me.

53. 53
Pretest by sitting down with someone as they go
through every step. Look for awkward or unclear
phrasing.

54. 54
Media options: video adds a lot, but also adds
more work for both you and the participants.
Take a picture that
symbolizes the role your
tracker plays in your
health/fitness goals
Tell us what ideal
health would look like
for you.

55. 55
Some questions are better suited for a photo,
which makes it easier to compare (and share)
responses.
A long road ahead. Before last year I never
thought I could be THAT
mom who was fit & healthy.
My tracker is a set of
tools I'm using to build a
healthier life for myself.

56. 56
When capturing real experiences or complex
ideas, video is better.
Karen, 26,
grad student
“Being able to walk upstairs without being out of
breath. Also, be at the pick of my thinking and
not feeling tired and drained because of ill
health. And I think that my tracker helps me with
that especially with sleep.”

58. 58
Diary studies generate a lot of data.
It is good to start small.
10 x 10 x 10 =questions participants entries
data points
1000

59. 59
Recruiting options include customers, friends,
Craigslist, social media and dscout.

60. 60
If recruiting through dscout, you’ll only pay
incentives for participants that finish. dscout
recommends paying $5-10 per entry.

61. 61
We looked for participants who have a fitness
tracker and were regularly using it as part of a
fitness or weight loss goal.

62. 62
Typical dropout rate is 15-25%, so over-recruit.
40
34

63. 63
You can view each applicant’s responses inside the
screener survey and rate them based on how good
they are.

64. 64
It’s easy to add participants if you do your own
recruiting. They get an email link for downloading
the app.

65. • Review & make notes from your data
as it’s coming in
• Be persistent with participants
• Ask them questions for clarity
• Do post-study interviews

67. 67
Watch the data as it comes in. This is the fun part.
What is your favorite
thing about your fitness
tracker?
Kayla, 27,
teacher
“My favorite thing is that by a single
click of a button, the display shows
me the time, date, exact steps I've
taken, my heart rate, steps climbed
and calories burned as opposed to
having to find my phone, unlock, open
the app...it's all right there, easy to
read. I like having it all right there on
the display. .

68. 68
Entries are reviewed individually.

69. 69
Take notes of interesting patterns and questions
to ask participants as you go through it.

70. 70
Moment of truth: are your questions holding up?
If not, can you fix it? Either way, learn for next time.

72. 72
Stay in touch with your participants. Send them
reminders if they haven’t submitted an entry. Be
friendly and persistant.

73. 73
Ask them questions if you want more clarity about
their answers. Be appreciative and interested in
what they have to say.

74. 74
Do follow-up interviews with the most engaged
participants. You already have rapport and can dig
deeper into their actual behaviors.

75. • Start by answering your initial questions
• Use the built-in analysis tools
• Identify patterns and/or exceptions
• Share your findings with real examples

76. 76
The good news is that by simply reviewing and
making notes on the data, you have already
started on the analysis.

77. What Further Analysis Should be Done?
That depends on many variables, such as:
• What your initial question/s were
• How surprised you are by the data that has come in
• How it fits into other research methods
• Your final deliverables
77

78. 78
The internal tools dscout offers are helpful.
Tagging can quickly reveal patterns.

79. 79
Multiple choice questions can be filtered. Here
we wanted to review the entries in which people
were not as satisfied with their tracker.

80. For Those Who Rated an Experience
with Their Tracker as “OK” or Lower:
• 30% were unhappy with some aspect of the band
(too bulky, not water resistant or something had broken)
• 25% were complaining about the charger or battery
• 15% had issues logging their food
(either through a secondary app or via their main device)
• The remaining 30% were a random mix of issues
80
Once the responses were filtered, it was quick to
manually categorize what remained.

81. 81
Transcribing video is time-consuming, but makes
the data much more accessible. Tagging is an
ideal way to remember interesting video.

82. 82
Downloading the data into a spreadsheet that can
be coded, refined, sorted and easily reviewed is
our preferred method if we intend to dive deep.

83. 83
If the original answer options were not accurate
enough, you’ll have to throw data away or recode it
yourself; here we recoded it.

84. Fitness Tracker
High-Level Findings

85. 85
Why did you get your first fitness
tracker?
Larry, 31
“I got a fitness tracker because
was about 300 lbs, on a diet,
exercising, wasn't losing
weight. I knew there was
something missing. I wanted to
learn how many calories I was
burning so could know how
little or much I should be
eating. That was the missing
link. I have lost 150 lbs, and the
only way did it was because I
started using Jawbone Up.”

86. People Love Trackers Because of What
They Learn About Themselves
Fitness trackers teach people things about their eating and exercise
patterns they’ve never known before.
They had made guesses, but they had guessed wrong.
People usually overestimated the exercise they were getting and
underestimated how much they were eating.
Knowing the actual numbers gives people control so they can get to
the outcome they want.
86

87. 87
What do you like least about your
tracker?
Amy, 38
“Now for least favorite. Even
though I loved a lot about all
the trackers: Omron and Fitbit,
the one thing I didn't like was
that after a while I got bored.
There's only so much
competing you can do, so
many steps you can take, but
they didn't do anything with the
data. It was great when I
reached my 10,000 steps for
the day, but then what? It
doesn't help me do anything
more with it.”

88. Once People Learn their Patterns, they
get Bored (and/or Frustrated)
Depending on how motivated people are to make changes to their
health behavior, they usually go through a slump after they learn their
patterns.
Changing exercise and eating pattens require significant changes to
one’s lifestyle, and knowing one’s patterns is only the first step.
People want more help getting to their goals.
88

89. 89
What does ideal health look like to you?
Mary, 42
“My ideal health would be to
feel better, not drag, and eating
better foods. It would help me
feel motivated to follow my
exercises I need a tracker that
would also [suggest what]
types of foods to eat. It knows
that I am running and this is the
food you should eat. That
would be so wonderful! That
would help me a lot even more
with my runs, my exercise, with
getting in shape.

90. Trackers Only Address a Fraction of
People’s Health Goals
When asked what their ideal health looks like, people talked about
both complex ideas (such as mood, energy level, endurance and
eating habits), and hard numbers (like weight and body fat).
Trackers only help with some of the aspects of ideal health.
People want their trackers to know their goals, learn their patterns and
offer suggestions to help them achieve their goals.
90

91. Digital Diary Study
Key Takeaways

92. Main Benefits of Using Diary Studies
• To collect rich stories about how and why people do things
• Get intimate glimpses of how a product fits into people’s lives
• Allows for tracking ongoing use of a product
• Cost-effective way to collect contextual insights
92

93. Tips for Building Your Own Diary Study
Before you can use it, you may have to sell it. Here are a few things to
keep in mind:
• It’s an extremely flexible tool; make it work for your needs
• Start small, let it prove it’s value to your organization and yourself
• Evangelize the insights and stories you collect to build more
support.
• You become an expert by doing. So dive in!
93

94. What Will You Do
with Diary Studies?

95. For a 4-min video about diary studies:
www.gotoresearch.com/speaking/sara/
Questions about diary studies? Ask!
sara@gotoresearch.com
@ maybanks

96. Thank you!
sara@gotoresearch.com
@ maybanks