Ana Chang and I presented this talk to PS SIG CHI on December 5, 2013. We also presented the material at Google and Chase on December 6th.

1. The RITE+Krug Method
Getting crazy good stakeholder involvement
in Agile user research
Jen McGinn & Ana Chang
December 5, 2013

2. Format
We’ll tell you our story
We’ll listen to your stories
We’ll give you summaries and links to
further reading
Hopefully, you’ll leave with something
helpful
And we’ll pick up some tricks from you 
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
2

3. Before we tell you our story, let’s start with you
Show of hands:
– How many of you are working with Agile development
teams?
– How many of you are conducting user research (feedback
sessions) with those Agile teams?
– How many of you are feeling really AWESOME about how
the dev team is involved in that research and that they
make changes as a result?
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
3

4. Late 2011: Traditional usability tests were conducted
I can’t believe he clicked
the blue button.
I assumed he would click
the gray button.
We should fix that!
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
4

5. Later 2011: The stakeholders didn’t buy into the results
The results are not
statistically significant.
This is not a typical user.
That’s not a realistic task.
I don’t agree with
your interpretation
of what that means.
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
5

6. The feedback came too late for our Agile environment
Current state of the
application
Proposed
new design
Current state of
the application
At the beginning of
the study
Jen McGinn and Ana Chang
Start of study
When the study results
are published
PS SIG CHI Monthly Meeting, Dec 2013
6

7. Now it’s your turn
• First names only
• Agile User Research only
• In less than a minute, tell us:
– What user research you are conducting now (method, frequency, # of
participants)
– One thing that is AWESOME about that
OR
– One thing that’s not working, that you wish you could fix
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
7

8. Ana had attended a conference
Hey, what about
RITE testing?
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
8

9. Great – so now what?
Rapid Iterative Test Environment (RITE)
– Developed by Dennis Wixon & the folks at Microsoft
games
– Can change the (live) system between participants
– Developers must attend the UT sessions
– Keep testing until show-stoppers are fixed
– Number of participants and time to conduct undefined
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
9

10. Early 2012: What’s been written on Agile + RITE?
• When I started looking for answers 18+ months ago:
• Lots and lots on Agile UX + design
• Not so much on Agile UX + user feedback
• RITE (described in general terms, but no specifics)
• Co-design
• Customer/user “proxies”
• However, there were some elements of Steve Krug’s approach
that stuck with me
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
10

11. Steve Krug … why does that name sound familiar?
http://www.amazon.com/Steve-Krug/e/B001KHCFUU/
11

12. The Krug Approach*
“A morning a sprint, that’s all we ask.”
• Test 3 users in a morning
• Invite everyone to attend: product owners, developers, designers,
writers, marketing …
• Debrief over lunch: 1 hour
– Only people who observed can attend/vote (sort of)
– Focus ruthlessly on most serious problems
• Report takes less than 30 minutes to write
– Just an email
– 2 page max., mostly bullet lists
* Originally presented by Steve Krug at the Boston UXPA 2013 Conference, May 2013
PS SIG CHI Monthly Meeting, Dec 2013
12

13. Our RITE + Krug Combination
• Elements from both RITE and KRUG
– Require that a stakeholder from dev, design and PM attend each
session
– Can change the prototype between participants
– 3 or 4 participants
– Findings discussed in 1-hour debrief
– Minimal “report” (bullets on a wiki page)
• Sessions went viral: 12+ observers
• Could test every as often as every 2 weeks
http://www.upassoc.org/upa_publications/jus/2013may/JUS_McGinn_May_2013.pdf
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
13

14. Artifacts: scaled down script
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
14

15. Artifacts: example wiki page with schedule and impact
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
15

16. RITE+Krug Tips & Tricks
• Before sessions: Hold a kickoff meeting with stakeholders to
get buy-in into the process and time required
• During: Keep a running list of observations and email them to
the stakeholders within 30 minutes of the last participant
• After sessions: Ask attendees of the stakeholder meeting to
review your list of observations prior to the meeting
• During debrief: Walk through the findings to get agreement
on what everyone saw (optional: agree to next steps to fix)
• After debrief: Immediately document the observations and
list of changes agreed to and send out/post on wiki
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
16

17. Let’s talk about Diana
The conversation has
changed from ‘what is it?’
to ‘how can we master it
and make it our own?’
- Diana DeMarco Brown (over lunch one
day)
http://www.amazon.com/Agile-User-Experience-Design-Practitioners/dp/0124159532/
PS SIG CHI Monthly Meeting, Dec 2013
17

18. Diana’s Agile Research Recommendations*
• Conduct research before the development cycle if possible
– But if you can’t, fit it in anyway
• Predictable and repeatable user feedback sessions during the cycle
• Complement user feedback sessions with fast, low-cost data
collection methods:
– Surveys
– Customer phone calls
– Web analytics
– Card sorts
– Un-moderated usability tests
* Originally presented by Diana Brown at the Boston UXPA 2013 Conference, May 2013
PS SIG CHI Monthly Meeting, Dec 2013
18

19. Jeff Gothelf’s Approach
• Continuous & collaborative
– Continuous: every sprint
– Collaborative: conduct
research together (dev, design,
PM, &research)
http://www.amazon.com/Lean-UX-Applying-Principles-Experience/dp/1449311652
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
19

20. Test everything
“In order to maintain a regular cadence of user testing,
your team must adopt a “test everything” policy.
Whatever is ready on testing day is what goes in front of
the users… you’ll find yourself taking advantage of your
weekly test sessions to get insight at every stage of
design and development.”
– Lean UX, Jeff Gothelf
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
20

21. What I do now: Continuous and Collaborative
• Collaborative
– Dev, Design, Doc, and PM stakeholders participate or observe
– User researcher will facilitate or advise
– Your opportunity to get “out of the building”
• Continuous
– Sessions take place at regular intervals, say once a sprint
– User researcher will recruit 3 – 4 target participants per session +
debrief same day
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
21

22. What else have people written on User Research + Agile?
Agile + RITE testing, finally (!) described in detail (2013):
• “The RITE Way to Prototype”, Shirey, Charng, & Nguyen
http://uxmag.com/articles/the-rite-way-to-prototype
Agile + Cognitive walkthroughs (2013):
• “Informal cognitive walkthroughs (ICW): paring down and pairing
up for an agile world”, Grigoreanu & Mohanna
http://dx.doi.org/10.1145/2470654.2466421
Agile + Wizard of Oz testing (2006):
• “Adding Usability Testing to an Agile Project”, Meszaros & Aston
http://dx.doi.org/10.1109/AGILE.2006.5
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
22

23. Agile & Lean UX resources that include user research
• “Adapting Usability Investigations for Agile User-centered Design”, Sy
(2007) http://www.upassoc.org/upa_publications/jus/
2007may/agile-ucd.pdf
• “Successful User Experience in an Agile Enterprise Environment”,
Federoff & Courage (2009)
http://link.springer.com/chapter/10.1007%2F978-3-642-02556-3_27
• Jeff Gothelf’s blog: http://www.jeffgothelf.com/blog/
• 5 tips for applying Lean UX to User Research, Jeff Sauro (2013)
http://www.measuringusability.com/blog/lean-research.php
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
23

24. Takeaways
• You and your team can incorporate user research into Agile
development environments
• Align research with the Agile development & design cycles
• Use lightweight research methods
• Use a few participants
• Require that key stakeholders attend (and/or conduct) user
research
• Get feedback at regular intervals (once a sprint)
• Get feedback on everything
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
24

25. Questions?
?
jen dot mcginn at oracle dot com & ana dot ramirez dot chang at oracle dot com
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
25

26. The RITE+Krug Method
Getting crazy good stakeholder involvement in Agile
user research
Jen McGinn & Ana Chang
December 5, 2013

27. Small, think-aloud study
Ease of Use
Large A/B study
Different versions of the UI
Jen McGinn and Ana Chang

28. A/B testing
• A validation technique that you can use when you have:
– Working code on the web
– Large numbers of participants
– A narrowly-defined problem
• “Which one of the designs results in fewer abandoned carts?”
• “Which one of the designs results in a higher click-through rate?”
“Unless you are a huge property like – like Bing or Google …
you’ll have more things to test than you have traffic
– Lean Analytics, Croll & Yaskovitz, Eric Reis, ed
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
28

29. “Formative” Feedback Sessions
• A discovery technique that you can use when you have:
– drawings, sketches, wireframes
– small numbers of participants
– questions about features, mental models, terminology, iconography,
usability of your prototypes, current practices, pain points, values,
design alternatives, or want to collect other qualitative data
“Initially, you’re looking for qualitative data … you’re speaking to
people … You’re exploring. You’re getting out of the building.”
– Lean Analytics, Croll & Yaskovitz, Eric Reis, ed
Jen McGinn and Ana Chang
PS SIG CHI Monthly Meeting, Dec 2013
29