Agile User Studies: Affordable, Iterative, Scalable, Team-Based User Feedback. Most Agile practitioners would agree that regular customer feedback is essential, but many teams struggle to obtain that information. Traditional usability tests seem too abstract, expensive, and time consuming. I will present a methodology that allows any team or practitioner to conduct affordable, iterative, team-based user studies. I will also share the result of applying this methodology at Cengage Learning over two years, multiple Agile teams, and dozens of projects.

1. Agile User Studies
Derek Poppink
User Experience Manager

2. Cengage Learning
Who we are
LIFELONG LEARNING RESEARCH
 Elementary and high school  Academic
 2- and 4-year colleges and  School
universities  Public
 Private/For Profit Colleges  Government
 Professional Training & Continuing
Education
 Annual revenue: $2 billion
 5,600 employees
 Operations in more than 20 countries

3. User Experience
What we do
http://www.limina-ao.com/approach/agile.html

4. User Experience
Why we do it
Peter Morville’s UX Honeycomb:
http://semanticstudios.com/publications/semantics/000029.php

5. Agile User Studies
Agenda
• Why User Studies?
• What is an Agile User Study?
• When
• Where
• What
• Who
• How
• Questions

6. Why User Studies?
•All sites have usability issues
•Serious issues are easy to find
•Watching users makes you a better designer

8. Big Honkin’ User Studies
http://www.laurasmith.info/UsabilityTest.jpg

9. Big Honkin’ User Studies
1-2 rounds of tests when site is nearly complete
1-2 days of tests per round
5-8 participants per round
Held on-site in room with one-way mirror
1-2 observers (product owner)
Dozens of problems prioritized by moderator
1 week to prepare report afterwards
$25,000 per round

10. Agile User Studies
http://www.uxforthemasses.com/blog/wp-content/uploads/2010/09/usability-testing-session.jpg

11. Agile User Studies
1 round of tests per sprint from concept to launch
1 morning per round (tests and debriefing)
3 participants per round
Held in conference rooms and remotely with WebEx
Observed by team
Ten problems and fixes prioritized by team
Findings and fixes distributed immediately afterwards
$4,000 per round

13. A Morning A Month
http://b.vimeocdn.com/ts/693/515/69351512_640.jpg

14. A Morning A Month
 Morning
 3 sessions (9, 10, 11)
 Debriefing over lunch
 Allows more people to attend
 Month (Sprint)
 Eliminates deciding when to test
 Finds enough problems to fix
 Affordable, iterative, scalable

15. Use Existing Spaces
http://portfolio.splendidnoise.com/

16. Or Test Remotely
 Conference room
 Laptop, projector, speakerphone, snacks
 Moderator, observers
 WebEx
 Remote participant
 Internet connection, phone
 Benefits
 Easier recruiting
 Easier scheduling
 No travel required
 Natural environment

17. Test What You Have

18. Test What
You Have

19. Test What You Have
Legacy
Materials products Mockups (Balsamiq)
Competitor products Wireframes (HotGloo)
Products in development Visual composites
Completed products Prototypes (HTML)

20. What Have We Tested?
Business & Company Resource Center Gale Virtual Reference Library
Business Insights: Global Illustrated London News
Books & Authors World Scholar: Latin America
Health Matters Slavery Anti-Slavery
Course Reader State Papers Online
Gale Admin Literature Resources from Gale
Nineteenth Century Collections Online National Geographic Digital Library

21. What Have We Tested?
Powerspeak PowerSearch
Galaxy Academic OneFile
Gale NewsVault Newsstand
Business Principles and Practices for the Science, Technology, Engineering, &
Salon and Spa Medicine (STEM)
Encyclopedia.com Opposing Viewpoints in Context
Public Library Websites HR Immersion Site

22. Recruit
http://farm7.static.flickr.com/6211/6351911040_9f7c92804b.jpg

23. Recruit
 Who?
 Representative users
 How many?
 Three per sprint, plus one backup
 How?
 Customers, personal networks
 Why?
 Amazon gift cards

24. Prepare Tasks
http://www.uqul.uq.edu.au

25. Prepare Tasks
 Pre-test questions
 Get participants talking
 Critical tasks
 Home – impressions
 Search – satisfaction
 Documents – outcomes
 Alternative designs
 Upcoming stories

26. Moderate
http://iutp.org/files/bunnyfoot/pics/blowup6.jpg

27. Moderate
 Tour guide
Give participant instructions
Keep things moving
 Therapist
Get participant talking
Show you are listening
Stay neutral
Respect participant, privacy

28. Observe
 Attend a session (or three)
 Avoid distractions
 Take notes
 Notice where user is confused or can’t complete tasks
 Make list
 Three most serious usability problems

29. Debrief
 Share observations
 Three most serious usability problems per observer
 Prioritize as a team
 Discuss easiest fix
 Tweak, don’t redesign
 Take something away
 Commit to changes

30. Agile User Studies
1 round of tests per sprint from concept to launch
1 morning per round (tests and debriefing)
3 participants per round
Held in conference rooms and remotely with WebEx
Observed by team
Ten problems and fixes prioritized by team
Findings and fixes distributed immediately afterwards
$4,000 per round