Turn Stakeholders into Data Collectors to to Uncover Testing Results… Quickly
•
1 recomendación•1,031 vistas
With all the data that usability testing produces, it can be difficult and time consuming to uncover the golden nuggets, and unfortunately you don't always have enough time to wait for the formal usability test report to impact the design process. Sound familiar? At Vistaprint, we've been piloting new ways of capturing and transforming test data into actionable insights to speed up the iterative design process. As a result we're finding that it not only increases our reaction time but improves collaboration and engagement across stakeholders.
Recomendados
Recomendados
Más contenido relacionado
Último
Último (20)
Destacado
Destacado (20)
Turn Stakeholders into Data Collectors to to Uncover Testing Results… Quickly
- 1. Turn Stakeholders into Data Collectors to Uncover Testing Results… Quickly Susan Rice & Lisa Spitz UX Center of Excellence, Global
- 2. Company Confidential Vistaprint © 2013 Overview 6/3/2013 2 • Who We Are • Identify the Problem • Describe the Idea • Initial Approaches • Revised Approach • Q&A + Discussion
- 3. Company Confidential Vistaprint © 2013 Who We Are 6/3/2013 3
- 4. Company Confidential Vistaprint © 2013 Vistaprint N.V. (Nasdaq: VPRT) 6/3/2013 4 We empower more than 15 million micro businesses and consumers annually with affordable, professional options to make an impression.
- 5. Company Confidential Vistaprint © 2013 Company Facts 6/3/2013 5 • 25 localized websites • 4,100 employees • 3 state of art manufacturing facilities • 21 offices worldwide
- 6. Company Confidential Vistaprint © 2013 6/3/2013 6 UX Center of Excellence
- 7. Company Confidential Vistaprint © 2013 The Problem 6/3/2013 7
- 8. Company Confidential Vistaprint © 2013 6/3/2013 8 BUILD DESIGN THINK CHECK
- 9. Company Confidential Vistaprint © 2013 6/3/2013 9 Session Notes Usability Report
- 10. Company Confidential Vistaprint © 2013 6/3/2013 10
- 11. Company Confidential Vistaprint © 2013 6/3/2013 11
- 12. Company Confidential Vistaprint © 2013 The Idea 6/3/2013 12
- 13. Company Confidential Vistaprint © 2013 Our Hypothesis 6/3/2013 13 Stakeholders Process Saved Time!
- 14. Company Confidential Vistaprint © 2013 Our Hypothesis 6/3/2013 14 Stakeholders Process Saved Time!
- 15. Company Confidential Vistaprint © 2013 6/3/2013 15 Don’t just sit there… Make yourself useful!
- 16. Company Confidential Vistaprint © 2013 Why Stakeholders? 6/3/2013 16 Collaboration Engagement Ownership
- 17. Company Confidential Vistaprint © 2013 Our Hypothesis 6/3/2013 17 Stakeholders Process Saved Time!
- 18. Company Confidential Vistaprint © 2013 Data Logging Practices 6/3/2013 18 • Problem Coding • Event Description • Event Description with Problem Coding • Event Description with Problem Coding & Video Time Stamps http://www.userfocus.co.uk/articles/dataloggingtools.html UPA 2004, Dr. David Dayton (Southern Polytechnic State University)
- 19. Company Confidential Vistaprint © 2013 Not Collaborative 6/3/2013 19
- 20. Company Confidential Vistaprint © 2013 Tomer’s Rainbow Spreadsheet 6/3/2013 20 http://uxdesign.smashingmagazine.com/2013/04/11/rainbow-spreadsheet-collaborative-ux-research-tool/ A Collaborative Lean UX Research Tool By Tomer Sharon
- 21. Company Confidential Vistaprint © 2013 Too High Tech, Mental Overload 6/3/2013 21
- 22. Company Confidential Vistaprint © 2013 Our Low-Tech Solution 6/3/2013 22
- 23. Company Confidential Vistaprint © 2013 Our Hypothesis 6/3/2013 23 Stakeholders Process Saved Time! ?
- 24. Company Confidential Vistaprint © 2013 Initial Approaches 6/3/2013 24
- 25. Company Confidential Vistaprint © 2013 6/3/2013 25
- 26. Company Confidential Vistaprint © 2013 Pilot 1: Approach 6/3/2013 26 • All stakeholders record observations on yellow sticky notes identified by participant # • At end of session, organize stickies by + / - according to site area
- 27. Company Confidential Vistaprint © 2013 Pilot 1: Lessons Learned 6/3/2013 27 • Errors • Inconsistencies • Redundancy • Too long to collate and make sense of the data
- 28. Company Confidential Vistaprint © 2013 Pilot 2: Approach 6/3/2013 28 • Stakeholders still record observations on stickies • Stickies were color coded by participant
- 29. Company Confidential Vistaprint © 2013 Pilot 2: Lessons Learned 6/3/2013 29 • Still too much redundancy • Stakeholders didn’t feel they were adding value
- 30. Company Confidential Vistaprint © 2013 Pilot 3: Approach 6/3/2013 30 • Stakeholders still record observations on stickies • UX actively culls observations during and between sessions • Identify # of participants making similar comments • Duplicate stickies are tossed
- 31. Company Confidential Vistaprint © 2013 Pilot 3: Lessons Learned 6/3/2013 31 • Requires one person to be there to manage stickies • Need more time between sessions • Redundancy is still an issue, but is managed on the fly
- 32. Company Confidential Vistaprint © 2013 What Worked 6/3/2013 32 • People were engaged • Stimulated discussion • Immediate readout of observations • Richer set of observations • Creating the usability test report was easier because the data was already compiled
- 33. Company Confidential Vistaprint © 2013 What Didn’t Work 6/3/2013 33 • Errors in participant identification • Inconsistent documentation • Tons of redundancy • No consistent way to obtain quantitative feedback • Observations were too general • We didn’t know what to record • Not saving enough time!
- 34. Company Confidential Vistaprint © 2013
- 35. Company Confidential Vistaprint © 2013 Revised Approach 6/3/2013 35
- 36. Company Confidential Vistaprint © 2013 6/3/2013 36 One size fits all NOT!
- 37. Company Confidential Vistaprint © 2013 Product Development Phases 6/3/2013 37 http://www.nngroup.com/articles/which-ux-research-methods/ Strategize Optimize Assess Goal: Inspire, explore and choose new directions and opportunities Inform and optimize designs in order to reduce risk and improve usability Measure product performance against itself or its competition Approach: Qualitative and Quantitative Mainly Qualitative (formative) Mainly Quantitative (summative) Typical Methods: Ethnographic field studies, focus groups, diary studies, surveys, data mining or analytics Cardsorting, field studies, participatory design, paper prototype and usability studies, desirability studies, customer emails Usability benchmarking, online assessments, surveys, A/B testing
- 38. Company Confidential Vistaprint © 2013 Two Types of Tests 6/3/2013 38 Simple Site Flows Complex Interactions All Studies
- 39. Company Confidential Vistaprint © 2013 All Studies 6/3/2013 39 All Stakeholders
- 40. Company Confidential Vistaprint © 2013 Two Types of Tests 6/3/2013 40 Simple Site Flows Complex Interactions All Studies
- 41. Company Confidential Vistaprint © 2013 Simple Site Flows 6/3/2013 41
- 42. Company Confidential Vistaprint © 2013 Simple Site Flows 6/3/2013 42 Key Takeaway: What Worked? Key Takeaway: What Didn’t Work?
- 43. Company Confidential Vistaprint © 2013 Two Types of Tests 6/3/2013 43 Simple Site Flows Complex Interactions All Studies
- 44. Company Confidential Vistaprint © 2013 Complex Interactions 6/3/2013 44
- 45. Company Confidential Vistaprint © 2013 Complex Interactions 6/3/2013 45 Key Takeaway: What Worked? Key Takeaway: What Didn’t Work?
- 46. Company Confidential Vistaprint © 2013 Complex Interactions 6/3/2013 46 P1 P2 P3 P4 P5 1. First they did … 2. 3. 4. 5. 1 Note taker
- 47. Company Confidential Vistaprint © 2013 Questions + Discussion 6/3/2013 47
- 48. Company Confidential Vistaprint © 2013 6/3/2013 48 General questions? Your test questions? Note taking ideas? Unique challenges?
- 49. Company Confidential Vistaprint © 2013 Thanks! 6/3/2013 49 Special thanks to the UX CoE team for their fantastic editing expertise… and for gathering supporting examples for the presentation. Susan Rice leads UX Design srice@vistaprint.com Lisa Spitz is a Senior UX Designer lspitz@vistaprint.com
Notas del editor
- To change the layout of any slide: Right click on the slide and scroll down to layout Choose the appropriate layout option If images and text need to be reset, right click and choose “reset slide”Slide Name: Title Slide
- Who We Are – 2 min (never starts on time)Identify the Problem – 3 minDescribe the Idea – 7 minShare Our Results – 13 minQuestions + Discussion – 15 min
- Vistaprint N.V. (Nasdaq: VPRT) empowers more than 15 million micro businesses and consumers annually with affordable, professional options to make an impression.
- UX CoE - Created in October 20117 UX Designers with over 75 years of combined UX experience2 UX ResearchersWe focus on delighting our customers by creating an effortless enjoyable experience based on user centered design, customer research, and iterative improvements.
- As our design process became more iterative, it meant that we needed customer feedback… FAST!
- We could no longer wait for the formal usability report, although still useful for documentation, executive buy in, etc.
- We needed design direction immediately to stay on schedule. Yet, each UX designer didn’t have the luxury of watching all 10 usability sessions.
- What to do…?
- High Level before getting into the details Utilize people and processes to save time in the iterative design processPeopleLow tech processCapture dataSort dataCapture trends across participants
- High Level before getting into the details Utilize people and processes to save time in the iterative design processPeopleLow tech processCapture dataSort dataCapture trends across participants
- We already have a lab at VPWe already have a lot of interest from stakeholders in observing usability testingSO why not make better use of their time!
- Collaboration – Sharing the workload to obtain resultsEngagement – Stakeholders are more involved in the usability testing processOwnership – Everyone owns the findingsSpeed! – Faster, better, more comprehensive feedback that can be used in an iterative design process – cuz we’re still a lean UX team!
- High Level before getting into the details Utilize people and processes to save time in the iterative design processPeopleLow tech processCapture dataSort dataCapture trends across participants
- To be clear, we are not the first ones to think about this. There is a lot of information out there on Data Logging techniques. Many leading to digital tools for quickly capturing data and turning it into a usability test report. There are different techniques for logging Usability Test Data in general. Often done by the usability practitioner him/herselfAt UPA 2004, Dr. David Dayton presented details of how 6 different practitioners (including himself) logged usability data. These were the 4 most popular types of Data Logging PracticesProblem Coding Record predictable events and sort them on the fly into one or more categories. Analyse the resulting quantitative data with statistical methods and compare to pre-defined benchmarks to assess the usability of the product. Event Description Records free-form handwritten notes to capture significant events and/or usability problems. Analyze notes post-test, group and categorize events, and rate their severity. Event Description with Problem Coding Code events into certain pre-set categories, and enter descriptive notes for later team review and discussion of the most significant problems.Event Description with Problem Coding & Video Time Stamps Capture the "story" of a test session in shorthand notes.
- With varying degrees on pros and cons depending on which approach taken
- Then there’s the Rainbow Speadsheet by Tomer Sharon – hi tech approach which uses a Google Doc for capturing test data. Each test stakeholder has access to and is responsible for entering observations. With it, you will be able to collaboratively observe UX research sessions with team members (or clients). You will be able to conduct research that involves the entire product team, with results that are turned around quickly and that team members will be committed to acting on. And all of this without writing a formal exhaustive research report that no one wants to read.Requires “homework” for familiarity and access to Google Docs. Ultimately, we wanted something low tech with less mental overload for stakeholders.
- But we were hoping for something more visible and interactive – where we could see trends throughout the course of the test sessions and quickly organize the visible data into a set of priorities
- High Level before getting into the details Utilize people and processes to save time in the iterative design processPeopleLow tech processCapture dataSort dataCapture trends across participants
- We took an iterative approach to field testing some ways of capturing data in a low-tech manner
- Let all stakeholders record all observations on yellow sticky notes identified by participant #Easy for stakeholdersAt end of session, organize stickies by + and – by site area
- Inconsistencies – example: funny quote documented by all, something deemed important only documented by UX Not consistently documenting across participants
- One example where we separated features and had stakeholders document observations and comments by + and -. Another example where we had observers document by step in the flow.
- Weren’t as engaged over time – due to redundancies
- One example where we separated features and had stakeholders document observations and comments by + and -. Another example where we had observers document by step in the flow.
- One example where we separated features and had stakeholders document observations and comments by + and -. Another example where we had observers document by step in the flow.
- So now what? How do we learn from our pilot tests to remove the redundancy and improve our time spent with the data yet add value for stakeholders? How do we squeeze more out of them?
- There are many different kinds of testsHere are a couple examples. We are mainly focused here in the middleYou may need to change your approach based on the kind of test, the research questions, and your place in the process
- Kind of test, research questions, place in process
- Kind of test, research questions, place in process
- Kind of test, research questions, place in process