Recommended
More Related Content
Featured
Featured (20)
The Quality Of Use Metric Stlux11
- 1. The Quality of Use Metric Mike Coble Coble Usability STLUX’11
- 2. Who Am I? • Coble Usability • User Experience Analyst • User-Centered Design Process Facilitator • Software for Knowledge Workers
- 3. Who Are You? • UX Professionals • Designers • Analysts • Project/Product Managers • Software Developers
- 4. Preview • Guiding Technology to Enhance Work • Quality of Use Metric • How it’s Assessed • What it Means • Validation & Claims • How it’s Used
- 5. Technology to Enhance Work Work Enhanced Work $ Technology
- 6. Tasks Work Enhanced Work $ Technology • Work is chunked into tasks.
- 7. Task Experience • How well does it go? • Effectiveness • Efficiency • Satisfaction
- 8. Task Experience Spectrum
- 9. Task Experience Spectrum Best Worst
- 10. Task Experience Spectrum Best It's actually very powerful. Worst
- 11. Task Experience Spectrum Best It's actually very powerful. It's annoying. Worst
- 12. Task Experience Spectrum Best It's actually very powerful. It's annoying. It's not obvious. Worst
- 13. Task Experience Spectrum Best It's actually very powerful. It's not obvious. It's annoying. I’d have to spend a lot of time with help to understand. Worst
- 14. Task Experience Spectrum Best It's actually very powerful. It's not obvious. I’d have to spend a It's lot of time with help to annoying. understand. I've had a little more experience, so it was easier for me. Worst
- 15. Task Experience Spectrum Best It's actually very powerful. I've had a little more experience, so it was easier for me. It's not obvious. I’d have to spend a It's lot of time with help to annoying. understand. Well, that didn’t work! Worst
- 16. Task Experience Spectrum Best It's actually very powerful. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's lot of time with help to annoying. understand. What am I supposed to put here? Worst
- 17. Task Experience Spectrum Best It's actually very powerful. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's lot of time with help to annoying. What am I supposed understand. to put here? Nice! Worst
- 18. Task Experience Spectrum Best It's actually very Nice! powerful. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's lot of time with help to annoying. What am I supposed understand. to put here? Wow! That’s impressive! Worst
- 19. Task Experience Spectrum Best Wow! That’s impressive! It's actually very Nice! powerful. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's lot of time with help to annoying. What am I supposed understand. to put here? I don’t understand this! Worst
- 20. Task Experience Spectrum Best Wow! That’s impressive! It's actually very Nice! powerful. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's I don’t understand this! lot of time with help to annoying. What am I supposed understand. to put here? What were they thinking? Worst
- 21. Task Experience Spectrum Best Wow! That’s impressive! It's actually very Nice! powerful. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's I don’t understand this! lot of time with help to annoying. What am I supposed understand. to put here? What were they thinking? You can’t be serious! Worst
- 22. Task Experience Spectrum Best Wow! That’s impressive! It's actually very Nice! powerful. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's I don’t understand this! lot of time with help to annoying. What am I supposed understand. You can’t to put here? be serious! What were they thinking? $#!^!!!!! Worst
- 23. Task Experience Spectrum Best Wow! That’s impressive! It's actually very Nice! powerful. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's I don’t understand this! lot of time with help to annoying. What am I supposed understand. You can’t to put here? be serious! $#!^!!!!! What were they thinking? It’s pretty easy once you figure it out. Worst
- 24. Task Experience Spectrum Best Wow! That’s impressive! It's actually very Nice! powerful. It’s pretty easy once you figure it out. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's I don’t understand this! lot of time with help to annoying. What am I supposed understand. You can’t to put here? be serious! $#!^!!!!! What were they thinking? Not bad. Worst
- 25. Task Experience Spectrum Best Wow! That’s impressive! It's actually very Nice! powerful. Not bad. It’s pretty easy once you figure it out. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's I don’t understand this! lot of time with help to annoying. What am I supposed understand. You can’t to put here? be serious! $#!^!!!!! What were they thinking? That was great! Worst
- 26. Task Experience Spectrum Best Wow! That’s impressive! It's actually very Nice! powerful. That was great! Not bad. It’s pretty easy once you figure it out. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's I don’t understand this! lot of time with help to annoying. What am I supposed understand. You can’t to put here? be serious! $#!^!!!!! What were they thinking? It seems to know what I want to do! Worst
- 27. Task Experience Spectrum Best Wow! That’s impressive! It's actually very Nice! powerful. It seems to know what I want to do! That was great! Not bad. It’s pretty easy once you figure it out. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's I don’t understand this! lot of time with help to annoying. What am I supposed understand. You can’t to put here? be serious! $#!^!!!!! What were they thinking? Why are we back here? We failed! Worst
- 28. Task Experience Spectrum Best Wow! That’s impressive! It's actually very Nice! powerful. It seems to know what I want to do! That was great! Not bad. It’s pretty easy once you figure it out. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's I don’t understand this! lot of time with help to annoying. What am I supposed understand. You can’t to put here? be serious! $#!^!!!!! Why are we back here? We failed! What were they thinking? I feel like an idiot! Worst
- 29. Task Experience Spectrum Best Wow! That’s impressive! It's actually very Nice! powerful. It seems to know what I want to do! That was great! Not bad. It’s pretty easy once you figure it out. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's I don’t understand this! lot of time with help to annoying. What am I supposed understand. You can’t to put here? be serious! $#!^!!!!! Why are we back here? We failed! I feel like an idiot! What were they thinking? Now I’m really annoyed! Worst
- 30. Task Experience Spectrum Best Wow! That’s impressive! It's actually very Nice! powerful. It seems to know what I want to do! That was great! Not bad. It’s pretty easy once you figure it out. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's I don’t understand this! lot of time with help to annoying. What am I supposed understand. You can’t to put here? be serious! $#!^!!!!! Why are we back here? We failed! I feel like an idiot! What were they thinking? Now I’m really This has annoyed! been a complete waste of time! Worst
- 31. Task Experience Spectrum Best Wow! That’s impressive! It's actually very Nice! powerful. It seems to know what I want to do! That was great! Not bad. It’s pretty easy once you figure it out. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's I don’t understand this! lot of time with help to annoying. What am I supposed understand. You can’t to put here? be serious! $#!^!!!!! Why are we back here? We failed! I feel like an idiot! What were they thinking? This has Now I’m really How about that! been a complete waste of time! annoyed! It’s just that easy! Worst
- 32. Task Experience Spectrum Best Wow! That’s impressive! It's actually very How about that! Nice! powerful. It’s just that easy! It seems to know what I want to do! That was great! Not bad. It’s pretty easy once you figure it out. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's I don’t understand this! lot of time with help to annoying. What am I supposed understand. You can’t to put here? be serious! $#!^!!!!! Why are we back here? We failed! I feel like an idiot! What were they thinking? This has Now I’m really I give up! been a complete waste annoyed! of time! Worst
- 33. Task Experience Spectrum Best Wow! That’s impressive! It's actually very How about that! Nice! powerful. It’s just that easy! It seems to know what I want to do! That was great! Not bad. It’s pretty easy once you figure it out. I've had a little more experience, so it was easier for me. It's not obvious. Well, that didn’t work! I’d have to spend a It's I don’t understand this! lot of time with help to annoying. What am I supposed understand. You can’t to put here? be serious! $#!^!!!!! Why are we back here? We failed! I feel like an idiot! What were they thinking? This has Now I’m really been a complete waste annoyed! of time! I give up! Worst
- 34. Quality of Use Metric 100 0
- 35. How is the Quality of Use Metric Assessed?
- 36. Quality of Use Metric Calculation Please place marks on the lines below to reflect your experience. Hard |--------------------------------------------------| Easy Frustrated |--------------------------------------------------| Satisfied Failure |--------------------------------------------------| Success The time it took: More |-------------------------|------------------------| Less As Expected
- 37. Quality of Use Metric Calculation Please place marks on the lines below to reflect your experience. Hard |--------------------------------------------------| Easy Frustrated |--------------------------------------------------| Satisfied Failure |--------------------------------------------------| Success The time it took: More |-------------------------|------------------------| Less As Expected
- 38. Holistic User Experience • Usability • Ease of Use • Ease of Learning • Traditional Software Quality • Performance • Validity • Reliability • Common Basis to Judge Impact on User
- 39. Human Factor • The metric does not replace human judgement.
- 40. Aggregate • Hard/Easy • Frustrated/Satisfied • Failure/Success • Time vs. Expectated
- 41. My Experience
- 42. My Experience • Balanced Formative/Summative Usability Tests • Learn about & measure the experience
- 43. My Experience • Balanced Formative/Summative Usability Tests • Learn about & measure the experience • “Bleeding Edge” • Fairly “low” Quality of Use
- 44. What does the Quality of Use Metric Mean?
- 45. Quality of Use Metric Meaning Quality of Use Interpretation Users will consistently accomplish their tasks, 90-100 will sometimes exceed users’ expectations. 80-90 Most users will easily accomplish their tasks. Users will accomplish their tasks, but many 70-80 rough edges will impact users -- barely acceptable. Many users will accomplish their tasks, but 60-70 many will reject -- serviceable. Most users will not succeed and will reject -- <60 varying degrees of unacceptable.
- 46. Quality of Use Metric Meaning Quality of Use Interpretation Users will consistently accomplish their tasks, 90-100 will sometimes exceed users’ expectations. 80-90 Most users will easily accomplish their tasks. Users will accomplish their tasks, but many 70-80 rough edges will impact users -- barely acceptable. Many users will accomplish their tasks, but 60-70 many will reject -- serviceable. Most users will not succeed and will reject -- <60 varying degrees of unacceptable.
- 47. Quality of Use Metric Meaning Quality of Use Interpretation Users will consistently accomplish their tasks, 90-100 will sometimes exceed users’ expectations. 80-90 Most users will easily accomplish their tasks. Users will accomplish their tasks, but many 70-80 rough edges will impact users -- barely acceptable. Many users will accomplish their tasks, but 60-70 many will reject -- serviceable. Most users will not succeed and will reject -- <60 varying degrees of unacceptable.
- 48. Quality of Use Metric Meaning Quality of Use Interpretation Users will consistently accomplish their tasks, 90-100 will sometimes exceed users’ expectations. 80-90 Most users will easily accomplish their tasks. Users will accomplish their tasks, but many 70-80 rough edges will impact users -- barely acceptable. Many users will accomplish their tasks, but 60-70 many will reject -- serviceable. Most users will not succeed and will reject -- <60 varying degrees of unacceptable.
- 49. Quality of Use Metric Meaning Quality of Use Interpretation Users will consistently accomplish their tasks, 90-100 will sometimes exceed users’ expectations. 80-90 Most users will easily accomplish their tasks. Users will accomplish their tasks, but many 70-80 rough edges will impact users -- barely acceptable. Many users will accomplish their tasks, but 60-70 many will reject -- serviceable. Most users will not succeed and will reject -- <60 varying degrees of unacceptable.
- 50. Quality of Use Metric Meaning Quality of Use Interpretation Users will consistently accomplish their tasks, 90-100 will sometimes exceed users’ expectations. 80-90 Most users will easily accomplish their tasks. Users will accomplish their tasks, but many 70-80 rough edges will impact users -- barely acceptable. Many users will accomplish their tasks, but 60-70 many will reject -- serviceable. Most users will not succeed and will reject -- <60 varying degrees of unacceptable.
- 51. Quality of Use is Correlated
- 52. Quality of Use is Correlated • Weighted user success rate
- 53. Quality of Use is Correlated • Weighted user success rate • Sales demo time explaining benefits
- 54. Quality of Use is Correlated • Weighted user success rate • Sales demo time explaining benefits • Training time on benefits rather than mechanics
- 55. Quality of Use is Correlated • Weighted user success rate • Sales demo time explaining benefits • Training time on benefits rather than mechanics • Documentation devoted to how to take advantage of the software
- 56. Quality of Use is Correlated • Weighted user success rate • Sales demo time explaining benefits • Training time on benefits rather than mechanics • Documentation devoted to how to take advantage of the software • Users’ time spent thinking about their work
- 57. Quality of Use Predicts User Adoption
- 58. Lab Results Predict Real World Use • Strong correlation on pre-release software • usability testing in lab (Quality of Use) • observation of actual work (Weighted User Success)
- 59. Lab Results Predict Real World Use • Strong correlation on pre-release software • usability testing in lab (Quality of Use) • observation of actual work (Weighted User Success) • Same relative impacts of issues in • Reliability (crashes) • Usability • Capability (missing functionality)
- 60. Prototype vs Software • Can’t test with paper prototype: • Performance & Reliability • Software QofU = PP QofU - performance - reliability
- 61. Experienced Observers Can Estimate QofU
- 62. Experienced Observers Can Estimate QofU • Team observed series of tests (iterative refinement)
- 63. Experienced Observers Can Estimate QofU • Team observed series of tests (iterative refinement) • During one iteration, observers asked to estimate QofU for each user/task.
- 64. Experienced Observers Can Estimate QofU • Team observed series of tests (iterative refinement) • During one iteration, observers asked to estimate QofU for each user/task. • Individual observers didn’t correlate well with specific user/task, but...
- 65. Experienced Observers Can Estimate QofU • Team observed series of tests (iterative refinement) • During one iteration, observers asked to estimate QofU for each user/task. • Individual observers didn’t correlate well with specific user/task, but... • Overall averages were virtually identical!
- 66. Target Users & Work
- 67. Target Users & Work • QofU = f(software, users, tasks)
- 68. Target Users & Work • QofU = f(software, users, tasks) • Can’t be determined by inspection • only by observation of users doing realistic work
- 69. Target Users & Work • QofU = f(software, users, tasks) • Can’t be determined by inspection • only by observation of users doing realistic work • If the software has well defined/understood target user profile and supported task profile • QofU = f(software) [users, tasks implied]
- 70. Self Calibrating • Experience rated relative to expectations • Allows Variation • user familiarity • task difficulty • software versions • Sub-Population Variance?
- 71. How many users? • 3 users = 48 data points
- 72. How many users? Calculated Quality of Use After N Users 100 80 • 3 users = 48 data 60 points 40 20 0 1 5 9 13 17 21
- 73. How is the Quality of Use Metric Used?
- 74. Refinement Quality of Use 100 80 60 40 20 0 Design 1 Design 2 Development Release
- 75. Dramatic Improvement 100 80 60 40 20 0 Before After
- 76. Other Measures • Objective Success • Time to Complete
- 77. Issue Impact
- 78. Issue Impact • QofU Gap = 100 - QofU • Estimate impact of each issue • distribute QofU Gap among issues • based on user/task QofU ratings and objective experience
- 79. Issue Impact • QofU Gap = 100 - QofU • Estimate impact of each issue • distribute QofU Gap among issues • based on user/task QofU ratings and objective experience Issue Value Overflow error displaying structures with rows selected 6.0 Labels on last row of each page are displaced down. 2.0 User expected nothing selected to mean use everything. 0.8
- 80. Guide for QofU Impact of Incidents Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 100 failure
- 81. Exercise 82 Usability Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 100 failure
- 82. Exercise 82 Usability Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 100 failure
- 83. Exercise 82 Usability Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 100 failure
- 84. Exercise 82 Usability Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 100 failure
- 85. Exercise 82 Usability Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 19 100 failure
- 86. Exercise 82 Usability Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 19 100 failure
- 87. Exercise 82 Usability Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 19 100 failure
- 88. Exercise 82 Usability Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 19 100 failure
- 89. Exercise 82 Usability Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 19 37 100 failure
- 90. Exercise 82 Usability Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 19 37 100 failure
- 91. Exercise 82 Usability Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 19 37 100 failure
- 92. Exercise 82 Usability Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 19 37 100 failure
- 93. Exercise 82 Usability Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 19 37 27 100 failure
- 94. Exercise 82 Usability Efficiency Impact Effectiveness Impact Satisfaction Impact No time lost 0 No impact on success 0 No frustration 0 1 second 1 May degrade results 1 Slightly displeased 1 5 seconds 6 Contributes to less Significantly disturbed 5 5 10 seconds 9 than full success Strongly annoyed 10 1 minute 17 Causes less than full Degrades impression 15 10 success 3 minutes 22 Deeply frustrated 20 Contributes to failure 15 5 minutes 25 Angry, Distressed 25 Major contribution to 1 hour 41 20 Completely infuriated 100 failure Causes failure 25 Certain to cause 19 37 27 100 failure 4 8 10
- 95. Estimate Software Qualities Gaps • Identify the Software Quality for each issue • E.g., Ease of Learning, Reliability,Validity • Sum impacts by software qualities Functionality Usability Reliability Quality of Use
- 96. Closing the QofU Gap Prioritizing Fixes • Estimate the cost of addressing each issue
- 97. Closing the QofU Gap Prioritizing Fixes • Estimate the cost of addressing each issue • Calculate the value/cost of each issue
- 98. Closing the QofU Gap Prioritizing Fixes • Estimate the cost of addressing each issue • Calculate the value/cost of each issue • Sort the issues by decreasing value/cost
- 99. Closing the QofU Gap Prioritizing Fixes • Estimate the cost of addressing each issue • Calculate the value/cost of each issue • Sort the issues by decreasing value/cost • Draw a line to set the cutoff • reach deadline • hit the target Quality of Use • optimize value/cost
- 100. Closing the QofU Gap Illustration Difficulty Value Cost /Cost Value Cum (hrs) Value Cum Cost Baseline: At Time of Test 72.0 240 Overflow error displaying structures with rows selected 6.0 1 6.000 78.0 241 Labels on last row of each page are displaced down. 2.0 2 1.000 80.0 243 User expected nothing selected to mean use everything. 0.8 1 0.825 80.8 244 … User confused that added Mol Weight column had 6 decimals 0.3 100 0.003 99.5 1250 (more than before). User tried to select "whole thing" and couldn't select query cell. 0.3 120 0.003 99.8 1370 User believed substituents wrong. 0.2 100 0.002 100.0 1470
- 101. Closing the QofU Gap Illustration Difficulty Value Cost /Cost Value Cum (hrs) Value Cum Cost Baseline: At Time of Test 72.0 240 Overflow error displaying structures with rows selected 6.0 1 6.000 78.0 241 Labels on last row of each page are displaced down. 2.0 2 1.000 80.0 243 Today User expected nothing selected to mean use everything. 0.8 1 0.825 80.8 244 … User confused that added Mol Weight column had 6 decimals 0.3 100 0.003 99.5 1250 (more than before). User tried to select "whole thing" and couldn't select query cell. 0.3 120 0.003 99.8 1370 User believed substituents wrong. 0.2 100 0.002 100.0 1470
- 102. Issue Impact: Benefits • Trade-off ALL further work • fix bugs, add features, change design, ... • Stay focused on the users needs during the "end game" • Making these estimates is time spent thinking about what meeting the users' needs means
- 103. Recommended Response 100 Interpretation Diagnosis Recommendation polish the implementation 90 works for users the implementation works through Usability Testing 80 refine the design & basically works for users with 70 the design basically works implementation through significant apparent obstacles Usability Testing 60 barely works for users with the design does not work reconsider the design through many significant obstacles, and should be reconsidered Paper Prototyping many not apparent the users' needs are not well reconsider the need through does not work for users understood Contextual Inquiry 0
- 104. Review • Build technology that enhances work by supporting peoples’ tasks • Measure how well a design/software supports the tasks • Guide software development to create an optimal user experience
- 105. Contact • Mike Coble • Coble Usability • www.cobleusability.com • mike@cobleusability.com
Editor's Notes
- This talk explains how I have developed and used the Quality of Use metric which has moved Usability from a matter of opinion to a matter of measurement.\n\n
- more of an analyst than a designer\nfacilitate contextual inquiry, paper prototyping, usability testing\ndesktop and web-based software for knowledge workers \n
- How many?\nOthers?\n
- First I&#x2019;ll talk a bit about ...\n
- I&#x2019;m talking about software being built to enhance some work. As opposed to entertainment software or software used by consumers buying products.\n\n
- Work can be chunked into tasks.\n
- When people try to accomplish tasks, it may go well or poorly. The task experience comes down to three aspects: effectiveness, efficiency, and satisfaction. \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- We can think & talk about a continuum of task experience from very good to very bad. \n\nWe&#x2019;re going to try an experiment. I&#x2019;m going to put up quotes from users trying to do tasks. I want you to think about how good or bad their experience was. I will be able to read the group consensus and put the quote on a spectrum from good to bad.\n\nI&#x2019;ll put up this spectrum to help you. When you see the quote just concentrate on the color where you want it placed.\n\nLet&#x2019;s start. <click> ... <clicks>\n\nGood. We have a reasonably shared notion of what is a good and bad Task Experience. You may have figured out I can&#x2019;t really read your minds and that isn&#x2019;t how we assess this. But it does illustrate that their is an intuitive continuum. If only we could quantify it... \n
- When the task involves use of a tool, we can call that task experience Quality of Use. \n\nWe report Quality of Use as a number between 0 and 100 meant to indicate how well a product, function, or design supports its intended users and uses.&#xA0; A Quality of Use of 100 means users are always completely successful and satisfied; they find doing work to be very easy and anything new takes much less time than they expect. A Quality of Use of 0 means users are always completely unsuccessful and frustrated; they find trying to accomplish anything at all to be impossibly hard and time consuming.\n\n
- \n
- Usability tests: We have people who reasonably match the target user profile work with the software to perform realistic tasks.\nAfter the user finishes each task we ask the user to complete a brief form reflecting their experience. It asks how hard or easy it was, how frustrated or satisfied they are, if they feel they failed or succeeded, and how the time it took compared to their expectations.&#xA0; They place marks on 4 lines.\n\nWe turn their marks into a number between 0 and 100. We do this for each task (we typically have a user work from 1 to 5 tasks) and for each user (we typically have 3 to 4 users for a round of testing). We average the scores from all the tasks with all the users. That's the number we report as Quality of Use for the specific software or design at that time with the target users and tasks. &#xA0;<click>\nThe form asks about, and the Quality of Use metric measures, the users' experience.&#xA0; Most people (users and developers) naturally tend to focus on the software, the thing, the solution and describe it as good or bad.&#xA0; Our focus is on the users' experience with the software. So we don't ask the user various questions about the software ("did it have clear wording?", "did it give the right answer?") but instead ask about the user's experience with the software ("were you successful?" "do you feel satisfied?").\n\n
- The Usability Testing protocol and Quality of Use metric incorporate all aspects of the users' experience.&#xA0; These aspects include not only ease of use and ease of learning but also performance, validity, and reliability traditionally associated with software quality.&#xA0; For example, if the software crashes during a usability test, the usability test continues while the user recovers from the crash (if possible).&#xA0; The whole experience -- including crashes, poor feedback, long run time, undiscoverable functions, etc. -- is evaluated in the Quality of Use.&#xA0; In this way, all issues with the software can be judged on an equal basis as to their impact on the user (in particular, we learn that not all crashes are equal -- some are annoying and some are devastating).\n\n
- Once we have calculated the Quality of Use Metric from the users&#x2019; marks, we always make a judgement whether it seems to accurately reflect the users' experience. The objective outcome may not match the users' perception. The facilitator may provide substantive assistance without which the participant may have taken much more time or may not have completed the task at all and the participant may or may not take that into account when they make their marks. The participant may feel they were successful (and mark accordingly) when they objectively were not (generally we don't tell them at all and certainly not before they make their marks).\n\n
- We don't look very closely at the individual components (hard/easy, frustrated/satisfied, failure/success, time vs. expected). Instead, we see them as 4 measures considering different aspects of the user's experience. I suppose there might be something of interest to look at if one product did well in terms of success but poorly in terms of satisfaction while with another product people felt satisfied although they didn't feel successful. But we've never felt the need to do this kind of analysis.\n\n
- The style of usability testing generally used attempts to be both formative and summative. In other words, try to facilitate in such a way to both learn about the experience and measure that experience as best we can.\n<click>\nThe designs and software I&#x2019;ve tested tend to be complex and we&#x2019;re working on moving the QofU from bad to good. Not sure how this holds up with trying to make good designs outstanding.\n\n
- The style of usability testing generally used attempts to be both formative and summative. In other words, try to facilitate in such a way to both learn about the experience and measure that experience as best we can.\n<click>\nThe designs and software I&#x2019;ve tested tend to be complex and we&#x2019;re working on moving the QofU from bad to good. Not sure how this holds up with trying to make good designs outstanding.\n\n
- \n
- Clearly, higher Quality of Use scores are better. But several years of experience have taught us what different numbers mean. \n\nCorrelating the Quality of Use score with other data from usability tests (such as objective indications of success and verbal characterizations of the experience) and with observation of real world use has allowed us to develop the following approximate scale:\n<click>\n90+ <click> 80+ <click> 70+ <click> 60+ <click> <60 \n\n(Coincidentally & conveniently, these roughly correspond to the ranges for assigning grades -- A, B, C, D, F -- in many schools.)\n\nBased on these interpretations, we often set a Quality of Use of 80 as a minimal requirement for software development.\n\n\n
- Clearly, higher Quality of Use scores are better. But several years of experience have taught us what different numbers mean. \n\nCorrelating the Quality of Use score with other data from usability tests (such as objective indications of success and verbal characterizations of the experience) and with observation of real world use has allowed us to develop the following approximate scale:\n<click>\n90+ <click> 80+ <click> 70+ <click> 60+ <click> <60 \n\n(Coincidentally & conveniently, these roughly correspond to the ranges for assigning grades -- A, B, C, D, F -- in many schools.)\n\nBased on these interpretations, we often set a Quality of Use of 80 as a minimal requirement for software development.\n\n\n
- Clearly, higher Quality of Use scores are better. But several years of experience have taught us what different numbers mean. \n\nCorrelating the Quality of Use score with other data from usability tests (such as objective indications of success and verbal characterizations of the experience) and with observation of real world use has allowed us to develop the following approximate scale:\n<click>\n90+ <click> 80+ <click> 70+ <click> 60+ <click> <60 \n\n(Coincidentally & conveniently, these roughly correspond to the ranges for assigning grades -- A, B, C, D, F -- in many schools.)\n\nBased on these interpretations, we often set a Quality of Use of 80 as a minimal requirement for software development.\n\n\n
- Clearly, higher Quality of Use scores are better. But several years of experience have taught us what different numbers mean. \n\nCorrelating the Quality of Use score with other data from usability tests (such as objective indications of success and verbal characterizations of the experience) and with observation of real world use has allowed us to develop the following approximate scale:\n<click>\n90+ <click> 80+ <click> 70+ <click> 60+ <click> <60 \n\n(Coincidentally & conveniently, these roughly correspond to the ranges for assigning grades -- A, B, C, D, F -- in many schools.)\n\nBased on these interpretations, we often set a Quality of Use of 80 as a minimal requirement for software development.\n\n\n
- Clearly, higher Quality of Use scores are better. But several years of experience have taught us what different numbers mean. \n\nCorrelating the Quality of Use score with other data from usability tests (such as objective indications of success and verbal characterizations of the experience) and with observation of real world use has allowed us to develop the following approximate scale:\n<click>\n90+ <click> 80+ <click> 70+ <click> 60+ <click> <60 \n\n(Coincidentally & conveniently, these roughly correspond to the ranges for assigning grades -- A, B, C, D, F -- in many schools.)\n\nBased on these interpretations, we often set a Quality of Use of 80 as a minimal requirement for software development.\n\n\n
- Clearly, higher Quality of Use scores are better. But several years of experience have taught us what different numbers mean. \n\nCorrelating the Quality of Use score with other data from usability tests (such as objective indications of success and verbal characterizations of the experience) and with observation of real world use has allowed us to develop the following approximate scale:\n<click>\n90+ <click> 80+ <click> 70+ <click> 60+ <click> <60 \n\n(Coincidentally & conveniently, these roughly correspond to the ranges for assigning grades -- A, B, C, D, F -- in many schools.)\n\nBased on these interpretations, we often set a Quality of Use of 80 as a minimal requirement for software development.\n\n\n
- I believe the Quality of Use Metric is correlated with a number of other measures of product success.\n<click>\nThe Quality of Use Metric seems to often match a &#x201C;weighted user success rate&#x201D;: count success, failure, partial success as half. In one unusually large study, we had a weighted success rate that exactly corroborated the Quality of Use Metric of 55. We rarely have a study large enough to do a meaningful comparison but we do feel that this generally holds.\n<click>\nIn sales demo situations, the lower the QofU rating, the more time the demonstrator will have to spend explaining the interface rather than the benefits.\n<click>\nI&#x2019;ve sat in on training classes and tracked how much time is spent explaining (and re-explaining) how to make the software work vs. explaining how to take full advantage of it.\n<click>\nAnd similarly, you can read how much of the documentation must be devoted to explaining mechanics vs benefits.\n<click>\nI suspect if we were to track the time users spend thinking about their work vs thinking about the software they&#x2019;re using to do their work it would be close to the QofU metric.\n
- I believe the Quality of Use Metric is correlated with a number of other measures of product success.\n<click>\nThe Quality of Use Metric seems to often match a &#x201C;weighted user success rate&#x201D;: count success, failure, partial success as half. In one unusually large study, we had a weighted success rate that exactly corroborated the Quality of Use Metric of 55. We rarely have a study large enough to do a meaningful comparison but we do feel that this generally holds.\n<click>\nIn sales demo situations, the lower the QofU rating, the more time the demonstrator will have to spend explaining the interface rather than the benefits.\n<click>\nI&#x2019;ve sat in on training classes and tracked how much time is spent explaining (and re-explaining) how to make the software work vs. explaining how to take full advantage of it.\n<click>\nAnd similarly, you can read how much of the documentation must be devoted to explaining mechanics vs benefits.\n<click>\nI suspect if we were to track the time users spend thinking about their work vs thinking about the software they&#x2019;re using to do their work it would be close to the QofU metric.\n
- I believe the Quality of Use Metric is correlated with a number of other measures of product success.\n<click>\nThe Quality of Use Metric seems to often match a &#x201C;weighted user success rate&#x201D;: count success, failure, partial success as half. In one unusually large study, we had a weighted success rate that exactly corroborated the Quality of Use Metric of 55. We rarely have a study large enough to do a meaningful comparison but we do feel that this generally holds.\n<click>\nIn sales demo situations, the lower the QofU rating, the more time the demonstrator will have to spend explaining the interface rather than the benefits.\n<click>\nI&#x2019;ve sat in on training classes and tracked how much time is spent explaining (and re-explaining) how to make the software work vs. explaining how to take full advantage of it.\n<click>\nAnd similarly, you can read how much of the documentation must be devoted to explaining mechanics vs benefits.\n<click>\nI suspect if we were to track the time users spend thinking about their work vs thinking about the software they&#x2019;re using to do their work it would be close to the QofU metric.\n
- I believe the Quality of Use Metric is correlated with a number of other measures of product success.\n<click>\nThe Quality of Use Metric seems to often match a &#x201C;weighted user success rate&#x201D;: count success, failure, partial success as half. In one unusually large study, we had a weighted success rate that exactly corroborated the Quality of Use Metric of 55. We rarely have a study large enough to do a meaningful comparison but we do feel that this generally holds.\n<click>\nIn sales demo situations, the lower the QofU rating, the more time the demonstrator will have to spend explaining the interface rather than the benefits.\n<click>\nI&#x2019;ve sat in on training classes and tracked how much time is spent explaining (and re-explaining) how to make the software work vs. explaining how to take full advantage of it.\n<click>\nAnd similarly, you can read how much of the documentation must be devoted to explaining mechanics vs benefits.\n<click>\nI suspect if we were to track the time users spend thinking about their work vs thinking about the software they&#x2019;re using to do their work it would be close to the QofU metric.\n
- I believe the Quality of Use Metric is correlated with a number of other measures of product success.\n<click>\nThe Quality of Use Metric seems to often match a &#x201C;weighted user success rate&#x201D;: count success, failure, partial success as half. In one unusually large study, we had a weighted success rate that exactly corroborated the Quality of Use Metric of 55. We rarely have a study large enough to do a meaningful comparison but we do feel that this generally holds.\n<click>\nIn sales demo situations, the lower the QofU rating, the more time the demonstrator will have to spend explaining the interface rather than the benefits.\n<click>\nI&#x2019;ve sat in on training classes and tracked how much time is spent explaining (and re-explaining) how to make the software work vs. explaining how to take full advantage of it.\n<click>\nAnd similarly, you can read how much of the documentation must be devoted to explaining mechanics vs benefits.\n<click>\nI suspect if we were to track the time users spend thinking about their work vs thinking about the software they&#x2019;re using to do their work it would be close to the QofU metric.\n
- Quality of Use predicts use over time. After the initial interest in a new product or release, use declines dramatically with lower Quality of Use. \n
- On one project where we did usability tests first and then later did field observations of people using the same software, we saw a very strong correlation between usability testing in a lab and observation of actual work by real end-users.&#xA0; Not only did the Quality of Use rating match the Weighted User Success in the real world. \n<click>\nBut also, the usability testing predicted relative impacts of reliability, usability, and capability issues.&#xA0; Contextual Inquiry with actual users doing their own work produced outcomes very close to what was predicted -- down to the relative impact of reliability, usability, and capability issues.\n\n
- On one project where we did usability tests first and then later did field observations of people using the same software, we saw a very strong correlation between usability testing in a lab and observation of actual work by real end-users.&#xA0; Not only did the Quality of Use rating match the Weighted User Success in the real world. \n<click>\nBut also, the usability testing predicted relative impacts of reliability, usability, and capability issues.&#xA0; Contextual Inquiry with actual users doing their own work produced outcomes very close to what was predicted -- down to the relative impact of reliability, usability, and capability issues.\n\n
- The Quality of Use metric works with both paper prototypes and software.&#xA0; Of course, some aspects of the users' experience cannot be tested with a paper prototype -- performance and reliability, in particular.&#xA0; On one project, the Quality of Use scores on the implemented software matched those on the paper prototype almost exactly -- there were no performance or reliability issues and the usability issues were just as predicted.&#xA0; On another project, there was a significant difference between the paper prototype Quality of Use and the implemented software Quality of Use -- on that project there were significant performance and reliability issues that perfectly explained the difference.\n\n
- \n
- \n
- \n
- \n
- It is not meaningful to speak of the Quality of Use of a piece of software without regard to the targeted users and work.&#xA0;\n<click>\n Quality of Use is not a function of the software alone but a function of the software, the users, and the tasks.&#xA0; \n<click>\nQuality of Use cannot be determined by inspection but only by observation of people who reasonably match the target user profile doing tasks that are realistic examples of work intended to be supported.&#xA0; \n<click>\nHowever, when a product has a well defined target user profile and supported task profile, then we can speak of the product's Quality of Use -- the user and task are implied.&#xA0;\n
- It is not meaningful to speak of the Quality of Use of a piece of software without regard to the targeted users and work.&#xA0;\n<click>\n Quality of Use is not a function of the software alone but a function of the software, the users, and the tasks.&#xA0; \n<click>\nQuality of Use cannot be determined by inspection but only by observation of people who reasonably match the target user profile doing tasks that are realistic examples of work intended to be supported.&#xA0; \n<click>\nHowever, when a product has a well defined target user profile and supported task profile, then we can speak of the product's Quality of Use -- the user and task are implied.&#xA0;\n
- It is not meaningful to speak of the Quality of Use of a piece of software without regard to the targeted users and work.&#xA0;\n<click>\n Quality of Use is not a function of the software alone but a function of the software, the users, and the tasks.&#xA0; \n<click>\nQuality of Use cannot be determined by inspection but only by observation of people who reasonably match the target user profile doing tasks that are realistic examples of work intended to be supported.&#xA0; \n<click>\nHowever, when a product has a well defined target user profile and supported task profile, then we can speak of the product's Quality of Use -- the user and task are implied.&#xA0;\n
- I believe the Quality of Use metric is somewhat "self calibrating" or "self leveling" because it asks the user to rate their experience relative to their expectations.&#xA0; This allows for variation in user familiarity, task difficulty, and software versions.&#xA0; For example, a user will have lower expectations the first time they see a new piece of software and will thus rate an objectively poorer experience less harshly.&#xA0; Similarly, they will adjust for their own strength or weakness in domain knowledge, the inherent complexity of the task (tuning a carburetor is inherently more complex than filling a gas tank).&#xA0; For this reason we don't have to control for variation as much as we otherwise might.\n\nOn one unusually large study we tested a system with 21 different users at 3 sites, 2 disciplines, variable seniority and variable frequency of use. On this particular study we found the Quality of Use rating virtually invariant across all these sub-populations. I believe that suggested that the Quality of Use rating (which was 55) for the tested tasks was a measure of the tested system itself and not the varying tasks strategies and experiences of the various sub-populations. Note that the different sub-groups didn&#x2019;t have equivalent objective experiences -- but their experience with the system relative to their differing expectations was quite consistent.\n\n
- Is three users enough?&#xA0; Most people are surprised to hear that we believe we can learn what we need from only 3 users.&#xA0; In terms of strict number of data points, remember that we're dealing with roughly 48 data points to determine a Quality of Use score (3 users x 4 tasks x 4 attributes) rather than 3.&#xA0; Also, the Quality of Use numbers are not the only data we get -- live observation is a rich source of data.&#xA0; The Quality of Use number is a way to encapsulate in a single number how well the software supports its intended users and uses.&#xA0; If the Quality of Use number ever seemed at odds with our impression of how things went (or at odds with objective indicators like task completion and elapsed time) we would take special action to understand or explain that.&#xA0; We've rarely had to.&#xA0; We've never used the Quality of Use measure for "close calls" -- e.g., is design A with an 82 better than design B with a 76?&#xA0; We're more interested in whether design A is "good enough", "needs refinement", or "needs rethinking".\n<click>\nThat unusually large study with 21 users also gave us the opportunity to look at how many users we would have needed to reach the "final" Quality of Use Metric.\n\nAfter 3 users we were 7 off; after 5 users we were 4 off; and after 7 users we were 2 off. Of course, we learned more from additional users and they did represent different disciplines and profiles. Also there was some politics involved that necessitated the large number.\n
- \n
- It is completely expected that software under development and refinement get low Quality of Use ratings. In fact, we don't really start worrying much about the absolute level of the rating before the software is close to shipping. The important thing is that the Quality of Use is trending upward as refinements are made.&#xA0; Why do we go ahead and have the user complete the form even in a very early paper prototyping when we don't really care about the score?&#xA0; Because we get useful comments from the user as the user considers different aspects of their experience.\n\n
- We have tested software and designs at various stages of development and release. We have seen Quality of Use scores range from as low as 12 to as high as 96 (these are averages for a round of tests with multiple users doing multiple tasks -- not just one user with one task). We have seen released software get a Quality of Use score as low as 25. \n\nDuring the course of development we can see quite dramatic improvements in Quality of Use. The pre-release software that had a Quality of Use of 12, was redesigned and released with a Quality of Use of 80. Other software that shipped at 95 was at 50 when first tested; another went from 50 to 94. Of course, a product may be first usability tested at varying stages of its life -- it may already be released, it may be thought to be ready for release, or it may be in very early paper prototyping.\n\n
- The Quality of Use metric we use is a subjective, quantified measure.&#xA0; There are other indicators of quality of use -- subjective or objective; quantitative or qualitative.&#xA0; We frequently use two other objective quantitative measures.&#xA0; One is "objective success", which is applicable when the task has a clear result that can be objectively judged successful or not (e.g., calculating an attribute).&#xA0; It is important to track this when it may differ from the user's own perception of success (in many domains -- e.g., computational chemistry -- a bad answer believed to be a good answer can be much worse than a bad answer known to be bad).&#xA0; The other objective measure we sometimes use is "time to complete".&#xA0; We almost always measure this but usually find (again, in domains I've worked) that the users' subjective impression of time is more important.\n\n
- So we had a Quality of Use rating for the software and a list of issues. These are commonly characterized as having low, medium or high severity. But we wanted to take it a step further and map the issues into the &#x201C;gap&#x201D; between the software&#x2019;s current Quality of Use rating and the ideal Quality of Use rating of 100. So we can see how to progress the software toward the ideal.\n\nSo the way we map the issues into the Gap is to estimate the Quality of Use impact of each issue based on the user/task QofU ratings. So that all the issues collectively account for the Gap. \n<click>\nSo now we can associate a numerical value with each issue that has a meaning: if the issue is completely addressed we estimate the Quality of Use of the software will improve by that amount.\n\n\n\n
- So we had a Quality of Use rating for the software and a list of issues. These are commonly characterized as having low, medium or high severity. But we wanted to take it a step further and map the issues into the &#x201C;gap&#x201D; between the software&#x2019;s current Quality of Use rating and the ideal Quality of Use rating of 100. So we can see how to progress the software toward the ideal.\n\nSo the way we map the issues into the Gap is to estimate the Quality of Use impact of each issue based on the user/task QofU ratings. So that all the issues collectively account for the Gap. \n<click>\nSo now we can associate a numerical value with each issue that has a meaning: if the issue is completely addressed we estimate the Quality of Use of the software will improve by that amount.\n\n\n\n
- Recently we have tried a more accurate approach to estimating the impact of each issue by looking at the individual incidents that make up an issue. During the review of each usability test, as we identify an incident, we also assess the impact of the incident in terms of efficiency, effectiveness and satisfaction using these guidelines. These are used to get a relative impact of each issue which is then scaled to fill the &#x201C;QofU Gap&#x201D;.\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- I was preparing a presentation (not this one) on my MacBook and kept getting small rotations of graphics. I guess it was feeling a multi-touch and decided that meant rotate. I became more careful and reduced the frequency of that happening. Once in a while I noticed many of the graphics&#x2019; labels were slightly off. I hoped it was just a display issue and would show up correct during the presentation. \n&#x2022; Finally, I happened to notice in the Inspector tool that one of those graphics was actually rotated to 359.6 degrees! I set it back to 0 and TaDa! It turned out that every one of those graphics was slightly rotated like that (I probably created most by copy/paste). So I went through the presentation and corrected each one. \n&#x2022; Later, I decided to see if I could turn off this rotation behavior. I had previously turned off two-finger actions in the Systems Preferences (I thought) so I looked in Keynote Preferences and couldn&#x2019;t find such an option. As I was writing this, I realized I just turned off 2-finger scaling (which messed me up in word documents) but not 2-finger rotation. So I went into System Preferences, turned it off and now that&#x2019;s not happening any more (but I&#x2019;ve already finished that other presentation).\n82\nUsability\n
- Once we have impact values for all the issues, we can categorize them in various ways to gain insight into how to move forward (e.g., staffing, training).\n\nFor example, we can characterize the issues as falling into various qualities -- like functionality, performance, reliability or usability. \n
- We can use the issue impacts to help us prioritize what to address next.\n\nWe work with the developers to come up with estimates for the cost of addressing each issue.\n<click>\nWe can then divide the value by the cost for a &#x201C;bang for the buck&#x201D; ratio.\n<click>\nIf we sort the issues by decreasing value to cost ratio, we&#x2019;ll get the shortest path to improving the Quality of Use.\n<click>\nWe can decide how far down the list to go based on various criteria.\n
- We can use the issue impacts to help us prioritize what to address next.\n\nWe work with the developers to come up with estimates for the cost of addressing each issue.\n<click>\nWe can then divide the value by the cost for a &#x201C;bang for the buck&#x201D; ratio.\n<click>\nIf we sort the issues by decreasing value to cost ratio, we&#x2019;ll get the shortest path to improving the Quality of Use.\n<click>\nWe can decide how far down the list to go based on various criteria.\n
- We can use the issue impacts to help us prioritize what to address next.\n\nWe work with the developers to come up with estimates for the cost of addressing each issue.\n<click>\nWe can then divide the value by the cost for a &#x201C;bang for the buck&#x201D; ratio.\n<click>\nIf we sort the issues by decreasing value to cost ratio, we&#x2019;ll get the shortest path to improving the Quality of Use.\n<click>\nWe can decide how far down the list to go based on various criteria.\n
- We can use the issue impacts to help us prioritize what to address next.\n\nWe work with the developers to come up with estimates for the cost of addressing each issue.\n<click>\nWe can then divide the value by the cost for a &#x201C;bang for the buck&#x201D; ratio.\n<click>\nIf we sort the issues by decreasing value to cost ratio, we&#x2019;ll get the shortest path to improving the Quality of Use.\n<click>\nWe can decide how far down the list to go based on various criteria.\n
- Here&#x2019;s an example of a subset of identified issues. Each has a Value which is the Quality of Use impact. Each has an estimate of the cost to address. The 3rd column is the Value to Cost ratio. The 4th column is the cumulative value in terms of the Quality of Use, starting from baseline current rating of 72, this shows what the QofU would be once that issue is addressed. The last column is the cumulative cost, starting from a baseline of what has already been spent.\n<click>\nWhen we graph this we get a nice picture showing how our Quality of Use should improve.\n\nSofter input from the development team we are able to make realistic goals for Quality of Use of the next or future sprints, iterations or releases.\n\n
- Estimating Quality of Use impact of all the issues offers many benefits.\n\nThe development team can always be thinking about how best to improve the software, whether that be adding a new feature, fixing an existing feature, polishing a design, improving performance, or whatever. They are all on the same playing field. \n\nIt is especially challenging to stay focused on "true north" -- meeting users needs -- during the "end game" of development. Pressures, demands, schedule, ...\n\nI especially like non-UX professionals helping to make these estimates because it keeps their head in the users&#x2019; experience.\n
- We also use the Quality of Use rating to give guidance about what technique to use next when first testing a software or design.\n
- \n
- \n