Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.
Meta-Analysis of Correlations Among Usability Measures Kasper Hornbæk & Effie Law University of Copenhagen & Eidgenössisch...
Background and aim <ul><li>Usability describes quality-in-use  </li></ul><ul><li>Usability may be measured in many ways, e...
Earlier work <ul><li>Many classifications of usability (e.g., ISO 1998, Seffah et al. 2006, Hornbæk 2006) </li></ul><ul><l...
Data collection <ul><li>Studies from 8 HCI journals and conferences </li></ul><ul><li>Consider as candidates original rese...
Analysis <ul><li>Measures in each study classified according to  </li></ul><ul><ul><li>ISO 9241-11: effectiveness, efficie...
Results:  which measures are used? <ul><li>Effectiveness is typically error rates or task completion </li></ul>
Results:  which measures are used? <ul><li>Efficiency is typically measured as time </li></ul>
Results:  which measures are used? <ul><li>Satisfaction is measured in many ways, but standard questionnaires are rarely u...
Results: are measures correlated? <ul><li>Effectiveness vs. efficiency: r = .247 ± .059 </li></ul><ul><li>May be interpret...
Results: are measures correlated? <ul><li>Task complexity does  not influence the  correlations </li></ul><ul><li>More com...
Results: are measures correlated? <ul><li>Effectiveness vs. satisfaction: r = .164 ± .062 </li></ul><ul><li>Preference is ...
Results: are measures correlated? <ul><li>Efficiency vs. satisfaction: r = .196 ± .064 </li></ul><ul><li>Interfaces that a...
Results: other findings <ul><li>We calculated the reliability of questionnaires </li></ul><ul><li>Homegrown questionnaires...
Discussion <ul><li>Two interpretations of the main result </li></ul><ul><ul><li>The half-empty interpretation: correlation...
Conclusion <ul><li>We find mostly small correlations among usability measures </li></ul><ul><li>These correlations are sha...
Próxima SlideShare
Cargando en…5
×

CHI2007: Meta-analysis of correlations among usability measurs

A presentation of a paper at CHI2007 on

  • Inicia sesión para ver los comentarios

CHI2007: Meta-analysis of correlations among usability measurs

  1. 1. Meta-Analysis of Correlations Among Usability Measures Kasper Hornbæk & Effie Law University of Copenhagen & Eidgenössische Technische Hochschule Zürich
  2. 2. Background and aim <ul><li>Usability describes quality-in-use </li></ul><ul><li>Usability may be measured in many ways, e.g., as task completion time, error rates, subjective satisfaction, perceived workload, etc. </li></ul><ul><li>Quantitative measures of usability play an important role in human-computer interaction </li></ul><ul><li>Our aim is to understand how usability measures relate, which will help to: </li></ul><ul><ul><li>Select measures for usability studies </li></ul></ul><ul><ul><li>Understand usability better </li></ul></ul>
  3. 3. Earlier work <ul><li>Many classifications of usability (e.g., ISO 1998, Seffah et al. 2006, Hornbæk 2006) </li></ul><ul><li>Studies of correlations among measures (e.g., Nielsen & Levy 1994, Sauro & Kindlund 2005) </li></ul><ul><li>Limitations of existing studies include: </li></ul><ul><ul><li>Contradictory findings (e.g., Nielsen & Levy 1994 vs. Frøkjær et al. 2000) </li></ul></ul><ul><ul><li>Mostly without access to raw data (e.g., Nielsen & Levy 1994) </li></ul></ul><ul><ul><li>Do not consider the variety of ways usability may be measured or moderator variables </li></ul></ul>
  4. 4. Data collection <ul><li>Studies from 8 HCI journals and conferences </li></ul><ul><li>Consider as candidates original research papers reporting usability measures </li></ul><ul><li>Contacted authors of these papers and end up with raw data from 73 studies </li></ul>
  5. 5. Analysis <ul><li>Measures in each study classified according to </li></ul><ul><ul><li>ISO 9241-11: effectiveness, efficiency, satisfaction </li></ul></ul><ul><ul><li>Hornbæk (2006)’s 54 types of measure </li></ul></ul><ul><li>Then correlations among types of usability measure are calculated </li></ul><ul><li>These correlations may then be integrated using techniques from meta-analysis (Rosenthal 1991) </li></ul>
  6. 6. Results: which measures are used? <ul><li>Effectiveness is typically error rates or task completion </li></ul>
  7. 7. Results: which measures are used? <ul><li>Efficiency is typically measured as time </li></ul>
  8. 8. Results: which measures are used? <ul><li>Satisfaction is measured in many ways, but standard questionnaires are rarely used </li></ul>
  9. 9. Results: are measures correlated? <ul><li>Effectiveness vs. efficiency: r = .247 ± .059 </li></ul><ul><li>May be interpreted as: </li></ul><ul><ul><li>6% variance explained </li></ul></ul><ul><ul><li>small (~.1) to medium (~.3) effect (Cohen 1969) </li></ul></ul><ul><ul><li>examples: </li></ul></ul>r = .229 r = .23
  10. 10. Results: are measures correlated? <ul><li>Task complexity does not influence the correlations </li></ul><ul><li>More complex measures (e.g., quality of outcome) attenuate correlations </li></ul><ul><li>Difference between errors-along-the-way (.441) and task-completion-errors (.155) </li></ul>
  11. 11. Results: are measures correlated? <ul><li>Effectiveness vs. satisfaction: r = .164 ± .062 </li></ul><ul><li>Preference is related to fewer errors </li></ul><ul><ul><li>Prefer (13% errors) vs. do not prefer (18% errors) </li></ul></ul><ul><li>Six studies measure both effectiveness (objective) and participants’ assessment of their effectiveness (subjective) </li></ul><ul><ul><li>“ were your answers to tasks: very good – very poor” vs. errors in task answers </li></ul></ul><ul><ul><li>Correlations of these measures are not significantly different from zero </li></ul></ul>
  12. 12. Results: are measures correlated? <ul><li>Efficiency vs. satisfaction: r = .196 ± .064 </li></ul><ul><li>Interfaces that are preferred are about 20% faster than non-preferred ones </li></ul><ul><li>Again no correlation between objective and subjective measures </li></ul>
  13. 13. Results: other findings <ul><li>We calculated the reliability of questionnaires </li></ul><ul><li>Homegrown questionnaires have low reliability, 6 are below .7 </li></ul>
  14. 14. Discussion <ul><li>Two interpretations of the main result </li></ul><ul><ul><li>The half-empty interpretation: correlations are small ->can not reduce usability to one measure (e.g., Sauro & Kindlund 2005) </li></ul></ul><ul><ul><li>The half-full interpretation: surprising with consistent correlations across very different studies </li></ul></ul><ul><li>Models of usability may be improved </li></ul><ul><ul><li>Subjective/objective, error types not separated in some usability models </li></ul></ul><ul><li>Recommendations for practice </li></ul>
  15. 15. Conclusion <ul><li>We find mostly small correlations among usability measures </li></ul><ul><li>These correlations are shaped by measure complexity and the use of subjective/objective measures, but not by task complexity </li></ul><ul><li>Suggests that models of usability need extension and that usability measures should not be combined into a single measures </li></ul>kash@diku.dk [email_address]

×