2. Level 2: Learning — the degree to which learning occurs as a result of the instruction

3. Level 3: Transfers — the transfer of learning to impact on job behavior

4. Level 4: Organizational performance — the impact the learning has on the organization

6. What’s the purpose?

7. A focus on improvement during development.

8. Level 2 Evaluations

9. Appeal (Level 1: Reaction)

10. Effectiveness

11. Efficiency

13. The sooner formative evaluation is conducted during development, the more likely that substantive improvements will be made and costly errors avoided (Reeves & Hedberg, 2003, p. 142).

14. Formative Evaluation Methods (Reeves & Hedberg, 2003)

15. Expert Review

16. User Review

17. Usability Testing

18. Alpha and beta prototypes and field trials

19. Expert Review

20. “Experts are anyone with specialized knowledge that is relevant to the design of your ILE” (Reeves & Hedberg, 2003, p. 145).

21. SMEs, instructional experts, graphic designers, teachers, and trainers

22. Interface Review Guidelines from http://it.coe.uga.edu/~treeves/edit8350/UIRF.html

23. User Review

24. Observations from one-on-ones and small groups

26. “at least 90% of all commercial Web sites are overly difficult to use….the average outcome of Web usability studies is that test users fail when they try to perform a test task on the Web. Thus, when you try something new on the Web, the expected outcome is failure.” —Jakob Nielsen

27. Nielsen’s Web Usability Rules

28. Visibility of system status

29. Match between system and real world

30. User control and freedom

31. Consistency and standards

32. Error prevention

33. Recognition rather than recall

34. Flexibility and efficiency of use

35. Help users recognize, diagnose, and recover from errors

36. Help and documentation

37. Aesthetic and minimalist design

38. Usability.gov

39. Ease of learning - How fast can a user who has never seen the user interface before learn it sufficiently well to accomplish basic tasks?

40. Efficiency of use - Once an experienced user has learned to use the system, how fast can he or she accomplish tasks?

41. Memorability - If a user has used the system before, can he or she remember enough to use it effectively the next time or does the user have to start over again learning everything?

42. Error frequency and severity - How often do users make errors while using the system, how serious are these errors, and how do users recover from these errors?

43. Subjective satisfaction - How much does the user like using the system?

44. Two Major Methods to Evaluate Usability

45. Heuristic Evaluation Process

46. Several experts individually compare a product to a set of usability heuristics

47. Violations of the heuristics are evaluated for their severity and extent suggested solutions

48. At a group meeting, violation reports are categorized and assigned

49. average severity ratings, extents, heuristics violated, description of opportunity for improvement

50. Advantages of Heuristic Evaluation

51. Quick: Do not need to find or schedule users

52. Easy to review problem areas many times

53. Inexpensive: No fancy equipment

54. Disadvantages of Heuristic Evaluation

55. Validity: No users involved

56. Finds fewer problems (40-60% less??)

57. Getting good experts

58. Building consensus with experts

59. User Testing

60. People whose characteristics (or profiles) match those of the Web site’s target audience perform a sequence of typical tasks using the site.

61. Examines:

62. Ease of learning

63. Speed of task performance

64. Error rates

65. User satisfaction

66. User retention over time

67. Elements of User Testing

68. Define target users

69. Have users perform representative tasks

70. Observe users

71. Report results

72. Why Multiple Evaluators?

73. Single evaluator achieves poor results

74. Only finds about 35% of usability problems

75. 5 evaluators find more than 75%

76. Reporting User Testing

77. Overall goals/objectives

78. Methodology

79. Target profile

80. Testing outline with test script

81. Specific task list to perform

82. Data analysis & results

83. Recommendations

84. Recent Methods for User Testing

85. CrazyEgg

86. Userfly

87. fivesecondtest

88. 10 Second Usability Test

89. Disable stylesheets