UserZoom worked in partnership with UX Alliance members; Telono, User Intelligence and Xperience Consulting to benchmark the User Experience across 4 countries and 4 mobile operators (Orange, Vodafone, T-mobile and Swisscom)

1. Webinar Four Country Mobile Operator Study: a competitive benchmark of leading mobile operator sites 19th January 2011

3. Study Background

4. What do we mean by UX Benchmarking? Why is it important?

5. Methodology and Technology

6. Findings and Opportunities

7. Questions and Comments

10. the monthly cost to surf the internet daily with an iPad

11. the fax number/address for a high street shop

12. To gain insight into the European mobile operator market

13. Introduction and review of UX Benchmarking methodology

15. What is UX benchmarking and why is it important? OK……. So how do we define UX benchmarking?

17. The process of comparing the business processes and performance metrics including cost, cycle time, productivity, or quality to another that is widely considered to be an industry standard benchmark or best practice.

19. It’s about how a person feels about using a system. It highlights the experiential, affective, meaningful and valuable aspects of HCI and product ownership, but it also covers a person’s perceptions of the practical aspects such as utility, ease of use and efficiency of the system.

23. What is UX benchmarking and why is it important? ‘You can’t manage what you can’t measure’ Attributed to Peter Drucker, Mgt Guru The question is: How are you actually measuring UX and defining your benchmark?

24. Methodology and Technology

26. Obtaining idea adoption from stakeholders

27. Geographic/demographic limitations

29. Testing large samples of geographically spread participants

30. Combining statistically-significant usability + user behavior data

31. Measuring Customer Experience cost-effectively

32. Getting actionableinsightsBehavior Usability Lab Studies Web Traffic Analysis Online UX Studies Attitudes Online Surveys Focus Groups Qualitative Quantitative

34. 100% on-demand, web-based software solution

36. In their natural context

37. From geographically spread locations

38. Users try to complete tasks(or goals) + answer questions

39. No human moderation needed

40. Our browser bar connects users with our secure serversOur method

41. Analyze data real-time using UZ Analytics Effectiveness + efficiency ratios

42. Analyze data using UZ Analytics Qualitative feedback collected. Easy visualization through ‘text clouds’

43. Analyze data using UZ Analytics

44. UserZoom Results - Behavioural

46. Task: Find a monthly iPad data plan

48. Perceived complexity of sites

49. Perceived consistency of sites

50. Clear where to start

51. Confidence in using the sites

52. Usability Satisfaction (SUS score)

53. What was liked most about the sites

56. Current mobile operator

57. Age

58. Individuals between 18-55 years of age were invited to complete the study.GENDER 45 females completed the study 33 males completed the study

60. 19 participants are 26-35

61. 25 participants are 36-45

63. Current mobile operator

64. Age

65. Individuals between 18-55 years of age were invited to complete the study.GENDER 47 females completed the study 53 males completed the study

67. 15 participants are 26-35

68. 41 participants are 36-45

70. Current mobile operator

71. Age

72. Individuals between 18-55 years of age were invited to complete the study.GENDER 45 females completed the study 55 males completed the study

74. 43 participants are 26-35

75. 22 participants are 36-45

77. Current mobile operator

78. Age

79. Individuals between 18-55 years of age were invited to complete the study.GENDER 65 females completed the study 35 males completed the study

81. 30 participants are 26-35

82. 31 participants are 36-45

85. Orange Spain (SP) had the lowest success rate across tasks 24%, the lowest participants satisfaction scores, and substantially lower preference (25%) vs. Vodafone (75%).

86. All mobile providers tested in each country received low usability satisfaction scores by participants.

87. The lowest system usability scale scores (SUS) were received by Vodafone Spain (51) and Orange Spain (40).

89. Top level overview of Task 1 across all countries with deep dive into Switzerland & UK results “Find the monthly cost to surf the internet daily with an iPad”

90. Task Metrics: iPad Monthly Plan

92. “Internet”

93. “Internet with your Notebook”

94. “iPad tariffs”

97. 16% clicked “Mobile” on the top navigation bar

99. 14% clicked “movil” on the top navigation bar

101. 31% clicked “Tienda Online” on the top navigation bar

103. 14% clicked “Internet” on the top navigation bar

105. 13% clicked “Abonnement” on the top navigation bar

107. 37% clicked “iPad” on the left navigation bar

109. Successful Participants spent an average of 5 minutes 24 seconds and 19 clicks completing the task. Task Effectiveness

110. Orange > iPad: UK The majority of participants were able to find the correct price for the plan (75%). Successful Participants spent an average of 1 minute 11 seconds and 7 clicks completing the task. Task Effectiveness

111. Top level overview of Task 2 across all countries with deep dive into Netherlands & Spain’s results “Locate a store address/fax number”

112. Task Metrics: Locating Store Address/Phone number

114. The T-mobile site provided the best results regarding this task, 77% of the participants was successful.

115. On average it took slightly over one minute to find the right information, with an average of only 4 page views and 8 clicks.

116. On the Vodafone website 52% was successful in achieving this information.

118. The Vodafone website was less successful, but only 28% of the participants regarded it to be difficult to acquire the needed information.

119. On the T-mobile website, people were far more successful, but also less satisfied with the level of difficulty to get to the right URL: 42% of the participants thought this was (very) hard to achieve.

120. Vodafone

122. Page not loading: Especially the Vodafone website was unable to load the required page. Remarkable is that people wait up to 5 minutes!

123. Information not in expected location: Both Vodafone and T-mobile present information in an unexpected manner. Participants have certain expectations on where and how to find the required information. In case these expectations are not met, people easily abandon the website.QUOTES “I was looking for it under Customer service and expected to be able to fill in my location and that it would automatically give the closest store location.” “ I was unable to open the page.”

125. They regard it as easy to use for inexperienced users and there is no need for extra help.

126. Nevertheless, the lack of success will have some influence on the use of the website.

131. The information presentation is not clear enough

132. The information, options and menus are difficult to understand

133. The information wasn’t in the expected location

134. Few search options

135. As it has been seen in the previous task users again had problems with the information provided. I have entered province “Madrid”, city “Madrid” and no results appeared with this search options. I have tried with different options and it always happened the same.

137. Users explained that all the site is focusing on the iphone whereas ipad information is quite difficult to find.

138. The system could be improve using an accessible and easy to find search engine. There is no way to understand what is going on here. There is no way to find anything. What a fuss!

140. The links to the information were not clear enough.

141. The site is very focused on selling products or services instead of giving clients the service that they need. I don’t have anything against Orange but its website is a disaster

142. Final Questionnaire – all countries: Which mobile operator do you prefer? Netherlands Switzerland Spain UK

143. Metrics: SUS Score The System Usability Scale (SUS) is a set of 10 questions that is used to measure the perception of a systems overall usability. Participants rate each question with a 1 to 5 scale (5 being strongly agree). 1. I think that I would like to use this system frequently. 2. I found the system unnecessarily complex. 3. I thought the system was easy to use. 4. I think that I would need the support of a technical person to be able to use this system. 5. I found the various functions in this system were well integrated. 6. I thought there was too much inconsistency in this system. 7. I would imagine that most people would learn to use this system very quickly. 8. I found the system very cumbersome to use. 9. I felt very confident using the system. 10. I needed to learn a lot of things before I could get going with this system.

144. Metrics: SUS Score After completing the tasks on each mobile provider website participants were asked to complete the SUS questionnaire. The averages for each question are used to calculate an overall score (0-100). SUS Score Guide*: 35=poor, 50=OK, 70=good, 85=excellent Bangor, A. (2009, May). Determining What Individual SUS Scores Mean: Adding and Adjective Rating Scale. Journal of Usability Studies, 4(3), 114-123.

145. Net Promoter Score (NPS®) After completing the four tasks, participants were asked how likely they would be to recommend each mobile provider’s website. Promoters Detractors - = % of Promoters (9s and 10s) Detractors (0 through 6) NPS

148. UK - although participants had notably higher task success rates on the Orange site (75% and 71%) vs. the Vodafone site (50% and 49%), when asked for site preference at the end of the study, the Orange website was selected only 8% more than the Vodafone site (54% vs 46%) respectively. Out of the Orange customers (n=17) more than half (9) preferred the Vodafone website.

149. Orange is not strong in Spain

150. Swisscom the national provider is stronger in Switzerland

152. As Vodafone and Orange were both tested in 3 of the 4 markets we can benchmark their experiences.

153. Vodafone outperforms Orange on aggregated SUS (53 versus 51.3)

154. Vodafone outperforms Orange on aggregated NPS (-55% versus -64.3%)

155. Task 1 success aggregated : Vodafone outperforms Orange (46% versus 42%)

158. Offices in Amsterdam and Helsinki, additional service offering in Sweden and Norway

160. We help companies implement a systematic user-centred design process, from user experience strategy and user research to interaction design and usability evaluations.

162. 2 offices (Barcelona)

163. 150 AI projects

164. 300 usability studies

165. 6000 users analyzed

166. 150 clients (15 countries)Usability and user experience experts, we create interfaces that suit user needs and objectives of your company. Intuitive and easy to use webs.

168. Back in 2005 the leading user experience companies from France, Germany, Spain, UK and US met in Barcelona with a single common goal in mind: how to provide effective international user experience services to our clients.

169. Having worked together on international projects for many years, we recognised that we were ideally placed to lead international user experience research. Thus, the UXalliance was born… www.uxalliance.com

172. 9 years experience in UX research & consulting

173. Offices in Sunnyvale (CA), London (UK) and Barcelona (Spain)