This document provides an overview of best practices for conducting mobile user experience research. It discusses the challenges of mobile research like varied devices, locations, networks and user experiences. It recommends testing across many devices and simulators, addressing network coverage issues, screening for data plans and experience. Field studies are suggested to understand true context of use. Tips include creating a "hot zone" for devices, addressing reflections, simplifying language and having prototype strategies. The document emphasizes understanding usage across many scenarios to effectively evaluate mobile products and identify opportunities for improvement.

1. Making the Leap from Web to Mobile Best Practices in Mobile User Experience Research Amy Buckner, AnswerLab Kris Mihalic, Yahoo! UPA 2009, June 12, Portland

2. Agenda

3. Introductions

4. State of Mobile

5. There are mobile phone subscribers worldwide 4.1 Billion Only of US mobile phones are iPhones (~5 Million) 2% Social networking on mobile phones is growing Y/Y 196% There are more mobile IM users than mobile business email users in the US 1.4x

6. Mobile Market Summary Globally, there are 800 million cars, 850 million PCs, 1.3 billion fixed landline phones, 1.4 billion credit cards, 1.5 billion TV sets – and 2.7 billion mobile phones – in use. (Experian 2008) eMarketer projects that worldwide spending on mobile advertising will reach a total of $19.1 billion in 2012 – up from 4.6 billion in 2008. Source: Experian 2008, eMarketer 2008

7. U.S. Mobile Web Users Growing Rapidly Source: m:metrics/comScore, 2009

8. Mobile Audience Size (# of users) In the U.S., the size of the Mobile Audience is 250 Million Source: m:metrics/comScore, 2009

9. U.S. Mobile Internet Usage Mobile Internet users are heavy users 60% access the Internet from their phone at least once/day 48% access the Internet more than once per day Source: TSM|TargetProfile 2007

10. Device Traffic Mobile phones dominate mobile Web traffic with 66%market share (US). BlackBerry and Smartphones (Windows Mobile, Palm, etc.) are heavy users of mobile Internet – combined 26% of traffic iPhone/iPod Touch has shown tremendous growth, capturing 8% of the mobile Internet market in the US Source: m:metrics/comScore, 2009

11. Widgets Improve Mobile Internet Experience Source: ‘Widgets Improve the Convenience of the Mobile Internet,’ Forrester, 2008

12. Group Exercise

13. Take out your cell phone and search for times that ‘Star Trek’ is playing in your neighborhood. Raise your hand when you have finished the task.

14. Discussion Points How long did it take? How did you do it? (web, app, SMS?) What problems, if any, did you have?

15. Mobile User Research Methods

16. Overview of Research Methods Quick and Dirty feedback Lab Studies Field Studies Online Surveys Basic Survey Survey with Behavioral Tracking (Keynote on iPhone only) Quantitative Behavioral Analysis

17. Quick and Dirty – What are obvious problems? Advantages Informal setting with Lo-Fi prototypes Fast iteration cycles Brain-damage check Finds critical usability issues in shortest time Challenges Examples Insider/stakeholder view Anecdotal, unstructured results Small sample size Difficult to convey real usage scenarios Cross-functional team feedback Internal alpha/beta

18. Lab Studies – Can they use it? Advantages Allows for probing by moderator Can record interactions Best for prototype testing Allows for real-time viewing of interaction – team engagement Challenges Examples Artificial environment Moderator bias; participant ‘pleasing’ Small sample size Difficult to test all hardware / network scenarios Usability study Participatory design sessions

19. Field Studies – What is the context of usage? Advantages Assess usage under real conditions Candid user feedback Delivers unknown use-cases Challenges Examples Small sample size Difficult to observe and probe Participant engagement difficult Requires robust product Diary study Community study

20. Online Surveys – What do they think of it? Advantages Can deliver statistically valid results Provides qualitative and/or quantitative data Geographical diversity (remote) Broad representation of devices Challenges Examples Difficult to observe and probe Out-of-context interaction, esp. with desktop survey Technical constraints Desktop surveys On-device survey Survey with behavioral tracking

21. Behavioral Analysis – How do they use it? Advantages Reflects what people do, not say Statistically valid results Shows all behaviors, rather than those confined to a single task Challenges Examples Probing / deep-dive difficult Context and intent unknown Attitudes and perceptions unknown Log data mining

22. Types of Prototypes Paper prototypes Mocks Paper and pencil Interactive prototypes Apps Flash Native, e.g. iPhone Browser-based Simple HTML prototype High-fidelity prototype (with session variables)

23. Team Exercise & Case Study

24. Yahoo! Go Mobile Application Search News Sports Weather Other Topics

25. Y! oneSearch integrated into Y! Go

26. Business Issue Drive mobile search uptake – Improved user experience can help accelerate growth of the service Improve relevancy – Understanding search intent can contribute to delivering more relevant results Product differentiation – New approach to search results page can deliver higher value to customers

27. Research Objectives Understand mobile search behavior in users’ daily lives Identify content users seekwhen conducting mobile searches Assess the context surrounding mobile searches Evaluate effectiveness of new product Identify opportunities to improve the user experience

28. Form into teams of 4. Develop a research plan for Yahoo! Go. You have 10 minutes.

30. Number of participants

32. Our Solution: One-Month Field Study (1) Daily Mobile Survey Digital Photos Voicemails & Pocket Card Daily SMS mini-survey Linked to phone number for voicemail Users sent photos of themselves or surroundings in the context of using oneSearch 9 users Daily voicemails Laminated pocket card with key questions

33. ‘Pocket Card’ Questionnaire

34. Sample Voicemail Diary Messages -- Winchelle Friday 7:39pm I used oneSearch today, was looking for a hospital – directions and address to it. I was getting off work and sitting in my car, trying to find directions. It was about 4:30. I liked that it came up with the website and with information about that. And the directions on how to go. I didn’t like that it could not find my work address – which is saved on my Yahoo! directions (recently gone) saved places. I think using the actual internet through the PC will help me better because it knows where the address is. I’m not sure why the address is not coming up in my oneSearch and that sucked. Saturday 8:06pm I didn’t use oneSearch today. I looked for info, looked for a store and I was at home. I wanted to know if the store was open STORE HOURS (9:45am). I used the internet with my phone and used Google to search. It was not very helpful because I wanted to see what time they opened and I couldn’t find it. I’ve used oneSearch in the past and I guess the results were the same but I guess it’d be nice to go to the site for that particular store. Monday 10:48pm I used oneSearch to find Expedia, was at work on my lunch break. It was about 12:20. I liked that I can find it quickly. I guess the website didn’t support my mobile phone though, so I didn’t like that. Would rather book a flight/hotel at home because its supported.

35. Our Solution: One-Month Field Study (2) Pre-Interview Mid-Check Interview Wrap-up Interview An initial 45-minute interview Verified technical capabilities of phone Explained research program 20-minute check-in phone interview Initial feedback Q&A Final in-person interview of 75 minutes Clarified voicemail reports Overall impressions of the product from the month-long usage

36. Screening Criteria: 9 Experienced Mobile Users Have an unlimited data plan for WAP services through their mobile carrier Use their mobile for text messages (SMS) daily and use mobile more than three times per week for activities that require the transmission of online data Have Cingular, Sprint, T-Mobile, or Verizon as their mobile carrier Have a primary mobile device capable of downloading and using Y! Go Live or work within San Francisco Mix of gender, age (23-46+),and typical method of commuting to work (4 users primarily drive, 5 primarily take mass transit)

37. Sliding Scale for Participant Incentives $100 for the initial in-person interview $50 for the mid-way phone interview $150 for the final in-person interview $3 each day a report is given

38. Natural environment key to context All research activities on the mobile device Mix of methods Engage users in program Breadth of insights – context, usability, impressions Why did we choose this method?

39. What did we learn? (1) Perceived value increased with usage Usage typically driven by lack of computer availability However, certain scenarios drove phone choice over computer: Social setting Privacy Convenience

40. What did we learn? (2) Mental model based on computer usage Speed and relevance of search results highest area of frustration Lack of understanding that results were customized by widget

41. Daily SMS Survey Results *Website not adapted for mobile device

42. What did we learn? (3)

43. Summary of Use Case Frequency Note: The 166 searches were conducted by 9 users

44. Shopping Business’ number, location, directions, and hours of operation “While driving home from work, I did a search for Fry’s electronics on the local search and it came up with all the different Fry’s in the area, their numbers, as well as directions and all that. The results were all helpful and fast in helping me find the one closest.” “Since we were already out shopping, I wanted to find where the closest Citibank was to where we were.” Pricing/Product comparisons “I was at Costco and comparing TV prices. I wanted to know how much these same TV’s would be at the Circuit City down the street.”

45. Dining A restaurant’s number, location, directions, and hours of operation “I just woke up and didn’t want to boot my computer up but I wanted to find out what time this breakfast place opened so I could know how soon I could get there.” “I was in my car and wanted their number so I could call and place an order to pick it up on my way home from work.” Ideas of places to eat “I was with my wife in the Haight and we wanted to find nearby restaurants so we could decide what we wanted to eat that was within walking distance.” “I was in the South Bay with some friends and we wanted to find a good sushi place nearby.” Reservations & Reviews “I was hoping to make reservations straight from the phone, or at least call them to place them.” “I was out with some friends and curious about whether the place nearby was any good.”

46. Traveling Transit schedules and numbers “I wanted to find out when the next MUNI but the MUNI’s site wouldn’t work on my phone.” “I wanted the number to call a taxi. I was outside and tired of waiting for the bus.” Tourist activities “I wanted to see what kind of ferry tours we could do before looking for another museum that would be fun for my daughter.” “I was looking for fun things to do while at Hermosa beach.” Maps and directions “I was trying to get a map of the Mt. Shasta area and I wanted to plan which route to take. It would be nice if the driving directions had live updates on traffic and road conditions.” Flight status / check-in “I was in the taxi to the airport and wanted to check my flight’s status.”

47. Business Implications Drive mobile search uptake – Distribution through partnerships, marketing campaigns, opening to third party services and developers Improve relevancy – Focused on improving results for specific use cases (e.g. local businesses, flights, etc.) Product differentiation – Federated search results, integrated user experience across multiple services (e.g. search results + maps)

48. Mobile Usability Research Challenges & Tips

49. Variety of Mobile Devices Challenges Tips Difficult to know and understand user experience across all devices Devices vary by model, browser type, carrier, and input type Moderator may be unable to help users through usability tasks – resetting prototypes, navigating – if interface is not well-known Design team should have a variety of devices simulators / devices on hand for testing Prototype designs should be tested across as many devices as possible Limit recruiting to only devices that have been tested Have a back-up plan with either a basic device or paper prototypes

50. Challenges Tips Mobile Devices are Small and . . . Well, Very Mobile Create a hot zone on the table, encouraging users to keep the device within a narrow frame Consider light source and potential reflection on the phone; continually adjust mid testing; turn-off lights Use remote-controlled video camera (with technician in back room) or have a second technician available in interview room Project the mobile screen on a larger monitor in interview room and back room Creatively screen out participants whose finger size may distract from findings (if non-touch screen) Users move the device around while interacting and explaining, often moving video display out of focus Reflectors on screens and smudges can make video all reflection Small device screens make it difficult to see what users are doing Large fingers and long fingernails can cause unintended device responses

51. Challenges Tips Varied Locations and Use Cases Mobile device usage occurs in a variety of places 59% of Americans check email while in the bathroom* Difficult to recreate the true experience in a lab setting Create opportunities for feedback in context of usage Conduct field studies on beta products MOBILE DEVICE ACCESS Source: Nielsen Mobile 2008

52. Challenges Tips Varied Carrier Network Coverage Network coverage is inconsistent; difficult to predict accessibility during lab testing Difficult to validate that out-of-town lab truly has adequate coverage Visit facility with colleagues who have various carriers to confirm network availability in lab From out-of-town labs, request: Reference clients from last mobile studies List of carriers confirmed to have consistent coverage At a minimum, check carrier coverage in various cities online

53. Screen during recruiting for: Unlimited data plans Willingness to send / receive text messages during study Rescreen again upon arrival for study Challenges Tips Calling / Data Plans Vary Widely Participants may be charged for mobile web access or sample texts sent during testing Participants may change data plan between recruiting interview and date of study

54. Challenges Tips Varied User Experience Most mobile device owners only utilize a small percentage of total device capabilities 43% of mobile subscribers do not use text messaging on a regular basis* Users may have no experience typing in a web address to access a site or with sending SMS messages (which may be critical to your study) If using a prototype phone for downloadable app, users may not be comfortable with it Pre-test participants to ensure appropriate device experience Mobile address typing Send SMS to recruiter Spend 2-5 minutes explaining how prototype phone works * Source: Nielsen 2008 ‘The Short Code Marketing Opportunity’

55. Other Helpful Tips Tell participants to bring recharger Simplify language (e.g., SMS vs. text message) Have a prototype strategy Back-end SMS text simulation Easy URL for access Back-up paper prototypes Index page for easy access to alternative flows Have plan for interruption from phone call or text (resetting session variables) Plan for getting screenshots for your report iPhone: Press hold button and home button at same time (saves into photo gallery) Use documented comps

56. Q&A

57. Thank You. Amy Buckner Managing Partner & Co-Founder, AnswerLab Kris Mihalic Head of Mobile User Experience and Design Research, Yahoo! Twitter: suikris