This presentation to FedEx stakeholders at their World Usability Day expo summarizes the tradeoffs between onsite / lab-based vs. remote user testing for usability studies and provides tips for both methods.

1. lab vs. remote usability testing presented 11/12/09 at FedEx World Usability Day Expo

2. Lokion is a boutique interactive agency. The depth of our services in strategy, design and technology outweighs our size, as do our clients and partners.

3. Lokion Services strategy user experience and usability information architecture business analysis content strategy quality assurance design user-sensitive design branded content creation and management multimedia production technology expert custom software development systems architecture ecommerce software implementation

4. Client Sampling AutoZone Viking Range Corporation FedEx International Paper Memphis Grizzlies First Tennessee / First Horizon Hilton Nokia Deluxe Corporation Hewlett Packard FedEx Institute of Technology Smith and Nephew ServiceMaster Clean AmeriSpec National Ornamental Metal Museum FedEx Kinko’s Accredo Lightyear Capital Harrah’s eDiets.com David Lusk Gallery Cellular South Jeppesen Scripps Network Stephens Mimeo.com Rhodes College Luminetx Corporation Strategic Financial Services SRA/Environmental Protection Agency Cap Gemini Horizon Ag mbi Bridges

5. Strategic User Experience

6. Strategy: People + Process Information Architecture (IA) Sitemapping Use Cases Wireframes Prototyping User Experience (UX) Thinkfactory Audience Research Persona Creation Usability Reviews Benchmark Studies User Testing Web Analytics Accessibility Audits Buzzwatch Business Analysis (BA) Business Requirements Gap Analysis Process Analysis and Improvement Training and Change Management Content Strategy (CS) Unified Strategic Plan for Content Content Gathering, Creation and Editing Content Style Guides Content Audits Content Management Systems and Training Search Engine Optimization (SEO) Personalization and Scenario Marketing Quality Assurance (QA) Test Plan QA Test Cases QA Management and Reporting QA Test Participation Regression/Smoke Testing User Acceptance Testing (UAT)

7. Why Lokion? Passionate user experience advocates Conducted 200+ usability test sessions including 28 remote test sessions Facilitated 44 user test sessions for FedEx including Ship Online, Billing Online, Critical Inventory Logistics and Investor Relations Lokion is 10 years old and has worked with FedEx consistently for 9 of those years. Other usability testing engagements include First Tennessee / First Horizon, Service Master, Cellular South, Viking Range, SkyTel, Ducks Unlimited and Rhodes College

8. Lokion and FedEx "We needed to redesign FedEx® Billing Online to be more useful than ever, and more efficient than the paper invoicing process. We selected Lokion Interactive because of their knowledge of FedEx technology and processes, and their design expertise. This was definitely the right choice, as they exceeded our expectations and project goals with their Prototype development and Usability management. We saw a measurable customer satisfaction improvement with the new design.“ -- Derrick Reives, Sr. Marketing Specialist "We had a great experience working with the Lokion team on the FBO Project, our first dUX team partnership on a usability study. Lokion provided excellent support in setting up the usability sessions and insightful feedback on each of the prototype revisions. Success metrics were determined pre-usability and their Usability Report included very detailed improvement measurements and actionable recommendations. We look forward to working with them more in the future.“ -- Jane Fulmer, Marketing Specialist Advisor "We recently worked with Lokion to perform usability on an improved user experience for an existing fedex.com application. The team at Lokion provided great value in recruiting the right customers, managing the project deliverables, facilitating the sessions and providing a top notch summary and detailed usability findings report. I strongly recommend their services!“ -- Beth Newby, Marketing Principal "Lokion was an effective business partner for every step of the prototype development and usability. We were able to get great information from local and remote customers without spending travel money. Their facilities and their facilitator were fabulous!“ -- Ann Crislip, Sr. Marketing Specialist

9. Lab vs. Remote Usability Testing VS

10. Lab-based Usability Testing Individual representative users physically come into a lab setting and perform a series of pre-defined tasks using a web site, application or prototype. One or more observers in the same room or an adjacent room record the level of success, usability issues encountered and time it takes to complete each task.

11. Usability Lab :: Typical Setup

12. Participant Room

13. Observation Room

14. Guinea Pig Syndrome?

15. “They make frequent testing easy, manage it well, and give great, timely results.” Lokion’s onsite usability lab — Vice President, Internet Development Online Financial Services First Tennessee

16. unobtrusive test environment

17. comfortable observation room

18. Remote Usability Testing Individual representative users stay in their own native settings using their own computers and perform a series of pre-defined tasks using a web site, application or prototype. One or more observers connect with the user over the phone and Internet to record the level of success, usability issues encountered and time it takes to complete each task.

19. Remote Testing :: Typical Setup

20. Tools of the Trade Integrated: Techsmith Morea / Uservue, Ethnio, Loop11, OpenVULab, TestLed Automated: Chalkmark, ClickTale, Keynote WebEffective, LabsMediClickHeat, m-pathy, OpenHallway, OptimalSort, RelevantView, Simple Mouse Tracking (SMT), Treejack, UserFocus, UserZoom, Vividence, Webnographer Recorded: Adobe Connect, Breeze, Camtasia, CamStudio, Co-Pilot, DataLogger, GoTo Meeting, Hello Direct Pro, LiveLook, LK Audio QuckTap, Note Pad, Skype, Trillian, WebEx, Windows Meeting Space

21. Other Variations Moderated / facilitated vs. automated / unattended Traditional agency recruiting vs. online scheduled or live native task recruiting Synchronous vs. asynchronous Quantitative vs. qualitative Baseline, iterative or comparative Electronic vs. paper High- vs. low-fidelity Static, interactive, mockup or live

22. Trade-offs :: Logistics and Technicalities Lab Remote

23. Comparison :: Logistics

24. Tradeoffs :: Interactions

25. Comparison :: Interactions

26. Emotional experience :: non-verbal cues

27. Tips and Considerations Both methods are valid. Remind repeatedly (emails, Outlook appointments and snail mail). Suggest arrival or install / connect 10 minutes prior to session. Make creative use of backups. Privacy agreements matter. Communication helps. Expect glitches (Plan B). Lather, rinse, repeat…

28. Tips :: Lab-based Usability Testing Include a map and parking instructions. It’s the little things (water, snacks, bathroom…). Cash is good….

29. Tips :: Remote Usability Testing Confirm email and phone. Include instructions and demo links. Installation is a hurdle. Practice muting. Amazon gift certificates only need an email address to fulfill.

31. contextual

33. focused

34. personalVS There is a place for both methods. They are complementary. Mix and match.

35. Resources :: Learn More “Comparative Study of Synchronous Remote and Traditional In-Lab Usability Evaluation Methods,” by PrakaashSelvaraj, Virginia Polytechnic Institute and State University, May 19, 2004. http://scholar.lib.vt.edu/theses/available/etd-05192004-122952/unrestricted/Thesis_Prakaash_Selvaraj.pdf “An Empirical Comparison of Lab and Remote Usability Testing of Web Sites,” by Tom Tullis, Stan Fleischman, Michelle McNulty, Carrie Cianchette, and Marguerite Bergel, Human Interface Design Dept, Fidelity Investments. http://home.comcast.net/~tomtullis/publications/RemoteVsLab.pdf “Remote Online Usability Testing: Why, How, and When to Use It,” by Dabney Gough and Holly Phillips, Boxes and Arrows, June 9, 2003. http://www.boxesandarrows.com/view/remote_online_usability_testing_why_how_and_when_to_use_it “Experience Remote Usability Testing: Examine study results on the benefits and downside of remote usability testing,” by VeldaBartek and Deane Cheatham, IBM, January 1, 2003. http://www.ibm.com/developerworks/library/wa-rmusts1/