True Intent: The Best Online Benchmark You've Never Measured
•Descargar como PPTX, PDF•
6 recomendaciones•4,260 vistas
UXPA 2013 Annual Conference - Wednesday July 10, 2013 by Ted Boren Remote testing can tell you how easy your site is to use... for selected tasks. Surveys tell you how visitors feel... but lack performance data. Web analytics tell you where visitors go... but not whether they actually succeed. "True intent" studies bridge these gaps and help your team learn what's really happening on your site, by asking real visitors why they came, tracking where they go, then allowing them to tell you if they succeeded. Work together to build affinity diagrams of intents and conduct a detailed failure analysis for even deeper insights that can shape your team's strategy for years.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (16)
Similar a True Intent: The Best Online Benchmark You've Never Measured
Similar a True Intent: The Best Online Benchmark You've Never Measured (20)
Más de UXPA International
Más de UXPA International (20)
Último
Último (20)
True Intent: The Best Online Benchmark You've Never Measured
- 1. Ted Boren User Research Lead The Church of Jesus Christ of Latter-day Saints THE BEST ONLINE BENCHMARK YOU’VE NEVER MEASURED TRUE INTENT
- 2. Is our site successful? GAPS IN BENCHMARKING
- 3. What do visitors want to do? (And can they do it?) Where do they go? (How long does it take?) What do they think of their experience? (Who are they anyway?) What you want to know about site visitors’ Experience …
- 4. Remote Usability Testing Web Analytics Online Surveys True Intent What you want to know about site visitors’ Experience …
- 5. “True Intent” in a Nutshell 1. Intercept live visitors. 2. Ask them why they came to your site. 3. Ask them to go do that. 4. Track where they go and how long it takes. 5. Let them tell you when they succeed— or give up. 6. Ask satisfaction and profile questions.
- 6. True Intent Studies are “Time-Aware” “Remote research opens the door to conducting research that also happens at the moment in people’s real lives when they’re performing a task of interest.” “Time-awareness in research makes all the difference in user motivation: it means that users are personally invested in what they’re doing because they’re doing it for their own reasons, not because you’re directing them to; they would have done it whether or not they were in your study.” ~ Nate Bolt and Tony Tulathimutte, “Remote Research,” page 10 (italics added).
- 7. CASE STUDY: True Intent Study for LDS.org* * The flagship website for the Church of Jesus Christ of Latter-day Saints, my employer
- 8. Give me the details on this True Intent thing! “WHEN, HOW, WHO?”
- 9. WHEN in the development cycle do I do this? During planning… … or after deployment.
- 10. BEWARE: Variation by hour, day, or special event Thursday… Friday… Saturday…Sunday…Monday… Personal study Find something for my family Thursday… Friday… Saturday… Sunday… Monday… Prepare a lesson Prepare a talk Search Intents on LDS.org
- 11. Thursday… Friday… Saturday…Sunday…Monday… Personal study Find something for my family Thursday… Friday… Saturday… Sunday… Monday… Prepare a lesson Prepare a talk BEWARE: Variation by hour, day, or special event Search Intents on LDS.org
- 12. TYPICAL STREAM: A Whole Week? Day? Month? Thursday… Friday… Saturday…Sunday…Monday… Search Intents Declining on the Weekend (spiking on Monday) Personal study Find something for my family Thursday… Friday… Saturday… Sunday… Monday… Search Intents Spiking on the Weekend Prepare a lesson Prepare a talk
- 13. So if in Doubt—Slow It Down and Spread It Out! • If you invite 100% of your visitors, you may finish too quickly, especially with: • Heavy traffic • Enthusiastic audience So: • Meter the rate (preferred)* • Or turn your survey off and on
- 14. WHO do I invite? (And how?) •Email from a pool? •Link on the page? •Pop-up on the page?
- 18. True Intent’s Golden Question: Why did you come to this website today? Please be as specific as you can.
- 19. Prepare a lesson. 80% of your responses may be as exciting (and detailed) as this one. But the other 10%...
- 20. My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook. ~ LDS.org visitor My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook. My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook. My grandmother died tonight and one of my friends was trying to comfort me. I wondered if there was anything on the site that might help her understand my grief is filled with hope and faith. I shared two articles with her via Facebook.
- 21. its my homepage. helps remind me who i am everytime i get on the computer. ~ LDS.org visitor
- 22. So when asking the Golden Question… Beware of Qualifiers! And Seize Today!
- 23. The One (and only) Task: “Do What You Came To Do.”
- 24. Visitors make their way through your site… ?
- 25. … but sometimes they may be hard to track. ?
- 27. The tool continues tracking time and path…
- 28. … until they say they are done.
- 29. Primary Post-Test Questions Goals Satisfaction & Relevancy Problems, Anything Else?
- 30. Secondary Post-Test Questions •Key pivots for slicing and dicing •Demographics / profile •System Usability Scale (SUS) or similar instruments… only if external comparisons are important and feasible •Be thinking ahead to future comparisons… how will you know you’ve improved? •What data will speak to your stakeholders? •But KEEP it SHORT!
- 31. Q: How long do you run this thing? A: “It Depends.” • Site volume • Percentage invited • Survey length • Audience engagement… and incentives* • I’ve finished in a day or two, or in 4-6 weeks. • Shoot for at least 300-400 responses: + More variety of audience and intents + More subgroups to slice and dice = More responses needed
- 32. A FEW QUESTIONS?
- 33. 300-400 responses later… “ANALYZE THIS!”
- 34. Basic Quantitative Stats Success Rate Average Satisfaction Ratings 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Abandoned task Completed task 1 2 3 4 5 6 7 I found what I wanted. It was easy to find what I wanted. Within the next few weeks, I'll probably study, share, or use the information I found… Strongly Agree Strongly Disagree
- 35. Breakouts: Stated Goals, by Age Range • Younger LDS.org visitors more likely looking for personal study or inspiration… • Older visitors come for these reasons too, but also more for utilitarian purposes. Personal Study Find something inspirational Find something for my calling Use the calendar, directory, maps, or leader tools 18 to 34 35 to 54 55 and older
- 36. Benchmarks: LDS.org Search Success Version 2 was better overall: • Especially for preparing a lesson… • But not so much for finding something inspirational… Prepare a lesson Find something inspirational Version 1 Version 2
- 37. ANALYZING QUALITATIVE DATA: The Affinity Diagram • A thematic, hierarchical view of qualitative data points. • A way to see what issues arise out of the data, instead of imposing a pre-determined structure.
- 38. Sample Affinity Diagram Excerpt: 1. I visit LDS.org often for inspiration and information 1.1. I want to be inspired and strengthened 1.1.1. I am looking for something to help me feel the Spirit 1.1.2. I want quick inspiration during a break in my day 1.1.3. I start on LDS.org to keep “spiritually minded” This is the data we start with... These are categories we generate. Intent: “I try to read a quote or watch a Mormon Message every time I get on the internet. It helps me stay spiritually minded.” (Female, 35 yrs) Intent: “I come every day as I start my browser, to keep me focused.” (Male, 62 yrs) Intent: “LDS.org is my homepage. It helps me remember who I am and what I stand for before perusing the internet.” (Female, 17 yrs) Intent: “Its my homepage. helps remind me who i am everytime i get on the computer” (Male, 25 yrs)
- 39. Preparing Data for the Affinity Diagram •Export the data to a spreadsheet •Create one row per comment*, including all data needed for context •Use “mail-merge” functionality to pull quotes and key contextual data (age, experience, success ratings, etc.) into a document formatted for printing the notes. •And then…
- 40. Meet Your New Best Friend… •Post-It Notes for Printers •Print a few merged notes onto regular paper to check alignment •Print a few at a time on manual feed to avoid jams
- 41. Organizing over 800 comments Naming and grouping 169 categories (11 at the top level) Conduct an Affinity Diagram Workshop
- 42. Finished diagram, with three levels of headings.
- 43. Record the Diagram Headings Only Record only the headings in a document or spreadsheet More Complete Associate each data point with its headings in a spreadsheet Formal Report Re-merge the headings and data back into a formatted document, with an introduction, findings, etc. REPORT Background Key Findings
- 44. A FEW QUESTIONS?
- 45. Analysis Steps LDS.org Example 1. Filter to those who a failed a specific task. * Looking for a location 2. Review all the data you have for each instance: • Stated intent • Path taken, search terms, etc. • Biggest problem • Other survey questions that might be relevant Trying to find the DC Temple Home > Maps < Home > Directory < Home > Temples > Find a Temple > Washington DC Temple Didn’t see Temples under Families Identify patterns and themes Temples not seen under “Families” Suggest solutions, or follow-up questions Broaden the heading. Conduct some tree testing. Failure Analysis
- 46. Failure Cases for LDS.org Almost all problem cases fell into one or more of the following categories: •Difficulty finding things in menus •Problems with lack of quotation marks in Search •Content lacking or spread across multiple sites
- 47. Mitigating Challenges It can be difficult to get enough context to conduct an effective failure analysis. This works best when you have: • Additional information derived from the URLs captured (e.g. search terms or other parameters) • Live pilot sessions to provide context for common experiences. • “Biggest problem” survey data. Be ready to accept that some paths will still be inexplicable.
- 48. What Next? •Conduct interviews or tests on key areas. •Adjust or create audience segments. Enrich personas. •Prioritize top intents for future development (or conduct a follow up survey to quantify themes from the affinity diagram). •Fix common problems from failure analysis. •Re-test after you’ve had new changes deployed for a while; compare quantitative results.
- 49. Archive It! But don’t just forget about it. True Intent data has a good shelf-life…
- 50. How Long Does This Take? • For 300-400 intents: • Creating the online study, collecting data, running quantitative stats: 40-60 hours for the facilitator* • Creating the affinity diagram: 20-40 hours for the facilitator* • Failure analysis: 20-60 hours for one analyst** • Total: 80-160 hours of your time, plus team time • Can be less with shortcuts • Should be less the second time around
- 52. “Thanks for leading this exercise. I will admit I was a bit skeptical going into these meetings (who isn’t skeptical of meetings that take you away from already-pressing deadlines?). But I’ve been really pleased with the process and results, and I think they will be of great help in determining our future strategy.” ~ Member of Team 1
- 53. “I mentioned this study in two different high-level meetings this week, and there is a great deal of interest and enthusiasm about the way the study was done and in seeing the results.” ~Member of Team 2
- 54. “Thanks a ton for your work. As I’ve said before, the results of the study will help inform our policies & site development going forward in big ways. We’ve learned a lot – I’ll share appropriate info with [other key stakeholders].” ~Member of Team 3
- 55. No time for all of this? Here are some ways to trim time and budget. SHORTCUTS
- 56. Shortcut #1: TRIM SCOPE Skip 1 or 2 of the 3 main activities: •quantitative benchmark •affinity diagram •failure analysis* Or collect the data for all 3, but wait on the analysis for 1 or more.
- 57. Shortcut #2: USE WORD CLOUDS •One cloud per open-ended question (or per set of closely related questions) •Dump all comments into a single text file •Standardize closely related word forms
- 58. “Please tell us why you came to this website today.”
- 59. “What was the biggest problem you encountered in your session today?”
- 60. “Anything else you'd like to tell us about your experience today?”
- 61. Conduct a “True Intent” Study 1. Intercept live visitors. 2. Ask them why they came to your site. 3. Ask them to go do that. 4. Track where they go and how long it takes. 5. Let them tell you when they succeed— or give up. 6. Ask satisfaction and profile questions. Shortcut #3: Conduct a “True Intent” Study SURVEY 1. Intercept live visitors. 2. Ask them why they came to your site. 3. Ask them to go do that. 4. Track where they go and how long it takes. 5. Let them tell you when they succeed— or give up. 6. Ask satisfaction and profile questions.
- 62. FINAL THOUGHTS
- 63. Evaluating success on an actual, self-defined task Large numbers of paths and times on task Feedback & profile data (qualitative & quantitative) True Intent
- 64. We could call this: A single-task remote unmoderated interview-guided usability test with integrated analytics and survey questions.
- 65. But it’s really all about: True Tasks True Success True Paths True Profiles True Satisfaction True Intent
- 66. FINAL QUESTIONS?
- 67. References & Links Official Website of the Church of Jesus Christ of Latter-day Saints: • http://lds.org Nate Bolt and Tony Tulathimutte, “Remote Research” • Book: http://amzn.to/18nEWtb • Supplementary Site:http://remoteresear.ch Jared Spool, “The KJ Technique”: • http://www.uie.com/articles/kj_tec hnique/ Caroline Jarrett, “Surveys That Work”: • http://rosenfeldmedia.com/books/s urvey- design/blog/do_incentives_help_to _improve/ Creative Commons Licenses: • Attribution, Non-commercial: http://creativecommons.org/lic enses/by-nc/2.0/ • Attribution, Non-commercial, No Derivatives: http://creativecommons.org/lic enses/by-nc-nd/2.0/
Notas del editor
- Who am I?I’m Ted Boren.I began my professional life as a technical writer, became frustrated with documenting poorly designed softwareI went back to school to earn my Masters Degree from the Department currently known as “Human-Centered Design and Engineering” at the University of Washington, where I focused on design and user research, and especially the use of thinking aloud in usability testing.I was then hired as a usability engineer at Microsoft, where I worked on mapping software, the MSN portal, and other projects.After about 6 years, I left Microsoft to work for my Church, the Church of Jesus Christ of Latter-day Saints, also known as the LDS or Mormon Church.Since working at the Church, I’ve played a number of roles in user experience -- information architecture, interaction design, and user research. I’m currently a User Research Lead.I’ve conducted lots of studies of various kinds in all that time, but some of my favorites– and yet little talked about in industry – are true intent studies. Hence this presentation. >
- Every team wants to measure their site's success, but how? They have lots of questions they want answered, questions like these
- We want to know a lot of things about our audience and their experience on our site…
- There are a lot of tools available to help us answer those questions…Remote Usability Testing: I can define some tasks and see how well people succeed on those… That’s a reasonable thing we do in formative studies all the time. But in a benchmark… I kind of want to know how well actual visitors do on THEIR ACTUAL TASKS…Web analytics: I know where visitors go on their own… but not what they intended to do nor whether they are satisfied… nor who they are…Online surveys: I know what respondents say… but not what they really do…In our work, and maybe yours, we needed something that combines elements of each of these, that ties them all together and crosses the gaps. True intent studies are that thing.I did not coin this phrase, but you’ll be hard-pressed to find a very detailed description of how to conduct a true intent study effectively. Hopefully this presentation helps.
- You’ll notice that while any survey tool can suffice for 1,2 and 6, you need a remote usability testing tool to accomplish 3-5.My demos today will use a remote usability testing tool called Loop11, but there are a number of other tools that would also work.Each has its own strengths in terms of price, feature set, and so forth; I’ll let the vendors make their case.I believe Loop11 and UserZoom are both represented here at the conference if you’d like to talk with them.Nate Bolt’s site, remoteresear.ch also has a list of other automated usability testing tools that could work. See my reference slide at the end for a URL.There. That’s the whole technique. If you’ve conducted surveys and remote automated usability tests before, you probably now know enough to go run a basic true intent study.This remainder of this presentation focuses on some techniques and shortcuts I’ve found to make them easier to conduct and to help you leverage the data you collect for more strategic impact.
- One important point to make is that True Intent studies are Time Aware.What does that mean and why does it matter?To quote Nate Bolt and Tony Tulathimutte:Tulathimutte = pronounced “TOO-la-tee-MOO-tee” (Per personal website http://tonytula.com/about/ )
- LDS.org is the flagship website for the Church of Jesus Christ of Latter-day Saints.The website’s primary audience is members of the Church, from lay clergy and teachers to the general membership.This audience is composed of members from many countries, age groups, education levels, and duration of membership.We have run several true intent studies to better understand the reasons people come to the site, what they search for, and what problems they have.Many of the examples in this presentation come from these studies, and are shared with permission. I use these studies because I know them best, rather than just making up data.Screenshot from live website on 4 July 2013: http://lds.org
- At what stage of the development cycle do I run a true intent study?Either during planning or after deployment.Sounds contradictory, but:You either want to use the data to set a baseline for what’s coming and to understand your audience before you start…Or you want to find out how well your new version performs, after you’ve deployed it.It is NOT something you’d typically do in the middle of development.Most true intent studies that I have done have been during planning.Planning photo by Flikr user “eclectic echoes” under CC license (attribution, non-commercial): http://www.flickr.com/photos/eclectic-echoes/6681499071/sizes/l/in/photolist-bbqqTR-h8aqS-4PFNU1-7Engva-7zaL96-7zaKE4-7zbi3Q-7zbf57-7z7T4M-7zem9o-7zbzmm-7zaA7V-7zbhqh-7zbzzY-9NyADu-8UQ6YD-7GEApW-CaPC1-7C2bLH-4u54mg-4u5kG2-4u9gqy-4u568V-4u9bsQ-4u5gvZ-4u9kBm-8m6ZGC-9HjHZF-7Kuftn-7Kuh8K-7KycVG-df7Wzi-8R5TNU-8R2LZF-5K8rJo-bYyXSb-5TUdpg-5u7qJ1-5u324K-8GBRC1-8GBRGh-buRkJx-buRkRB-8CtU46-4qzGyk-dZmVSG-dZmVEE-7JJ7aU-7BXziW-6QABZR-8d48o1/ Deployment screenshot from live website on 4 July 2013: http://lds.org
- But when specifically? You have to be careful.Prepare a lesson and a talk for Sunday services clearly spike on the weekend – especially on Sunday.Personal study, finding inspirational things, and looking for things for family are lower on the weekend.Finding things for family and personal study take a sharp increase on Monday (maybe in relation to weekly family night, maybe in relation to renewed commitment following Sunday meetings?)How would your interpretation of the data be different if you gathered data only on Sunday? On Monday?
- Or Monday? Now Personal study tops the list, and more people are finding something for their families than preparing a talk…
- Better to gather for a whole week—a typical stream--both to catch the full spectrum of activity and also to be able to see the weekly rhythm. Your site may have other patterns on a shorter or longer time-scale. What are they likely to be? Daily? Weekly? Monthly?Data like this could be invaluable in helping optimize your publishing schedule and frequency.
- If you are not sure what your “typical stream” look like, better play it safe and collect data over a longer period—at least a week.A challenge with this is that most tools have a limit on how many participants you can use in a single study.This means that if data comes in too fast, you will use up your quota before you’ve completed the period you are shooting for—you may end up with jusr Sunday or Monday data.So you may need to slow it down.As a rule of thumb, shoot for 300-400 responses if you want to build an affinity diagram, as we’ll explain later.More if you want to segment the quantitative data, or if you suspect your visitors have extremely diverse motivations for coming.Ways to “slow it down”:“Meter” the response rate (e.g. invite only 5%)Turn the survey on and off to spread the sample.** (Can be risky. Make sure users don’t get invited to a temporarily disabled study, and make sure they don’t get invited twice.)(Needed to slow it way down to under 2% on one high-traffic site in order to not fill up before we got the full week that we wanted.Needed closer 100% on another low traffic site in order to get the numbers we needed in a month.)Photo by Flikr user WSDOT under Creative Commons License (attribution non-commercial-non-derivative): http://www.flickr.com/photos/wsdot/5241927335/sizes/l/in/photolist-8Zdfnv-KU5uU-aUY3pc-aUXPPg-aUXw5D-aUXw5x-aUXw62-aUXw5P-aUY3p4-aUXw5Z-aUXw6g-aUXPNZ-4tcZi3-4FH2KY-6j1DUY-6igWWU-6CvLZE-5ejE2J-8Zdfm4-2r7e17-6icviX-dmvDdU-9e5w51-6ihdvj-dFxnMi-782cwk-78QgCa-4m3Ue-x8L7R-x8KG7-x8Khh-x8JGy-x8JZQ-6toUo8-6toUqK-6toU2t-6toUfc-6j1AuN-4dzhU7-9NGabs-boSCTr-bwFwMZ-9M2DZZ-9NKJWL-9CBrC9-9NC5eF-9NH11D-dTs6eV-9NAsCc-9NGYu8-7S5bQX/
- Email: PROS: control over timing, easy to meter, maybe easier to get permission, no code or content changes on page. CON: lack immediacy of intent and might not represent real site audience. Not Time Aware.Link: PROS: Fast results, require only a minor content change but no script. CONS: An on-page call to action may be difficult to meter and tends to draw a more enthusiastic crowd.
- The much maligned pop-up invitation typically yields the best cross-section of visitors and intents – it takes advantage of time-awareness. CONS: Need to put some script on the page, may need to meter responses.This screenshot is from a Loop11 study. Invitation greys out the rest of the screen and shows a pop-up invitation with text you can customize. I like to keep it short, but reassure them it will be short and simple. Read example text.
- Depending on your tools and how you set up the study, participantsmay be taken to an instruction page , or instructions may appear in a modal dialog.Keep the instructions as brief as your stakeholders will allow; just explain the basics.
- When they accept the invitation, you show them the first and most important question: Why are you here?Let’s assume a visitor to LDS.orggave this response…
- Let’s drill in on this Golden Question a little bit more. Remember this is time-aware research, so you are interested in what they are doing right now. Not what you do “in general”Not the worst/best experience you ever hadToday, right now.Also ask them to be specific, and don’t try to limit their responses.Let’s look at a few real responses from an LDS.org true intent study to see why it’s important to focus on Today, and to ask for specifics without trying to restrict them too much.
- Not very specific after all, even though we asked for it. A lesson on honesty? Kindness? Looking for a lesson outline? A video clip? What?
- Read first. Did anyone just feel something shift inside? Why? You don’t know this person…The open-ended format opens the possibility of getting something richand potentially moving like this, not just a checkbox saying “Look for articles.”I was asked recently if it would be a good idea to limit responses to forty characters or something like that, to make it “easier to analyze the data.”Where is “the right place to cut this message off? 40 characters? 100? 200? If you were on this team, which part would you want to lose:The difficult reality this visitor faces? Her heartfelt response to it? How the site helped? What channels she used to share what she found?No, I would want ALL of it.If you’re going to ask the question, you might as well let them speak.
- What’s interesting is that in this particular study, one of our first, we specifically asked people to not participate if they came only becauseLDS.org was their homepage. We thought that wouldn’t be “interesting.”Thankfully many people ignored that request.We learned that many—men, women, older users, teenagers--had set LDS.org as their homepage as a reminder of their faith and values before venturing online.This actually helped explain the bounce rate on the homepage in a way the team had not anticipated, and underlined the imporatnce of LDS.org not only as the SITE’s homepage, but as an individual or FAMILY’s homepage.This can have serious consequences for design and content on the homepage—and we would have missed it. As it was, we probably under-represented this intent in our sample because of our misguided prohibition.
- Beware of qualifiersCan backfire and exclude interesting responsesHard to know what you loseSeize today!
- After asking the Golden Question, it’s time to set visitors loose.For this kind of test, there is only one task: Continue to do whatever you were going to do. I typically don’t ask any more profile or other questions before this task; I want to capture the primary data first, and quickly: Their intent, and then their performance. Profile, demographics, etc. are all secondary and can wait until afterwards.Depending on your tools and setup, tasks are either docked at the top of the screen or appear in a modal dialog.There is also some way for the visitor to indicate whether they have completed the task or give up.The system starts tracking the visitor’s time and path.In our case, I am trying to find information about the Washington DC Temple…Let’s say I start clicking around in the menus…
- Temples is listed there under Resources and Family, but there’s so much other stuff I don’t see it. I try Tools instead…
- I may be tempted by Maps or Directory, so maybe I click on one of these. If so, great – the tool can track the click (and by inference that they opened the menu). But if not, I may not know they opened the menu because the tool may not track that kind of on-page flyout behavior.Another frequently problematic case for tracking is authentication, which might cause an error or drop the path if the system requires a sign-in.Or video files might not play properlyOr… something elseSo test out your site in advance, identify problems, and work with the vendor and your dev team to work out the wrinkles.Sometimes (as in the case of the flyout menus), there may be nothing you can do. Decide if it’s OK to move ahead based on the information you CAN collect.For today, let’s assume I go back to that previous menu and click “Temples”.
- I come to a landing page about LDS Temples in general.There are multiple paths to the correct destination from here.Could search, or click “Find a temple” to see a list, or could look on a photo gallery or map.The tool will track each page that loads and associate it with this session. This path data can become the basis not just of page count stats later, but also of a detailed failure analysis.
- I’ve eliminated a few steps to save time, but here I am on the page for the Washington DC Temple, with tabs for the temple schedule, closure dates, and so forth. Here I might decide to click “Task Complete.”The tool would stamp the time elapsed and whether I clicked “Abandon” or “Complete” -- my first quantitative metrics have been collected: success, time on task, and path.Contrast with a usability test, where the researcher may step in and say “You’re done” or decide behind the scenes whether the attempt was successful or not – or web analytics, where you really don’t know whether they feel they’ve succeeded at all – here, you get the judgment straight from the horse’s mouth.They decide based on their own criteria whether they themselves succeeded with their own task.Hard to argue with that…
- After they are done, it’s time to find out more about them and their experience.Goals:A constrained question about why they came to the site. (Quick stats, compared to the richer qualitative question earlier.)Satisfaction and Relevancy Scales:Note that degree of success can be different from ease of use… or relevance of content.Some additional freeform questions:“What was the biggest problem you encountered in your session today?” This can also be very helpful in conducting your failure analysis.“Is there anything else you would like to say about your experience today?”These will often shed additional light on intents that were not shared in the opening Golden Question.
- (Turns out we didn’t need incentives in our case.) You may need incentives for your audience. If so, Caroline Jarrett has some excellent advice for you.Search for “Caroline Jarrett incentives.” See for example: http://rosenfeldmedia.com/books/survey-design/blog/do_incentives_help_to_improve/
- Loop11 and other tools come with out of the box reports for quick reports, but I often want to summarize, combine, and select key data in Excel.Quick and Easy Quantitative Charts:Success ratesQuantitative Survey QuestionsCan also report time data, but beware. (Thank you UXPA reviewer, for pointing this out.)Task times tend to have a lot of variability, compared to a more traditionally scripted usability test. This is probably best attributed to:High variability in intent. Some people are doing long tasks; others are doing short tasks.Getting distracted or multi-tasking. People aren’t constrained by a lab environment, so they might just follow wherever their fancy leads them. This can affect paths as well as time on task.BUT COMPARED TO WHAT? This is where those demographics and other secondary questions come in handy.
- By asking those demographic questions, you not only improve your understanding of the audience,but now you can break out success and satisfaction metrics based on those characteristics.In this case it’s also broken out by specific tasks from our Goals question, and a pattern emerges. (Read)You can imagine other useful breakouts—by device, frequency of visiting the site, gender—whatever is important in helping you segment your audience and their needs.
- The other key comparison could be between versions (old vs. new, or different designs testing live)We can not only tell in this case that version 2 performed better—we can tell which goals had a special advantage (prepare a lesson) and which were not helped much, or even hindered (finding something inspirational).As with any good quantitative research that tells you WHAT happened, this raises a question of WHY that happened…Your freeform comments and failure analysis might provide some insight, or you might need to run a follow-up study.
- Numbers are Great, but A Key Deliverable Can Be the Affinity DiagramYour automated tools are not going to spit this out for you…A thematic, hierarchical view of qualitative data points.A way to see what issues arise out of the data,instead of imposing a pre-determined structure.Used extensively in "contextual inquiry" to help organize large amount of field data.Useful for organizing other qualitative data too—like survey responses.Three levels of hand-written headings (pink at the top, green, and blue),with original comments on printed post-its at the bottom (yellow).Also some incidental team comments and questions (purple).
- Just one section, edited for brevity and privacy, but this is true to themeaning of the actual affinity diagram.We’ve seen this first quote earlier, from a 25 year old man.> Turns out there are more like this – from people of both genders and many different ages.> We group them together to create a category around the idea of staying “spiritually minded” on the internet.> Other comments are grouped to form related categories, like (read above)> These are grouped into higher level categories around being inspired.> The key thing to note is that none of these categories existed before we started the affinity diagram.We are NOT starting with the categories and then stuffing data into them.It starts with the data, bottom-up.
- * I use the three main freeform comments, all thrown in together, because in my experience they ALL shed light on intent.
- Team members sometimes breathe a huge sigh of relief when they realize they won’t be hand-writing these!(not removable labels or other sticky substitutes)May be hard to find in storesAvailable online, for example shop3m.com(search for product ID: 70071213287 for 100 sheets)
- We won’t get into the details of conducting the workshop; if you’ve done these for contextual inquiry or other applications, it’s very similar.Also some descriptions online, including many aspects of the KJ technique described by Jared Spool on the UIE website http://www.uie.com/articles/kj_technique/ Why involve others? Why not just do it yourself?QuickerImproves buy-inDrives the data deep into the minds of key stakeholders. Sometimes the biggest impact from the study comes from this exposure.Who do I invite?Those who have responsibility for requirements and design(product managers, designers, analysts, etc.)Others who need to hear the “voice of the customer”—whose buy-in do you need?5-10 people maxHow much time do I block out?I typically schedule 2 four-hour periods on consecutive days.People will wince, but they will thank you later.Photos by Tom Johnson, an LDS Church employee. All in the photos were also employees.
- Final diagram has three levels of headings, with smaller categories grouped into larger categories.Take time to review the full diagram as a team, vote on key sections to discuss, and have a conversation about insights and next steps.An electronic version created afterwards – 60 pages for the fully formatted report.Photo credit: Ted Boren (FTE of LDS Church)
- You may want to keep the wall-sized affinity diagram around. It took a lot of effort to create and can be a great artifact to bring out form time to time at key decision points.But for on-going use, you need a smaller form factor, something you can easily duplicate and distribute. There are several options. (Read above.)The “More Complete” option seems to strike a good balance of time, effort, and value… You have all the headings, comments, and original data points, but don’t have to wrestle with the mail merge, which can get tricky.
- Sometimes it’s possible to review in detail the experience of those who failed.* For example those who identified a common task on your survey, or whose clickstreams include parameters like search. (If the number failing is small, you can expand it to include some who succeeded but were dissatisfied.)Tool support for clickstream analysis variesEven when it’s available, clickstream analysis is not always useful for cases like this where there are so many possible “correct” paths for the task.Decide how important this kind of analysis is before you choose your tool.
- We analyzed in detail 145 “problem cases” (29% of all cases).These were people who abandoned their task or indicated that they had a lot of trouble.
- Data from our original LDS.org true intent study in 2011 is still being requested and is informing ongoing design and audience research.Photo by Ted Boren, FTE LDS Church, presenter
- For 300-400 responses:Creating the online study, collecting data, running quantitative stats:40-60 hours for the facilitator (more if you do a lot of subgroups and comparisons)Creating the affinity diagram:20-40 hours for the facilitator (depending on how many notes you have and how complete your report needs to be)Failure analysis:20-60 hours for one analyst (depending on failure rate, site complexity, and depth of analysis)
- Rush me and you get a lousy miracle…100 hours is actually a pretty small price to pay to understand why people come to your site, whether they are successful, and how to help those that fail.But don’t take my word for it. Here are some unsolicited comments I received by email after three different projects.
- Three quotes from key stakeholders on three very different sites and studies…
- (skip if short on time)
- (skip if short on time)
- But if you’re in a crunch, you can cut some corners.Sometime time or money is the crunch. Sometimes there are technical problems and you can’t get a remote usability testing tool to work with your site.
- Skip 1 or 2 of the 3 main activities:quantitative benchmark (but this does not take that much additional time, once you’ve collected the data)affinity diagram (but this is often the main point…)failure analysis (often this is what drops out)
- Many free and cool word cloud tools out thereFaster and cheaper than an affinity diagram and failure analysis (but less rich, and more risky)(Also can make nice backdrops for report title and section slides. This one is from search terms on LDS.org. Done in Tagxedo.One cloud per open-ended question(or per set of closely related questions)Dump all comments into a single text fileStandardize closely related verb and noun forms (“homepage” not “home page”, “find” not “finding, found, finds”), etc.
- Play with layout and other options to look for patterns and make a point.Look at the prominent words; filter comments by those words. Look for themes.Can also be interesting to compare to the affinity diagram, and possibly use as a thumbnail representation of the themes.In this case, many of the main reasons reported for visiting LDS.org can be seen from this single word cloud (see wordle.com), based solely on word frequencies in the first question:Prominent are “Talk, Scripture, Lesson, Home, Page (and homepage), Read, Study, and Prepare.”This reflects several of the top categories in the affinity diagram.
- Wordle results for the free-form “What was the biggest problem you encountered in your session today?”Prominent are “Find, Search, and Looking.” This reflects several of the top categories in the affinity diagram around menus and searching.
- Wordle results for the free-form question: “Is there anything else you'd like to tell us about your experience today?”Prominent are “Love, Like, Great.” This reflects a top category in the affinity diagram around having a good experience on the site. This is significant because we did not actively solicit praise (as opposed to us actively asking for “biggest problems”).
- If you are more interested in why people come than how well they do, you could transform your “study” into a simpler “survey” – steps 1-2, 6.Your study becomes an intent survey (intents, satisfaction in general, plus profile questions).(Can even include the Golden Question question in other surveys or studies,then analyze when convenient.)You wouldn’t be conducting a performance benchmark, but you would gather reams of valuable data about your audience and their needs.The omitted steps are about collecting performance data, which is valuable but can add a level of complexity in terms of tools and data analysis.If you only did steps 1, 2, and 6 you can still gain a lot of valuable insight.
- I’ve heard it said before that one drawback to remoteunmoderated testing is that you can’t conduct an interview-guided test, where you base your scenarios in the participant’s responses to a pre-test interview. In a way, true intent studies in fact allow you to do this. You are evaluating performance on an actual task they define themselves in a pre-test survey.You collect large numbers of paths and times on tasks, as you would with analytics—but with the great advantage of knowing what they are trying to do when you analyze the data—and whether they felt they succeeded. With web analytics, the lack of this crucial context can be crippling.You gather satisfaction and profile data about the visitors, not only so you can profile them, but so you can slice and dice the other metrics along segments that matter to you (by device, by experience, by gender, age, language, etc.)
- You could say a true intent study is a single-task remote unmoderated interview-guidedusability test with integratedanalytics and survey questions.
- You could say a true intent study is a single-task remote unmoderated interview-guidedusability test with integratedanalytics and survey questions.But I prefer to say true intent studies are about asking real site visitors to complete their own tasks, then tell you how you did. In other words, you get true tasks, true paths, true success, and true true satisfaction with true user profiles tossed in, should you choose to ask for those details.