Rich Search Results for Low Literacy Users - UXPA 2013

Malcolm Kemeny
Nicole Kerber
Kathryn Summers
Noel Alton
Megan McKeever
UXPA International - July 2013
Rich Search Results for Low Literacy Users
Thursday, July 11, 13

What is Low
Literacy?
What are Rich
Results?
Findings
Agenda
Rich Search Results for Low Literacy Users Kemeny, Kerber, Summers, Alton, McKeever
Research
Overview

What is a Low Literacy User?

Corporate
What is a Low Literacy User?
• 43% of Americans are considered low literacy and
read at or below an 8th grade level.
• Literacy is defined as “using printed and written
information to function in society, to achieve
one’s goals, and to develop one’s knowledge and
potential.”
• Measured using a REALM (Rapid Estimate of
Adult Literacy in Medicine) test.
REALM Test - Side A

Corporate
Recruiting Low Literacy Users
• Contacted various nonprofits and literacy
groups in Baltimore with little success before
settling on ad-hoc recruiting.
• Target group is unlikely to have consistent
contact method and are difficult to schedule.
• Network recruit - participants bring their
family/friends to participate.

Working With Low Literacy Users
• Tread carefully.
• Ask, “Do you think someone else might have problems using this
website? What do you think they would find confusing?”
• Be respectful and friendly.
• Build a relationship.
• Be prepared with all materials to get session completed quickly.

Web Use by Low Literacy Populations

Corporate
Low Literacy Web Use
• Read every word over scanning the page.
• Become disoriented on the page easily - scrolling is
difficult.
• Accept what they see online as truth without verifying
or reading any additional content.
• Less likely to access information online.
• Easily confused on pages.
• Make decisions without the benefit of context.

What are Rich Results?

Corporate
• Information seeking aid incorporated into search
engines.
• Built to provide answers to queries on the primary
results page.
• Presented above the rest of the search results in large,
visually distinct, unambiguous text.
• Increases user satisfaction by presenting relevant
content where it can easily be found.
• Relatively new. Google started presenting rich results in
early 2010.https://support.google.com/webmasters/answer/99170?hl=en
Rich Search Results

Corporate
• Style of results vary by topic.
• Usually come from a “do” search or a
simple question such as, “What day is
thanksgiving?”
• 20-30% increase in click through rates
for websites that support rich results.
http://www.wpromote.com/blog/seo/smx-advanced-2012-day-1-part-1/
Rich Search Results

Corporate
Rich Search Results
• Rich results are still in their
infancy and not fully supported.
• Search engines must continue to
improve support for misspellings,
phrasing, and formatting.

Search Behavior Study

Purpose:
• Understand the presentation of rich results in a search engine results page (SERP) as it
relates to information retrieval by low literacy users.
Test Plan:
• 30 low literacy participants, who read at/below an 8th grade reading level, were recruited.
• Literacy levels were determined by administering the Rapid Estimate of Adult Literacy in
Medicine (REALM) test.
• Participants were asked to complete 4 tasks using one of three search engines; Google,
Bing, and Yahoo!. Each participant used one of the three engines exclusively throughout
the test. All participants were tested with a Tobii t60 (remote) eye tracker.
• Each task was designed to increase the likelihood of seeing rich results.
• $40 cash incentive for one hour of participation.
Methodology

Methodology
Test Scenarios:
1.You and a friend want to go see the movie Looper tonight at the Landmark theatre.
What are the showtimes for tonight?
2.Your daughter has a question on her homework about the word fortuitous. You want
to make sure you give her the right answer. Look up what the word means on the
web.
3.You are planning your families’ thanksgiving dinner. Look up what day Thanksgiving is
this year.
4.Your son has been invited to a birthday party at Port Discovery. You want to call and
ask about parking. Find the phone number for Port Discovery.

Participant Demographics
College Graduate
3%
Some High School
33%
No High School
3%
High School Graduate
37%
Some College
23%
Participant Education Levels
3rd Grade and Below Grade 4-6 Grade 7-8
18
9
3
Participant Reading Levels
60+
3%
50-59
27%
40-49
10%30-39
37%
18-29
23%
Participant Ages
• 30 Total Participants.
• 56% access the Internet primarily
through a mobile device.
• 60% rated themselves as very
comfortable using the Internet.

Results Overview
• When some users reached a results page with rich results, they were able to
understand, interact with, and learn from rich results
• Identified THREE key behaviors among participants which limited their ability to
learn from rich results -
1. Participants used natural language queries, which search engines had trouble
providing rich results for.
2. Participants relied heavily on autocomplete, which often led them down the
wrong search path due to misspellings.
3. Participants often overlooked rich results when they were displayed.

Success with Rich Results

Corporate
• Users who saw, understood and
interacted with the rich results were
generally pleased with the
information.
• Users were visibly surprised that they
were able to find the information they
were looking for easily. Users
interacting with rich results did not
click further to verify the information,
and overall task time was reduced.
Top: P01 - Alvin
Bottom: P16 - Earlene

P17 - Kennis

P07 - Danielle

P20 - David (offset)

Top: P26 - Victoria

Bottom: P06 - James

Issues with Rich Results

Issues with Rich Results
Task Found and Reported Overlooked
Movie Showtime 1 time 7 times
Define:
Fortuitous
4 times 5 times
What day is
Thanksgiving?
8 times 7 times
Phone number
lookup
6 times 5 times
Rich Results were displayed 45 times out of a possible 120.
Rich Results were discovered and used 19 times out of 45.

Difficulty with Computers
• The participants during testing that tested as very
low literacy had a great deal of difficulty with the
study tasks -
• 3 participants tested as reading below a 3rd
grade level
• 9 participants tested as reading between a 4th
and 6th grade level.
• One participant entered ‘Pour Disrovr’ while trying
to find the phone number for Port Discovery’. He
later ended up looking at the Pottery Barn site.
“It’s not really hard, I
just gotta be more
specific to actually get
to where I want to go. ”

Difficulty with Computers
•Some test participants were not able to understand the goals of each
task. These users had trouble during the test and many became
visibly frustrated.
•Other users were familiar with web use but not comfortable using
search tools. Once participant began every search with WWW. in the
search bar.
•Existing search tools do not adequately support the needs of users
new to the web.

Natural Language Queries

Corporate
• User queries frequently matched
the language used by the test
moderator in the prompt.
• These natural language queries
often resulted in unsuccessful
searches.
Participant Search for Task Prompt of “...find the phone number for Port Discovery”
`

Corporate
RECOMMENDATIONS: Search
engines should work on
providing rich results for
natural language queries. This
will simplify information
seeking for users of all literacy
levels.

Autocomplete

Autocomplete
• Users relied heavily on search engine autocomplete to help guide search.
• Autocomplete was used as a way to help users avoid typing and spelling, often a
difficult activity. Users would type one letter, look to the autocomplete, type
another letter and check again.
• Autocomplete failed as user’s queries moved further from the correct spelling.
• Some users would look at the keyboard as they typed, missing the autocomplete
suggestions altogether.

Corporate
RECOMMENDATION:
Autocorrect should support
error correction as the user
types. A search for
“Thanksgaving” should still
return an autocomplete
suggestion and rich results for
Thanksgiving.
Autocomplete

Corporate
RECOMMENDATION: Participant
queries that receive autocorrect
results should support rich results.
Search engines currently support
autocorrection, showing rich
results for these queries can help
with information retrieval.
Autocomplete

Overlooking Rich Results

P13 - Joanne

P05 - John

P02 - Kevon

P06 - James

Overall

• Rich results can be particularly beneficial to low literacy users, who tend to settle on
the first answer they find in a search.
• There were THREE behaviors from participants which impacted their ability to use
rich results -
1. Participants used natural language queries, which search engines had trouble
providing rich results for.
2. Participants relied heavily on autocomplete, which often led them down the
wrong search path due to misspellings.
3. Participants often overlooked rich results when they were displayed.
Overall

Overall
1. Search engines should move to provide
rich results for natural language queries.
2. Search engines should
unobtrusively support live autocorrect
inside of the autocomplete options.
4. As rich results become more common,
users will start to expect rich results.
2. Search results displayed with autocorrect
should support rich results.

MALCOLM KEMENY NICOLE KERBER
KATHRYN SUMMERS
KSUMMERS@UBALT.EDU
NOEL ALTON
NTALTON@GMAIL.COM
MEGAN MCKEEVER
MCKEEVER.MEGAN@GMAIL.COM
THANK YOU
MKEMENY@PAYPAL.COM NKERBER1@GMAIL.COM

Rich Search Results for Low Literacy Users - UXPA 2013

Recomendados

Recomendados

Más contenido relacionado

Más de Nikki Kerber

Más de Nikki Kerber (6)

Último

Último (20)

Rich Search Results for Low Literacy Users - UXPA 2013