Improving Forms with User Experience Testing and Eye Tracking
1. Improving Federal Forms with User
Experience Testing and Eye Tracking
Jennifer Romano Bergstrom, Jon Strohl,
Andrew Hale, Sarah Keaton
May 17, 2014
AAPOR | Anaheim, CA
@romanocog #AAPOR
2. 2
@romanocog #AAPOR
+ emotions and
perceptions =
UX
Usability = “the extent
to which a product can
be used by specified
users to achieve
specified goals with
effectiveness,
efficiency, and
satisfaction in a
specified context of
use.” ISO 9241-11
2
3. 3
User Experience Design (P. Morville): http://semanticstudios.com/publications/semantics/000029.php
User Experience
@romanocog #AAPOR
3
4. 4
OBSERVATION
AL
+ Ethnography
+ Time to complete task
+ Reaction time
+ Selection/click behavior
+ Ability to complete tasks
+ Accuracy
IMPLICIT
+ Facial expression analysis
+ Eye tracking
+ Electrodermal activity (EDA)
+ Behavioral analysis
+ Linguistic analysis of
verbalizations
+ Implicit associations
+ Pupil dilation
EXPLICI
T+ Post-task satisfaction
questionnaires
+ In-session difficulty ratings
+ Verbal responses
+ Moderator follow up
+ Real-time +/- dial
UX Data
4
@romanocog #AAPOR
5. Traditional UX research
is good at explaining
what people say and do,
not what they think and
feel.
Why should we measure implicit?
@romanocog #AAPOR
8. 8
User Experience Design (P. Morville): http://semanticstudios.com/publications/semantics/000029.php
@romanocog #AAPOR
Plain language
*Public Law 111-274. Retrieved from http://www.plainlanguage.gov/plLaw/
• President Obama
signed the Plain
Writing Act of 2010
requiring federal
agencies to use “clear
Government
communication that
the public can
understand and use.”
– Find what they need
– Understand what
they find
– Use what they find
to meet their needs
8
9. Jarrett, C., & Gaffney, G. (2009) Forms that Work: Designing Web Forms for Usability. Morgan Kaufmann
Government forms matter!
@romanocog #AAPOR
9
Poor forms lead to:
• Increased errors
• Lower response rate
• Higher nonresponse
• Loss of time
• Increased frustrations
• Poor user experiences.
10. Jarrett, C., & Gaffney, G. (2009) Forms that Work: Designing Web Forms for Usability. Morgan Kaufmann
Good forms = good UX
@romanocog #AAPOR
10
11. Issue: Instructions are not read, and the starting point is not clear
*Hochheiser, H. & Shneiderman, B. (2000). Performance benefits of simultaneous over sequential menus as task
complexity increases. International Journal of Human-Computer Interaction, 12(2), 173-192.
UX Best Practice: Use step-by-step menus for simple forward-moving tasks.*
“It’s not intuitive at all for where I’d go to
find information.”
Users select their desired job to plan a
career path.
• Keyword search
• Select from a series of dropdowns.
UX Findings:
• Instructions are not read
• Users enter their current job, not
desired job.
• It is not clear that users have the
option to either search or enter job
details when entering a starting job.
Fixation count gazeplot shows the participant looks back and forth
between the two input options
@romanocog #AAPOR
11
12. Issue: The layout does not encourage left-to-right processing
UX Best Practice: Allow users to enter information in ways that are logical to them.
@romanocog #AAPOR
12
Version 1 PrototypeVersion 2
Users enter their household TV viewing for one week.
13. Issue: The location of navigation buttons cause confusion
*Niemela, M. & Saarinen, J. (2000). Visual search for grouped versus ungrouped icons in a computer interface. Human
Factors, 42(4), 630-635.
UX Best Practice: Clearly differentiate navigation elements from one another.*
Users enter the job
title or keyword that
they want to search
for and click the
arrow pointed to the
right to run the
search.
UX Finding:
• The “Save this Search” button is closer to the box where the job title is entered than the
arrow icon that runs the search.
• Some participants incorrectly clicked on “Save and Search” to continue.
After clicking on the Save This Search button, one participant said: “Hmmm, create a new account and
I actually don’t want to create a new account…Maybe I can just scroll down and have a look here.”
@romanocog #AAPOR
13
14. Issue: It is not clear how to get started and enter data on the form
1: You want to compare the average cost of a public university to the average cost of a private university. Using this site,
approximately how much more is a 4-year private university likely to cost?
2: You’re deciding between the University of Virginia Main Campus and Georgetown, and you’re interested in graduation
rates. Which university graduates a higher percentage of students who enroll?
UX Best Practice: Ensure clickable parts of the form appear clickable.
Users must click “Add a School” to get started.
They select a college or university and enter
financial data.
UX Findings:
• The “Add a school” button does not look like an
actionable button. It is not clear that this is how
to start.
• The form did not look like it was clickable.
• Users were unsure how to proceed once they
had successfully brought up two schools side-
by-side.
10
17
2
15
10
4
Task
Accuracy
Public/Private Task1 Graduation Rate Task2
Success
Fail
n/a
@romanocog #AAPOR
14
15. Issue: The text on the home page is dense, and the form is not visible.
UX Best Practice: Ensure that the location of page elements do
not create the illusion that users have reached the bottom of a
page when they have not.1 Facilitate scanning by using clear
headings, short phrases and sentences, and short paragraphs. 2
Users enter information into the form and are shown their
cancer risk.
UX Findings:
• Users do not read the dense text.
• The form to enter information is below the fold, and the
layout creates the illusion that the bottom of the page
has been reached.
“It seems like a lot of information. I would skip past it and use the tool.”
“I see blah, blah, blah. A lot of information and I don’t really care for it.”
“I didn’t see it because I didn’t scroll down. If I saw this on the first
page, I would have been much happier.”
1: Ivory, M.Y., Sinha, R.R., & Hearst, M.A. (2000, June). Preliminary findings on quantitative measures for distinguishing
highly rated information-centric web pages. Proceedings of the 6th Conference on Human Factors and the Web. Retrieved
November 2005, from http://www.tri.sbc.com/hfweb/ivory/paper.html.
2: Morkes, J. & Nielsen, J. (1997). Concise, SCANNABLE, and objective: How to write for the Web. Retrieved November
2005, from http://www.useit.com/papers/webwriting/writing.html.3: Rosenholtz, R., Li, Y., Mansfield, J. & Jin, Z. (2005).
@romanocog #AAPOR
15
16. See Bristol, Romano Bergstrom & Link, AAPOR 2014 for more about the UX across devices.
• Not necessarily the same UX across all devices
• Different issues occur with different modes
• Match user expectations and survey objectives
• Conduct UX testing with real users
Assess the UX across devices.
@romanocog #AAPOR
16
17. Thank you!
• Twitter: @forsmarshgroup,
• LinkedIn: http://www.linkedin.com/company/fors-marsh-group
• Blog: www.forsmarshgroup.com/index.php/blog
Jennifer Romano Bergstrom
@romanocog
jbergstrom@forsmarshgroup.com
AAPOR 2014| Anaheim, CA