Final usability report for a graduate class at University of Baltimore comparing two different discovery tools for the library.

1. USABILITY TESTING REPORT:
DISCOVERY (SEARCH) TOOLS OF ENCORE AND SUMMONS
Authors: Heather Mathieson, Debbie Shultz, Nikki Kerber
IDIA 642 - Lucy Holman
May 12, 2010

2. Table of Contents
Executive Summary
Participants
Methodology
Results
Recommendations
Appendices
 Heat-map
 Wireframes
 Consent Form
 Faculty Screener
 Graduate Student Screener
 Script
 Task List

3. Executive Summary
This report contains the results of a user research project conducted between April 14 and May 12, 2011, to
develop recommendations for the University System of Maryland and Affiliated Institutions (USMAI) Library
Consortium in their consideration of several commercial discovery tools to purchase for use in USMAI
libraries. Before a product is purchased, it was determined that usability testing should be performed on the
four main tools under consideration: EDS, Primo, Summons, and Encore. Dr. Lucy Holman’s IDIA 642
Research Methods class at the University of Baltimore conducted the testing as their final project.
To gain insight into each tool’s usability in terms of both negative issues and positive attributes, we
conducted usability testing on Summons and Encore using four participants: two graduate students and two
faculty members. Participants selected had to be from a university or college within the state university
system of Maryland. To conduct the test, we used the Tobii T60 Eyetracker and Software and Morae, a
usability testing and market research software, at the University of Baltimore Usability Research Lab. This
report concerns the findings of our usability testing.
What Are Discovery Tools?
Universities within the state university system of Maryland currently use Research Port, developed by the
University of Maryland at College Park as a front end for Federated Search. (Federated Search is an
information retrieval technology that allows the simultaneous search of multiple searchable resources.) The
university system of Maryland is planning to replace Research Port by purchasing a commercial Discovery
tool.
Currently, when a user logs onto a library website to search for information, he or she faces a daunting task:
how to find something doing independent searches on 75 article databases, local digital content on the
library server, the library catalog, and electronic collections across libraries.

4. Executive Summary (cont.)
This is where Discovery tools come in. Discovery tools, which are middleware that is used as a searching
front end, make searching faster and easier. By entering a single query, Discovery tools index content
simultaneously from a variety of sources such as library catalogs, article databases, and electronic
collections. Discovery tools are available as open source, but the most robust ones come from commercial
vendors.
Summary of Findings
In general, our tests showed that Summons was easier to use than Encore. Both the quantitative data and the
qualitative data proved that this was true. The quantitative data, which measured such things as the task
completion rate, the number of searches needed to complete each task, and the ocular fixations and scan
paths, indicate that participants took longer and needed more searches to complete a task in Encore than
they did in Summons. The qualitative data, which primarily included an evaluation of participant comments
and observations of body language and facial expressions, led us to infer that participants were more
confused and frustrated in Encore than they were in Summons.
Both tools were lacking in feedback to the user, especially in cases in which a search fails because of a
misspelled word. Neither tool offers adequate search hints for frustrated users. However, Encore had many
more issues of major or moderate severity, while most of Summons’ issues were minor. Perhaps the most
serious error that occurred in Encore had to do with the links to additional content that were located on the
right-hand side of the page. Not one user saw these links right away and without clicking on them, search
after search failed. The “Results” section explains the findings in more detail.

5. Executive Summary (cont.)
One of the most surprising findings had to do with one participant’s comments as he was searching. He
spoke about how he tries to use the subject matter terms to get to the right Library of Congress catalog,
indicating that he applies the traditional thought patterns of a library catalog to his search. Discovery tools
are supposed to provide a different search experience for the user than that provided by a traditional library
catalog search. This made us question whether the two tools we tested are providing that experience, or
whether it is a case of users beginning to change old paradigms as they become more familiar with
Discovery tools.
Recommendations
For most issues, our recommendations include suggestions to reorganize screen elements to make
searching less confusing, easier, and faster. Adding more user feedback in the form of search tips and “do
you mean” prompts when a word is misspelled would make the user experience less frustrating. In some
cases, a simple cosmetic change might make things clearer.

6. Participant Overview
User 1 User 2 User 3 User 4
Gender Female Female Male Female
Age 18-29 18-29 60+ 50-59
UB Affiliation Graduate Student Graduate Student Faculty Faculty
Frequency
>10 times per 4-10 times per
Conducting >5 classes 1-2 classes
month month
Research
Website used first Langsdale library Langsdale library
Google Google
in research website website
Academic Search Academic Search
Academic Search Premier; ABI/ Premier; ABI/
Online Databases
Premier; Inform; Research Port Inform; Lexis
Utilized
Psychinfo PyschInfo; Nexis; Research
EBSCO; Business Port
Source Premier

7. Participants (cont.)
User 1 User 2 User 3 User 4
-- If none, which
-- -- -- --
statement?
** Statement
about online #2 #1 #2 #1
databases
Online Research
Good Good Good Excellent
Skills
Length of time
1-2 hours 31-60 minutes 0-30 minutes 0-30 minutes
researching
Frequency
1-3 times per 1-3 times per <1 time per
visiting UB <1 time per week
week week week
library’s website
** Note: The statement numbers, #1 and #2, refer to the appropriate statements
on the screeners. Please refer to the screener to read those statements.

8. Methodology
RECRUITMENT
Graduate students and faculty of the University of Baltimore community were contacted about
participating in a usability test of two discovery search tools. Team members, as well as Dr. Lucy
Holman, used a variety of ways to contact prospective participants, including but not limited to e-mail
blasts, individual e-mails, the UB Students Daily Digest, the UB Faculty Daily Digest and word of
mouth. A total of four participants who were all affiliated with the University System of Maryland were
found to participate in the usability test. Two participants were UB faculty members and two
participants were UB graduate students; all were over the age of 18. Participants were tested in
sessions that ran approximately 60 minutes each.
PREPARATION
Prior to the test dates, the team reviewed two discovery tools, Encore and Summons, to determine how
each tool worked. A screener, test-script, and consent forms were written and duplicated for each
participant. Projects were created in each of the testing software tools, Tobii and Morae.
TESTING PROCEDURE
Each testing procedure began with introductions and a description of the project. The participant was
asked to fill out an informed consent form. A screener was also given to the participant to collect user
profile information (as shown in the profile matrix in the previous section). Each of the four
participants was asked to perform five tasks in each of the two discovery tools, Encore and Summons.
The specific tasks included combinations of searching for articles, books, and audio-visual materials.
A Task List is included in the Appendix, pages 48-50.

9. Methodology (cont.)
Participants were met and welcomed into the reception area of the Usability Lab. The members of the group
took turns being the moderator for each participant. Each participant was set up in the testing room and
logged into the system. If Tobii was used, it was calibrated to the user’s eye movements.
During the test, the moderator followed a written script and emphasized two important points at the
beginning and throughout the test: first, that we were testing the discovery tools and not the user or the
user’s ability to succeed; and second, that because of the test environment, participants might not be able to
fully complete a task. Participants were also encouraged to talk aloud and comment on anything they
wanted to during the test.
The other two team members sat in the Observation Room behind a one-way mirror. They took notes,
observed the users’ actions, and made note of users’ comments. They also recorded any quantitative data.
After the test, participants were asked to summarize their reactions to the two Discovery tools and provide
us with any additional feedback that could help us in our evaluations.

10. Methodology (cont.)
CRITERIA OF USABILITY
We used the following criteria as a measure of the usability in Encore and Summons:
Quantitative Measures
Measure of time participants needed to complete each task
Task completion rate
Number of searches needed to complete each task
Average number of searches needed to complete each task
Qualitative Measures
Conversations with participants after the usability test and answers from the screener questionnaires
Comments from participants
Facial expressions
Body language

11. 11 Results
Pages 15-24 provide an extensive results overview. Please refer to
the Task List, located in the Appendix, pages 48-50, to get
acquainted with all of the tasks we asked our users to complete.

12. Results
Features of Encore and Summons that Support Users
Tool Good Usability Practices Current Features of Web
Both Tools Allow users to use search criteria in a logical way Left side bar with various search options
to further narrow search results produced by a displayed when searching articles in Encore.
basic search. This paradigm will fit users’ mental
models of the search process.

13. Results
Features of Encore and Summons that Support Users
Tool Good Usability Practices Current Features of Web
Summons Allow users to select as much criteria for an Material Type field on the Advanced Search
advance search with as little clicks or selections as screen includes an option that combines books
possible. This way users will spend less time and journals together.
searching and more time looking through
meaningful search results.
Encore Provide a clean and simple user interface, clear of Encore provides a clean and simple user
advertisements, banners, or other added features interface that is fairly intuitive to navigate.
that do not provide a purpose in the search
process.

14. Results
Features of Encore and Summons that Support Users
Tool Good Usability Practices Current Features of Web
Summons Give users plenty of search criteria options in Summons offers a nice Advanced Search form.
order to help narrow their search from the
beginning.

15. Results
Error Severity
The most critical level - The user was unable to correctly complete
Major
task.
Moderate Significant problems caused for the user.
Minor annoyances that slow the user down during the tasks. The user
Minor
leaves frustrated.
Good Indicates that an object is well designed.

16. Results
Tool Issue Results Level Frequency
Encore Users were blind in most cases It took most users anywhere from 5 to 10 Major 4
to finding and/or clicking on minutes to locate this link. One user never
“Other Sources” (WorldCat and found it. Until the user located and clicked
Link+) “Other Sources,” searches failed.
Encore Tool does not clearly show how The user could not understand why certain Moderate 1 user failed
it uses search terms results were listed. The results list did not to complete
highlight keywords to see why certain data task;
was included. Sometimes data was included
that did not appear to contain any of the 3 users had
search terms. significant
problems
Encore Irrelevant search results Search results sometimes seemed completely Moderate 2
irrelevant to the search terms entered. The
“Did you mean” entries often appeared to be
somewhat or very unrelated to what the user
was looking for.
Encore Tool manipulates keywords It appeared that the tool treated the whole Moderate 4
search string as individual keywords and not
specifically as Title, Author, or Subject.

17. Results (cont.)
Tool Issue Results Level Frequency
Both Both tools are unforgiving of When a user misspelled a word, the search Moderate 2
Tools spelling errors. failed. No feedback was provided to bring
the misspelling to the user’s attention.
Unless the user noticed the misspelling and
corrected it, additional searches could also
fail.
Both Neither tool offers search hints Encore did not provide any assistance or Moderate 4
Tools for frustrated users suggestions to users when a search failed. It
did not offer any suggestions as to how the
user could get an unmanageable list (too
many items) down to a manageable list.
Summons offered search tips but they were
below the fold of the Advanced Search page
and users did not always see them.
Both Tool offers multiple ways to Clicking in various search fields produced Moderate 4
tools search for items. search results pages each with a different
look/feel and with different filtering options.
Navigating through these search options
required the user to click multiple times.

18. Results (cont.)
Tool Issue Results Level Frequency
Encore Links for Catalog, Images, and Placement of these links at the very top of Minor 2
Articles are hard to find and the screen made them hard to locate. Their
behavior can be confusing behavior is sometimes puzzling. For
example, when the user clicks Articles, then
clicks Advanced Search, the Articles link
disappears and only the Catalog link
displays, making the user wonder whether
the search is actually being performed on
the catalog.
Summons Users appeared to hesitate when clicking
The tool has two Submit
this button as if unsure of which one to use.
buttons on the Advanced Minor 3
Some clicked the top button; others clicked
Search form.
the bottom.
Not obvious when the tool Because users did not see the “No entries
displayed “No entries found” on found” message, they were unsure whether
Summons Minor 2
the top of the Advanced Search the search was still processing or had
form. finished.
Encore Confusing search results Search displayed “No Catalog Results Minor 1
Found,” at the top of the page; at the bottom
of the page, it displayed a whole series of
results (in one case over 40,000 articles).

19. Results (cont.)
Tool Issue Results Level Frequency
Summons Not obvious when the tool Because users did not see the “No entries Minor 2
displayed “No entries found” on found” message, they were unsure whether
the top of the Advanced Search the search was still processing or had
form. finished.
Both Screens in both tools were It was not always clear where users should Minor 3
tools cluttered, with options for go when starting a search.
searching on the left-hand
side, the center, and
sometimes on the right-hand
side of the page.
Both Discovery tools are not Participant tried to use subject matter terms Minor 1
tools providing a different search to get to the right Library of Congress
experience catalog, applying the traditional thought
patterns of a library catalog to his search.
Discovery tools are supposed to provide a
different search experience for the user than
that provided by a traditional library catalog
search.

20. Results
Number of searches for each task
Encore Summons Encore Summons
User 1 User 2
T1 6 1 T1 6 2
T2 7 1 T2 6 2
T3 4 2 T3 4 2
T4 4 1 T4 5 1
Encore Summons Encore Summons
User 3 User 4
T1 10 1 T1 7 2
T2 10 2 T2 5 1
T3 9 6 T3 6 4
T4 3 2 T4 10 1
Overall, users had the most problem with successfully
completing task 1 in Encore and task 3 in Summons

21. Results
How successful were users in completing tasks?
Participant completed all of the components of a certain task successfully.
Complete
Example: Participant found many articles on Artificial Intelligence that were no
more than five years old and are peer-reviewed.
Participant only completed some of the components of a certain task.
Incomplete
Example: Participant successfully found an article but did not save an article for
retrieval.
Participant was not able to complete any portion of the task.
Failed
Example: Participant did not find a known item: Beowulf

22. Results
How successful were users in completing tasks?
Failed Tasks
15%
Failed Tasks
35%
Completed Tasks
45%
Incomplete Tasks
Completed Tasks 25%
60%
Incomplete Tasks
20%
Encore Summons

23. Results
Average Time on Task in Encore and Summons
Based on our findings, Task 2 in Encore gave users the most problem. It took an average of 7:50
minutes for users to complete the task or end the task before asking to move to the next task.
Similarly, it took users an average of 5:15 minutes to complete or end Task 3 in Summons.

24. Results
Time on Task in Encore and Summons
User 1 User 2 User 3 User 4
Encore
Task 1 4:31 2:44 5:40 2:51
Task 2 7:39 5:29 14:25 3:49
Task 3 1:30 3:49 9:50 4:45
Task 4 4:43 4:39 4:31 3:27
Task 5 0:14
Summons
Task 1 0:42 0:48 1:44 2:44
Task 2 0:40 0:42 4:03 0:50
Task 3 2:41 2:40 7:18 8:21
Task 4 1:29 2:14 6:16 2:22
Task 5
In general, it took users much more time to complete or end tasks in Encore than in Summons. Interestingly, Task 2 in
Encore gave users the most problem, as their completion times are higher.
In Summons, users were much more successful in completing tasks. Interestingly, Task 3 gave users the most trouble
with time rates varying between 2:40 - 8:21 minutes, however these times do not compare to the problems Task 2
gave users in Encore. Comparing Task 2 in Encore and Summons, users cut their time rate by more than half.

25. Interesting Observations
The observations described below include those things that we felt were important or interesting enough to
make a note of while we were conducting our testing.
•
Only one participant found the + button in the Format field of the Advanced Search screen in Encore.
•
Only one participant typed in “peer reviewed” as a search term in the advanced search criteria. The
other participants were unsuccessful in filtering articles by this criteria because there was no specific
option and it didn’t occur to them to include it as a search term.
•
One participant had no successful searches in Encore because this participant never located the “Get
More Results in: Link+ and WorldCat” on the right-hand side of the screen. It appeared from our
observations that the right-hand side of the screen was totally ignored during the usability testing for
this tool with this participant.
•
Only one participant was successful in saving the article for retrieval in Task 5. The other participants
found the article but did not save.
•
Participants who used a basic search and then narrowed their search results using additional search
criteria seemed to have better results with both tools (provided that they had already clicked Link+ or
WorldCat in Encore) than those who started with Advanced Search.
One of the most surprising findings we came across in our usability testing had to do with one participant’s
comments as he was searching. He spoke about how he tries to use the subject matter terms to get to the
right Library of Congress catalog, indicating that he applies the traditional thought patterns of a library
catalog to his search. Discovery tools are supposed to provide a different search experience for the user
than that provided by a traditional library catalog search. This made us question whether the two tools we
tested are providing that experience.

26. Recommendations
Encore Recommendations
Highlight words in results page to show user how terms are being used.
All results should appear below the primary search field.
Use better labeling such as “Get More Results in: Link+ and WorldCat.” If no search results are found,
this should be located where the results would be. If results are found, move this to the top left sidebar
where there is a higher probability a user would look for more information on the page.
Summons Recommendations
Have only one Submit button on Advance Search Form page.
Make the words “No Entries Found” in a larger font size and in the color red, located below the search
fields where results would normally appear, so that user has a clear understanding that their search
returned zero results and that the search has completed.
Common Recommendations
Have a Modify Search Link below the search field to edit search or make changes to the search directly
from the Search Results page.
Have a list of the top search term suggestions.
Spelling suggestions are needed so that a search does not always result in “No Entries Found”.
Clear call-out button for help or instructions on how to use the discovery tool. Perhaps a video tutorial
or PDF guide on how to use the system. Help link/button should also be clearly labeled on all internal
web pages.

27. Overall Recommendation
Should a purchasing decision need to be made between Encore and Summons, our group feels that
Summons would be the better discovery search tool for the graduate students and faculty members
of University of Baltimore, and as a whole for the University System of Maryland and Affiliated
Institutions (USMAI). While some changes should be made to Summons before it is integrated into
USMAI, we feel that these changes are minor in comparison to the issues Encore would need to
address. The results show that many users felt Summons was a lot easier to use compared to Encore.
Users successfully completed 60% of the five tasks in Summons versus users only completing 45% of
the tasks in Encore. The overall failure rate of successfully completing tasks shows an even greater
disparity; users testing Summons only had a 15% failure rate compared to Encore’s 45% failure rate.
We feel that if our recommendations were taken into consideration, the failure rate in Summons
would drop significantly.

28. 28 Appendices

29. 29 Heat Map
Heat Maps show how long each part of the screen has been looked at, as well as
the main areas of the website users looked at and focused on.

30. Heat-Maps
These heat maps show that users have a right-side “blindness” in both Encore and Summons. In other
words, most users did not notice that there was a portion on the right side bar in Encore that said
“Other Sources”. We assume this “blindness” occurs because advertisements and other banners
normally appear on the right hand side on webpages. On average, it took participants approximately
1:34 minutes to find “Other Sources” in Encore. User 4 in particular never even found the “Other
Sources” side bar. A similar pattern of “blindness” was also found in Summons. For example, none of
our participants looked to the right sidebar in Summons to see what features or links were available.
User 2 even commented that her field of vision was in the center of the screen and to the left.

31. 31 Wireframes

32. Wireframes
The following wireframes were created in response to our recommendations, based on both of
the discovery search tools we tested. We envisioned a discovery search tool that incorporates
all of our recommendations. Our wireframes also include the positive features we found in
both systems.

34. Wireframes

35. Wireframes

36. Wireframes

37. 37 Consent Form

39. 39 Faculty Screener

40. Faculty Screener
1.! What!is!your!age?
!!18!–!29!!!!! !!!30!–!39!!!!! !!40!–!49!!!!! !!50!–!59!!!!! !!60+
!
2.! How!often!do!you!conduct!research,!for!classes!or!personal!interest?!
!!Never! !!Rarely! !!Two!or!three!times!per!month!
! !!Four!to!ten!times!per!month! !!More!than!ten!times!per!month
!
3.! When!first!conducting!research,!which!of!the!following!do!you!typically!go!to!first?!
!!School!library!website!for!resources
!!Online!Search!Engine!(please!specify)!!_______________
!!Other!!___________________________
!
4.! Which!online!databases!do!you!use?!(Check!all!that!apply)!
!!Academic!Search!Premier! !!ABI/Inform!!!!! !!MasterFile!Premier!
! !!PsychInfo! !!Ebsco! !!Other:!!_______________________
_____!!None/I!do!not!use!online!databases!to!search!for!information!
!
5.! If!you!selected!“none”!!from!question!4!above,!what!is!your!reasoning?!
!!I!never!find!what!I’m!looking!for.!!!!
! !!It!is!too!difficult!to!use.!
! !!Other!(Please!Specify):!!___________________
!
6.! If!you!use!the!library's!search!tools,!which!statement!best!reflects!how!you!feel!about!your!use!of!
online!database!tools:!
!!I!can!locate!all!of!the!appropriate!resources.!
! !!I!am!missing!some!of!the!resources!available!to!me.!
! !!I!am!not!able!to!use!the!search!tools!to!find!what!I!need.
!!I!was!not!taught!how!to!use!these!tools.
!
7.! How!would!you!rate!your!searching!skills?!
!!Poor! !!Fair! !!Good! !!Excellent
!
8.! On!average,!for!how!long!do!you!do!research!in!one!sitting?!
!0!to!30!minutes!!!!! !!31!to!60!minutes!!!!! !1!to!2!hours!!!!! !More!than!2!hours
!
9.! How!often!do!you!visit!your!library’s!website!to!locate!resources?!
!!2+/day!!!!! !!1/day!!!!! !!4+/wk!!!!! !!1"3/wk!!!!! !!less!than!1/week!!!!! !!Never

41. 41 Graduate Student Screener

42. Graduate Student Screener
1.! What!is!your!age?
!!18!–!29!!!!! !!30!–!39!!!!! !!40!–!49!!!!! !!50!–!59!!!!! !!60+
2.! What!is!your!major!and!concentration?!
Major:!__________________________________!
Concentration:!________________________________!
3.! In!your!academic!career,!how!many!classes!have!required!you!to!conduct!research?!
!!0!!classes!!!!! !!1"2!!classes!!!!! !!3"4!classes!!!!! !!More!than!5!classes
4a.! When!first!conducting!research,!which!of!the!following!do!you!typically!go!to!first?!
!!School!library!website!for!resources!!!!! !!Online!Search!Engine
4b.! If!you!selected!Online!Search!Engine!in!Question!4a.!what!search!engines!do!you!normally!use!
for!conducting!research?!(Check!all!that!apply)!
!!Google! !!Yahoo!! !!Bing/MSN! !!AskJeeves! !!AltaVista
!!Other!(Please!Specify):!!________________
5a.! Which!online!databases!do!you!use?!(Check!all!that!apply)!
!!Academic!Search!Premier!!!!! !!ABI/Inform!!!!! !!MasterFile!Premier!!!!!
! !!PsychInfo! !!Ebsco! !!Other:!!___________________
!!None/I!not!use!online!databases!to!search!for!information
5b.! If!you!selected!none!from!question!5a!above,!what!is!your!reasoning?!
!!I!never!find!what!I’m!looking!for.!!!!
! !!It!is!too!difficult!to!use.!
! !!Other!(Please!Specify):!!___________________
6.!!!!If!you!use!the!library's!search!tools,!which!statement!best!reflects!how!you!feel!about!your!use!of!
online!database!tools:!
!!I!can!locate!all!of!the!appropriate!resources.!
! !!I!am!missing!some!of!the!resources!available!to!me.!
! !!I!am!not!able!to!use!the!search!tools!to!find!what!I!need.
!!I!was!not!taught!how!to!use!these!tools.
7.! How!would!you!rate!your!searching!skills?!
!!Poor! !!Fair!!!!!!!! !!Good!! !!Excellent
8.! On!average,!for!how!long!do!you!do!research!in!one!sitting?!
!!0!to!30!minutes!!!!! !!31!to!60!minutes!!!!! !!1!to!2!hours!!!!! !!More!than!2!hours!!
9.! How!often!do!you!visit!your!library’s!website!to!locate!resources?!
!!2+/day!!!!! !!1/day!!!!! !!4+/wk!!!!! !!1"3/wk!!!!! !!less!than!1/week!!!!! !!Never

43. 43 Script

44. Script
Introduction
M: Good <time of day>, thank you for making time in your schedule to participate today. My name is <name>, and I am a part of a group
working on a research study to learn about discovery (search) tools for the University System of Maryland and Affiliated Universities (USMAI).
We have asked you here <time of day> to help us evaluate two discovery tools through your interaction with them.
At a basic level, a discovery tool is library search engine that allows you to search multiple databases, along with the library's catalog in one
search interface. During today’s session, you will be using <name discovery tool> and <name discovery tool> to complete some basic tasks
that a <student/faculty member> might carry out during their search process.
Please realize that we are not testing your ability. Instead, we’re testing the effectiveness of the tool. Do you feel comfortable in proceeding?
P: Participant acknowledges
M: Very good, if you will, please sign our release form here. It says:
“I, the undersigned, agree to be part of a usability study conducted at the University of Baltimore. As a participant. I agree to be videotaped
and to have my activities on the computer recorded. I allow my comments and observations about my experiences to be become part of the
findings of the usability study.”
P: Participant signs release form.
M: Thank you. Before we begin with the usability test, we could like you to fill out this basic questionnaire in order to get an idea of your
background in searching online databases and online search engines for resources such as books, articles, or audiovisual items. (Hands the
participant the screener).
P: Participant fills out screener and hands it back to the moderator.
M: Thank you.

45. Script (cont.)
System Introduction and Calibration (Tobii)
M: In order to start the usability test we need to set up the eye-tracking software. This computer system is equipped with eye-
tracking software, which is a device for measuring eye positions and eye movement which is helpful in analyzing how you use the
discovery tools. Before we begin, we will need to set up a session in the software. Would it be OK if I used your first name to
identify this session?
P: Participant responds.
M: O.K. <name of user>.
The next step is to calibrate the eye tracker to your eyes. During this calibration and throughout this session, please relax and
continue to look at the screen. While we will be conversing throughout the session, please try to keep your focus on the screen
rather than looking at me. Do you have any questions?
P: Participant responds.
M: OK, here we go… (start and run through calibration)
FAILURE
M: OK, the system had a little trouble picking up all the information it needed for calibration. I would like to recalibrate the areas
of the screen on which the system had some trouble. Are you ready?
P: Participant responds
M: OK – Here we go
SUCCESS
M: Excellent! I will now open the first discovery tool.

46. Script (cont.)
Alternate paragraph if using Morae:
(Will the Morae session be set up prior to beginning the test, or will we have to set it up before the user starts working with the
tool?)
M: The computer system you will be working on today is equipped with Morae software, which is used for usability testing and
user experience research. It records your interactions with the computer so they can be analyzed after your session. I’m now
going to set up a session in the software. Would it be OK if I used your first name to identify this session?
P: Participant Agrees
Familiarization
M: Here is the first discovery tool you will be working with. As you can see, <name of tool> includes options that allow you to
search library databases for books, articles, and other types of materials.
Overview of Tasks
M: To gauge the usability of the tool, we will be asking you to work your way through some tasks that we would expect a
<student or faculty member> to do. These will be tasks that you may or may not have done before.
Because of the test environment, you may not be able to FULLY complete a task and may be stopped at the point of a login or
click a link that is unavailable. If you can’t find something or if it doesn’t make sense, just tell us, and we can inform librarians
who are making decisions about the tools.
Remember: We are testing the discovery tools, not you. There is no right or wrong answers, no right or wrong way to do things.
Every action you take, no matter how you may feel about how it turns out, helps us to evaluate the product. Do you have any
questions before we begin?
P: Participant responds

47. Script (cont.)
Tasks – Discovery Tool 1
Insert Discovery Tool Tasks and Scenarios.
Tasks- Discovery Tool 2
M: Here is the second discovery tool that you will be working with. As you can see, <name of tool> includes options that allow
you to search library databases for books, articles, and other types of materials.
To gauge the usability of this tool, we will be asking you to work your way through the same tasks that you did using the previous
tool. Again, because of the test environment, you may not be able to FULLY complete a task and may be stopped at the point of a
login or click a link that is unavailable. If you can’t find something or if it doesn’t make sense, just let us know. Do you have any
questions before we begin?
P: Participant responds
Insert Discovery Tool Tasks and Scenarios.
Conclusion
M: Those are all of the tasks we have today. Thank you for your assistance in evaluating the discovery tools; we appreciate your
time completing them for us. Your participation today will help us make recommendations that can make a difference to the UB
community. Do you have any further questions about the study before you leave today?
P: Participant responds.
M: (Answers questions if any are asked). Once again, thank you for spending your (morning/afternoon) with us today. As a thank
you, please accept this gift of appreciation. (Hand over $10 Starbucks gift card)

48. 48 Task List

49. Task List
Task Script
1. Find Your professor has asked you to read the Seamus Heaney translation of Beowulf. Find out if a copy is
known item available in the library and, if so, where it is located. If a copy is NOT available, to check out, are
(book) there any other options for obtaining or accessing the book?
Alternate wording for faculty: You are interested in reading Seamus Heaney’s translation of Beowulf.
Find out if a copy is available in the library and, if so, where it is located. If a copy is NOT available,
to check out, are there any other options for obtaining or accessing the book?
2. Find Your professor suggested that you consult an article entitled “Modernism and the Harlem Renaissance” that
known item was published in American Quarterly for a paper you’re doing. You want to see if it’s available full-text from the
(article) & save library’s databases. Once you have found the article, can you save that information for later use?
article citation
Alternate wording for faculty: There is an article you recall seeing in American Quarterly on “Modernism and
the Harlem Renaissance” that you think you might assign as a class reading. You want to see if it’s available
full-text from the library’s databases. Once you have found the article, can you save the information to pass on
to your students?
(Full citation, do not give to subject: Baker, Houston A. (1987). Modernism and the Harlem Renaissance.
American Quarterly, 39 (1), 84-97.)
3. Conduct You are interested in learning basic Portuguese for an upcoming trip to Brazil. You want to see if the library has
topic search for any elementary Portuguese language textbooks or audiovisual materials
print or AV
materials

50. Task List
Task Script
4. Conduct a You have to research Artificial Intelligence for a project, and your professor has asked you to use a variety of
topic search for resources including books and journal articles. The articles should be no more than five years old and be peer-
all types of reviewed. As a preliminary assignment she wants you to report on how many relevant books and articles you
materials found.
Alternate wording for faculty: You are researching recent developments in Artificial Intelligence. As a
preliminary step, you want to compile a list of relevant books and peer-reviewed articles from the last five years
that are available through your library.
5. Retrieve
article You’re ready to use the Modernism article you found earlier, and you want to retrieve it.