Three universities will discuss assessment of discovery systems. Texas State University-San Marcos believed a discovery tool wouldn’t offer value beyond existing cross-platform searching tools. After two years of testing, we changed our minds and learned many lessons. We will present strategies and recommendations for conducting usability testing and surveys, testing and customizing interfaces, and post-implementation management. Florida International University and the University of North Florida have both implemented or trialed discovery systems in the past year and have been working on strategies to assess these systems. We will share what we have done so far, and then would like to open the floor to see what assessments attendees have planned or performed.

1. Electronic Resources & Libraries 2013
AGAINST INSTINCT
Choosing Discovery & Making it Work
Liane Taylor, Continuing Resources Librarian
Arlene Salazar, Reference/Instruction Librarian
Lisa Ancelet, Head, Reference & Instructional Services
Albert B. Alkek Library
Texas State University – San Marcos
March 18, 2013

2. ABOUT US:
TEXAS STATE UNIVERSITY-SAN MARCOS
5th Largest University in Texas
(…but you’ve never heard of us.)

34,000 FTE
High Undergraduate (~30k FTE)
 18 Master’s Programs (~3600 FTE)
 11 Doctoral Programs (~400 FTE)


http://www.txstate.edu/about/history-traditions.html
Main Library + Music Library +
Satellite Campus Library

3. THE SEARCH FOR THE PERFECT
SEARCH TOOL THAT WILL SOLVE
ALL OF OUR PROBLEMS FOREVER
July 2009 - August 2011

4. THE TEAM:
TRANSFORMATION & COMPOSITION
Federated
Search
Team
Discovery
Search
Team
The Perfect Mix
(Balance is key)
• Reference/Instruction +
Acquisitions/Cataloging
• For “One-Search” +
Against “One-Search”
BFFs
• Met weekly for two years
(some more than 2 yrs)
http://www.flickr.com/photos/1000photosofnewyorkcity/6018490089/

5. WHEREFORE EBSCO DISCOVERY SERVICE?
We considered comparing EBSCO
Discovery Service vs Summon,
however…
•
•
•
•
EBSCO Link Resolver
EBSCOHost-centric (75% of databases)
Clear Gateway to Subject-Specific
Databases
December 2009: EBSCO Discovery Service
Beta Test Site

6. EDS VS EBSCO POWERSEARCH: WOULD
STUDENTS EVEN NOTICE THE DIFFERENCE?
PowerSearch is FREE, EDS is NOT.
What does EDS give us that we don’t
already have?
NOTE: Image not to scale

PowerSearch combines most of our databases


Transferred most db to EBSCO, except for several where
native interfaces were clearly stronger (i.e., EngVillage)
…but PowerSearch does not include Catalog, IR,
other vendors

7. WHAT DOES SUCCESS LOOK LIKE?
(ORIGINAL GOALS VS VALUE TEST)
Original Goals




EDS Value Test
Provide enhancements to the
library’s catalog such as facets
and tags (discovery tools)
Provide the option for patrons
to search multiple resources
simultaneously (federated
search)
Improve patron awareness of
the many and varied
electronic resources to which
the library provides access
Provide customized search
options for various disciplines
or academic departments
?
EDS is valuable if a
searcher can consistently
type an unsophisticated,
non-boolean search query,
and as a default, with no
sophisticated strategy,
retrieve non-EBSCO
results, including books
from the Catalog (if not an
article-only search),
within the top 50 results.

8. THE LONG AND
WINDING ROAD

Customizing, Testing, Tweaking
Search Mode Parameters
 Facets
 Relevancy Ranking Tweaks


Waiting, waiting (Dec 2009-April 2011)

Listservs + Webinar

12. THE WINNER: OPTION 1 (EDS)

137 Respondents

37 faculty members (27%)


67 students (undergrad + grad) (48.9%)



46 preferred EDS
PROBLEM: Most were relying on number of hits or
differences in the interface of one over the other.
Why Option 1?


26 preferred Option 1 (EDS)
After taking out interface differences that could be fixed,
results that include books from the catalog and journals
articles.
Given the feedback comments, it was clear that either of
the options were better than the existing search option.

13. FOCUS GROUPS

13 Participants (mix of undergrad, grad, fac)

Led group through 5 searches total, comparing:
EDS to PowerSearch
 EDS to Catalog


Post-Search Feedback Questions, examples:

Hard was this to do?



If “Hard” or “Very Hard” – Why?
Were the results same -> very different?
Which do you prefer?

14. AS CLEAR AS…
For most of the questions, all
interfaces were rated “About
the Same” in terms of
preference & performance
BUT!
“Overall, which option
did you prefer
when searching for
articles?”
5=The Same 5=EDS
2=Neither 1=PowerSearch*
(*likely mistake!)

15. READING (PROJECTING?) THE FINE PRINT
“I was impressed with option 1 and its one stop
for everything”;
 “Option 1 is more connected with the library
catalog which helps especially if you want one
place to find everything.”
 “I like being able to immediately see the cover of
whatever book im looking at. Both had equally
relevant articles, but I prefer books to PDF's”;
 “When looking for books, option 1 was better
because it had the book plus reviews on the book
when it came up”.


16. WHERE DO WE PUT IT?
Development of new homepage search widget
 If your site has a tabbed search widget, we looked
at it.

17. WHAT DO WE CALL IT?

18. “FINAL” SEARCH BOX

19. IT AIN’T OVER ‘TIL…WELL, EVER.

Implementation & Promotion

Advisory Group

Continued Customization

21. IMPLEMENTATION | INSTRUCTION
http://libguides.txstate.edu/startyourresearchatalkek

22. ADVISORY GROUP
Members for an advisory group were brought together
to continue to shape EDS.

23. AGAINST INSTINCT, MAKING IT WORK
Our guts tell us two things: IT MUST BE
PERFECT. IT MUST BE COMPLETE.
Your discovery tool is neither.

Manage Expectations!


It’s a STARTING POINT, not an END-ALL, BE-ALL,
SEARCH TOOL FOREVERYONE
It can’t have everything in it!

Many databases aren’t well-suited to be included!