Presentation at MLA annual conference on MiSearch, MLA ID 08-pap-479-mla

1. Assessing an adaptive,
profile-based PubMed
search tool
Marisa Conte
meese@umich.edu
NLM Associate Fellow
University of Michigan Health Sciences Libraries

2. MiSearch
 Adaptive biomedical literature search tool
 Utilizes implicit relevance feedback
 Builds statistical profile to predict which results will be most
relevant to user

3. Objective
 To compare the performance of a locally-
developed third-party PubMed search tool
to PubMed in terms of user satisfaction,
search success and efficiency

4. Collaboration

5. Methodology
 N = 11 librarians (12 recruited)
 Tasks derived from mediated search files
 Each subject completed 2 tasks with each
search tool
 Each task completed by 3 subjects with
each search tool
 Pre-training conducted by tool developer

6. Data collection
 Web-based surveys
 Demographic survey
 Task-specific survey
 PubMed search histories
 MiSearch data logfiles

7. Sample questions
 In patients with diabetes mellitus, does
increasing hemoglobin A1c reduce the risk
of amputation and kidney failure?
 What are the attitudes, beliefs and socio-
cultural factors that influence cervical
cancer screening in the US? What about
specifically among immigrant groups? And
I’m only interested in research from 1990 –
present.

8. Limitations
 Convenience sample
 Unclear survey questions
 Study tasks too complicated
 Problems with data collection

9. Sample results – Query 2
Average
satisfaction MiSearch PubMed
Search activity 1 1.7
Tool 1.3 1.7
Speed 1.3 1.7
Accuracy 1.3 2
Completeness 1.7 2
1 = low

10. Overall satisfaction with searches
60
50
% responses
40
30
PubMed
20 Misearch
10
0
High Fair Low
User satisfaction

11. Overall satisfaction with tools
70
60
% responses
50
40
30 PubMed
20 MiSearch
10
0
High Fair Low
User satisfaction

12. Suggestions
 Improve database response time
 Add more filters
 Add a “reset” function
 Make search details more transparent

13. Next steps
 Librarians suggested changes which:
 were easy to implement
 dramatically improved the tool’s functionality
 First usability test – helped establish formal
usability protocols and heuristics
 New roles for librarians

14. New roles for librarians
 Usability experts
 Heuristic evaluation of tools
 Portal development
 Formal usability studies
 Weblog analyses
 Education, promotion, dissemination
 Research collaborations

15. Resources
 MiSearch: http://misearch.ncibi.org/
 NCIBI: https://portal.ncibi.org/gateway/
 States, D.J., Ade, A.S., Wright, Z.C., Bookvich,
A.V. & Athey, B.D. (2008) Misearch Adaptive
PubMed Search Tool. Bioinformatics.
PMID 18326507

16. Acknowledgments
 Jean Song, Nirit Glazer, Barbara Mirel, David States
 Librarians from UM’s Health Sciences and Shapiro
Science Libraries
 National Center for Integrative Biomedical Informatics
(NIH grant #U54DA021519)
 National Library of Medicine
This research was supported in part by an appointment to the NLM
Associate Fellowship Program sponsored by the National Library of
Medicine and administered by the Oak Ridge Institute for Science and
Education.