Evaluation of search interface features for bioscience literature
•Download as PPTX, PDF•
2 likes•954 views
Presentation at the HCIR-2011 workshop by Anna Divoli (University of Chicago) and Alyona Medelyan (Pingar). Title: Search interface feature evaluation in biosciences
Recommended
Recommended
More Related Content
Viewers also liked
Viewers also liked (11)
Similar to Evaluation of search interface features for bioscience literature
Similar to Evaluation of search interface features for bioscience literature (20)
Recently uploaded
Recently uploaded (20)
Evaluation of search interface features for bioscience literature
- 1. Search interface feature evaluation in biosciences Alyona Medelyan joint work with Anna Divoli (University of Chicago) HCIR-2011 20.10.2011
- 2. Talk Overview Search interfaces and their features Search tasks on the web and in bioscience Experiment description Side-by-side evaluation example Data collected Results
- 3. autocomplete facetted refinement results preview related searches search expansions
- 4. Talk Overview Search interfaces and their features Search tasks on the web and in bioscience Experiment description Side-by-side evaluation example Data collected Results
- 5. Search Tasks and their Types in Web Search (Kellar et al. 2007) weather exchange rate Fact finding … Information gathering email banking grad schools Browsing travel plans shopping … Transactions … blogs Other news …
- 6. Search Tasks and their Types in Bioscience I’d like to find out what kind of animal models of huntington’s disease are out there Fact Finding I need to collect publications by others on connexins & how they relate to our studies Information Gathering I want to find out whether there are any new publications on the mechanism that underlies Golgi cisternal maturation in yeast Browsing
- 7. Research Questions and Hypotheses Which search interface features are useful 1 for searching the biomedical literature? Hypothesis Users prefer different interface features depending on the search task Which approaches to facetted navigation 2 work best for this domain? Hypothesis It’s better to display dynamically computed sets of facets than a complete hierarchical list
- 8. Talk Overview Search interfaces and their features Search tasks on the web and in bioscience Experiment description Side-by-side evaluation example Data collected Results
- 9. The Study Exploratory short study with 6 bioscientists 2 faculty, 2 postdocs, 2 PhD students Q&A on 3 search types in their work, queries, resources, systems 10-15min in person sessions Main study with 10 bioscientists 2 faculty, 7 postdocs, 1 PhD student Email & 1-2hr in person sessions Side-by-side comparison of anonymysed search interface features Per participant: 1 baseline and 1 own query
- 10. Talk Overview Search interfaces and their features Search tasks on the web and in bioscience Experiment description Baseline query Side-by-side evaluation example connexin Data collected Results
- 11. 1. Autocomplete A B C D E F G F G
- 12. 2. Search Expansions A B
- 13. 3. Faceted Refinement - links A B C E D
- 14. 3. Faceted Refinement - checkboxes F G I H
- 15. 4. Related Searches A B C D E F G
- 16. 5. Search Results Preview A C
- 17. Talk Overview Search interfaces and their features Search tasks on the web and in bioscience Experiment description Side-by-side evaluation example Data collected Results
- 20. Talk Overview Search interfaces and their features Search tasks on the web and in bioscience Experiment description Side-by-side evaluation example Data collected Results
- 21. Usefulness ratings for interface features & search tasks Autocompletion br Positive ff ig Neutral Negative Query expansions br 1 participant ff ig Facetted refinement br ff ig Related searches br ff ig Search results preview br ff ig
- 22. Summary of Findings, Participants’ Comments Autocomplete is less important: “we feel pigeonholed by suggestions” Facets are useful: “we focus and refine the search all the time” Choose facets wisely: “a large number of facets is overwhelming” Checkboxes are better than links: “we want to select multiple values” Aesthetics are important but what really matters is the content
- 23. Conclusions Which search interface features are useful 1 for searching the biomedical literature? Facets and results preview are useful for any search task Other features are more useful for browsing Which approaches to facetted navigation 2 work best for this domain? Few, query-oriented facets with specific values, in checkboxes! Alyona.Medelyan@pingar.com Anna.Divoli@pingar.com
Editor's Notes
- Contacted Anna after reading Marti Hearst’s book and seeing a screenshot of Anna’s interface, very similar to one that Pingar came up withSuggested to do a research together around interface, usability and searchThrough Anna’s background we focused on bioscience
- We looked specifically at 5 features that guide users in their search from input of the query to evaluating the results
- Autocomplete, search expansion, facetted refinement, related searches, results preview on PubMed – commonly used system in bioscience
- To add an additional dimension to our research we looked at search tasks.
- A study by Kellar shows that nearly a half of search tasks related to transactions. The remaining tasks can be grouped equaly into ff, ig and br.
- We found that in bioscience the differentiaon is somewhat difficult… It’s hard to know which query relates to each task.These are examples of real search needs descriptions, and how we classified them based on Kellar’s search tasks.
- Based on existing literature we found that interface features are studied extensively independently from each other, but comparison of their usefulness for search has been missing.We chose to focus on the bioscience domain to have specific findings. Bioscience is one of the verticals, in which Pingar plans to specialize their software for.Our hypothesis was that the usefulness of features depends on the search task.For Pingar specifically we were also interested in testing various approaches to computing and representing facets.Our hypothesis was that dynamic set of facets is more effective and checkboxes are better than links.
- The Experiment itself took place primarily at the University of Chicago
- We run an exploratory short study with 6 bioscientists.We explained them the 3 types of search and asked them for examples of such searches they use for their work: queries and resources/systems. We used these findings to set up the main study with 10 bioscientists, all of them are researchers in various biological areas: Developmental, Molecular, Cell, Evolutionary, Transcriptional and Systems Biology, as well as Biochemistry, Immunology, Genetics, Population Genetics, and Neuroscience. We emailed them to aks for 4 different queries they use, each with a note on what information they are looking forWe selected two baseline queries and one personal query from each participant. For each query we took screenshots in different systems and isolated the part of the system that shows the interface feature in question (logos were removed).
- During each session, which lasted 1-2 hours, we showed participants a PowerPoint presentation with screenshots for one baseline query (of their choice) and for one of their own queries (Table 1). We asked them to rate overall usefulness and aesthetics using a 5 point Likert scale for different handlings of the features. We alsoasked them to rank the systems in order of preference.
- The way different systems implement autocomplete varies a lot. They display different values in different order and provide various types of information in addition to the suggestions.
- We have collected A LOT of data. It took weeks to summarize and analyze it properly.
- We used spreadsheets which doesn’t fit onto one slide even if I minimize it.This is the kind of information we collected about each user – first line of the table.
- The remaining lines contain the responses only. Each color is a different interface feature.These are the kind of responses we have collected from the users. This one is for autocomplete.
- We have summarized the results in order to find answers to our hypothesis.
- This figure summarizes how participants judged the usefulness of interface features for their queries. The length of each bar equals the number of ratings. All 4 browsing participants liked autocomplete, whereas 5 out of 6 participants with info gathering queries rated it as neutral and 1 out of 6 as not useful. This shows a clear difference in usefulness of this feature. For query expansions participants expressed positive or neutral opinions for browsing and mixed opinions for the other search types. Facetted refinement was rated mostly useful for all search types with equal neutral scores for information gathering. Related searches got predominantly neutral or negative reactions, except for browsing, for which they are equally spread. Not surprisingly, it’s useful to see document previews for all search types. Comments analysis shows that snippets are better for browsing, whereas summaries with full sentences for finding specific or specialized information.The results are intuitive and participant’s comments confirm these. Based on these findings, we encourage the interface designers to give priority to “green” elements of the search display and do not bother much about related searches for biologists. Most of them told us that their searches are usually specific and even correctly suggested related searches are not of interest. Access to query expansions is important to experts, but should not be a default feature.
- Here are most important findings from our study:Our results agree with those previous research that facets are well received by bioscientists, but autocomplete is less important.But while previous findings have indicated that a large number of facets are needed per query, as implemented in Semedico, we found that 9 out of 10 did not want to see a large number of facetted groups. The large number of choice and the inevitable redundancy overwhelmed them. Our results also confirm findings in that users prefer selecting multiple suggestions using checkboxes.Overall, participants told us that aesthetics are important (and need to be “good enough” to use a system) but what really matters is the content.