Measuring electronic resource availability final version
•Descargar como PPT, PDF•
1 recomendación•320 vistas
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (12)
Destacado
Destacado (8)
Similar a Measuring electronic resource availability final version
Similar a Measuring electronic resource availability final version (20)
Measuring electronic resource availability final version
- 1. Sanjeet Mann Measuring Electronic University of Redlands SCELC Research Day Resource Availability March 5, 2013
- 2. Armacost Library infrastructure •ILLIAD interlibrary loan system •Full text targets (databases, ejournals) (79,757 unique titles) •Serials Solutions 360 Link •A&I databases (~77,000 titles indexed) •Innovative Interfaces catalog/proxy
- 3. Why do electronic resource errors matter? Costs Frustrated expectations Undermined confidence Complicated instruction
- 4. Research question "How often does full text linking work?"
- 5. Availability studies Sample of items Available? Yes/No Error? Order encountered Probabilities Prioritize fixes
- 6. Development of the availability technique •Print material availability card catalog user surveys (Reviewed in Mansbridge 1986, Nisonger 2007) •Linear sequence (De Prospo 1973) •Branching model (Kantor 1976) •Applied to e-resources 500 articles from 50 high impact journals (Nisonger 2009)
- 7. OpenURL performance •OpenURL-based reasons for availability error (Wakimoto et al. 1998) •“Digging into the Data” on link resolver failure (Trainor and Price 2010) •NISO Initiatives: KBART, IOTA, PIE-J (Chandler et al. 2011, Glasser 2012, Kasprowski 2012)
- 8. Usability studies focusing on e-resources •Database link pages (Fry 2011, Ponsford et al. 2011b) •Resolver menus (O’Neill 2009, Imler & Eichelberger 2011, Ponsford et al. 2011a) •Discovery services (Williams & Foster 2011, Fagan et al. 2012) •Entire process
- 9. Methodology 400 citations 4 questions X 10 databases X 10 results [18:11] redlandsreference: what is your research topic? Arts & Humanities [18:11] meeboguest59808: RILM Oral Motor Activity MLA Philosopher’s Index [18:11] redlandsreference: Is this for a Communicative Disorders class? Social Sciences America: History & Life EconLit Sociological Index Sciences Biological Abstracts ComDisDome,
- 10. Link testing http://goo.gl/606us
- 11. Error coding • What is an error? • Six error categories • Updated criteria
- 12. Armacost Library failure points: P vs. O
- 13. Error details 1: Proxy errors Domain missing from forward table Domain missing from SSL Certificate Timeouts trying to establish connection
- 14. Error details 2: Source errors Missing metadata Erroneous metadata (e.g. rft.date=0001-01-01)
- 15. Error details 3: Knowledge base errors Title not selected in knowledge base Title selected, but in poorly chosen collection Knowledge base holdings do not reflect access entitlement (embargo, back issues, etc.)
- 16. Error details 4: Link resolver error Confusion between two similar titles Unusual OpenURL syntax
- 17. Error details 5: Target errors Content not loaded (supplement, embargo) Records concatenated from full text and non-full- text databases Server downtime
- 18. Error details 6: ILLIAD errors Unicode metadata not displayed properly rft.title used for both book title and article title, affects chapters and dissertations
- 19. Results Error rate: 150/400 (38%) Overall availability: 250/400 (62%) Locally downloadable: 104/400 (26%)
- 20. Sampling Necessary sample size for a yes/no condition is determined by: To use this, you need: •Availability rate from a small pre-test •Choose acceptable % confidence (95%) •Choose acceptable margin of error (+/- 5%) Plug values into the formula… •p = 0.625 (250 / 400 successes) •1-p = 0.375 (150 / 400 errors) •C = 0.95 (95% confidence) •Zc = 1.96 (statistical textbook or http://www.measuringusability.com/pcalcz.php) •E = 0.05 (5% error) I could have just used 360 citations…
- 21. Confidence Your confidence in a study of a particular sample size is given by: I could have just used 360 citations…
- 22. Discussion
- 24. Local Availability / Errors by Discipline
- 25. Solutions Edit proxy forward table
- 26. Solutions Upgrad e ILLIAD
- 27. Solutions Customiz e Serials Solutions
- 28. Solutions Simplify result screen interface and terminology
- 29. Summary •400 citations obtained through likely keyword searches of 10 A&I databases •62% availability / 38% error rate (98% confidence, +/- 5%) •26% downloadable full text •Responses include fixing proxy, kb holdings, interfaces, upgrading systems •Strengths: quant + qual data, very flexible (n=100 allows 85% confidence) •Weaknesses: Does not account for issues with interfaces, searching or evaluation faced by actual users
- 30. Towards availability testing with live students • More barriers: o confusing interfaces o difficulty formulating searches and evaluating sources o login errors • How to test: o cognitive walkthrough + recorded task protocols o analysis informs information literacy and interface design • Deliverables: o availability % o branching model o usability report
- 31. Questions Further reading: http://goo.gl/4fu47 Data set: http://goo.gl/606us skatdesign.co m
Notas del editor
- This is the online version of my presentation given March 5, 2013 at SCELC Research Day, Loyola Marymount University.
- This diagram presents an overview of Armacost Library’s e-resource discovery infrastructure. Five systems (proxy server, source database, knowledge base/link resolver, target database and ILL system) must work together using common standards for students and faculty to be able to discover full text.
- Electronic resource errors cost libraries in terms of unrealized value on paid-for content that cannot be accessed, and in terms of staff time spent on troubleshooting. Unnecessary ILL requests also add staff costs and IFM/copyright charges. Errors frustrate student and faculty expectations and undermine library staff confidence in the accuracy of their own systems for day-to-day use. Scarce physical and fiscal resources are already compelling libraries to justify their relevance to their campuses; unavailable e-resources only fuel skeptics’ concerns. Errors also require instruction librarians to take precious course time away from higher-order thinking skills to explain technical workarounds and search mechanics in greater detail.
- My research study asks the question, how often can Armacost Library users get to the full text of sources they find in abstracting and indexing databases? My study includes, but is not limited to, investigation of OpenURL linking. I operationalized “availability” as two separate factors: students’ ability to download the full text of a source, and the likelihood that users would receive an error as opposed to finding that a source was available in any way (via download, in the physical library, or via ILL)
- Availability studies are a systems analysis research method designed to find out why libraries are unavailable to supply materials to readers, and prioritize troubleshooting efforts. The method was first used in an academic library in 1934 (Gaskill). Investigators generate a sample of items and attempt to retrieve them from the stacks, or download them online. All unavailable items are classified according to the reason why they could not be obtained. Problems can be sorted in the order that a student would encounter them, and assigned probabilities of occurring based on their frequency in the sample. Ideally, librarians would then fix the most frequent problems first.
- Nisonger and Mansbridge’s review articles give a succinct overview of the availability technique and findings from numerous studies. De Prospo, Kantor and Nisonger have also contributed significantly to our knowledge of this research method.
- In addition to the literature on availability studies, research on OpenURL performance was also relevant to my study. These investigations focus on one source of error – the library’s knowledge base. Researchers tested samples of OpenURL links to determine proportions of available and erroneous items. Problems frequently involved the metadata “supply chain” linking publishers, database vendors and knowledge base providers. Several NISO initiatives have sought to improve the quality of e-resource metadata to reduce the frequency of metadata-related errors.
- Many library website usability studies have focused on how students access electronic resources. These studies focus on interface design and vocabulary issues that affect electronic resource availability. Researchers have used a variety of usability methods, including task protocols and cognitive walkthroughs. Studies have either isolated parts of the library’s online presence, or glanced over the entire process a student would use, as in Kress’s study of the reasons why students might place an unnecessary ILL request for an article contained in a subscribed e-journal.
- I collected a sample of 400 citations by identifying 4 actual student research topics (mentioned in our reference transactions) and searching the topic keywords in each of 10 A&I databases covering a variety of subject areas. I attempted to retrieve the full text of the first 10 search results from each database (I did not modify the default sort order or page to subsequent result screens in order to more accurately simulate student research behavior)
- For each of the 400 items tested, I recorded bibliographic metadata in an Excel spreadsheet (see Google spreadsheet link). I also collected “incoming” (from source A&I database to link resolver, see yellow “find full text” link in screenshot) and “outbound” (from link resolver to target full text database, see red circles in screen shot) OpenURLs for each item and pasted them into the spreadsheet. Finally, I recorded ability to download full text and availability as two separate yes/no parameters. (An item could be either available or erroneous. Not all available items were available via full-text download) After testing all items, I went back and assigned a category of error to each unavailable item.
- Error categories require judgment calls on the part of the investigator. Many errors (such as incorrect publisher metadata) are not evident at their point of origin, only detectable by problems that occur later in the retrieval process. I developed six error categories roughly matching the five systems involved in e-resource retrieval. The next seven slides present an overview of the categories and examples of common errors of each type. The availability and openURL-testing literature contains some discussion of what constitutes an unavailable or erroneous resource. I chose to treat ILL requests as a normal part of the retrieval process, rather than as a failure of the library to obtain all items a user might need (something which is no longer possible even for libraries with the most comprehensive collection development policies) I also specified that each item must link directly to a screen providing HTML or PDF full text; screens such as the one pictured here where the link leads to a list of items could confuse students, so I counted it as an error.
- A side-by-side comparison of causes for error in the print and online environments demonstrates the additional complexity of conducting research in an online environment.
- Failure can be localized to the proxy server because the full text target database’s domain is missing from the proxy server forward table, because the proxy server SSL certificate does not contain the full text target database domain, or because the proxy server slowed the connection significantly, causing the web browser to time out. User logins are another source of error (not tested in this study)
- The source A&I database can cause problems due to its interface design (not tested in this study) or because metadata are missing or erroneous. This can cause the link resolver to fail or delay the processing of ILL requests. Our ILL staff frequently need to verify requests with duplicate information in the article and journal title fields or nonsensical dates (e.g. “0001”). Libraries that have configured their ILL system to automatically send requests (e.g. ILLIAD Direct Request) will experience slower performance as erroneous requests are flagged by the system for human intervention.
- Library staff are responsible for selecting titles and collections in knowledge bases that reflect their subscription entitlements. Errors occur if the entire title, or the starting and ending range of the library’s holdings of that title, are either selected when they shouldn’t be (“false positive” error) or not selected when they should be (“false negative” error). Sometimes the same title is listed in multiple collections; library staff must choose the collection with the most complete metadata or risk errors such as the missing article-level link illustrated here (the “SCELC Wiley-Blackwell Collection” lacked information necessary to achieve article level linking throughout the collection, while a different collection with the same titles contained that information) Publishers and knowledge base vendors can also contribute to problems at this stage, when a publisher does not notify the knowledge base vendor in a timely manner of publication changes, or when a knowledge base vendor does not accurately reflect the publisher’s embargo or other information pertaining to access.
- Link resolvers and target databases contributed relatively few errors in my study. Most link resolver errors involved a failure to draw a match between the requested citation and the item in the target resource. This could be due to idiosyncratic metadata or even to variation in libraries’ cataloging practices. In this example, an article from Costerus , a journal not held online, was then run as a catalog title search, which matched on an issue that had been cataloged as a serial monograph. (This problem is likely incomprehensible to our undergraduates)
- Target databases most commonly generated errors because of missing content (either because their publisher agreement forbids loading that content or because they had not notified the link resolver of an embargo). Interface issues represent another source of error not tested in my study. One provider’s tendency to concatenate records for the same item from multiple databases (one containing full text, one containing only an abstract) created problems when the full text record was consistently “hidden” in favor of the abstract-only record (which was consistently targeted by the link resolver)
- Many errors manifested at the point of submitting an ILL request. Articles with foreign-language characters in the title did not display properly because the then-current version of ILLIAD did not support Unicode. (A subsequent upgrade fixed the problem). Also, when ILLIAD received an OpenURL that only used rft.title, it listed the field twice, in both journal name and article name. Our ILL staff frequently referred these issues to me because they were not sure which was the journal title (used to select the correct OCLC record to request)
- These sample findings can be generalized to the entire population of all e-resources at Armacost Library with over 95% confidence and +/-5% margin of error (see next slide) Out of every 100 citations, I would expect to find: 38 errors significant enough to prevent a student from obtaining full text or successfully placing an ILL request 34 potentially successful ILL requests 2 items available from the physical collection 26 full text downloads
- Full-text availability and the presence of error are yes/no (Bernoulli or Binomial) outcomes. Statistical textbooks give the formula for determining the sample size for a binomial population. You will need to conduct a pre-test first to obtain values for the proportion of successful and unsuccessful outcomes. You can choose confidence (c) and error (E) values arbitrarily. The lower the values, the less strong your study, but the easier it is to conduct because you can use a smaller sample. The value Zc is found in a table online or in a statistical textbook. It is related to your confidence: the higher your confidence, the greater Zc becomes.
- Rewriting the equation to solve for Zc gives you this equation, which lets you state your level of confidence in a study of a particular sample size. Look up the Zc value in the table of standard normal distributions or online to determine the confidence probability. Note that small, convenient samples can still obtain a reasonably high confidence probability.
- Most of the sources I tested were articles, but dissertations and book chapters generated a disproportionately great number of errors.
- Most of my errors occurred at the source database, knowledge base or ILL stages.
- There was considerable variation by discipline. The music searches produced results that were rarely downloadable full-text and frequently triggered errors, while history searches frequently led me to a seamless full-text download. Several factors could be influencing these results. Different databases may have different metadata standards and publishers and are distributed by different vendors. Some databases index a lot of hard-to-obtain items like conference proceedings, while others mostly consist of journal articles. Some vendors augment their A&I search results with “linked full text” PDFs from another database on the same platform. This type of direct comparison is not useful for assigning responsibility for error, but is interesting for subject librarians who need to know the challenges students in their liaison areas may be facing.
- I attacked some simpler solutions immediately after finishing my study. I added missing domains to the proxy forward table…
- Upgraded ILLIAD to get Unicode support…
- Corrected holdings in Serials Solutions…
- … and worked with our web team to make the link resolver result screen easier to understand.
- Availability studies are a flexible technique to get quantifiable information about students’ access to full text. My study was a “simulated” study because I tested the access myself, rather than using actual library patrons.
- No studies have attempted an electronic resource availability study with library patrons so far. Such a study would add several potential causes for error. The study would need to incorporate usability methods, for example, conducting a cognitive walkthrough of the path Armacost Library users could follow to research a particular topic, then observing students trying to search that same topic.
- Follow these links to view my literature review and dataset. Email me with questions at sanjeet_mann@redlands.edu