Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Evaluating a new electronic service using mixed methods
Similar a Evaluating a new electronic service using mixed methods (20)
Último
Último (20)
Evaluating a new electronic service using mixed methods
- 1. Evaluating a new electronic service using mixed methods Presentation to QQML, 2012 Denise Turner and Sue Myer Teesside University
- 2. Outline of Session • Background • Introduction to LibGuides • Evaluation stage • Opportunity for your questions and feedback
- 3. Background • Wide variety of resources available overwhelming for students • Subject guides just list of links
- 7. LibGuides • Creates a one stop shop for a subject • Interactive multimedia • Integrated help at the point of need • Customisable • Linked to modules in the VLE
- 8. Questioning LibGuides • How to justify an ongoing subscription? • Are they being used? • What do people think about them? – Academic staff – Librarians – Students
- 9. LibGuides Usage – top 25 Forensic Science Graphics Interior Architecture and Interior Design Media Youth Studies September 2011- April 2012 Sociology Midwifery and Women's Health Total hits = 161,465 Teaching in the Lifelong Learning Sector Radiography, Imaging and Ultrasound Accounting and Finance Computer Games and Animation English Physiotherapy Law Paramedic Science Education Criminology History Early Childhood Studies Sport and Exercise Computing Occupational Therapy Nursing Psychology Business and Management 0 2000 4000 6000 8000 10000 12000 14000 16000
- 10. Variation in usage • Some guides not being used heavily • Top guide: – Business - 13,471 hits • 81st guide: – Food & Consumer Safety – 44 hits • Some content areas within guides not being used
- 11. Business Guide - Sections Page Total Welcome 4977 Finding Journal Articles 3519 Journals 1285 Electronic books 1116 Finding Books 815 More Resources 760 Referencing 327 Useful Websites 254 Statistics 157 Getting Started 140 Off Campus Access 98 Darlington 23 Totals 13471
- 12. Google Analytics • Only introduced May 2012 • Who are the users? – Initial results – majority coming from Teesside • How are users getting to site? – Nearly 50% users going directly to libguides • How long are users spending on the site? – Average visit 3 mins 38 secs
- 13. • Less than 2% on a mobile • Only 3.5% from search engine • 55% referred to site • 95% of referrals come from Library website
- 14. • Excellent site! • I will be using for my assignments • Useful informative and easy access what more could a student want? • Really handy BUT • Less cluttered • Search box could be more specific
- 15. Feedback Form Limitations • Only 24% added comments • Collating individual emails • Who are they from? • Too many feedback forms on the site – confusion with feedback for information literacy sessions – “more time to be shown”
- 16. Academic Staff “I just wanted to say how good the Lib guides are. They really help my teaching and final year projects :)” • Learning and Teaching Conference 1. Should LibGuides be extended to include study skills? 2. How can we involve academic staff?
- 17. Should LibGuides be extended to include study skills?
- 18. Answers to: How can we involve academic staff? • Who decided how many sub divisions eg SCM 2 guides not enough? Sohsc many but overlap between them. • Who decides subject for guide areas can we influence this eg research methods or generic health not just nursing if new research doesn't cover this? • Name link • Build in area for staff suggestions
- 19. Librarians • Focus group with librarians: – What are the positive aspects of LibGuides? – What are the negative aspects of LibGuides? – What do the statistics tell us? – What should we test with students?
- 20. Positive feedback from focus group
- 21. Negative feedback from focus group
- 22. What do statistics tell us? • Perhaps reduce number of subject specific guides eg Nursing, Engineering • Common pattern in section usage across the 6 most popular guides for each School • Can see impact of training sessions in usage statistics • Finding journal articles 2nd highest on each - wording makes it easy to find? • Are headings meaningful? • More resources hides expensive databases • Low book usage • Usage stats for databases don't reflect actual usage - students accessing from elsewhere or bookmarking
- 23. Students • Have the guides lived up to our expectations? • Can students find the information that they need? • Will we find some of problems that other research has found?
- 24. How far have we got? • Usability study protocol developed – based on University of Washington Libraries, USA • Technical set up sorted out – using Camtasia in designated room – Student workers • Tasks agreed • Studies to be carried out this week
- 25. Tasks for usability study 1. Find a libguide for Forensic Science 2. Find the electronic book: Dingwall, G. (2006) Alcohol and crime. Willan. 3. Locate the following journal article and open the full text: Low, B. and Laffey, D. (2011)"Is Twitter for the Birds? Using Twitter to Enhance Student Learning in a Marketing Course." Journal of marketing education 33(2) pp.183-192.
- 26. Tasks for usability study 4. Find the database Proquest Nursing Journals and open the factsheet. 5. Find out why Katy Demoily has recently been in a newspaper.
- 27. Conclusion • Evaluation has developed from original project plan • Pleased with usage • Evaluative approach could be used for other resources and services • Work in progress!
Notas del editor
- – overview of the sessionThis session will introduce you to a project that we have been running at Teesside University to introduce a new electronic service at Teesside University. In September we introduced LibGuides a hosted web based services to support information literacy and so this presentation will be about the various methods we have been using to evaluate this new service and reflecting on our approach. So we will start by explaining the background and what LibGuides are and how they have been developed. We will share with you how we have tried to gain information from stakeholders and what information we have gained so far (eg on their usage) and what we plan to do next.There will be an opportunity for your questions in the end.
- Denise – Before we implemented Libguides, we had links to all of the resources we had for a subject in subject guides, but these were very static, and the structure of our web site meant that the help for the resources was separate from the actual resources. The guides themselves were just a list of links and the headings for the sections of the subject guides had be agreed as each subject area needed to use the same headings.
- So for example this is what the business subject guide – looked like. As I’ve said just a list of links to various types of resources. There was no focus on a task a student might want to perform they needed to understand what online databases via Search It meant – there was information below to explain a little more but do people read it?Students got to their subject guides from the navigation on the left – because of the timing of the project teaching material had already been designed telling students to click on the link to Subject Guides and colleagues were reluctant to replace this with a LibGuide link and so for this year we kept both the original Subject Guides and introduced the new LibGuides.
- BusinessLibGuideThis is what the new Business LibGuide looks like. For anyone not familiar with LibGuides the new service uses a library content management system to present information tailored to a specific discipline. It is very easy to create guides and all staff in my team were involved where in the past only a small number of the team would have had the skills needed to create a webpage like this. We aimed to bring together all of the most useful resources for a subject, along with instructions, tutorials and details of the academic librarians supporting that subject - in effect creating a one stop shop.We have one of these replacing each of our original subject guides. Initally we created a template which was used by all subjects, this speeded up conversion to the new guides and meant that core information is only input once and then cascaded to all LibGuides. We are now working on developing them to be more discipline specific. The structure isTab-based, organised by a mixture of different formats of information and tasks. This is the welcome screen: our strategy is to push relevant info at students through the VLE rather than to expect them to navigate our web site. Each Blackboard module will link to the relevant subj guide. We are using RSS feeds to add the latest news to this page and a calendar of events that updates to show the latest month . This is also the opportunity to introduce the subject team.
- Through areas like Finding books we can push key texts to students. Through the e-books – area we can promote the eversion of heavily used print titles , as these can be fairly hidden from students. In the section we are looking at now we can point out major journals collections and online for that discipline together with factsheets, tutorials etc. shows students how to reference this type of material – in a relevant style for their school.
- Denise –So in summary, some of the positive aspects of libguides.Customisable so able to tailor them to the needs of a subject in consultation with academic staff
- Denise We have been monitoring usage of LibGuides since they were introduced and take up has been good. 161,465 hits so far in comparison to the Old subject Guides which over the same time period last year = had 41,747 hits. this years usage of the old guides has dropped to just 10,725 a drop of just over 74%. (74.31%)
- Statistics from Sep-April.Show graph of under-used guides – take to librarians meeting?Compare a few well-used guides – top 5 – any trends?
- Limitations of libguides statistics – no idea of the path that users taking or time spent or how reached the site – so implemented Google Analytics. Any results?
- One of the early decisions we made on planning the evaluation of LibGuides was to use a feedback form. This is a facility that LibGuides gives you and building a form is very easy. We implemented a feedback form on every LibGuide. Asking users to rate the LibGuide out of 5 based on how useful it was. Space was also available to add comments. Despite high usage we have only received 85 feedback forms from the site. None from the Business School who had the most used guide! Most respondents scored the guides highly and based on the score we could have got we achieved almost maximum scores. Some students also left comments and many of these were also positive. Plus there were some useful suggestions about how we could improve guides.
- However, only 24% of the returned forms included comments and where the score was low there was often no comment at all which was frustrating. Unlike other methods we had used for forms in the past like Survey Monkey of Google Docs the responses came as individual emails and so even with the low response it was time consuming to collate the feedback. We didn’t know who was returning the forms and so We thought this would give us a good opportunity to obtain feedback on the Libguides – but it hasn’t proved as useful as we had initially hoped. For next year, we may not abandon feedback but will consider using our own survey form which will mean feedback will automatically be collated.
- Swap to Sue – going now from general feedback to our evaluation with specific user groups, beginning with academic staff.Introduced guides to them before we launched them to students – but only interested staff attended. Feedback from those staff was very positive. Subject librarians publicised the guides with their schools – again feedback was positive, but we don’t know whether the staff then promoted the guides with students.We gave a presentation on LibGuides to our Teesside Learning and Teaching Conference this year. We asked whether libguides should be extended to include “study skills” and How can we involve academic staff?
- When we asked whether libguides should be extended to include “study skills” these were the areas that were mentioned.When we looked at the feedback from the L&T Conference even though only one group mentioned Dissertation, there was lots of feedback that could be encompassed in a guide to writing a dissertation eg report writing, time management – we have a separate project working on this. So we found this feedback useful.
- We didn’t find these responses particularly useful – quite random statements that didn’t always link to the questions, but maybe that this was due more to the limitations of the conference session:Limitations of methodology eg range of attendees (also included learning technologists and ICT), difficulty of engaging them with actual question within short timescalePut into groups and asked as a group to write on post it's and struggled to get feedback so 5 mins is not enough. Could have asked for longer session but an hour too long?In hindsight should have asked them to start individually and then discuss(NHS not able to access due to firewalls. Problems with social media and ICT departments.)
- Focus group earlier this month.
- Strong agreement on positives – especially that they’re easy to create, also easy to use, brings everything together, and looks good
- More varied comments – concerns over info overload, content being hidden, “dumping ground” (negative side of easy to create). Also finding a balance between shared template and freedom to ‘do your own thing’
- A lot of these thoughts were what we’d already decided before the session – but it worked well to allow colleagues to see this for themselves from the evidence rather than just tell them – eg have fewer guides.Questions about whether the headings are meaningful especially More Resources were addressed by the usability study
- Students are the most important user group. When we thought about what we wanted to learn from them, these were our key questions:Expectations – look back at advantages that we mentioned at the start of this presentation and seeing if reality matchesCan students find what they need? Do students find them overwhelming? (info overload concern of librarians)Shared problems – reference:LibGuides Usability TestingCustomizing a Product to Work for Your UsersRachel Hungerford, Lauren Ray, Christine Tawatao, Jennifer WardUniversity of Washington Libraries – Seattle, Washington USA https://digital.lib.washington.edu/dspace/bitstream/handle/1773/17101/UWLibGuidesUsability-2010LAC-tawatao.pdfTesting carried out in 2009.Also research from:
- Usability protocol – based on research from Uni of Washington, but also Nielsen’s web usability advice and our own experience of testing our web site several years ago.Also informed by all our evaluation up to this point – aspects that we particularly want to know about such as whether students find the tabs and sub-tabs easy to use Specific tasks - As well as 5 Tasks, X and Os exercise when students indicate useful and less useful parts of pages, also more general questions about how use libguides – up to 1 hour in total.
- Tasks – start with easy (basic) question2. and 3 - Key tasks that want students to be able to do – find expensive resources –ebook and journal article
- Two tasks based on specific concerns:4. Do students realise what the resource icons do (for factsheet) – do they understand that online databases are within Finding Journal Articles5. Newspaper database beneath More Resources – does this hide them? (particular concern of librarians)Early days – can only report on 2 students experience – but demonstrated keenness to use search box rather than navigate the site. Also just as Google Analytics is giving us more than we expected, these usability studies are also providing us with much richer data.
- Work in progress – not yet completed usability study but also needs to be iterative process (make changes, test, make changes ….)
- 20 mins here - Sue - We are at an early stage and so value both your questions but also your feedback