Presentation given with Brian Merison (BILD) at EFODL International Conference on May 2007.

1. European Self-Evaluation Tool for eLearning ‘ Evaluation for quality improvement’ EDUCATION AND CULTURE LEONARDO DA VINCI EFODL International Conference May 2007 Brian Merison – The British Institute for Learning & Development, UK Edwige Morin – VIDEOSCOP - Université Nancy 2, France

3. Better Quality learning? Results Learning experience

4. First challenge : S coping the project Who are the key stakeholders? Who will carry out the evaluation? ? Wider Public Third Parties (government agencies etc) External Facilitators External Providers Company Department Learning Department Company Management Learner

5. Second challenge: Using a quality model © EFQM Excellence Model

6. Level 1: Reactions – a ‘Smile Sheet’ completed by the learner. Level 2: Learning outcomes – attempts to measure what has been learned (skills, knowledge and/or attitudes) Level 3: Transfer of learning to the everyday environment – changes in behaviour Level 4: Results for the organisation such as increased productivity, increased sales, lower costs etc. Third challenge: Using an evaluation model The Kirkpatrick model

7. Constructing the framework A range of stakeholders ? – an unanswered question – who will carry out the evaluation ? EFQM Excellence Model Kirkpatrick evaluation Model The Learner

8. Constructing the framework A range of stakeholder interests The Learner Questionnaire Designer Learning resources Learning processes Results

9. Learning Resources Learning Processes Learning Results Main Criteria Process 1 Process 2 Process 3 Criteria A questionnaire based on a three level structure Process 4 Sub-Process a Sub-Process b Sub-Process d Sub-Process e Sub-Process c Sub - Criteria Statements Questions / Statements

10. Learning RESOURCES Part I 5 Criteria 3 2 3 3 2 79 Items of evaluation Information Electronic learning environment Physical environment Face to Face environment Learning materials 19 23 12 12 13

11. Learning PROCESSES Organisation of services Knowledge evaluation Part II 4 Criteria 4 3 17 12 85 Items of evaluation Learning activities 7 42 Learner Support 3 14

12. Learning RESULTS Part III 3 Criteria Learner’s competencies 1 4 45 Items of evaluation Learner’s knowledge 2 7 Ability to learn 5 34

13. Question Applicable/Not applicable Rating Priority of improvement Related to main crtieria / criteria/ subcriteria How important is this topic for you in this course ? What is your evaluation of this topic in your course ? Is improvement necessary ? 1.1 Accessibility 1.1.1 Access to relevant information on learning opportunities 1 2 3 4 1.1.2 Means to personalise the information given to meet my needs and profile 1 2 3 4 1 2 3 4 1.2 Description of the learning supply 1.2.1 Availability of up to date information on the course 1 2 3 4 1 2 3 4 Yes No

14. Tool functionalities The stakeholders The learners SEVAQ tool administrator SEVAQ tool Enables the designer with instalment and setting the tool The entity who designs the questionnaire to be applied to the learner The final recipients of the results. The people for whom the evaluation is really set-up. Three functions in the tool 1 – designing the questionnaire 2 – producing 3 – consulting results 1 – Securing the purpose of the evaluation 2 – Building the questionnaire framework to be applied 3 – Applying the questionnaires 6 – Building an action plan for improvement 5 – Analysing the results 4 – Getting the results The questionnaire Designer

15. Home page Answers to a questionnaire Create and manage questionnaires Manage the tool

16. The designer using the sevaq tool Create and manage questionnaires Create and manage learners Quick link menu

20. - select the subcriteria - select the questions - write new questions - validate the questionnaire Create a new questionnaire > Step 3

22. Three choices 1 – completely anonymous (one login and pwd) 2 – anonymous amongst a targeted public belonging to the same course (one login and pwd for one course) 3 - identified for each learner (one login and pwd/ learner) Learners & groups

23. - choose your users or groups - publish your questionnaire - receive an invitation email Link questionnaire to learners or groups

24. - choose the questionnaire to answer - answer the questions - add your comments - fill up the profilling - validate your answers Answer a questionnaire

27. EFODL Project manager Helga Van Heysbroeck hvheysbr @ vdab .be www. efodl .net Brian Merison brian @ british -learning.com Edwige Morin Edwige .Morin@ univ -nancy2. fr For additional questions, a personal test log-in or further dialogue: WWW.SEVAQ.COM Thank you for your attention!