Presentation to a workshop joint with Walter Kugemann to the EFQUEL Academy just prior to the EFQUEL Innovation Forum

1. Quality in Virtual Schools and
Colleges – a view from VISCED
Paul Bacsich (paul.bacsich@sero.co.uk)
VISCED – www.virtualschoolsandcolleges.info
1 EFQUEL Academy

2. Introduction
2 EFQUEL Academy

3. Topics
• Historical overview
• The Pick&Mix system
• Supplementary criteria, local criteria and slices
• Scoring, evidence and documentation
3 EFQUEL Academy

4. Benchmarking and Quality in e-
learning: My part of the story
Overview and history
4 EFQUEL Academy

5. Benchmarking e-learning (UK)
• Foreseen in HEFCE e-learning strategy
• Higher Education Academy oversaw it
• Four phases – 82 institutions (2005-07)
• Justified entry to Pathfinder and Enhancement
programmes - and useful for JISC bids
(Curriculum Design etc)
• Was leveraged into refresh of learning and
teaching strategy (Leicester etc)
5 EFQUEL Academy

6. What is benchmarking? (HEFCE)
“*for+ individual institutions [to] understand
their own positions on e-learning, to set their
aspirations and goals for embedding e-
learning… to benchmark themselves and their
progress against institutions with similar goals,
and across the sector [world]”
Can replace e-learning by “distance learning”,
“OER”, “podcasting” ,etc
6 EFQUEL Academy

7. Methodologies: Pick&Mix
• First used for Manchester Business School 2005
• Has been used in 36 institutions, mainly new
universities and university colleges
– Including 4 very diverse institutions in Wales (2007-09)
– And 5 in the Distance learning benchmarking Club (UK,
Sweden, Canada) plus two more in UK (2009-12)
– And basis for the Critical Success Factors schemes in
two EU projects: Re.ViCa (2007-09: developments fed
back into Pick&Mix) and now VISCED (2011-12)
– Also an OER version of the scheme ready for use
• Wording slightly revised for schools and colleges
7 EFQUEL Academy

8. Pick&Mix
Criteria and metrics
8 EFQUEL Academy

9. Criteria
• Criteria are “statements of practice” which are scored
into a number of performance levels from “bad” to
“best” to “excellent”
• These statements are in the public domain – to allow
analysis, refinement etc (cf.OER)
• Pick&Mix has over 80 criteria…
– But typically a set of 20-30 is chosen – this range is based
on analysis from the literature (ABC, BS etc) and experience
in many scoring meetings
9 EFQUEL Academy

10. Pick&Mix Metrics
• Use a 6-point scale (1-6) – ignore 6 if you wish
– 5 (cf Likert, MIT90s levels) plus 1 more for “excellence”
• Backed up by continuous metrics where possible
• Also contextualised by narrative
• The 6 levels are mapped to 4 colours in a “traffic
lights” system (red, amber, olive, green)
10 EFQUEL Academy

11. Pick&Mix
Three sample criteria
11 EFQUEL Academy

12. P01 “Adoption” (Rogers)
– not used now
1. Innovators only
2. Early adopters taking it up
3. Early adopters adopted; early majority taking
it up
4. Early majority adopted; late majority taking it
up
5. All taken up except laggards, who are now
taking it up (or retiring or leaving)
6. First wave embedded, second wave under way
(e.g. m-learning after e-learning)
12 EFQUEL Academy

13. P10 “Training”
1. No systematic training for e-learning
2. Some systematic training, e.g. in some projects and
departments
3. Institution-wide training programme but little
monitoring of attendance or encouragement to go
4. Institution-wide training programme, monitored and
incentivised
5. All staff trained in VLE use, training appropriate to job
type – and retrained when needed
6. Staff increasingly keep themselves up to date in a “just
in time, just for me” fashion except in situations of
discontinuous change
13 EFQUEL Academy

14. P05 “Accessibility” – tough one
1. e-learning material and services is not accessible
2. Much e-learning material and most services conform to
minimum standards of accessibility
3. Almost all e-learning material and services conform to minimum
standards of accessibility
4. All e-learning material and services conform to at least minimum
standards of accessibility, much to higher standards
5. e-learning material and services are accessible, and key
components validated by external agencies
6. Strong evidence of conformance with letter & spirit of
accessibility in all countries where students study
14 EFQUEL Academy

15. Pause for questions
15 EFQUEL Academy

16. Extensions, local criteria and
slices
16 EFQUEL Academy

17. Extensions
• Pick&Mix started with 20 core criteria – but now
there are over 80 – and a different core (2.0)
• There is room for more supplementary criteria –
and further analyses are always being done to
ensure topicality
• Web 2.0 aspects are already covered
• But it is mainly the institutions who have kept
and will help to keep the system up to date
• And some EU and other projects
17 EFQUEL Academy

18. Supplementary criteria - examples
• They include:
– IT reliability
– Market research, competitor research
– IPR
– Help Desk
– Management of student expectations
– Student satisfaction
18 EFQUEL Academy

19. Slices
• As well as benchmarking the whole institution it is
wise in large institutions to look at a few “slices”:
• Useful to give a context to scores
• Slices need not be organisational or subject-based
– Distance learning (is a natural one)
– Thematic or dimensional like HR, costs…
19 EFQUEL Academy

20. Scoring
• Scores are usually in the range 1-5, point 6 is
exceptional and cannot be planned for
• Focus not only on best practice but also bad
practice and real practice
• Do not try to ensure that your institution
scores 5 overall
• 6 sometimes has aspects of 1
20 EFQUEL Academy

21. Use in Distance Learning
(such as Virtual Schools)
details
EFQUEL Academy
21

22. Distance learning
• A variant “mood” of the default Pick&Mix was
developed to benchmark distance learning
• This was used in the Distance Learning
Benchmarking Club:
– University of Leicester
– Thompson Rivers University
– Lund University
– Royal Swedish Technical Institute (KTH)
• and also at the University of Gotland
• Most recently at University of Northampton
EFQUEL Academy 22

23. Evidence and documentation
• Normally institutions want a final report but but the
aim is NOT to recreate the QAA-style SED or DATs –
the report is an outcome of the benchmarking
process not the start of it
• Have a “file” for each criterion
• Some narrative will be needed
• Institutions normally group criteria according to their
own L&T strategy or in terms of “owning”
departments
– We also supply some standard groupings based on MIT90s
but few use these
23 EFQUEL Academy

24. Documentary support
• There are many reports available – many but not
all are public, others are available to those
participating in Pick&Mix
• The HE Academy wiki “was” a good source of
information and the Academy blog structure “was
to be” maintained indefinitely…. Not now
• We store all Pick&Mix key material separately
from the HE Academy
• There are now many papers and a bibliography is
available
24 EFQUEL Academy

25. Ongoing work
• Evolving Pick&Mix into a benchmarking and
quality scheme for virtual schools:
– With specialist help from EFQUEL
– And a research team at KU Leuven
– And correlation with iNACOL scheme in US
• Version of Pick&Mix ready to benchmark OER
uptake (developed with Bieke Schreurs, OUNL)
• Variant moods for accessibility and
employability (Wales)
EFQUEL Academy 25

26. Thank you for listening
email for this work is
paul.bacsich@sero.co.uk
Benchmarking E-Learning in UK Universities: Lessons from and for the
International Context , published in
http://www.openpraxis.com/files/Bacsich%20et%20al..pdf
http://www.virtualcampuses.eu/index.php/Bibliography_of_benchmarking
and
http://www.mendeley.com/groups/1075191/benchmarking-e-learning/
26 EFQUEL Academy