The presentation considers (a) general principles of peer-review, (b) remote peer-review of e-learning programmes. The presentation was created for an ITC/ILO training on the ecbcheck (http://ecbcheck.efquel.org) quality label, at its campus in Torino, 28.09.2012

1. Online Peer-Review
... of e-learning programmes
ECB-Check Training
Anthony F. Camilleri
ITC/ILO Turin – 28.09.2012
www.efquel.org

2. What‘s in a Name?
Audit Expert-Review
Objective Subjective
Precise standards Broad guidelines
Verifies account of client Interprets account of client
Requires detailed knowledge of Requires detailed knowledge of the
administrative procedures subject area concerned
standards-based
review
www.efquel.org

3. Who is a peer?
• an expert in the field
• an expert in what field?
• e-learning!
• what is e-learning?
www.efquel.org

4. Features of an Expert-Peer
familiar with
best-practice
experience in
course design and delivery
natural communicator
sense of mission
comparative perspective
www.efquel.org

5. Inter-subjectivity
• different subjective perceptions, taken from
different viewpoints, give an objective
view of reality
• the quality basis of an external review
• requires consensus
consensus is not a diplomatic nicety
but an essential pre-requisite of quality
www.efquel.org

6. Communicating with the institution
Communication starts
with self-assessment!
• This is not a one-way exercise!
– Has the institution given enough information
to allow a successful review?
– What don‘t you understand, and who can
answer your questions?
• Be active in schedule-design
www.efquel.org

7. Conducting a Review
Familiarise yourself with the standards
1. Understand which standard covers which area (sometimes
they can overlap).
2. Understand their relation to each other.
3. Make sure you have a vision of successful and non-
successful completion of each standard – based on
comparative experience
www.efquel.org

8. Inspect the Course
„Most Quality Managers are Magicians―
Look Here
www.efquel.org don‘t look here!

9. Inspect the Course
• Log onto the LMS
• Look through every part of the course:
– Lessons
– Help-pages
– Tests
– Forums / support pages
– Interactive material
– What else?
www.efquel.org

10. Fill in your Forms
• Your forms are a letter to the agency
COMMUNICATE!
www.efquel.org

11. Communicating with the agency
„I don't know anything about
art, but I know what I like―
Gelett Burgess
www.efquel.org

12. Communicating with the agency
„I don't know anything about
the standards, but I know
quality when I see it―
Unnamed reviewer
www.efquel.org

13. Communicating with the agency
Standards of Proof
• Some credible evidence
• Preponderance of evidence
• Clear and convincing evidence
• Beyond reasonable doubt
www.efquel.org

14. Communicating with the agency
In your Report
• Be yourself
Bad: The institution showed....
Good: The review team saw / found /
observed....
www.efquel.org

15. Communicating with the agency
In your Report
• Be specific
Good: The institution showed....
Better: The review team found multiple and
consistent examples of
www.efquel.org

16. Communicating with the agency
In your Report
• Say what you know
Bad: The institution lied....
Good: The review team found
inconsistencies between evidence (x) and
interview (y)
www.efquel.org

17. Communicating with the agency
In your Report
• Give your opinion (where relevant)
Bad: The institution is... / or NOTHING
Good: We suspect, It seems likely that,
Given the evidence available, etc...
www.efquel.org

18. Communicating with the agency
In your Report
• Give your reasoning
Bad: There is no quality.
Good: When we consider (X), (Y) and (Z),
we find it impossible to say there is quality
www.efquel.org

19. Communicating with the agency
In your Report
• Be CLEAR
Bad: The structure of the LMS is in need of
improvement.
Good: The LMS barely functions – it is littered
with wrong links, the help-function is unusable
and most sections still need to be populated
www.efquel.org

20. Communicating with the agency
In your Report
• Link Effect with Cause
Bad: Quality systems are in place, but there
is no evidence of iterative improvement.
Good: Quality systems are not effective, due
to lack of iterative improvement procedures.
www.efquel.org

21. One last thing
Remember that
Quality Labels should act as
Quality-Enhancers
not only transparency tools
offer a path towards quality improvement
www.efquel.org

22. Hvala!
Thank-you for your attention
Questions?
Anthony F. Camilleri (anthony@kic-malta.com)
Presentation available from: http://www.slideshare.net/anthonycamilleri/
Released under a Creative Commons Under the following conditions:
Attribution ShareAlike 3.0 Slovenia License Attribution — You must attribute the work in the
manner specified by the author or licensor (but
You are free: not in any way that suggests that they endorse
• to Share — to copy, distribute and transmit you or your use of the work).
the work Share Alike — If you alter, transform, or build
• to Remix — to adapt the work upon this work, you may distribute the resulting
work only under the same or similar license to
this one.
www.efquel.org