The ongoing discussion on the issue of the quality of academic eLearning criticizes the dominating culture of “quality” linked to the industrial production, mostly based on rationalization and conformity to pre-defined standards, that requires the collection of massive quantitative data, with major interest on educational outputs as a vision of system’s productivity (Ehlers & Schneckenberg, 2010, Ghislandi, 2008, 2012). Instead, quality of education requires reflection and deep understanding of complex contextual elements, interactions and relational dimensions that are essential and often invisible to traditional assessment tools. In this paper, through the presentation of an eLearning course as case study, we attempt to show how participatory/constructivist evaluation can become a key practice to support the quality of an eLearning experience from the point of view of the learner. In fact, as it emerges from the analysis, this open form of evaluation has an enormous potential to address practices towards the values/concepts underlying meaning making processes inside a transformative learning culture. Building on this results, we contend that the evaluation of quality needs to integrate methods that open up the sense of practices and values to the participant. To this regard, we also discuss how qualitative constructivist approaches to evaluation can make an important contribution drawing on the coherence found between its epistemological and ontological assumptions and the idea of new cultures of quality evaluation where the participants build the own values and concepts of goodness.

1. 5th International Conference of Education, Research
and Innovation
Madrid, 19-21 November 2012
Patrizia Ghislandi – Juliana Raffaghelli
University of Trento

2. A) eLearning evolution and
the problem of evaluating
quality
B) Reconceptualizing
Quality of eLearning
From Standards to a culture of Quality
Fourth Generation Evaluation: understanding
the epistemological approach to evaluation of
eLearning quality.
C) A Case Study
The context of the research: evaluating
quality inside the eLFO12 Learning
Community
The Methodological Approach
D) Conclusions and
Tools for a participatory evaluation of
eLearning quality
Debate!
Results

3. eLearning evolution and the problem of
evaluating quality
Traditional
eLearning: delivery
of online contents
Evolution: Open,
networked Learning

4. Reconceptualizing Quality of
eLearning
…The evolution of quality has
been influenced by industrial
models
○ Predefined Standards
○ Measurements on finished
products
○ Quantitative Evaluation
○ Intangible aspects (relation,
communication) in a second
place for the evaluation tools
○ The need to accountable to
external referees
○ The gap between research and
educational practice
○ A vision of evaluation as
certification, objectivistic
quantitative paradigm
4

5. Reconceptualizing Quality
of eLearning
 Diverse Cultures of Quality are
underpinned by diverse values:
 Exceptional/Original: the
value is on the uniqueness
 Distinctiveness: not for all
 Excellence: The highest
levels of performance
 Fitness for Purpose: doing
what has been planned
 Inclusive: all people can
participate
5

6. Who says what QeLHE is?
ISO/IEC 19796
Q UNESCO Quality for all
SLOAN-C MODEL –USA-
EFQUEL –UE-
Sistematic
Approaches CENTRO VIRTUAL PARA EL
Access? DESARROLLO DE ESTÁNDARES
Excellence? DE CALIDAD PARA LA EDUCACIÓN
Innovation SUPERIOR A DISTANCIA EN
Inclusiveness?
AMÉRICA LATINA Y EL CARIBE
6

7. Quality is a Complex issue
4- Diverse 2- Diverse Methods of Analys
Meanings benchmarking
guidelines
Pedagogical
standard
Organizational
Economical
Qualitative Evaluation
Technological
Quantitative Evaluation
Q
Methodological
5-Diverse Levels of
Analysis
Individual
Group
3- Diverse times
institutional
Socio-cultural of intervention for the analysis
in itinere
1-Diverse Perspectives ex ante
The teacher ex post
The Student
The institution
The external evaluators
7

8. Reconceptualizing Quality
of eLearning
 Quality is not an intrinsic,
universal value
 It is very much about the methodology of
evaluation,
 And the substantial
epistemological principles
and values underlying the process of
evaluation

9. Fourth Generation Evaluation
(Guba & Lincoln, 1989)
 Understanding the epistemological approach to evaluation of
eLearning quality
 Four generations of Evaluation : There is a wise, well informed,
objective, aseptic, external
EVALUATOR
 1) Measurement There are unaware, badly
 2) Observation informed, too emotionally
engaged EVALUATED PEOPLE
 3) Judgement (and the processes/products
coming from them
There is an expert, open to share,
 4) Joint Reflection committed EVALUATOR
There are committed, willing to
and Transformation learn, EVALUATED

10. Understanding the epistemological approach
to evaluation of eLearning quality
 Four generations of Evaluation (Guba&Lincoln, 1989)
There is a wise, well informed, Quantitative
 1) Measurement objective, aseptic, external Methods
 2) Observation EVALUATOR
 3) Judgement There are unaware, badly External
informed, too emotionally Evaluation
engaged EVALUATED
Qualitative
There is an expert, open to
Methods
 4) Joint Reflection share, committed
EVALUATOR
and Transformation
There are committed, willing Self- Peer
to learn, EVALUATED Evaluation
External, agreed, frameworks
of reference (EFQUEL, 2007)

11. The eLFO12 Learning Community

12. Defining our Case Study
 The decisive factor in defining a study a
case study is the choice of the individual
unit of study and the setting of its
boundaries (…)the individual unit may be
studied in a number of ways, for instance
qualitatively or quantitatively, analytically or
hermeneutically, or by mixed methods. This
is not decisive for whether it is a case study
or not; the demarcation of the unit’s
boundaries is. (Flyvbjerg, 2011, p. 301)

13. Defining our Case Study
a)  The decisive factor
the educational process and in defining a study a
strategies to improve it b) an eLearning setting
Self-Peer study is the choice of the individual
case Moodle
platform
Evaluation: eLFO12: eLearning
unit of study and the setting of its interoperable
boundaries (…)the individual unit may bewith Web 2.0
Trainers’ Log course for training tools
AdAstra
the trainers of
studied in a number of ways, for instanceSocio-
Rubrics
Interviews
operators of social
qualitatively or quantitatively, analytically constructivist or
approach to
hermeneutically, andor by mixed methods. Thisadoption of
health care the
educational
is not decisive for
c) a professional learning whether it is a case study
Technologies
community composed by operators
orhealth and social services willing of the unit’s
of not; the demarcation
boundaries is.as trainers
to improve their skills (Flyvbjerg, 2011, p. 301)
Committement University of Trento
–UNITN- /Direction of Health Care
(Province of Trento) –PAT-

14. The Process •The log-book as part of learning
activities to collect live reflections on
the learning process.
•A suite of rubrics as meta-evaluation
instruments to support reflection on
teaching and learning quality
Rubrics •Interviews with participants to deepen
on the emerging perceptions, ideas
and reflections in the former
instruments.
exploring
and
constructing
eLearning
Quality
The learner’s
Interviews Becoming aware of values and
log-book
practices for quality
Making visible the invisible
Transform

15. The Log-book
 Having introduced the course as a “socio-
constructivist” oriented, the tutors asked the
students to be active and collaborative ( a
quality value)
 it was registered during the initial phases of
the course, an important amount of discontent
with the approach: to much “chaos”
 A lesson on the pedagogy of socio-
constructivist approach
 Reflections and discussion
 Becoming more participant and enganged:
aware of the importance of collaboration

16. Table 1 - Sloan-C pillars the connected organization of the adASTRA rubrics .
Sloan-C pillars and and the connected organization of the adASTRA rubrics .
The AdAstra Rubrics
 A set of rubrics, elaborated by the University of Trento research unit
(Ghislandi & Pedroni, 2011), on the basis of SLOAN-C MODEL
Sloan-C Pillars Rubric adASTRA
Rubric Analysis, Design,
Learning effectiveness Syllabus, Community of Practice,
Screencast, Portfolio
Student satisfaction Rubric Feedback Students
Faculty satisfaction Rubric Feedback Teacher
Cost effectiveness Not developed
Access Rubric Accessibility

17. The AdAstra Rubrics
 Dimensions of Quality Analized
Dimension Description
Organization Analyses how the several course activities planned and scheduled
are implemented consistently.
Resources Analyses how the resources offered for learning are considered by
learners, in the sense of completeness and coherence.
Teaching Analyses how the teaching methods and strategies implemented are
Methods and considered in the sense of effectiveness (the achievement learning
Strategies goals as well as to maintain learners’ motivation).
Comunication Analyses how the several synchronous and asynchronous
instruments of communication are adopted (the usage is coherent
with the learning goals and requirements)
Collaboration Analyses how the students interact between them to achieve learning
goals
Technologies Analyses how the technological arrays are used, in the sense of the
facilitation of learning.
Assessment Analyses how the assessment is implemented, particularly taking into
account the coherence between the type of assessment activities and
what the course offered to learners.
. The rubrics where implemented through 2 meetings and across 2 months of the course (1 month devoted to the
learning design and 1 month to the delivery of the exercise module of peer learning), one at the beginning and
another at a closing phase, being the researchers available to discuss with teachers and students if doubts
arisen on the concepts driven.

18. Interviews: reflecting on quality
evaluation
Participatory meta-evaluation: the rubrics as instruments
for the reflection on course quality (excerpt on 1 of 4 dimensions)
Dimension Results from Interviews (*)
(*) the students transcriptions in Italian have been translated into English by the
research team
“It is really important to have something written that helps you to avoid
mistakes when planning a course, and also to reflect on what had been done
Coherence previously (…) to solve problems and to make changes” [eL5]
“the level of detail has to be correlated with the dimension of course quality
that you are to analyze, and this was perfectly represented into the rubrics”
[eL4]
“Diversely from the student side rubrics, the teacher side rubrics lack of items
that recall how the teacher has reacted to critical issues, how did she feel
with regard to those critical issues, so the teachers subjective perspective is
missed somehow” [eL3]
“It could have been introduced some question regarding the methodology
adopted in the design phase or even better questions that allowe to establish
if the teacher has followed coherently a pedagogical approach” [eL1]
“The student rubric misses something regarding the relationships and the
collaborative work…or this should be more evident….these represents a
positive factor for the groups and single learning don’t they? [eL5].

19. eLFO12 Case Study: a participatory approach to the
evaluation of Quality

20. Discussion of Results (1)
 The three moments of evaluation showed how the
evaluation process worked progressively in
addressing understanding and learning towards the
values/concepts of the ELFO12 group
 The breakthrough in this process was the moment in
which the learners were given the opportunity to
revisit (self-evaluate) their own learning in the
context of socio-constructivist theoretical framework
 The rubrics where considered coherent, complete,
effective, though rather less efficient. But this meta-
evaluation of the tools was given in the context of
utility and use of the same: a socio-constructivist and
technologically mediated learning activity/course.

21. Discussion of Results (2)
 The quality was evaluated not in the
sense of external parameters of
performance but in terms of the
coherence and authenticity of the course
learning model and the learners
awareness and engagement within it. It
is crucial to point out that both steps or
“moments” of quality evaluation.

22. CONCLUSIONS
 The selection of qualitative methods,
 a phenomenological approach based on narrative self-evaluation, peer-
evaluation and meta-evaluation,
 emphasizes the interest on processes and on the
empowerment of learners AS COMMITTED EVALUATED
 This logic studies the topic within its context, uses an emerging design
that accounts for reality as subjective and multiple, lessen the distance
between “official” evaluators and participants (Denzin & Lincoln, 2011).
As a result,
the evaluation process encompasses a
transformational (participatory/innovative)
opportunity for the engaged individuals/
institutions.(Creswell, 2007; Mertens, 2009).

23. CONCLUSIONS
 this transformational perspective is the
kernel of a quality learning culture:
 a human group that take part of a learning
experience as a deep, reflective experience,
connected to the own professional/ personal
identity
○ For which purposes do I learn? What can I do with
this learning?
 not just for accomplishing activities, recalling
information, and obtaining credentials (course
diploma).
We call this become insiders of the
culture of quality.

24. Thank you for your interest!
For communications:
patrizia.ghislandi@unitn.it