Universities are increasingly looking at Learning Gains as a means for the effective measurement of student progress and the contribution they make to their students’ learning and development. However, there remains much to understand about the validity and reliability of these measures used. This paper seeks to probe the relationship between how students understand and interpret the learning gains they experience and the proxy measures of learning gain such as assessment marks universities use. The findings are based on an analysis of nineteen semi-structured interviews of UK distance learners. The paper will present key findings and discuss their significance in respect to how to reconcile students’ own experience of gain and proxy measures of gain and the assumptions on which learning gain measures are predicated. Simon Cross, Jekaterina Rogaten, Duygu Bektik, Denise Whitelock, Bart Rienties, The Open University, UK

1. Simon Cross, Jekaterina Rogaten, Duygu Bektik,
Denise Whitelock, Bart Rienties
Institute of Educational Technology, The Open University
SRHE Conference / 7 December 2017
How do students understand
and interpret learning gains?
Perspectives from in-depth
interviews of UK distance
learners

2. PRESENTATION
OVERVIEW
Using data from interviews undertaken in May
2017 at the Open University UK, this presentation
will probe the relationship between how students
understand and interpret the learning gains they
experience and the meaning and significance they
give to metrics in use for measuring learning gain.
ABC Learning Gains Project

3. PRESENTATION
OVERVIEW
Using data from interviews undertaken in May
2017 at the Open University UK, this presentation
will probe the relationship between how students
understand and interpret the learning gains they
experience and the meaning and significance they
give to metrics in use for measuring learning gain.
ABC Learning Gains Project

4. 07/12/2017
Learning Gains
As TEF develops we will
incorporate new common
metrics on engagement … and
learning gain, once they are
sufficiently robust.
Government Green Paper
Policymakers are now called on
to provide evidence to inform
questions such as: How do
students’ knowledge, skills and
work-readiness change and
improve through their experience
of higher education?... Traditional
measures for evaluating student
performance remain essential
…but they do not provide all of
the evidence needed to address
these questions. And this
evidence is more important than
ever…
HEFCE
Average gain score needs to be
interpreted with caution and
there remain many outstanding
issues that need investigating.
Pascarella et al. 2011.

5. 07/12/2017
1 How do students understand and interpret
learning gains they experience?
2 How do students understand and interpret
measures of learning gain currently in use?
3 To what extent can these two be reconciled?
Research questions
How do students understand and interpret learning gains?

6. 07/12/2017
Semi-structured interviews with nineteen part-time
distance learners resident in the UK lasting 30-60 minutes
• Student perception of gain and progress
• Cognitive, behavioural and affective change
• Relationship between grades and progress
• Study expectations
• Work relevancy
Methodology and
participants
How do students understand and interpret learning gains?

7. 07/12/2017
First stage Interim analysis based on review of
interviewer notes, second listen to interview and coding of
interview transcripts.
Methodology and
participants
How do students understand and interpret learning gains?
Student 8 Female High Lowest STEM
Student 19 Female Low Medium Arts/Soc.Sci
Student 5 Female Low Highest Edn/Lang
Student 10 Male High Medium Arts/Soc.Sci
Student 12 Female Low Lowest STEM
Student 18 Female High Highest STEM
Attainment grouping
as measured by
assessment marks
Progress grouping
as measured by
assessment marks
Gender Faculty
Sources of quotations used in presentation

8. 07/12/2017
Emerging themes
How do students understand and interpret learning gains?
OWNERSHIP OF
LEARNING
RELATIVE VALUE
OF GRADES
WORK-LIFE
• LEVEL OF
STUDYLEVEL OF STUDY

9. 07/12/2017
Level of Study
‘Student 8’
How do students understand and interpret learning gains?
[Initially] it was very much
like study for study
purposes [but] about a
year and half into it, my
mind-set changed, it was
like I enjoy what I’m doing
and it’s giving me
something tangible. This
year was the first year
when … it reflected back
on my day job. The
outcome was a decision to
increase study intensity…
I [feel] that I have the
knowledge, it’s the time
that I was missing… I don’t
think for me it’s grade itself
is… it’s actually the
knowledge. [Getting the
grade] is what’s needed
this year for me not to give
up so that I had an
opportunity to do the exam
and face the next step.
Looking back Current module
I have decided that I will
care about my grades
next year and try to get
the highest scores I
physically can just to
see [how well I can do] if
I don’t have that
additional 60 credits.
Looking ahead

10. 07/12/2017
Relative Value of Grade
Explaining lower grades
How do students understand and interpret learning gains?
There were things in the
first two modules that I
already knew [but] when I
come to this year [then] I
would say probably 95%
of what I’ve learned … I’ve
never heard of. So, I think
this has been a real
turning point for me.
 Reassessment and reprioritisation
 Focus on elements relevant to own
interests
 Expended the oxygen of prior learning
 Expectation that modules will become
tougher and grades harder to achieve
 Module does not directly build on last
 Comparatively less effective learning
design
 Greater study workload
[This year] was my turning
point… Your TMA
(continuous tutor marked
assessment) is not
everything. It’s not, it’s
supposed to be what you
actually physically know
yourself inside… I think
that’s really important.

11. 07/12/2017
Owning the process
How do students understand and interpret learning gains?
I sort of challenged [the TMA
score], not ‘challenged’ it really,
but asked more questions. [I’ve
learned] don’t be afraid to.. I
thought I possibly would have
done better, you know maybe 5
or 10 marks here and there so I
have challenged. [The tutor
response] helped me… I like to
read it, dissect it [and] this year
the feedback I’ve got has
included things that I possibly
might have missed.
Despite the drop in marks, I still
think my understanding has
deepened and now … I feel
more engaged in [my learning]
actually, not just ticking off what I
need to do
Initiating discussion for
feed-forward
Counter-intuitive deeper
engagement

12. 07/12/2017
Work-Life Reference (Points)
How do students understand and interpret learning gains?
The way I think, the way that I
possibly act at times, my life
feels different now … I can talk
confidently to people.
It’s just when you learn
something you [then] become
aware of, for instance, either
news, work itself, everyday life;
which again kind of changes
your perspective and then it
allows you to properly build your
own confidence because you
understand things
Becoming confident Becoming aware
I notice that my approach to
things outside of the academic
context is different … I judge
[gain] on the assignment side
and the out-of-academic context
side …as well.

13. 07/12/2017How do students understand and interpret learning gains?
OWNERSHIP OF
LEARNING
RELATIVE VALUE
OF GRADES
WORK-LIFE
• LEVEL OF
STUDYLEVEL OF STUDY
Emerging relationships

14. 07/12/2017
Reconciling perceptions of
learning gain?
• Many factors impact on the quality of assessment that
students choose to submit.
• Students’ perception of ‘learning gain’ change as they
progress. Level 2 seems to be a key turning point.
• Students and assessors can interpret and give meaning to
the same grade in different ways
• Whilst assessors set the assessment criteria, it is the student
who decides its relevancy to their learning.
• Becoming an independent learner means taking ownership of
the meaning and use-for-learning of the assessment process
• Students (co-)determine the significance of a grade through a
process of contextualisation across study, social and work
lives.
How do students understand and interpret learning gains?

15. 07/12/2017
Concluding remarks
Significance and importance of
qualitative research
Short and medium term goals
An A+B+C Approach
How do students understand and interpret learning gains?
Email: simon.j.cross@open.ac.uk