CDE seminar conducted by Dr Gwyneth Hughes, Senior Lecturer in HE, Institute of Education. In this session Dr Gwyneth Hughes, a CDE Fellow, drew on her CDE research on ipsative assessment and a JISC funded project that she is leading at the IOE to explore why it is useful to analyse feedback for distance learners. It demonstrated a feedback analysis tool that has been developed as part of the project. Gwyneth, a CDE Fellow, teaches on Higher Education programmes within the Lifelong and Comparative Education department including the MBA in Higher Education Management. She also supervises doctoral students. She has undertaken research and published on a range of topics including: ipsative assessment, formative feedback, identity, blended learning, e‐learning, gender inclusivity, widening participation, online collaborative work, web 2.0, learning technologies and reflective practice.

1. Analysing feedback:
opening the black box
Dr Gwyneth Hughes, Institute of
Education, London

2. The purpose of assessment is to select learners for
further study and/or careers.
Agree/disagree
The purpose of assessment is to facilitate learning.
Agree/disagree
2

3. Outline
Why analyse feedback-some current issues
Using a feedback analysis tool (Assessment Careers JISC funded
project at the IOE)
Using assignment cover sheets for dialogue on feedback
3

4. Some current issues with feedback in
HE
• What is feedback?-received not just given out (Nicol &
Macfarlane-Dick, 2005)
• Feedback rarely scrutinised, when it is practice is inconsistent
• Lack of distance learner engagement with and understanding
of feedback (Lizzio & Wilson, 2008)
• Inefficiency in terms of effort and impact
• Transmitted feedback creates dependency on teacher (Carless
et al. 2011)

5. • Praise is not helpful to students unless accompanied by
detail on why the praise is deserved. Encourages
dependency on praise (Hattie & Timperley, 2007).
• Lack of information about progress but this could be
motivating –especially for distance learners
(Hughes, 2011)
• Formative feedback often encourages a focus on short-
term grade improvement rather than longer-term learning.
5

6. Assessment Careers Project: taking
action on feedback
• Year 1: Baseline and feedback analysis tool
• Year 2 (2012/13): Five pilot programmes including 2
distance programmes
• Year 3 Institutional implementation
Assessment Careers: Institute of Education website:
www.ioe.ac.uk/assessmentcareers and video clip:
http://youtu.be/VSaGbPoXPh0

7. Tool to identify feedback categories
P1 Giving praise
P2 Recognising progress (ipsative feedback)
Critique
• C1 Correction of errors
• C2 Factual criticisms
• C3 Criticism of approach
Giving advice
• A1 Specific to content current assignment
• A2 General skills in current assignment
• A 3 For future assignments
Q Clarifications and questions
O Other unclassified statements
Adapted from Orsmond & Merry, 2011 including Hughes, 2011.

8. Analysing feedback categories
• The score is the number of times a classification
appears in the feedback
• The default unit for analysis was the sentence
• Where a sentence contains clauses that make
distinct points, it was split into separate
clauses, each of which was classified separately.
• Neutral comments that for example describe the
piece of work, but do not make any judgement are
unclassified.

9. You have clearly developed your thinking on how you might use
technologies in your teaching but as I mentioned in the previous
feedback, you have not reflected on your experience of using
technology to identify the potential benefits and challenges.
You make your rationale for innovation clear, but less clear is the
learning theory that underpins your pedagogic design. You have
now explained X’s framework, but how does your design draw on
this?
When you do mention Y’s model you do so uncritically, but you have
responded to the feedback on your draft that you should discuss the
limitations of applying model Z to your design.
Try to critique all models and frameworks in future.
9

10. You have clearly developed your thinking on how you might use
technologies in your teaching but as I mentioned in the previous
feedback, you have not reflected on your experience of using
technology to identify the potential benefits and challenges.
You make your rationale for innovation clear, but less clear is the
learning theory that underpins your pedagogic design. You have now
explained X’s framework, but how does your design draw on this?
When you do mention Y’s model you do so uncritically, but you have
responded to the feedback on your draft that you should discuss the
limitations of applying model Z to your design.
Try to critique all models and frameworks in future.
10

11. The structure of the essay was very strong, with a clear introduction
which nicely contextualised the general problem of extraction
industries and the interest in social environments vs the needs for
quite „quick and dirty‟ (my phrase) data collection. The general aims
of REA was well described, although I did have questions while
reading this about how representative cultural samples could be
drawn quickly, and what „representative‟ meant in this context. In
terms of limitations I felt that the essay had maybe missed a trick – if
a closer comparison with the aims and debates in conventional
ethnography had been conducted, then this may have opened a
space for thinking about the extent to which REA can achieve those
aims or not. The issues raised in the conclusion about ethics and
trustworthiness for instance could have been discussed in some
detail ... Overall, the essay was very clear and well written, but I think
that the absence of relating this very particular method to the broader
issues in ethnography perhaps made it limited in the level of analysis
that it provided of REAs. An extremely interesting and competent
11 essay all the same though.

12. Data
• Analysed formative and summative assessment
feedback for modules on 5 postgraduate programmes
at the IOE (total 228 pieces)
• Recorded the total number of comments in each
category and the average per script
• Ranked the categories to obtain a feedback profile at
programme level as well as an aggregate profile of
the 5 programmes.

13. Baseline feedback profile
Total comments for 5 programmes
Summative assessment (N= 165)
Category of feedback Average per script Rank
P1 Praise 4.4 1
P2 Ipsative (progress) 0 (negligible) 5
C1-C3 Critique 2.7 2
A1-A3 Advice for current or 1.9 (mostly for current 3
future assignments assignment)
Q Questions and clarification 0.1 4
requests
13

14. Baseline feedback profile
Total comments for 5 programmes (N= 165)
Category of feedback Average per script Rank
P1 Praise 4.4 1
P2 Ipsative (progress) 0 (negligible) 5
C1-C3 Critique 2.7 2
A1-A3 Advice for current or 1.9 (mostly for current 3
future assignments assignment)
Q Questions and clarification 0.1 4
requests
14

15. Overall you have written a really comprehensive and detailed review of
the document. Your strength lies in your fantastic ability to
contextualise the document with your political and pedigogical
references. […] What do you understand by the term disruptive in this
context? […] You seem to suggest ( or I am interpreting wrongly) that
disruptive in this context is bad? […] Finally, you quote and cite almost
in every sentence- but what is your personal view of 'the rise of K-12
blended learning‘ in primary and secondary education? Where is your
personal voice? It scares me - or at least having a purely distance
online learning component for such students worries me. What about
you?
15

16. What do students do with feedback?
Formative n=50 Summative n=35
didn't respond to the question 4 3
no feedback received 0 1
nothing 4 5
read it 3 10
revised assignment 32 3
systematically took forward 7 13
16

17. Overview of interventions
Pilot 1 MA Education, Health Promotion and International Development
Pilot 2 MRes in Educational and Social Research
Pilot 3 MA Clinical Education
Pilot 4 MA/MSc Psychology of Education
Pilot 5 PGCE Primary
Approx. 400 students and 30 staff in total.

18. Assignment cover sheets
Thinking about the feedback on your draft of this essay, please
indicate what the key points were and what action you took to
respond to this feedback to help you prepare for and write this essay
I would like tutor feedback on:
1.
2.
3.
18

19. Is ipsative feedback appreciated by
students?
“I made an effort to act on it (formative feedback). This
was acknowledged in the feedback on the final
assignment.”
From another student:
“I like the fact that the feedback demonstrates that the
tutor is aware not only of my current level of
achievement, but my progress.”
19

20. Principles for Action on Feedback
1. Feedback is directed towards students' concerns
2. Students are encouraged to see the value of feedback for both short-
term and longer-term learning
3. Assessors initiate dialogue with students about feedback their work
4. Curriculum designers and assessors make it clear to students how
learning accumulates over a programme
5. Programme teams are aware that feedback has different purposes
and agree which categories of feedback are appropriate for their
students
6. Students have experience early and frequent formative assessment
opportunities which are designed into their programme
20

21. Conclusion
Findings so far:
• Feedback profiling is useful for getting programme teams
talking about feedback for both short and longer term
learning.
• Assignment cover sheets are a useful way of engaging
student in discussion of feedback.
• Feedback principles are useful for opening up discussion
of feedback practice at the institution level.
• www.ioe.ac.uk/assessmentcareers

22. References
Assessment Careers project Baseline Report see:
www.ioe.ac.uk/assessmentcareers
Carless, D. Slater, D.Yang, M. and Lam, J. 2011. Developing sustainable
feedback practices. Studies in Higher Education 36, no.4: 395-407.
Gibbs, G. & Simpson, C. (2004) Conditions Under Which Assessment
Supports Students’ Learning. Learning and Teaching in Higher
Education, 1: 3-31
Hattie, J. & Timperley, H. 2007. The Power of Feedback. Review of
Educational Research 77 no. 1: 81-112.
Hughes, G. (2011) Aiming for Personal Best: a Case for Introducing
Ipsative Assessment in Higher Education Studies in Higher Education
22 36 (3): 353 – 367

23. Lizzio, A. & Wilson, K. 2008. Feedback on assessment: student’s
perceptions of quality and effectiveness. Assessment & Evaluation in
Higher Education 33 no.3:263-275.
Nicol, D. & Macfarlane,-Dick, D. 2006. Formative assessment and
self-regulated learning: a model and seven principles of good feedback
practice. Studies in Higher Education 31 no. 2: 199-218.
Orsmond, P. & Merry, S. 2011. Feedback alignment: effective and
ineffective links between tutors’ and students’ understanding of
coursework feedback. Assessment & Evaluation in Higher Education.
36(2): 125-126.
23