Automatic feedback for motivation and self-regulation

Automatic feedback for motivation and
self-regulation
Professor Denise Whitelock
The Open University, Walton Hall,
Milton Keynes MK7 6AA, UK
denise.whitelock@open.ac.uk

I hate marking but want the tasks and feedback to
assist student learning
DMW UOC Open Seminar May 2013

The e-Assessment and automatic feedback
Challenge
• Constructivist
Learning –
Push
• Institutional
reliability and
accountability –
Pull
. DMW UOC Open Seminar May 2013

www.storiesabout.com
www.storiesabout.com/cr
eativepdp
c.mckillop@rgu.ac.uk

MCQs: Variation on a theme
Example of LAPT Certainty-Based
Marking, UK cabinet ministers demo
exercise showing feedback,
University College, Tony Gardner-
Medwin
Drug Chart Errors and Omissions,
Medicines Administration Assessment,
Chesterfield Royal Hospital

MCQs: High Stakes Assessment
Example of practice Thinking Skills
Assessment" (TSA) question,
Admissions Interview,
Cambridge Assessment, Steve Lay
Example of practice Thinking Skills
Assessment" (TSA) feedback,
Admissions Interview,
Cambridge Assessment, Steve Lay

Scaffolding and High Stakes assessment
• Math for Science
• Tutor less course
• Competency led
• No point to cheat
• Web home exam
• Invigilation technologies

Self diagnosis
• Basic IT skills, first
year med students
(Sieber, 2009)
• Competency based
testing
• Repeating tests for
revision
• Enables remedial
intervention

Characteristics Descriptor
Authentic Involving real-world knowledge and skills
Personalised Tailored to the knowledge, skills and interests of each
student
Negotiated Agreed between the learner and the teacher
Engaging Involving the personal interests of the students
Recognise existing skills Willing to accredit the student’s existing work
Deep Assessing deep knowledge – not memorization
Problem oriented Original tasks requiring genuine problem solving skills
Collaboratively produced Produced in partnership with fellow students
Peer and self assessed Involving self reflection and peer review
Tool supported Encouraging the use of ICT
Elliott’s characteristics of Assessment 2.0 activities

Authentic Assessment: Building e-portfolios
on a chef’s course
food preparation for e-portfolio, Modern
Apprenticeship in Hospitality and Catering,
West Suffolk College, Mike Mulvihill
Evidence of food preparation skill for e-
portfolio, Modern Apprenticeship in
Hospitality and Catering, West Suffolk
College, Mike Mulvihill

Peer Assessment and the WebPA Tool
• Loughborough (Loddington et al,
2009)
• Self assess and peer assess
with given criteria
• Group mark awarded by tutor
• Students rated:
• More timely feedback
• Reflection
• Fair rewards for hard work
• Staff rated:
• Time savings
• Administrative gains
• Automatic calculation
• Students have faith in the
administrative system

Mobile Technologies and Assessment
• MCQs ,PDAs
Valdiva &
Nussbaum(2009)
• Polls,instant surveys
• Simpson & Oliver
(2007)
• Draper (2009) EVS

Gains from Formative Assessment
• Mean effect size on standardised tests between 0.4 to
0.7 (Black & Williams, 1998)
• Particularly effective for students who have not done
well at school
http://kn.open.ac.uk/document.cfm?docid=10817
• Can keep students to timescale and motivate them
• How can we support our students to become more
reflective learners and engage in formative assessment
tasks?

Collaborative formative assessment with
Global Warming
DMW, Institute of Educational Technology, September 1997DMW, Institute of Educational Technology, September 1997

Global Warming

Next: ‘Yoked’ apps via BuddySpace
Student A
Student B
(‘yoked’, but
without full
screen sharing
required!)

Global Warming: Simlink Presentation

LISC: Aily Fowler
• Kent University ab-initio Spanish module
• Large student numbers
• Skills-based course
• Provision of sufficient formative assessment
meant unmanageable marking loads
• Impossible to provide immediate feedback
• leading to fossilisation of errors

The LISC solution: developed by Ali Fowler
• A CALL system designed to enable students
to:
• Independently
practise sentence
translation
• Receive immediate
(and robust)
feedback on all
errors
• Attend
immediately to the
feedback (before
fossilisation can
occur)

How is the final mark arrived at in the
LISC System?
• The two submissions are unequally weighted
• Best to give more weight to the first
attempt
• since this ensures that students
give careful consideration to the
construction of their first answer
• but can improve their mark by
refining the answer
• The marks ratio can vary (depending
on assessment/feedback type)
• the more information given in the
feedback, the lower the weight
the second mark should carry

Heuristics for the final mark
• If the ratio is skewed too far
in favour of the first
attempt…
• students are less
inclined to try hard to
correct non-perfect
answers
• If the ratio is skewed too far
in favour of the second
attempt…
• students exhibit less
care over the
construction of their
initial answer

What about emotional support in the
feedback?
• Difficult at times to receive written feedback
• Not just a cognitive response
• How can Bales help?

Coding into Categories
• Bales analysis
• Psychology 1950s
• Analyses talk
• Includes socio-emotive categories
• Flander’s (1970)categories inappropriate as also
includes classroom control

Bales Categories
• Four main groupings
• A. Positive reactions; agreeing and boosting the other
person
• B. Directing/teaching
• C. Questions: requesting information, clarification etc
• D. Negative reactions: disagreement

Coding the tutor comments
Categories Specific Examples
Positive Reactions
A1
A2
A3
1. Shows solidarity
2. Shows tension release
3. Shows agreement
Jokes, gives help, rewards others
Laughs, shows satisfaction
Understands, concurs, complies, passively accepts
Attempted Answers
B1
B2
B3
4. Gives suggestion
5. Gives opinion
6. Gives information
Directs, proposes, controls
Evaluates, analyses, expresses feelings or wishes
Orients, repeats, clarifies, confirms
Questions
C1
C2
C3
7. Asks for information
8. Asks for opinion
9. Asks for suggestion
Requests orientation, repetition, confirmation, clarification
Requests evaluation, analysis, expression of feeling or
wishes
Requests directions, proposals
Negative Reactions
D1
D2
D3
10. Shows disagreement
11. Shows tension
12. Shows antagonism
Passively rejects, resorts to formality, withholds help
Asks for help, withdraws
Deflates others, defends or asserts self
Bales’ Interaction Process

Identifying trends: H801
0 5 10 15 20 25
A Pass 1
A Pass 2
A Pass 3
A Pass 4
B Pass 1
B Pass 2
B Pass 3
B Pass 4
C Pass 1
C Pass 2
C Pass 3
C Pass 4
D Pass 1
D Pass 2
D Pass 3
D Pass 4
Bales'InteractionalCategoriesateachPassLlevel
Number ofIincidences
Graph to show conflated Bale’s categories against mean number of incidences in H801 scripts

Identifying trends: H801
5.96
17.13
5.73
1.61
A
B
C
D
Pie Chart to show the mean number of incidences per pass per
conflated Bales' Interactional Category for all four levels of
pass in H801 scripts
Key:
A = Positive reactions
B = Responses
C = Questions
D = Negative reactions

Identifying trends
A
B
C
D
A
B
C
D
A
B
C
D
Pie Charts to show the mean number of incidences per conflated Bales Interactional Category
for ‘Pass 1’ and ‘Pass 4’ in the following courses:
Key:
A = Positive reactions C = Questions B = Responses D = Negative reactions
Pass 4
Pass 1
B820 S103
A
B
C
D
A
B
C
D
A
B
C
D
H801

What is Open Mentor?
• “An open source mentoring tool for tutors”
• “Open source” = free and easy to use, and
to embed in an institutions infrastructure
and working practices
• “mentoring” = designed to help people
learn how to give feedback effectively,
through reflection and social networks
• “tutors” = primarily intended for teaching
staff, but with clear applications for those
involved in quality

How Open Mentor handles comments
• “Good work”
• “Yes, well done”
• “Yes, but is this useful?”
• “Can you explain what you
mean”
• “This does not follow”
• A = positive reactions
• A = positive reactions
• B = attempted answers, and
not a positive reaction
• C = questions
• D = negative reactions

Explaining OpenMentor’s Rules
• Four categories
• A – Positive Reactions
• B – Attempted Answers
• C – Questions
• D – Negative Reactions

‘A’ - Positive Reactions
Category Examples of Rules Examples of comments
A - Positive Reactions
1. Shows solidarity A1 ...excellent... Excellent Conclusions.
A1 ...(good|comprehensive)... Good, you are drawing on hard facts here.
A1 ...nicely... Very nicely stated. Your analysis is thorough and your
conclusions consistent regarding the attractiveness of the budget
airline sector. This is a good example of critical thinking.
A1 ...well presented... Very well presented diagram with interesting information.
A1 ...effective use... Effective use of the case material here.
A1 …well (structured|stated)… Report very well structured.
A1 ...(well|clear)(ly)*
(structured|structure|summary|
summarised|presented|presentation)...
The corporate vs. business unit strategy is well presented and
nicely tied to strategies.
A1 ...reasonable.... A reasonable structure as listed in your table of contents.
A1 ...useful point(s)... Generally useful points in this section.
2. Shows tension release A2 ... a helpful...
A2 …(thanks|thank you)…
3. Shows agreement A3 ...yes... Yes, the intellectual reactions are both real.
A3 ...indeed... Indeed – if it has one basic strategy it is surely differentiation,
though it still has to control costs.
4. Praise then direction A4 good...but... Good model, good quote, but be careful about what industry you
analyse ??

‘B’ Attempted Answers
Category Rules Examples of comments
B - Attempted Answers
B4 …perhaps… Perhaps even better here to explain the link in your mind between
"analysis of stratgeies" and "strategic issues".
B4 …requires…
B4 …take care… Take care with your STEP analysis not to make it too industry
focussed.
B4 …useful to… Innovation is closely linked to structure and culture- it would be
useful to see some book 6-8 concepts here too.
B4 …you (might|ought)… You ought to have explicitly stated these.
B4 Don’t|never … Don’t introduce new frameworks just for the sake of it in the
conclusion. The conclusion should be pulling together what went
before.
B4 Please (see|refer to|look at)... Please make sure to read and understand the question correctly
5. Gives opinion B5 I (am|think)... I think I can see where you are going, though a numbered report
format might have demonstrated the approach better
B5 This is.... this is an introduction rather than a “summary”
B5 ...sounds...like... This sounds as if it could be very popular!!
B5 ...not sure... I am not sure about the balance between the environmental
analysis and the review of the resources, capabilities (power,
culture, structures and systems) as raised in the question.
B5 I (thought|agree|suggest)… I thought it was because they did not need any external input and
saw a significant market sector they could address themselves.
B5 I (do|don’t) think… I don’t think this exercise has helped to develop your analysis. I
also think that the development of the perspectives is superficial
6. Gives information B6 …(demonstrates|shows) this….
B6 Also… Also, cross link to Leadership issues, Pettigrew on Strategic
Thinking too
B6 ...Q1... etc Q1 = 59/100

Is the rule set generic?
Comments Classified from Test Data
0
10
20
30
40
50
60
70
80
90
100
B820 Test
Set 1
B820 Test
Set 2
B820 Test
Set 3
A850 M878 S809
Test Data Set
%ofcommentsclassified

OpenMentor Transfer: JISC funded
• JISC funded project
• Transfer OpenMentor
technology to King’s and
Southampton
• What changes are
needed for cross
institutional use?
• Identify strengths and
limitations of OM for
training tutors

Transferring OM to other HEIs
• Transferred to Southampton and Kings
London
• Participating Tutors given face to face training
• King's College:
• 3 tutors.
• 25. learning experts at TEL forum
gave feedback after a demonstration
• Southampton
• 3 tutors.
• Interviews and questionnaire
• Open University
• 3 distance education tutors
• Questionnaire and epistolary
interviews
• 113 students in a Masters course
focussing on Innovation in eLearning
and 5 tutors.

Lessons learned after completion of first
cycle of trials
• Open Mentor’s theoretical framework was
robust enough to facilitate and encourage
dialogue and reflective activities
• Tutors positive about the system’s functions
to support provision of feedback
• Suggestions for change
• a module for user authentication and
management
• the development of OM reports to help tutors
to progress towards the ideal ‘state’ of
feedback provided.
• used for training purposes as an academic
development tool.
• Our contact details, blog and references -http
://omtetra.ecs.soton.ac.uk/wordpress/

What can we learn from modelling
tutors marking to construct a formative
e-assessment tool?
• Open Comment project builds on the work of
OpenMentor
• Free text entry for History and Philosophy students
• Immediate feedback (in context) to students
• Influenced by ELIZA (Weizenbaum, 1963)

Open Comment
addresses the problem
of free text entry
• Automated formative
assessment tool
• Free text entry for students
• Automated feedback and
guidance
• Open questions, divergent
assessment
• No marks awarded
• For use by Arts Faculty

Causal models of explanation
• First step:
• Identification of question types where students
exhibit causal reasoning
• Looked for questions with:
• Salient variables
• Description of these variables
• Identification of trends
• Identification of relationship between the variables
i.e. causality

Praise for effort and not just ability
• Praise for ability per se
can hinder learning
(Mueller & Dweck,
1998)
• Praise = being clever
• Negative feedback now
without ability
• Disempowering and
demoralising

Mueller & Dweck (1998)
• Raven’s Matrices (IQ)
• First test pupils praise either for effort or ability
• Second test most difficult
• Third test medium difficulty. Score up 1 points for pupils
praised for effort. Down 1 point ability

Stages of analysis by computer of students’ free text
entry for Open Comment: advice with respect to
content (socio-emotional support stylised example)
• STAGE 1a: DETECT ERRORS E.g. Incorrect dates,
facts. (Incorrect inferences and causality is dealt with
below)
• Instead of concentrating on X, think about Y in order to
answer this question Recognise effort (Dweck) and
encourage to have another go
• You have done well to start answering this question but
perhaps you misunderstood it. Instead of thinking about
X which did not…….. Consider Y

Computer analysis continued
• STAGE 2a: REVEAL FIRST
OMISSION
• Consider the role of Z in your
answer Praise what is correct
and point out what is missing
Good but now consider the role
X plays in your answer
• STAGE 2b: REVEAL SECOND
OMISSION
• Consider the role of P in your
answer Praise what is correct
and point out what is missing
Yes but also consider P. Would
it have produced the same result
if P is neglected?

Final stages of analysis
• STAGE 3:REQUEST
CLARIFICATION OF KEY POINT 1
• STAGE 4:REQUEST FURTHER
ANALYSIS OF KEY POINT
1(Stages 3 and 4 repeated with all
the key points)
• STAGE 5:REQUEST THE
INFERENCE FROM THE
ANALYSIS OF KEY POINT 1 IF IT
IS MISSING
• STAGE 6:REQUEST THE
INFERENCE FROM THE
ANALYSIS OF KEY POINT 1 IF IT
IS NOT COMPLETE
• STAGE 7:CHECK THE CAUSALITY
• STAGE 8:REQUEST ALL THE
CAUSAL FACTORS ARE
WEIGHTED

Where are we now?
• Opening up with Open Source
• Moving towards vision and not losing sight of it through
tool adaptation
• More work to do for Arts
• Open Comment - pedagogical model open to test
• Feedback
• Changing pedagogy
• Another handle on misconceptions

Open Comment drivers for reflection
• Students are able to find facts similar to X
• Know how X might be disputed
• Are able to make predictions about X
• Know how to use X in an argument
• Know how far X can be pushed
• Supported with tools and strategies for effort

SAFeSEA
Professor Denise Whitelock
Professor John Richardson
Professor Stephen Pulman
An automated
tool
supporting
online writing
and
assessment
of essays
providing
accurate
targeted
feedback
SAFeSEA: Supportive Automated Feedback for
Short Essay Answers

About SAFeSEA
• Effect of summarisation
• What are the beneficial
factors?
• Correlate measures of
learner activity and essay
improvement
• Effect of hints
• http://
www.open.ac.uk/iet/main/research-

OpenEssayist: SAFeSEA Web application for
summarisation-based formative feedback
DMW UOC May 2013

Key words and phrases visualized in the essay context. Sentences in
light-grey (green) background are key sentences as extracted by the
EssayAnalyser (the number at the start of the sentence indicates its
key-ness ranking); bigrams are indicated in bold (red) and boxed.
DMW UOC May 2013

The structural elements of the essay can be used jointly with
the key word extraction to highlight relevant information within
specific parts of the essay, here the introduction (and the
assignment question)
DMW UOC May 2013

Key words and phrases as separate lists
DMW UOC May 2013

Dispersion of key words across the essay
http://www.open.ac.uk/iet/main/research-scholarship/research-projec
DMW UOC May 2013

Can we find ways of using graph visualization
techniques on the key words and key sentences, to
make them helpful and meaningful to students?
DMW UOC May 2013

SAFeSEA
• Support for essay
writing
• Shape landscape of
eLearning and
Learning Analytics
• Improves the
student experience
• Support advances in
NLP

Feedback
• Students must decode
feedback and then act on it
Boud (2000)
• Students must have the
opportunity to act on
feedback Sadler (1989)
• Gauging efficacy through
action

Badge System: Mozilla

Characteristics Descriptor
Authentic Involving real-world knowledge and skills
Personalised Tailored to the knowledge, skills and interests of each
student
Negotiated Agreed between the learner and the teacher
Engaging Involving the personal interests of the students
Recognise existing skills Willing to accredit the student’s existing work
Deep Assessing deep knowledge – not memorization
Problem oriented Original tasks requiring genuine problem solving skills
Collaboratively produced Produced in partnership with fellow students
Peer and self assessed Involving self reflection and peer review
Tool supported Encouraging the use of ICT
Elliott’s characteristics of Assessment 2.0 activities
A d v i c e f o r A c t i o n

The 4Ts Pyramid
Tool Development
Training of staff
Transformation
Tasks
Transfer
Learning

National Union of Students’ Principles of Effective
Assessment Times Higher Education, 29th
January
2009
• Should be for learning, not simply of learning
• Should be reliable, valid, fair and consistent
• Should consist of effective and constructive feedback
• Should be innovative and have the capacity to inspire
and motivate.
• Should be conducted throughout the course, rather
than being positioned as a final event
• Should develop key skills such as peer and reflective
assessment

Final thoughts
• There is a growing consensus in the field of assessment that
times are changing and that assessment needs to become
more embedded/central in the teaching learning cycle
(Hatzipanagos & Rochon 2011).
• Our project provides another phase in this type of research
where the balance of socio emotive content contained in
feedback cannot be ignored (Draper, 2009).
• Feedback that encourages the student to actively change
their ideas and ways of organising their answers and
discourse within a given subject domain is what is required
and advocated by Whitelock (2011) as “advice for action”.

“Advice for Action”, Whitelock (2011)
• Helping students
find out what they do
not know and how
to remedy the
situation can avoid
the trauma of
assessment
• Are we on the way
with new e-tools?

References
• Van Labeke, N., Whitelock, D., Field, D., Pulman, S. & Richardson, J.
(2013) ‘OpenEssayist: Extractive Summarisation & Formative Assessment
of Free-Text Essays’. Workshop on Discourse-Centric Learning Analytics,
3rd
Conference on Learning Analytics and Knowledge (LAK 2013), Leuven,
Belgium
• Whitelock, D., Gilbert, L., Hatzipanagos, S., Watt, S., Zhang, P., Gillary, P.
& Saucedo, A. (2012) Supporting tutors with their feedback using
OpenMentor in three different UK Universities. 10th
International
Conference on Computer Based Learning in Science, CBLIS 2012,
Barcelona, Spain. 26-29 June 2012.
• Whitelock, D., Gilbert, L. & Gale, V. (2011) ‘Technology-Enhanced
Assessment and Feedback: How is evidence-based literature informing
practice?’ International Computer Assisted Assessment Conference,
DeVere Grand Harbour Hotel, Southampton, 5/6 July 2011. http://
caaconference.co.uk/wp-content/uploads/WhitelockB-CAA2011.pdf
• Whitelock, D. (2010) Activating Assessment for Learning: are we on the
way with Web 2.0? In M.J.W. Lee & C. McLoughlin (Eds.) Web 2.0-
Based-E-Learning: Applying Social Informatics for Tertiary Teaching.
IGI Global. pp. 319–342.

References (2)
• Whitelock, D. & Watt, S. (2008) ‘Putting Pedagogy in
the driving seat with Open Comment: an open source
formative assessment feedback and guidance tool for
History Students.’ CAA Conference 2008,
Loughborough University, 8/9 July 2008, edited by
Farzana Khandia pp. 347-356 ISBN 0-9539572-7-6
http://kn.open.ac.uk/public/document.cfm?docid=11638
• Whitelock, D. & Watt, S. (2007) e-Assessment: How an
we support tutors with their marking of electronically
submitted assignments? Ad-Lib Journal for
Continuing Liberal Adult Education, Issue 32, March
2007 pp 7-9, ISSN 1361-6323.

References (3)
• Whitelock, D. (2006) Electronic Assessment: Marking,
Monitoring and Mediating Learning. In McAndrew, P.
and Jones, A. (eds) Interactions, Objects and
Outcomes in learning. Special Issue of International
Journal of Learning Technology. Vol. 2, Nos 2/3 pp
264-276.
• Whitelock, D. & Watt, S. (2006) OpenMentor: opening
tutors eyes to the written support given to students in
their assignments. JISC Conference 2006,
Information & Communication Technology in
Education and Research. International Conference
Centre, Birmingham, 14 March 2006.

Automatic feedback for motivation and self-regulation

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (20)

Similar a Automatic feedback for motivation and self-regulation

Similar a Automatic feedback for motivation and self-regulation (20)

Más de eLearn Center (Universitat Oberta de Catalunya)

Más de eLearn Center (Universitat Oberta de Catalunya) (13)

Último

Último (20)

Automatic feedback for motivation and self-regulation

Notas del editor