Moving Towards Programmatic Assessment: Challenges and Opportunities

Moving towards programmatic
assessment
challenges and opportunities
Lesley Southgate
St Georges Medical School
University of London

Background
• Over the past 30 years there has been
significant change in approaches to
assessment for the UK medical
profession and wider.
• From entry to medical school through
postgraduate training and on to
established practice, regular assessment
has become a fact of life.
14/09/2014 lesley.southgate@dial.pipex.com

Background
• The days are gone when progress to the
next stage was determined largely by exit
examinations comprising elements that
were sometimes poorly constructed,
unrelated to the taught curriculum and
which could not be challenged.

Background
• During the period of change the focus
has principally been on improving the
quality of methods of assessment and
developing approaches to assessing
performance in practice

Background
• And more recently, it has become usual
to combine various assessment
instruments into a programme of
assessment which may support a period
of education or training over several
years

Principles for good assessment design
• Modern assessment programmes typically
contain a range of assessment methods
combined in a programme of assessment
developed as part of a curriculum. They are
selected in the light of the purpose and
content of the assessment
• Van der Vleuten CP, Schuwirth LW. Assessing professional
competence: from methods to programmes. Med Edu 2005
Mar;39(3):309-17.. C.vanderVleuten@educ.unimaas.nl

Changes have come…..
One of the cornerstones of the reform of
assessment for UK undergraduate and
postgraduate training, is the acceptance that a
student/trainee will be assessed both by
examinations, and in the workplace, within a
programme of assessment methods which,
taken together, ensures assessment of each
domain of Good Medical Practice. (UK
regulator guidance)

Changes have come
This enables a rich picture of the performance of
the student/trainee to be assembled and
considered in the light of the type of decision
that is to be made about career progression,
learning needs, and professional
development.
• http://www.gmc-uk.org/guidance/good_medical_practice/contents.asp

The purpose of assessing in the workplace
In educational settings assessment for learning should take
priority over assessment of learning.
A programme of assessment should aim at building n:n
relationships: each competency domain should be informed
from various assessment sources and each assessment source
should be used to inform about several competency domains.’
The GMC and other national bodies are currently grappling
with this issue
Programmatic assessment: From assessment of learning to assessment for learning.
Schuwirth & Van der Vleuten Medical Teacher 2011:33:478-485

Introduction and context
• From the beginning briefly
– The UK scene from 1995………
• GMC Good Medical Practice effective from 1995
• Latest edition 2013
– The GMC performance procedures
• A form of programmatic assessment introduced 1997
– PMETB 2005-2010
• Principles for assessment
• The UK foundation programme

UK GMC standards: Good Medical
Practice
• Good clinical care
• Maintaining good medical practice
• Teaching and training, appraising and
assessing
• Relationships with patients
• Working with colleagues
• Probity
• http://www.gmc-uk.org/guidance/good_medical_practice/duties_of_a_doctor.asp

(for example) Good medical practice
• Good clinical care must include:
– adequately assessing the patient's conditions,
taking account of the history (including the
symptoms, and psychological and social factors),
the patient's views, and where necessary
examining the patient
– providing or arranging advice, investigations or
treatment where necessary
– referring a patient to another practitioner, when
this is in the patient's best interests

UK GMC performance procedures
Late 1997/2004
• Assessment of poorly performing doctors
– At risk of losing licence to practice.
• Questions we asked
– What are the standards all practising physicians must
reach whether they are in training or fully registered
– What evidence must we collect to demonstrate that
the standards have been reached
– And where could we collect the evidence about
performance

Which standards, what evidence,
where…….
• The conclusions we reached, and the design for the
performance procedures, now well established in
law, informed thinking in the establishment of the
workplace assessments now in use in UK
postgraduate training.
Southgate, L., Cox, J., McAvoy, P., McCrorie, P.,et al. (2001)
The General Medical Council’s Performance Procedures: peer review of
performance in the workplace. Medical Education, vol.35 , Issue Supplement
s1 Pages 1–78
When enough is enough: a conceptual basis for fair and defensible practice
performance assessment. Schuwirth LW1, Southgate L, et al. Med Educ. 2002
Oct;36(10):925-30.

PMETB principles for assessment
A working paper from the Postgraduate Medical Education
and Training Board 14 September 2004
Lesley Southgate and Janet Grant
In this paper, an assessment system refers to an integrated set of assessments
which is in place for the entire postgraduate training programme and
which supports the curriculum. It may comprise different methods, and be
implemented either as national examinations, or as assessments in the
workplace. The balance between these two approaches principally relates
to the relationship between competence and performance. Competence
(can do) is necessary but not sufficient for performance (does do), and as
experience increases so performance based assessment in the workplace
becomes more important
http://evavalpa.org/modulos/modulo_04/principles__assessment.pdf

• Principle 1 The assessment system must be fit for a
range of purposes
• Principle 2 The content of the assessment will be
based on curricula for postgraduate training which
themselves are referenced to all of the areas of Good
Medical Practice The blueprint from which
assessments in the workplace or national
examinations are drawn will be available to trainees
and educators in addition to assessors/examiners
• Principle 3 The methods used within the programme
will be selected in the light of the purpose and
content of that component of the assessment
framework.

• Principle 4 The methods used to set standards for
classification of trainee’s performance/competence
must be transparent and in the public domain
• Principle 5 Assessments must provide relevant
feedback
• Principle 6 Assessors/examiners will be recruited
against criteria for performing the tasks they will
undertake

• Principle 7
• There will be Lay input in the development of
assessment
• Lay opinion will be sought in relation to appropriate
aspects of the development, implementation and
use of assessments for classification of candidates.
• Lay people may act as assessors/examiners for areas
of competence they are capable of assessing.
• Principle 8
• Documentation will be standardised and accessible
nationally

• Principle 9
• There will be resources sufficient to support
assessment
• Resources will be made available for the
proper training of assessors

And: the workplace
• The workplace enables the trainee to encounter
and resolve common and important clinical
problems in real time.
• The level of performance expected will depend
on stage of training and feedback about progress
• It gives an opportunity to observe a trainee
demonstrate understanding of what it means to
adopt Good Medical Practice as the basis for all
aspects of professional life.

From the GMC…to summarise……
One of the cornerstones of the reform of assessment for
UK postgraduate training, is the acceptance that a
trainee will be assessed both by examinations, and in
the workplace, within a programme of assessment
methods which, taken together, ensures assessment
of each domain of Good Medical Practice. This
enables a rich picture of the performance of the
trainee to be assembled and considered in the light of
the type of decision that is to be made about career
progression, learning needs, and professional
development.
• http://www.gmc-uk.org/guidance/good_medical_practice/contents.asp

The purpose of assessing in the
workplace
In educational settings assessment for learning should take
priority over assessment of learning.
A programme of assessment should aim at building n:n
relationships: each competency domain should be informed
from various assessment sources and each assessment source
should be used to inform about several competency domains.’
The GMC are currently grappling with this issue
Programmatic assessment: From assessment of learning to assessment for learning.
Schuwirth & Van der Vleuten Medical Teacher 2011:33:478-485

Early days
UK FOUNDATION PROGRAMME

The UK Foundation programme
from 2007 onwards
• The Foundation Programme is a two-year
generic training programme which forms the
bridge between medical school and
specialist/general practice training.

Early days: Assessment in the
Programme
• Purpose
– Determine fitness to
progress to next stage of
training
– Identify trainees in
difficulty
– Provide focused feedback
consistent with CQI
– Meet needs for
accountability
• Four Methods
– mini-Clinical Evaluation
Exercise (mCEX)
– Directly Observed
Procedures (DOP)
– Case-Based Discussion
(CbD)
– Peer Assessment (mini-
PAT)
• Refined versions of
traditional measures

•
The toolkit for workplace assessment
9/14/2014
DOPS MiniCex CbD MSF
Prof Dame Lesley Southgate
lsouthga@sgul.ac.uk
Purpose Observation
Observe
and assess
the conduct
of a
practical
procedure
Observation
Observe
and assess
a clinical
encounter
Conversation/discussion
Discuss an outcome/
output from workplace
activity using a record
the trainees has made a
contribution to
Review by
others/colleagues
Professionalism
Interpersonal
skills/Team working
Communication
Takes
place
Process
Reviewed
and
documented
with
feedback in
the moment/
as it is
happening
Process
Reviewed
and
documented
with
feedback in
the moment/
as it is
happening
Outcome/output
Discussing, explaining,
justifying aspects of the
report/record/result.
Including aspects of
professionalism
Reflecting on
comments of others
within the
framework of
constructive
feedback

Assessment: early days
• Foundation Programme Year 1
– 3640 trainees had at least one of instruments
completed
– 2929 submitted at least one encounter for all four
tools
– There were 7 to 11 questions per instrument
• All used a 6-point scale where 4 is “meets expectations”

Encounters: Complexity
70
60
50
40
30
20
10
0
mCEX CbD DOPs
Low
Average
High

Early days: References
• Med Educ. 2009 Jan;43(1):74-81. doi: 10.1111/j.1365-2923.2008.03249.x.
• Initial evaluation of the first year of the Foundation Assessment Programme.
• Davies H1, Archer J, Southgate L, Norcini J.
• Adv Health Sci Educ Theory Pract. 2008 May;13(2):181-92. Epub 2006 Oct 12.
• mini-PAT (Peer Assessment Tool): a valid component of a national assessment
programme in the UK?
• Archer J1, Norcini J, Southgate L, Heard S, Davies H.
• Med Educ. 2008 Oct;42(10):1014-20. doi: 10.1111/j.1365-2923.2008.03162.x.
• Specialty-specific multi-source feedback: assuring validity, informing training.
• Davies H1, Archer J, Bateman A, Dewar S, Crossley J, Grant J, Southgate L.

Early warnings 1
• The FAP was implemented in a very short time-frame in
response to a central mandate and there was understandably
significant concern about feasibility and the time it would
require. Despite this, a mean of 16.6 case-focused
assessments were submitted by each F1 trainee, although
40% of these were submitted in the last 6 weeks. It is likely
that this reflects anxiety about achieving low scores early on
in the year. Although the programme explicitly states that
some scores of < 4 would be expected early in the year, this
represents a major cultural shift in assessment.
• Med Educ. 2009 Jan;43(1):74-81. doi: 10.1111/j.1365-2923.2008.03249.x.
• Initial evaluation of the first year of the Foundation Assessment Programme.
• Davies H1, Archer J, Southgate L, Norcini J.

Early warnings 2
• It is also important that training is directed at
all the health professionals involved in
assessments and that it includes senior
trainees and nurse specialists. In order to fully
meet the PMETB principles, not only will
assessors need to be trained, but there will
need to be systematic processes in place to
provide them with feedback on their
performance

Feedback, judgement and training
assessors
• ‘What was striking during these discussions was the
expectation among trainees that WPBA should be
about helping them to become better doctors and their
corresponding openness to feedback. However,
expectations are dashed by a system that is seen to be
open to bias and corruption, with assessors who are
untrained and too busy, and which is thus failing to
deliver high quality, honest feedback. In turn, the
enormous potential benefit of helping trainee doctors
learn from their performance is being lost’.
• Abigail Sabey, Centre for Learning and Workforce Research
University of the West of England, Bristol

fast forward to 2013
• Foundation year 1 (F1) enables medical graduates to
begin to take supervised responsibility for patient
care and consolidate the skills that they have learned
at medical school. Satisfactory completion of F1
allows the relevant university (or their designated
representative in a postgraduate deanery or
foundation school) to recommend to the GMC that
the foundation doctor can be granted full
registration.
• http://www.foundationprogramme.nhs.uk/pages/home/curriculum-and-assessment/
curriculum2012

2013
• Foundation year 2 (F2) doctors remain under clinical
supervision (as do all doctors in training) but take on
increasing responsibility for patient care. In particular
they begin to make management decisions as part of
their progress towards independent practice. F2
doctors further develop their core generic skills and
contribute more to the education and training of the
wider healthcare workforce e.g. nurses, medical
students and less experienced doctors.
curriculum2012

The UK Foundation programme 2013
• At the end of F2 they will have begun to demonstrate
clinical effectiveness, leadership and the decision-making
responsibilities that are essential for hospital
and general practice specialty training. Satisfactory
completion of F2 will lead to the award of a
foundation achievement of competence document
(FACD) which indicates that the foundation doctor is
ready to enter a core, specialty or general practice
training programme.
curriculum2012

And what happened to the workplace
assessments?
• SLEs (structured learning events) will continue to use the established set
of tools of mini-clinical evaluation exercise (mini-CEX), acute care
assessment tool (ACAT) and case-based discussion (CbD) and the forms
will focus on constructive feedback and action plans.
• Trainees may link SLEs and other evidence to curriculum competencies in
order to demonstrate engagement with and exploration of the curriculum.
The trainee has to make a judgement as to the evidence needed
• Supervisors should sample the evidence linked to competencies in the
ePortfolio. It is not necessary to examine all the competencies to
determine a trainee’s engagement with the curriculum and to make a
judgement on the trainee’s progress
• http://www.jrcptb.org.uk/assessment/Documents/STAR%20report%20fina
l%2029%20April%202014.pdf

A happy ending? 2014
• Supervised learning event Recommended minimum number* Direct
observation of doctor/patient interaction:
• Mini-CEX
• DOPS
• 3 or more per placement*
• (minimum of nine observations;
• at least six must be mini-CEX)
• Case-based discussion (CBD)
• 2 or more per placement*
• Developing the clinical teacher
• 1or more per year

Moving Towards Programmatic Assessment: Challenges and Opportunities

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (8)

Similar a Moving Towards Programmatic Assessment: Challenges and Opportunities

Similar a Moving Towards Programmatic Assessment: Challenges and Opportunities (20)

Más de MedCouncilCan

Más de MedCouncilCan (20)

Último

Último (20)

Moving Towards Programmatic Assessment: Challenges and Opportunities