Dr Cruickshank

2. Prof Maggie Cruickshank
Professor MB ChB MD FRCOG Division of Medical and
Dental Education and Medical Sciences

3. Work place assessments for
training in colposcopy
Professor Maggie Cruickshank
University of Aberdeen

4. Objectives
• Explain the principles of good assessment
• Describe the workplace-based assessment tools
• Consider useful feedback
• Consider descriptors in workplace assessment
tools
• Reflect on the challenges related to assessment
of performance

5. Principles of Assessment
An effective assessment system should:
• Provide feedback for trainees
• Clarify action plans
• Help assessors to measure trainee performance
• Help assessors decide on competency

6. Features of Good Assessment
• Reliability
• Validity
• Feasibility
• Educational impact

7. What is the best method of
assessing Colposcopy practice?
…is it valid/reliable?

8. Reliability
• Does the test consistently measure the
same thing?
• Expressed as a number between 0 and 1
 1 = you are making it up
 0 = useless

9. Validity
• Does the test measure what it is supposed
to measure?
…not always easy to define or demonstrate!

10. Miller’s pyramid
Knows
Knows how
Shows
Does

11. Competence vs Performance
• Competence-based assessment
measures what doctors can do in
controlled representations of professional
practice
• Performance-based assessment
measures what doctors actually do in their
professional practice

12. Knows
Knows how
Shows
Does
Written exams
OSCE Competence
Knowledge
Performance
Action
valid
reliable
Content validity
Concurrent validity
Construct validity
Predictive validity
Test-retest reliability
Inter-rater reliability
Internal consistency
OSCE: Objective Structured Clinical Examination

13. Knows
Knows how
Shows
Does
Competence
Knowledge
Performance
Action

14. What type of question?

15. • Critical to learning and has a significant
influence on achievement
• Feedback is the largest influence on
achievement in learning or training
• Feedback alone is effective in 71% of
education studies
• Observation of a clinical performance
enhances learning
(Hattie 1999; Veloski 2006)
Formative Assessment and Feedback

16. Definition of Workplace-based
Assessment (WPA)
“The assessment of working practices based
on what doctors actually do in the
workplace, and is predominantly carried
out in the workplace itself”

17. Workplace-based Assessment (WPAs)
Methods
• Structured scoring systems to ensure systematic
review and feedback on colposcopy
performance
• Mini-Clinical Evaluation Exercise (mini-CEX)
• Directly Observed Procedural Skills (DOPS)
• Case-based Discussion (CbD)

18. Mini-CEX
• Focuses on formative assessment of
clinical skills
• Responds to the assessment problems of
the traditional long case
• Responds to the educational problems of
the workplace

19. Mini-CEX
• Assessor observes a trainee with a patient
• Usually a very specific focused task
– Take a history
– Do a colposcopy examination
– Explain results
– Form a management plan
– Not all components are assessed (i.e., mini)
• Assessor scores performance on a structured rating scale
• Trainee is given feedback
• Takes 5-10 minutes

20. Complexity of Low Average High
case:   
Please grade the following areas using
the scale below
Below
expectations
Borderline Meets
expectations
Above
expectations
U/C*
1 History Taking 1 2
 
3

4

5 6
  
2 Examination Skills       
3 Communication Skills       
4 Clinical Judgement       
5 Professionalism       
6 Organisation/Efficiency       
7 Overall clinical care       
*U/C Please mark this if you have not observed the behaviour and therefore feel unable to comment
Anything especially good? Suggestions for development
Agreed action plan:

21. Why multiple MiniCEXes ?
• Improve the reliability of any single
assessment
• Completes picture of performance
• Must agree in advance of patient
interaction and document
• Must be relevant to colposcopy training

22. Direct Observation of Procedure
• Assessor observes a trainee undertaking a
procedure
– Routine colposcopy examination or procedure
– Assessor rates performance on a rating scale
(check list)
– Takes as long as procedure (minutes)
– Trainee gets instant feedback
– Multiple encounters

23. Direct Observation of Procedure
• Assesses technical competency of a
particular procedure
• Assesses when trainee is ready to move
on to independent practice for a procedure
• Combined with other work place
assessments to determine competency for
independent practice

25. Case-based Discussion
• Structured format for discussing a clinical case
• Helps to record conversation about, and
presentations of, cases by trainees
• Designed to assess clinical decision making and
the application or use of medical knowledge
• Allows the trainee to discuss why they acted as
they did

26. Strengths of WPA
• Sample widely across the training curriculum
• Multiple raters
• Formative and summative
– Used with more reliable assessments to inform
high stakes decisions
• Reinforce educational culture where
feedback for learning is the norm
• Identify struggling trainees early

27. Assessment in training
Assessment1
Assessment2
Assessment3
Assessmentx
Competency 1 • •
Competency 2 • •
Competency 3 •
Competency x •

28. Limitations of WPA
• Not reliable enough to stand alone
• Low scores = “failure”
• Early success reduces motivation
• Weaker trainees least likely to seek
feedback
• Time
• Availability of trainers

29. Effective feedback is the key to the
success of these assessment
methods

30. Questions ?