The document discusses competency-based medical education across the continuum of training. It begins by outlining the speaker's conflicts of interest in assessment-related organizations. The objectives are then stated as discussing the roles of assessment in a competency-based program, developing approaches to effective assessment, and integrating assessment into Memorial University's education system. Milestones and entrustable professional activities are presented as frameworks to assess competency development longitudinally. Challenges of assessment across training levels and principles of competency-based assessment are also addressed.
The Value of Competency-based Medical Education Across the Continuum
1. The Value of
Competency-based
Medical Education
Across the Continuum
Eric S. Holmboe
2. Conflicts of Interest
• Employed by the American Board of
Internal Medicine
• Serve on the boards of the National
Board of Medical Examiners and
Medbiquitous (both non-profit)
• Receive royalties from Mosby-Elsevier
for a textbook on assessment
2
3. Objectives for Today
• Discuss the roles of assessment in a
competency-based medical program
• Develop approaches to help local educators
effectively use performance-based
assessment methods and tools
• Discuss how performance-based assessment
can be integrated into an education and
assessment system for Memorial University.
3
4. Nostalgialitis Imperfecta
Syndrome characterized by the following signs
and symptoms:
– “When I was an student…<insert superlative>”
– “Medicine was so much better 25 years ago”
• Reality: Not really…
– “Younger physicians today are less
professional, skilled, etc. because of <insert
favorite complaint>”
6. Change in Performance Over
Time
Lower Performance All Outcomes
Choudhry NK, Ann Intern Med, 2005;142:260-73
7. With your immediate
neighbors, discuss what
competency-based education
and training means to you?
8. Competency-Based Medical
Education
Is an outcomes-based approach to the
design, implementation, assessment and
evaluation of a medical education program
using an organizing framework of
competencies
the unit of progression is mastery of
specific knowledge, skills and attitudes
Frank, JR, Snell LS, ten Cate O, et. al. Competency-based medical
education: theory to practice. Med Teach. 2010; 32: 638–645
9. So What are the Outcomes
and
Who Determines Them?
The Profession?
The Public?
Policy Makers?
10. Determining Outcomes: Perspectives
The Profession?
– The “core” of a discipline?
– Competence in the Can MEDS roles? Safe,
effective, patient-centered care?
The Public?
– Trust that a doctor can do certain things?
Policy Makers?
– Meeting the needs of the complex and aging
Canadian health population?
11. Traditional versus Competency-based:
Start with System Needs
Frenk J, et al. Health professionals for a new century: transforming
education to strengthen health systems in an interdependent world.
Lancet. 2010 11
13. Implications of CBME
Curriculum and assessment follows from the
competencies and outcomes, not vice versa
Requires:
– Definition of milestones of competency
• What does competency look like?
– Robust assessment methods, tools & systems
14. Educational Program
Variable Structure/Process Competency-based
Driving force: Content-knowledge Outcome-knowledge
curriculum acquisition application
Driving force: process Teacher Learner
Path of learning Hierarchical Non-hierarchical
(Teacher→student) (Teacher↔student)
Responsibility: content Teacher Student and Teacher
Goal of educ. encounter Knowledge acquisition Knowledge application
Typical assessment tool Single subject measure Multiple objective measures
Assessment tool Proxy Authentic (mimics real tasks of
profession)
Setting for evaluation Removed (gestalt) Direct observation
Evaluation Norm-referenced Criterion-referenced
Timing of assessment Emphasis on summative Emphasis on formative
Program completion Fixed time Variable time
Carraccio, et al. 2002.
20. KSA Framework
K = Knowledge
S = Skill
Information gathering skills
– Interview, physical examination, communication
Ability to use knowledge and information
– Problem solving, clinical judgment
Management skills
– Diagnosis, treatment, patient education,
counseling, procedural skills
A = Attitudes
Professionalism, humanism
21. RIME Model
Developed at USUHS
– Lou Pangaro and Gordon Noel in the 1980s
for use in third year medical student
clerkships in internal medicine
“Synthetic” Model
– Reporter
– Interpreter
– Manager
– Educator
22. Dreyfus Developmental
Model
• Novice – Don’t know what they don’t know
• Advanced Beginner – Know what they don’t know
• Competent – Able to perform the tasks and roles of the
discipline – restricted breath and depth
• Proficient – Consistent and efficient in performance of the tasks
and roles of the discipline - know what they know and don’t know
• Expert – In depth knowledge concerning the discipline – often
rule based – know what they know
• Master – Expert who relishes the unknown, or the situation that
breaks the rules – who the experts go to for help – don’t know
what they know 1
as presented by Leach, D., modified by Nasca, T.J.
American Board of Internal Medicine Summer Retreat,
August, 1999
23. Competence Learning
Performance is exhausting
Sense of what’s important may be lacking
Volume of various aspects can be
overwhelming
Competent model of decision making:
– “is a detached, deliberative, and sometimes
agonizing selection among alternatives”
Dreyfus and Dreyfus, 1986
26. Milestones Definition
A significant point in development
Merriam-Webster
A scheduled event signifying the completion of a
major deliverable or a set of related deliverables.
mariosalaexandrou.com
27. Milestones and Trajectories
Milestones should enable the trainee,
program and the regulatory bodies to
know an individuals trajectory of
competency acquisition.
The focus is developmental
28. Approaches to Developing
Milestones 1
Discrete
– Defining different behaviors in a domain of
competence at each stage of training
Continuous
– A similar ability modified at each stage of
training to reflect increasing complexity or
sophistication
– Parameters:
• Setting, players, complexity, supervision
1
From Jason Frank, RCPSC
28
29. Approaches to Developing
Milestones 1
Narrative
– Detailed descriptions of stages of
development of competency by domain
– Short essays and vignettes that describe a
“competence story”
EPAs(entrustable professional activity)
1
From Jason Frank, RCPSC
29
30. Patient Care
ACGME Developmental Milestones Approximate Assessment
Competency Informing Time Frame Methods/Tools
ACGME Competencies Trainee to
Achieve
Stage
Clinical skills Historical Data Gathering Standardized
and 1. Acquire accurate and relevant history 6 months patient
reasoning from the patient in an efficiently Direct
customized, prioritized, and Observation
Manages hypothesis driven fashion Simulation
patients using 2. Seek and obtain appropriate, verified, 9 months
clinical skills of and prioritized data from secondary
interviewing and sources (e.g. family, records,
physical pharmacy)
examination 3. Obtain relevant historical subtleties 18 months
that inform and prioritize both
differential diagnoses and diagnostic
plans, including sensitive,
complicated, and detailed information
that may not often be volunteered by
the patient
Sub-
competency
31. Communicator Milestones Project
Stage 1 Stage 2 Stage 3 Stage 4
Greets the Demonstrates Demonstrates Demonstrates Demonstrates
patient in a rule-based proficiency; proficiency; proficiency &
way; some adapts to efficiency;
Simulation or adaptation; many Across
role play routine contexts; spectrum of
clinical complex practice
clinical
An Introduction to CBME – Frank, Snell, Harris, Holmboe 2012 31
32. Patient Care
The resident is demonstrating satisfactory development of the knowledge, skill and attitudes needed to advance in training.
He/she is demonstrating a learning trajectory that anticipates the achievement of competency for unsupervised practice that
includes the delivery of safe, timely, equitable patient-centered care.
____ Yes ____ No ____ Marginal
32
33. Milestones Benefits
Provide the learner with a clear path of
progression. There are no surprises.
Allow for rich formative feedback. Learners
know where they are and where they need to
go.
Define specific behaviors that can focus
assessment.
34. Milestones Challenge
Synthesizing milestones into larger global
representations of competency that reflect those
activities that define the profession.
These activities have been described as
entrustable professional activities or EPAs.
35. Entrustable Professional Activities
EPAs represent the routine professional-life
activities of physicians based on their
specialty and subspecialty
The concept of “entrustable” means:
– ‘‘a practitioner has demonstrated the necessary
knowledge, skills and attitudes to be trusted to
independently perform this activity.’’1
1
Ten Cate O, Scheele F. Competency-based postgraduate
training: can we bridge the gap between theory and
clinical practice? Acad Med. 2007; 82(6):542–547.
36. Entrustable Trainee Activities
ETAs, or entrustable resident or student
activities, can help to define important
benchmarks in a trainee’s development
ETAs in a training program may mean:
– A trainee has demonstrated the necessary
knowledge, skills and attitudes to be trusted to
perform this activity without constant or direct
supervision.
37. Why “ETAs” to Assess Competence?
Sampling of events that:
- are critical moments in medical training
- inform developmental progression
- faculty and leaders already implicitly assess
- are manageable for busy training programs
– are logical of assessment for stakeholders
Supported by generalizability theory
– 8-12 focused assessments can potentially allow
a generalized statement of competency
38. Synthesize to Physician trusted to meet
Analyze to
Educate and the health care needs of
Understand
Evaluate the population
Competency Milestones LANDMARK in EPA in
Training Practice
Medical MK1
Knowledge
MK2 Lead a resident Lead a health
care team care team
Patient Care PC1
PC2
Professionalism Prof1
Prof2 Care for clinic
patients with Practice
Interpersonal ISC1 distance
Skills independently
ISC2 supervision
Systems-based SBP1
Practice
SBP2 Complete an audit Lead Quality
of a panel of Improvement
Practice-based PBLI1 clinic patients initiative
learning
PBLI2
Shared Mental Models and Frameworks
40. Entrustments in Newfoundland
With a neighbor(s), discuss an
entrustment you make either with
medical students or post-graduate
trainees
How do you arrive at this entrustment
judgment?
A paradigm shift! This is not tinkering around the edges!
In the CBME approach, medical education is organized around “competencies”, and not just scientific medical knowledge as in our traditional approach.
In this way of thinking, our definition of competence is changed. It is multidimensional and contextual. This also leads to the relative terms “dyscompetence” and “supracompetence”.
I specifically have a box around the 18 month milestone in preparation for the next slide…
1.2 Initiating the interview Pre-Clerkship 1 Pre-Clerkship 2 MD 1 MD 2 Junior Resident 1 Junior Resident 2 Senior residency (entry to practice) 1 Senior residency (entry to practice) 2 Advanced Practice 1 Advanced Practice 2 a) Greets the patient, and caregivers, and confirms how s/he/they would like to be addressed 3. Demonstrates fully, but may be rule-based in application. 2. Simulation/Role play/Standardized patient 4. Demonstrates proficiency and efficiency and is able to adjust to the specific patient or context 3. Routine clinical 4. Demonstrates proficiency and efficiency and is able to adjust to the specific patient or context 5. Across the spectrum of professional practice 4. Demonstrates proficiency and efficiency and is able to adjust to the specific patient or context 5. Across the spectrum of professional practice 5. Demonstrates innovation, improvisation and deliberate use of best practice 5. Across the spectrum of professional practice
This example uses a narrative stream for patient care and demonstrates how narrative milestones could be used in a FasTrack type system. You will notice the language under the narrative stream asks for a judgment about the trainee’s trajectory. Given that there are 23 narrative streams which equates to about 3 – 4 for each general competency, you could envision a trainee progressing at different rates for those streams. Asking for a summative judgment allows a program to identify that a trainee may be lagging in some areas, but is still felt overall to be developing appropriately.
This would address the criticism of being reductionistic
Landmarks are a little different than EPA’s because they occur during training … Why use them to assess competence? They are a sampling of events that … Mention how the Reed Williams article supports this concept of focused assessment of a particular skill/set of skills allows for a generalized statement of competency for that skill
Here is how this may actually develop for any given learner. This does introduce the concept that competency development is not limited to the small window of formal training. One challenge that CBME will need to address is what will be the minimum needed to successfully complete residency or fellowship training?