21. Population
(e.g Under or
Postgraduates)
•Sample Size
•Power Calculation
•Criterion for
Statistical Significance
Used
Tim
Change)
Intervention
Statistical Conclusion Validity is the extent to
whether the statistical analysis can correctly
identify a change if it is present
Population
•Was there a risk of a
type II error?
Population
•Were all outcomes
correctly powered?
Population
22. Time (Which can be hours e,g pre and post test or up to years)
Change (Which can be measured in a variety of ways)
Intervention
Internal Validity is the extent to which the intervention can reliably be ascribed to
have effected the change
Population
Population
Population
bit.ly/mededvalidity
23. Threats to Internal Validity are best
analysed by considering an intervention
and control group
•History
Another event happens at
the same time as the
intervention
•Selection
There are pre-existing differences
between the groups
•Regression to the Mean
Extreme pre-test baseline performance
may mean change happens by chance
(e.g. unusually good or bad at outset)
•Maturation
There is already a pre-existing trend for
change
•Instrumentation
There are changes in the methods of measurement
(e.g data collectors become more experienced)
•Testing
Pre-testing changes the performance of
participants (e.g. promotes information seeking)
•Differential Attrition
Drop out from the study is not equal
between groups
Intervention Group
Control Group
24. Time (Which can be hours e,g pre and post test or up to years)
Change Y (Which can be measured in a variety of ways)
Intervention
A
Population
Construct Validity relates to the association between the concept you
are investigating and the measures you use to test it
25. Time (Which can be hours e,g pre and post test or up to years)
Change (Which can be measured in a variety of ways)
Intervention Population A
External Validity is the extent to which the intervention is reliable in different
populations (this may in terms of participants, settings, times etc)
26. Time (Which can be hours e,g pre and post test or up to years)
Change (Which can be measured in a variety of ways)
Intervention
Population B
Hello my name is Damian. And I am a fraud. Well maybe an imposter… I am at a conference on critical care and emergency medicine but am at my heart a humble paediatrician. I am talking on educational evaluation but am neither experienced enough or wish to be known as an educationalist. And at an event which reveals TED lectures I neither can pull of Simon’s Professorial ‘pauses’ or have the charm of Chris Nickerson. But being a imposter is a good thing. Imposters aim to understand what people are expecting to see. They have to know a concept and make it translatable so that both the expert and non-expert are drawn in. I hope this gives them perhaps a different insight.
Hello my name is Damian. And I am a fraud. Well maybe an imposter… I am at a conference on critical care and emergency medicine but am at my heart a humble paediatrician. I am talking on educational evaluation but am neither experienced enough or wish to be known as an educationalist. And at an event which reveals TED lectures I neither can pull of Simon’s Professorial ‘pauses’ or have the charm of Chris Nickerson. But being a imposter is a good thing. Imposters aim to understand what people are expecting to see. They have to know a concept and make it translatable so that both the expert and non-expert are drawn in. I hope this gives them perhaps a different insight.
Hello my name is Damian. And I am a fraud. Well maybe an imposter… I am at a conference on critical care and emergency medicine but am at my heart a humble paediatrician. I am talking on educational evaluation but am neither experienced enough or wish to be known as an educationalist. And at an event which reveals TED lectures I neither can pull of Simon’s Professorial ‘pauses’ or have the charm of Chris Nickson. But being a imposter is a good thing. Imposters aim to understand what people are expecting to see. They have to know a concept and make it translatable so that both the expert and non-expert are drawn in. I hope this gives them perhaps a different insight.
My hope then its to try and lay out a framework so that anyone can look at the an initiative or practice changing intervention they are undertaking and determine what, if any, benefit it has. Clearly education is a challenging area as if it had been cracked we would have needed to here the fantastic talks from the previous two speakers.
I want to hit you early on with my first big take-home point. Education is everywhere. Virtually every interaction you have in the workplace can lead to learning. “We are always learning”. Really nice mantra but it doesn’t half make life difficult for those of us trying to work out which learning had the greatest effect. In some respects we have become sloppy in not putting the immunisation of a child “an intervention” on a par with a ultrasound teaching course “equally an intervention”. I can feel eyes beginning to roll already so I hope the fact the The description of the mode of delivery of the intervention was reported for only 52% of educational studies by Pino 2012.
You need to think of everything that you do with a health care professional as an intervention
There are a number of models/frameworks/strategies for examining education or training interventions. The most popular, although very much debated, was that created by Kirkpatrick. Its really important to recognise this was not a healthcare model although the obvious face validity led it to become very popular. It is still used in systematic reviews of educational initiatives to this day.
Its pretty simple. Its starts with determining a participants reaction to a training programme.
A then asks for a demonstration of learning.
Followed by a test of whether there has been a change of behaviour and finally
The results, or more precisely the return-on-investment should be described. The term kirkpatrick ‘levels’ are commonly used but to be fair to Donald Kirkpatrick he didn’t use the term levels in his original paper. However the concept of a hierarchy has become implicit in the use of the system.
Since Kirkpatrick many people have made modifications to make it more healthcare centric. Most retain a stepwise approach and often don’t really deviate from the underlying premise suggested.
One of the divergent schematas was proposed by Hakkennes 2006 where the level structure is taken out and example measures are given given the system a bit more weight. I’d like to take a pause here to consider a vital aspect of educational evaluation my second take home point. Does it matter if the content if crap is the outcome is good? Ross Fisher a paediatric surgical consultant with a strong interest in presentation skills would look at this slide and weep. Is this a good slide? Not really – are you honestly taking anything home from this other than what I have said? This highlights a real issue in the challenge of delivery and content. It is almost certain that the two previous speakers lectures will be rated highly – is that because of the speaker, their material, a bit of both. I’m pretty sure some of the speakers could talk to absolute rubbish and still be educational. Natalie May is going to give a talk with no text at all but having seen her present I am sure that the learning from that lecture will be fantastic. The idea of learning styles is becoming increasingly discredited so I am very glad Victoria gave such an expansive appraisal of the types of learning.
This very thought prompted Sarah Yardley 2012 to examine a number of educational papers which utlised a outcome focused modal and discredit them. Just concentrating on outcomes was in her opinion akin to performing research on clinical drugs and not assessing potential side effects.
Yardley wants us to move from a situation of did a specific outcome occur.
To what ere the outcomes of this intervention?
This emphasises why I thinking about education as intervention is important. Importantly it helps to determine what you don’t think is education (find dave davis once said some asked himi: if you don’t think lectures are effective why do you lecture so much?
And we know this is really important. There is evidence that when problem based learning is was introduced patient centred outcomes such as clinical performance increased but outcomes in more traditional outcomes fell (Philip Davies 2000)
Statistical Conclusion Validity is the extent to whether the statistical analysis can correctly identify a change if it is present
Internal Validity is the extent to which the intervention can reliably be ascribed to have effected the change
External Validity is the extent to which the intervention is reliable in different populations (this may in terms of participants, settings, times etc)
Construct Validity relates to the association between the concept you are investigating and the measures you use to test it
(This does NOT mean the concept of the intervention is what has actually delivered the change or that the outcome measured is relevant to the intervention)
Medical Education research often has difficulty in comparing two exactly matched populations which differ only by case and control (as the environment of study often mixes participants). Demonstrating external validity may mean different power calculations are needed for a different setting.
The need to ensure adequate pre-testing (but avoid confounding by that very testing improving performance) often results in sample sizes which are difficult to obtain
Concepts in medical education are often based on research which may not have had adequate internal validity to begin with. Therefore demonstrating adequate construct validity (which requires a conceptual framework) while maintaining the integrity of internal validity can be difficult.
Concepts underpinning construct validity may only be relevant in certain populations making subsequent external validation difficult
Half of respondents were actually unable to access the videos in their place of work.