Presentation VOV Lerend Netwerk, Gent, Belgium, Sept 2012 Title: One or Two Things About Evidence-Based Management

1. Postgraduate Course
One or two things about
Evidence-Based Management
Pitstop Evidence-Based HR, VOV lerend netwerk,
Gent, 27 september 2012

2. Postgraduate Course
Evidence-Based Management?
VOV leden, n= 86

3. Evidence-Based Practice
Postgraduate Course
1991 Medicine
1998 Education
1999 Social care, public policy
2000 Nursing
2000 Criminal justice
???? Management?

4. Definition
Postgraduate Course
Evidence-based management means making decisions
about the management of employees, teams or
organizations through the conscientious, explicit and
judicious use of four sources of information:
1. The best available scientific evidence
2. Organizational facts, metrics and characteristics
3. Stakeholders’ values and concerns
4. Practitioner expertise and judgment

5. Four sources
Postgraduate Course

6. Four sources
Postgraduate Course

7. Postgraduate Course
Trust me: 20 years of management experience!

8. Bounded rationality
Postgraduate Course

9. Bounded rationality
Postgraduate Course

10. Het feilbare brein
Postgraduate Course
 Seeing order in randomness
 Mental corner cutting
 Misinterpretation of incomplete data
 Halo effect
 False consensus effect
 Group think  Confirmation bias
 Self serving bias  Authority bias
 Sunk cost fallacy  Small numbers fallacy
 Cognitive dissonance reduction  In-group bias
 Recall bias
 Anchoring bias
 Inaccurate covariation detection
 Distortions due to plausibility

11. Postgraduate Course
if you’re
breathe into a bag
hyperventilating

12. Postgraduate Course
elderly people who have give them a drug that
an irregular heartbeat are reduces the
much more likely to die of number of
coronary disease irregular beats

13. How 40,000 cardiologists can be wrong
Postgraduate Course
In the early 1980s newly introduced anti-
arrhythmic drugs were found to be highly
successful at suppressing arrhythmias.
Not until a RCT was performed was it realized
that, although these drugs suppressed
arrhythmias, they actually increased mortality.
The CAST trial revealed Excess mortality of
56/1000.
By the time the results of this trial were
published, at least 100,000 such patients had
been taking these drugs.

14. Errors and Biases of Human Judgment
Postgraduate Course
Doctors and managers hold many erroneous beliefs,
not because they are ignorant or stupid, but because
they seem to be the most sensible conclusion
consistent with their own professional experience!
available evidence.

15. Problem II: false information
Postgraduate Course
 Half of what you learn will be shown to be either dead
wrong or out-of-date within 7 years of your graduation;
the trouble is that nobody can tell you which half
 Sackett: remember that your teachers are full of crap,
just like your parents.

16. Evidence-based?
Postgraduate Course
1. Incompetent people benefit more from feedback than
highly competent people.
2. Task conflict improves work group performance while
relational conflict harms it.
3. Being intelligent is a disadvantage for performing low
skilled jobs.

17. Evidence-based?
Postgraduate Course
 Competentie management
 Excellente zorg
 Kwaliteits management
 Het nieuwe werken
 Kennis management
 Magnet, Investors in People
 Plain Tree, Healing Environment
 Balanced Score Card / INK
 Lean / Six Sigma / TOC

18. Postgraduate Course
Don’t be a parrot!
Think critically about experience,
question your assumptions, and
challenge what you think you know.
(Show me the evidence!)

19. Postgraduate Course

20. 5-step approach
Postgraduate Course
1. Formulate a focused question
2. Searching for the best available evidence
3. Critical appraisal
4. Turning evidence into practice
5. Monitor the outcome

21. Postgraduate Course
Step 3: Critical appraisal of studies

22. Postgraduate Course
Best available evidence?

23. Research designs
Postgraduate Course
What is the BEST car?

24. Research designs
Postgraduate Course
Which design for which question?

25. Twee type vragen
Postgraduate Course
Effect vs Non-effect

26. Type vraag: effect
Postgraduate Course
Werkt het?
Werkt het beter dan ....?
Effect Heeft het een effect op ....?
Wat zijn de succesfactoren voor ....?
Wat is nodig om het te laten werken?

27. Type: non-effect
Postgraduate Course
Needs: Wat hebben mensen nodig, waar hebben ze
behoefte aan?
Attitude: Wat denken of vinden mensen van ...?
Experience: Wat zijn de ervaringen van mensen met ...?
Prevalence: Hoeveel mensen / organisaties ....?
Procedure: Hoe kunnen we .... implementeren?
Explanation: Waarom werkt het?
Economics: Hoeveel kost het (tijd en geld)?

28. Which design for which question?
Postgraduate Course
Explanation

29. Postgraduate Course
Best research design?

30. Best available?
Postgraduate Course

31. What is evidence?
Postgraduate Course
 Evidence is not the same as ‘proof’ or ‘hard facts’
 Evidence can be
- so strong that no one doubts its correctness, or
- so weak that it is hardly convincing at all

32. Postgraduate Course
The best available evidence =
Studies with the highest internal validity
Studies with the highest external validity

33. Internal validity
Postgraduate Course
internal validity = indicates to what extent the
results of the research may be biased and is
thus a comment on the degree to which
alternative explanations for the outcome found
are possible (confounding).

34. Confounding
Postgraduate Course
Confounding is the idea that a 3rd variable can distort or
confuse (or confound..) a relationship between two other
variables. For instance, when factor X causes disease Y,
that relationship could be confounded by factor C that is
associated with both factor X and disease Y. C would be
an alternative explanation for the relationship observed
between X and Y.

35. What are the confounders?
Postgraduate Course
1. Shoe size & quality of handwriting
2. Body length & body weight
3. Number of storks & birth rate
4. Smoking youngsters & better lung function

36. Reverse causation
Postgraduate Course
?
Charismatic leaders
Successful companies

37. Levels of internal validity
Postgraduate Course
1. Were there enough subjects in the study?
2. Was a control group used?
3. Were the subjects randomly assigned?
4. Was a pretest used?
5. Was the study started prior to the intervention or event?
6. Was the outcome measured in an objective and reliable way?
6x yes = very high (A)
5x yes = high (A)
4-3x yes = limited (B)
2x yes = low (C)
1-0x yes = very low (D)

38. Levels of internal validity
Postgraduate Course

39. Levels of internal validity
Postgraduate Course
It is shown that …
It is likely that …
There are signs
that …
Experts are of the
opinion that …

40. Postgraduate Course
Best available evidence:
external validity

41. External validity: generalizability
Postgraduate Course
Always ask yourself to what extent the evidence
is generalizable to your situation:
Ecological validity: Is your organization so different from
those in the study that its results may be difficult to apply?
Population validity: Is your population so different from
those in the study that its results may be difficult to apply?

42. Generalizability
Postgraduate Course
Keep in mind:
What works in one narrowly defined setting
might not work in another,
but some psychological principles
are generalizable to all human beings.

43. Postgraduate Course
Step 4:
Turning evidence into practice

44. Postgraduate Course

45. Feasible?
Postgraduate Course
 organizational facts and characteristics
 cultural aspects
 stakeholders’ values and concerns
 political aspects
 financial aspects /cost-effectiveness / ROI
 priorities
 change readiness / resistance to change
 implementation capacity
 timing

46. Tot slot:
Postgraduate Course
Focus on the decision making process!
(not the outcome)