Digital Transformation in the PLM domain - distrib.pdf
Evidence-Based Management presentation
1. Evidence-Based Management
Helping Managers to Make Better Decisions
Denise M. Rousseau
H.J. Heinz II University Professor of Organizational Behavior
Carnegie Mellon University
Pittsburgh, PA USA
3. - Link to presentations
- Teaching material
- Checklists, tests
- Chapters EBMgt handbook
- Answer all your questions
4. What is Evidence-Based Management?
EBMgt is the practice of making organizational
decisions based upon conscientious use of
1. Science-based principles & knowledge
2. Valid & relevant organizational and business facts
3. Critical thinking aided by decision supports
4. Ethical considerations (i.e., effects on stakeholders)
5. Why Should We Care about EBMgt?
Better Decisions by Using Practices that Work (and
avoiding those that don’t!)
Defensible Decisions that Stand Up to Scrutiny (using
best evidence and best process)
Developing Expertise throughout a Career
(experience can be a poor teacher--bad habits!)
20 years of valid experience is different than 1 year of experience
repeated 20 times!
8. 1. Use of Scientific Findings
Chesley Sullenberger, USAIR pilot, has been a
visiting scholar at UC Berkeley’s Collaborative
for Catastrophic Risk Management since 2007
Does research on how to make decisions to
maintain safety despite technological
complexity and crisis conditions
9. 2. Reliance on Reliable and Valid
Organizational Facts
Has written and analyzed aviation accident
reports for over 20 years
10. 3. Mindful Decision Making:
Becoming Decision Aware
Used Decision Aids to Support Good Decision: As Sully
considered what decision to make that day, he had his
copilot review and follow all checklists on board
relevant to crash landings
Formal Education to Prime His Skills: Sully is a graduate
of the U.S. Air Force Academy and holds masters
degrees from both Purdue University in Industrial
Psychology and the University of Northern Colorado in
Public Administration
11. 4. Ethics and Responsibility to
Stakeholders
The last person to leave the plane, Chesley
Sullenberger twice walked the plane’s aisle to
check all passengers were off
Sully’s last act onboard was to grab the
passenger list. Used on-shore to verify rescue
of all passengers and crew
12. In Sullenberger’s Own Words…
“One way of looking at this might be that
for 42 years, I've been making small,
regular deposits in this bank of
experience, education and training.
And on January 15, the balance was sufficient
so that I could make a very large withdrawal.”
14. EBMgt is a means to
improve decision quality.
It’s a career, not a course.
15. The Zeitgeist
Evidence-based practice movements abound in
medicine, education, and public policy
Management research from
psychology, engineering, operations research (ETC.)
yields 1000s of studies annually
Internet (scholar.google.com) gives ready access
Innovative companies now hiring “chief evidence
officers”
Public demands accountability (quality decisions that
are defensible)
16. EBMgt Overcomes Limits of Unaided Decisions
Bounded Rationality Critical Thinking
The Small Numbers Decision Supports
Problem of Individual
Experience Research
• Large Ns > individual
Prone to See Patterns experience
Even in Random Data • Controls reduce bias
The “Human” Problem Evidence-Based Practice
17. Five Good EBMgt Habits
1. Get evidence into the conversation
2. Acquire evidence in your area of Practice
3. Learn to find evidence
4. Become “Decision Aware”
5. Reflect on decisions’ ethical implications
18. Five Good EBMgt Habits
#1 Get evidence into the conversation
Regularly ask “what’s the evidence…?”
Illustration- Discuss with your seatmates…
What’s a practice in your organization that you suspect might
not be NOT evidence-based?
19. Five Good EBMgt Habits
#2 Acquire Evidence in Your
Area of Practice
Focus on Action Principles Where Science is Clear
(On-going Practice of A Professional Manager)
Focus on Business Facts based on valid metrics
relevant to your decisions
(On-going Practice of Responsible & Transparent
Organization)
20. What is evidence?
Evidence is not the same as „proof‟ or „hard facts‟
... can be
- so strong that no one doubts its correctness, or
- so weak that it is hardly convincing at all
21. Don’t confuse
Evidence of effect (do!)
Evidence of no effect (don‟t!)
No evidence of effect (research!)
22. Five Good EBMgt Habits
#2a Learn Use of Scientific Evidence
Focus on Action Principles Where Science is Clear
Rely on Science-based Sources
Example: Locke’s Handbook of Organizational Behavior
(access electronic copy for free)
Peer-reviewed research, especially meta-analyses
Reduce dysfunctional variations in practice
Build effective routines, procedures, checklists
23. Five Good EBMgt Habits
#2a Learn Use of Scientific Evidence
Best Scientific Evidence is
based on large N (sample size of
people/organizations)
well-controlled studies with comparison groups
&/or longitudinal data
peer-reviewed
25. Five Good EBMgt Habits
#2b Use Reliable and Valid Business Facts
Best Business Facts are
large numbers sampled relative to population
(not single or isolated cases, e.g. sales/# sales calls)
linked to context (season, location, #users, etc.)
provide key indicators for business decisions
26. Five Good EBMgt Habits
Illustration--Discuss with your seatmates…
What indicators does your organization most
commonly use to make important decisions?
Are these the “best business facts” you need to
make these decisions?
What indicators would be more useful, if you
could get them??
27. Help Learner How to Interpret Business Facts
# Medication errors in Unit 1 were 200% greater in 2011 than Unit 2’s. Is patient
safety worse in Unit 1? Depends on number of unsafe incidents divided by #
patients or # procedures—needs a control.
Mike has w/10 subordinates & 20% turnover while Kim has 55 employees & 10%
turnover. Is retention better in one? Hard to determine. Small N’s have greater
bias and are more variable.
McDonald’s stores average 300+% turnover/year. Does Mickey D. have a problem?
Depends on industry comparison and business strategy.
Company A managers focus decisions on monthly cost, downtime and revenues.
Company B managers focus on service quality, employee retention and
profitability by customer category. So what? B’s more diverse performance
criteria can promote attention to longer-term and growth-oriented outcomes.
A’s narrower economic focus can promote shorter-term thinking.
28. Five Good EBMgt Habits
#3 Learn to Gather Evidence
Structure and pose a managerial question
Search for best available evidence (check out Google
Scholar or CEBMa website)
Critically appraise information found
Apply relevant case information to decision
Write down the decision made, assumptions, and
expected outcomes
Evaluate outcomes over time
29. Five Good EBMgt Habits
Gathering Evidence is a 5-step approach
1. Formulate an answerable question (PICOC)
2. Search for the best available evidence
3. Critical appraise the quality of the found evidence
4. Integrate the evidence with managerial expertise
and organizational concerns and apply
5. Monitor and evaluate the results
30. Getting the (Scientific) Evidence
What kind of evidence are we looking for?
Studies with a design that best answers your
question
Studies with the highest level of evidence
33. Levels of internal validity
It is shown that …
It is likely that …
There are signs
that …
Experts are of the
opinion that …
34. Practice Searching for Evidence
Using “Google Scholar”
Learning through play !
Try all buttons
Make lots of mistakes
Have fun!
Just do it!
35. Five Good EBMgt Habits
#4 Become “Decision Aware”
Identify different kinds of decisions you face.
What kinds of approaches are used to them?
Why?
How can you determine whether you made a
“good decision” when you cannot know the
outcome? (The answer to this question is what is
known as “decision quality”)
36. Five Good EBMgt Habits
Awareness Calls Attention to Decision Process.
Proper Processes Improve Decision Quality
What is the process for making the decision?
Different processes work better…
- for routine decisions (create validated checklists and action plans)
- for decisions with known unknowns (systematic sequence of
considerations)
- for decisions with unknown unknowns (pilot-tests and trial/
experiment)
Decisions have an “aftermath” and a “pre-math” that a
good manager actively manages. Is the decision well-
managed? Help make it so.
38. Five Good EBMgt Habits
#5 Reflect on Decision’s Ethical
Implications
Who are stakeholders for this decision?
Possible effects?
How might the decision be altered to optimize positive
stakeholder effects and reduce negative?
39. Resources
www.cebma.org
For teaching support
For access to journals
For help desk in support of evidence-searches
denise@cmu.edu
moral support and pointer knowledge!
40. Turning evidenceEvidence into Practice
Turning into practice
& Practice into Evidence
Evidence based management:
closing the gap between research and practice
41. Turning evidence into practice
Got Evidence? References
J. Ehrlinger, K. Johnson, M. Banner, D. Dunning, J. Kruger. (2008) Why the unskilled are unaware: Further
explorations of (absent) self-insight among the incompetent. Organizational Behavior and Human Decision
Processes, 105,(1) pg. 98
E.A. Locke (ed.), Handbook of Principles of Organizational Behavior, 2nd edition, 2009. Malden, MA: Blackwell.
D. M. Rousseau (2012) Oxford Handbook of Evidence-Based Management, New York.
D.M. Rousseau, D.M. & E. Barends (2011) Becoming an evidence-based manager. Human Resource
Management Journal, 21, 221-235.
D.M. Rousseau, J. Manning & D. Denyer (2008) Evidence in Management and Organizational Science:
Assembling the field’s full weight of scientific knowledge through reflective reviews. Annals of the
Academy of Management, 2, 475-515.
R.C. Schank, D. Llyras & E. Soloway (2010) The future of decision making. New York: Palmgrave Macmillan.
J.F. Yates. (2003). Decision management. San Francisco: Jossey-Bass.
J.F. Yates & M.D. Tschirhart (2006). Decision making expertise. In K. A. Ericsson, N. Charness, P. J. Feltovich, &
R. R. Hoffman. (Eds.). Cambridge handbook of expertise and expert performance (pp. 421-438). New York:
Cambridge University Press.
J.F. Yates, E.S. Veinott & A.L. Patalano (2003). Hard decisions, bad decisions: On decision quality and decision
aiding. In S. L. Schneider & J. C. Shanteau (Eds.), Emerging perspectives on judgment and decision research
(pp. 13-63). New York: Cambridge University Press.
44. CAT: Critically Appraised Topic
A Critically Appraised Topic (CAT) is a structured, short (3
pages max) summary of evidence on a topic of
interest, usually focused around a practical problem or
question. A CAT is like a “quick and dirty” version of a
systematic review, summarizing the best available research
evidence on a topic. Usually more than one study is included
in a CAT.
Examples: http://www.cebma.org/presentations/
47. Asking the Right Question?
Does team-building work?
Does leadership development training work?
Does management development improve the
performance of managers?
Does employee participation prevent
resistance to change?
Is 360 degree feedback effective?
48. Answerable Question: PICOC
P = Population
I = Intervention (or success factor)
C = Comparison
O = Outcome / Objectives
C = Context
49. Answerable Question: PICOC
Scenario: You are a consultant, your client is an
insurance company, there are plans for a merger, you
have heard that the other company has a different
culture, you want to know if this will effect the outcome
P = Organizations with a different corporate culture
I = Merger
C = Organizations with a similar corporate culture
O = Long term profitability
C = Profit organizations, competitive market
50. Search terms
Operationalise your Pico elements!
O = long term profitability?
Share holder value? Return on investment? Return on
assets? EBIT? Employee productivity? Profit margin?
Competitive position? Corporate image? Innovation
power? Market share? Customer satisfaction?
51. Searching Evidence
The problem with finding evidence:
the abundance of literature
53. Searching Evidence
Evidence-based Searching
In a systematic and transparent way
searching for the “best” evidence
Part of EBMgt where decision maker is not a
‘subject matter expert’
54. Searching Evidence
What kind of evidence are we looking for?
1. Studies with a design that best suits the
research question
2. Studies with the highest level of evidence
59. Search Strategy
Why do we need a search strategy?
Promotes deeper learning about your
question
Leads to better yield of quality research.
Saves time in the long run.
60. Search strategy
Two search strategies
(Don’t forget to use pointer knowledge along the way)
Snowball method Building blocks method
61. Snowball method
Starting from one book or article, you search
for other literature on the same topic.
Snowballing to older publications by finding out which
publications were used by the author (see bibliography of
book or article).
Snowballing to more recent publications by finding out
how often that book or article has been
cited by other authors (see Web of
Knowledge or Google Scholar).
62. Building blocks method
Keyword 1 Keyword 2 Keyword 3 Keyword 4
Synonyms or Synonyms or Synonyms or Synonyms or
related terms related terms related terms related terms
• …. • …. • …. • ….
• …. OR • …. OR • …. OR • ….
• …. • …. • …. • ….
• …. • …. • …. • ….
AND AND AND
63. Search terms
Example: You are a consultant, your client is an
insurance company, there are plans for a merger, you
have heard that the other company has a different
culture, you want to know if this will effect the outcome
P = Organizations with a different corporate culture
I = Merger
C = Organizations with a similar corporate culture
O = Long term profitability
C = Profit organizations, competitive market
64. Select Keywords
P = Organizations with a different corporate culture #2
I = Merger #1
C = Organizations with a similar corporate culture
O = Long term profitability #3
C = Profit organizations, competitive market
1. Underline the most important keywords
2. Number the order of importance
65. Select keywords
The keywords of your search query may be enough.
If not, select more words by using:
synonyms
alternate spelling, translations
related terms / words / subjects
narrower or broader terms
corporate culture: organizational behavior/character, corporate identity
merger: acquisition, take-over, fusion, combination, unification
profitability: profit, advantage, return on investment, shareholder value