Healthcare leaders often make bad decisions due to a lack of statistical understanding. This session will remind attendees that simple comparisons of two data points or comparisons to goals and targets can be misleading. Control charts allow us to better validate project success and make better ongoing management decisions.
It’s far too easy for improvement facilitators to draw incorrect conclusions about the success of their Lean event or Six Sigma project if they are simply comparing before and after performance. Likewise, healthcare leaders make bad decisions when they are likewise comparing two data points (today versus a previous month or year or today versus a target).
Basic Statistical Process Control (SPC) methods, like control charts, are a simple and proven alternative.
Key Learning Objectives
1) Understand some of the common pitfalls in the creation and use of performance measures in various healthcare settings
2) See statistical chart analysis methods that allow for the best management decision making, such as knowing if we are improving and if a "bad day" requires investigation or if it is merely "noise" in the system's performance
3) Connect key principles of Lean management and the Deming philosophy into modern KPI and metrics management
By the end of this session attendees will
1) Understand the importance of "control charts" for management decision making
2) Be able to create and interpret a basic management control chart
3) Know of other resources for more learning
Mark Graban is author of the Shingo-Award winning book "Lean Hospitals: Improving Quality, Patient Safety, and Employee Engagement." Mark is also co-author, with Joe Swartz, of "Healthcare Kaizen: Engaging Front-Line Staff in Sustainable Continuous Improvements" (also a Shingo recipient) and "The Executive Guide to Healthcare Kaizen."
He serves as a consultant to healthcare organizations through his company, Constancy, Inc and is also the Chief Improvement Officer of the technology company KaiNexus.
Mark has a B.S. in Industrial Engineering from Northwestern University and an M.S. in Mechanical Engineering and an M.B.A. from the Massachusetts Institute of Technology’s Leaders for Global Operations Program. Mark and his wife live in San Antonio, Texas.
Mark Graban SHS 2014: Two Data Points Are Not a Trend: Using SPC to Manage Better
1. #shs2014
@MarkGraban
Two Data Points Are Not a Trend:
Using SPC to Manage Better
Mark Graban
VP of Innovation & Improvement Services, KaiNexus
Slides & Audio: www.MarkGraban.com/SHS2014
2. Key Management Questions
• How are we performing?
– Are we getting better or worse?
• What action should we take?
Some rights reserved by Marco Bellucci
3. Or Not Take Action
“Management must understand the
theory of variation: If you don’t
understand variation and how it comes
from the system itself, you can only
react to every figure.
The result is you often overcompensate,
when it would have been better to just
leave things alone.”
W. Edwards Deming
4. My Most Favorite Book Ever
Donald J. Wheeler, PhD
http://www.spcpress.com/
Amazon: http://bit.ly/wheeler-book
6. Comparisons of 2 Data Points
Fatalities per 100 Million Vehicle
Miles Traveled (U.S. & CT)
2
U.S.
1.5
1
CT
0.5
0
1992
1995
1998
2001
2004
2007
2010
7. Need to Look for Trends
“You don't want to make
a big conclusion based on
just one year.”
– Jonathan Adkins of the Governors
Highway Safety Association
“Office Space”
12. 2 Data Points Lack Context
• Total triage cycle time was reduced by 23 minutes
• Total ED-IP cycle time was reduced by 33 minutes
• Average LOS decreased from 97 to 61 minutes
Source: Case study, “Harris Methodist saves $648,695 through SIPOC process changes”
13. What About the Other Data Points?
“The average patient satisfaction
increased from 87.2 to 89%”
17. Applications of SPC Charts
•
•
•
•
•
•
Monthly employee attrition %
Daily % of patients discharged before 11 am
Daily lab test turnaround times
Weekly patient satisfaction scores
Daily % of on-time case starts
Daily % of patients who arrive late / no-show
19. Deming’s 7 Concepts of Variation
1. All variation is caused – specific reasons
2. There are 4 types of causes:
1.
2.
3.
4.
Common causes
Special causes
Tampering
Structural
3. Managers must distinguish amongst these
– Each one requires different managerial actions
20. Deming’s 7 Concepts of Variation
4. For special causes, get timely data
5. For common causes, all data are relevant
– In-depth knowledge of the process being
improved is needed – statistics, flow
charts, Pareto, stratification analysis, DOE
6. When all variation is common cause, the
system is said to be “stable” and “predictable”
7. SPC limits let a manager predict future
performance with some confidence
21. Responding to Daily Changes?
180
160
140
120
100
80
60
40
20
0
Daily Length of Stay Average
KB
KB
Kick
Butt
PT
Praise
Team
PT
Are we helping? Is this process stable?
3/1/12
3/2/12
3/3/12
3/4/12
3/5/12
3/6/12
3/7/12
3/8/12
3/9/12
3/10/12
3/11/12
3/12/12
22. So we should do
nothing?
“Don’t just do something,
stand there.” -- Deming
24. “Western Electric” Rules (1956)
• 8 consecutive points on same side of mean
• 6 consecutive points moving same direction
• 14 alternating up/down points in a row
• Any single point above or below 3-sigma LCL or UCL
– Full rules http://bit.ly/WErules
25. Step 1: Initial Data – LeanBlog.org
• Generally need 20 data
points to calculate
control limits
26. Step 2: Mean & MRs
• Calculate mean of the
first 20 points
• Calculate the moving
range of the first 20
points
– Ex: =ABS(E5-E4)
28. Step 4: Add Control Limits
• Calculate “MR-bar”
– Average of the 1st 19
MRs
• Calculate Control Limits
– LCL = Mean – 3*(MR bar)/1.126
– UCL = Mean + 3*(MR bar)/1.126
34. Testing for Process Shifts
• If you made a change that you expected to improve the
system, use a control chart to test the hypothesis
Daily TAT
35
35
30
30
Process Shift
25
25
20
20
15
15
10
10
5
5
52 52
49 49
46 46
43 43
40 40
37 37
34 34
31 31
28 28
25 25
22 22
19 19
16 16
7
7
13 13
4
4
0
10 10
1
1
0
36. NOT Understanding Variation Leads To…
• Pressuring people to get better results by
working harder within the same system
• Wasting time looking for explanations of a
perceived trend when nothing has
changed
• Taking other actions when it would have
been better to do nothing
• Not focusing on systemic improvements
37. Isn’t it always the system?
It’s (almost) always the system.
38. Q&A / Contact Info
• President, Constancy, Inc.
– www.constancy.us
• VP of Professional Services, KaiNexus
– www.KaiNexus.com
• Founder, LeanBlog.org
– mark@leanblog.org
• Twitter @MarkGraban
• Books: www.MarkGraban.com
40. The Funnel Experiment
• Lloyd Nelson, 1987
– Suspend a funnel on a stand a
few inches off the ground
– Drop 50 marbles
x
41. A “Stable” System
• Does NOT mean:
– Zero variability
– System meets customer
requirements
• It only means:
– Causes of variation are basically constant over time
42. We Have to Try Harder!!!
• 4 different rules for adjusting the funnel
No adjustment
Adjust relative
to last position
Adjust relative
to center
Learn more – online simulator at http://www.symphonytech.com/dfunnel.htm
Notas del editor
Don’t turn off your phones and computers…. Tweet and shareI will post a link to an audio recording of my talk at www.MarkGraban.com/SHS2014 by March 1, 2014
Question from Lloyd S.Nelson, who worked with Deming -- This is a trick question!
Story from a reader of my blog… a story that illustrates this point perfectly. Management wastes too much time chasing every up and down (or wastes the time of people who are expected to give an “explanation” for each data point. Reacting to every data point usually INCREASES variation in a process and its results.
This book is so good, you should go online right now, download the Kindle version, leave my talk and spend 50 minutes reading it
The slide title is a Wheeler quote from the book – data must have context to be meaningful. It must be understandable. It must be more than just a comparison between actual and target.What is a “quality panel”? Keep in mind this is supposed to be a public metric in the LOBBY for visitors to see. What is the scale? What is a good score? What’s a good score compared to others? Why is the target what it is? The target is SUSPICIOUSLY close to the actual. Comparing a number to a target provides very little context… so does comparing a number to last year or last quarter… as we often see in the news.
Test scores are down, teen smoking rates are up… Big changes aren't necessarily a signal -- Small changes aren't just noise. 2.9% decline might be somewhat trivial (but good), while a 42% increase in CT *might* be statistically significant. Each state’s “JUMP” could be statistical noise. Look at the chart (made from NHTSA data) – are our roads safer (chart) or more dangerous (headline). A 42% increase in CT – what’s going on there?
Adkins is completely right… we can’t make big conclusions based on two data points. We don’t want to “jump to conclusions” (ala the mat from Office Space)
Two data points are used all the time in the news, especially politics. At least this context shows a “margin of error.” How is that same concept useful in our use of data at work? It’s weird that the time scale goes right to left. What looks like an increase from 43 to 47% disapproval, could statistically be a drop from 46% to 44%... Therefore, the media and politicians should not overreact to every up and down in the data.
This is what’s jokingly called an “Executive time series” with just two data points. -- Take into account noise in the system and common cause variation the picture is much less clear
It’s hard to know from two data points if things are statistically better… Case study link http://www.caldwellbutler.com/index.php/case-studies/harris-methodist-southwest
Here is the more complete chart from the case study… Consulting firm case study from a hospital… the early and late data point comparison…. Or you can try the linear trend. But by SPC standards, this is a stable process… not statistically valid improvement
The more complete method should really include TWO paired charts. We’re looking for a signal in the actual data AND unusual variation from period to period (the bottom MR chart)To create the MR chart, we calculate the “MR bar” – the average of the MR data points (the first 19 MRs if we used 20 data points for the limit calculations). MR-bar is the green line on the MR chart on this slide. The UCL (red line) is 2.67 * MR-bar. The LCL is zero (the MR can’t be negative since it’s an absolute value). We apply the same western electric rules to evaluate if the MR chart is in control (like one data point above the UCL, etc.).
Why is the MR chart helpful? It finds signals that might be missed from just using the X chart. Why was there a small spike in our revenue? (this is masked data from a wine bar business)
Driving to work…. How long does it take to get there each day?Read more: http://www.dtic.mil/dtic/tr/fulltext/u2/a238399.pdfStuctural variation = seasonality, for example
“Management is prediction” – DemingIn a stable system, we can’t just exhort people to do better… we have to change the system and that’s management’s responsibility.
“Some managers do not get the concept of variability in a process. This example is similar to one I experienced in a hospital. During a meeting of the board, a consulting heart surgeon was presenting data on AMI occurrence. The data showed a normal variation over several months, with an aggregate trend downward. Several members of the board, including the CEO, COO, and the hospital’s process expert voiced concerns that one month’s values were above the average then went down in the following month. This pattern repeated itself, and the individuals wanted to know why all the months did not show a value below the average. The surgeon was well versed in the principles of statistical process control, and he attempted to explain as did I. Alas, to no avail..”
Here’s a basic control chart that shows a stable process. This means the future data points are predictable (within 23 and 53 or so). The mean (average) is about 38 in this chart.You could use a chart like this for any time of management data…
These are the most basic rules that would indicate you have something other than common cause variation (or noise). Any one of these rules helps detect a special cause signal.
This is real data (weekday data) from my website, leanblog.org… if you have more than 20 data points, you could use the first 20-25 to calculate control limits. This approach, because of the data required to calculate limits, works better for daily or weekly data unless you have a lot of historical data.
In a stable system, we wouldn’t have any data points outside the control limits… that’s the most straightforward case. But, what if we DO have a data point outside the control limit(s)?The simplest rule to look at first is to see if there’s a single data point ABOVE the UCL or BELOW the LCLTopic that people didn’t like? Website down? Stop SOPA DayWe re-calculate limits based on removing that out of control data point
Special cause of Jan 2 – holiday weekend, so that was eliminated. Not the most stable process, but can generally predict daily page loads would be between 1200 and 2200 without there being any special cause January 6 was a particularly popular post about the need for Kaizen in healthcare – slightly above the UCL
The first two data points that jump out are the two ABOVE the UCL – there’s likely a special cause that’s actually worth looking for
Here, we see a group of more than 8 consecutive data points ABOVE the mean – that’s a sign of a special cause
We recalculate new control limits based on the first 20 data points… and continue evaluating to look for special causes
This is how I would test if a client is not only sustaining gains, but also improving
Read about the funnel experiment in Out of the Crisis -- http://books.google.com/books?id=LA15eDlOPgoC&pg=PA327&lpg=PA327&dq=lloyd+nelson+funnel+experiment&source=bl&ots=MsiysUhKJq&sig=GkN-H-tyg5R1MFN2u26Rj4pJjiM&hl=en&sa=X&ei=Dxy_UpE0j-zYBdfMgYAD&ved=0CEQQ6AEwAw#v=onepage&q=lloyd%20nelson%20funnel%20experiment&f=falseSee this online simulator: http://www.symphonytech.com/funnelexp.htm