Quantitative Approaches to
Improve Healthcare Access and Quality, Rocky Mountain INFORMS Chapter Meeting, A panel presentation, featuring the work of:
Linda LaGanga, Ph.D.,Steve Lawrence, Ph.D.,
C.J. McKinney, Ph.D. Candidate1,
Antonio Olmos, Ph.D., Michele Samorani, Ph.D. Candidate
(Mental Health Center of Denver,
University of Colorado-Boulder,
University of Colorado-Denver,
University of Northern Colorado)Thursday, March 17, 2011
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
INFORMS Rocky Mtn Presentation 03-17-11
1. Quantitative Approaches to
Improve Healthcare Access and Quality
Rocky Mountain INFORMS Chapter Meeting
A panel presentation, featuring the work of:
Linda LaGanga, Ph.D.1,3
Steve Lawrence, Ph.D.2
C.J. McKinney, Ph.D. Candidate1,4
Antonio Olmos, Ph.D.1
Michele Samorani, Ph.D. Candidate2
1. Mental Health Center of Denver
2. University of Colorado-Boulder
3. University of Colorado-Denver
4. University of Northern Colorado
1
Rocky Mountain INFORMS, March 17, 2011
2. Healthcare Issues we address
To overbook or not?
If we schedule them, will they come?
What would Deming do to improve
healthcare?
To achieve efficiency and effectiveness
of healthcare
2
Rocky Mountain INFORMS, March 17, 2011
3. Where is our work developed and
documented?
Experience and data from
the Mental Health Center of Denver
Community mental health center serving over 14,000
people per year
Surveys and interviews of other healthcare
providers/systems
Presented at INFORMS annual conferences
Other conferences:
Production & Operations Management Society
Decision Sciences Institute
Mayo Clinic Conference on OR/Systems Engineering in
Healthcare
American Evaluation Association
3
Rocky Mountain INFORMS, March 17, 2011
4. Read more about it…
Decision Science Journal (May, 2007)
Journal of Operations Management
(2010, in press)
Conference presentations and proceedings at
http://www.outcomesmhcd.com/Pubs.htm
Research posters on the wall
opposite this room
4
Rocky Mountain INFORMS, March 17, 2011
5. Appointment Scheduling and
Overbooking
Clinic Overbooking to Improve Patient Access
and Increase Provider Productivity
LaGanga, L. R., & Lawrence, S. R. (2007).
Clinic overbooking to improve patient access and
provider productivity.
Decision Sciences, 38(2), 251 – 276.
5
Rocky Mountain INFORMS, March 17, 2011
7. Model Assumptions
Number of patients booked, K:
E(K) = SK = N
S = Show rate, N = target n of patients
K = N/S
Patients scheduled at even intervals throughout
the day
T = N/K = S
Inter-appointment times compressed by the show rate
Patients arrive on time with probability S
Patient service times deterministic
Added variability in final version
7
Rocky Mountain INFORMS, March 17, 2011
8. Overbooking: Best Case
10 appointment slots / session; 50% show rate
Regular Time Overtime
Time Slot 1 2 3 4 5 6 7 8 9 10 11 12 13
Start Time 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6
Best Case Expected number of patients (5) arrive, evenly spaced
Arrivals A1 X2 A3 X4 A5 X6 A7 X8 A9 X10
Service D1 D3 D5 D7 D9 No overtime
Waiting No patients wait
5 patients seen; no provider idle time; no patients wait; no clinic overtime
8
Rocky Mountain INFORMS, March 17, 2011
9. Overbooking: Bunched Early
10 appointment slots / session; 50% show rate
Regular Time Overtime
Time Slot 1 2 3 4 5 6 7 8 9 10 11 12 13
Start Time 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6
Case 1 Expected number of patients (5) arrive, bunched early
Arrivals A1 A2 A3 X4 A5 X6 A7 X8 X9 X10
Service D1 D2 D3 D5 D7 No overtime
Waiting W2 W3 W5 W7
5 patients seen; no provider idle time; 4 patients wait; no clinic overtime
9
Rocky Mountain INFORMS, March 17, 2011
10. Overbooking: Late Arrival
10 appointment slots / session; 50% show rate
Regular Time Overtime
Time Slot 1 2 3 4 5 6 7 8 9 10 11 12 13
Start Time 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6
Case 2 Expected number of patients (5) arrive, one late arrival
Arrivals A1 X2 A3 X4 A5 X6 A7 X8 X9 A10
Service D1 D3 D5 D7 I D10 OT
Waiting No patients wait
5 patients seen; 10% provider idle time; no patients wait; 10% clinic overtime
10
Rocky Mountain INFORMS, March 17, 2011
11. Overbooking: Bunched Late
10 appointment slots / session; 50% show rate
Regular Time Overtime
Time Slot 1 2 3 4 5 6 7 8 9 10 11 12 13
Start Time 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6
Case 3 Expected number of patients (5) arrive, bunched late
Arrivals A1 X2 A3 X4 X5 X6 A7 A8 A9 X10
Service D1 D3 I I D7 D8 D9 OT
Waiting W8 W9
5 patients seen; 20% provider idle time; 2 patients waiting; 20% clinic overtime
11
Rocky Mountain INFORMS, March 17, 2011
12. Overbooking: Extra Arrival
10 appointment slots / session; 50% show rate
Regular Time Overtime
Time Slot 1 2 3 4 5 6 7 8 9 10 11 12 13
Start Time 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6
Case 4 More patients arrive (6) than expected (5)
Arrivals A1 A2 A3 X4 A5 X6 A7 X8 A9 X10
Service D1 D2 D3 D5 D7 D9 OT
Waiting W2 W3 W5 W7 W9
6 patients seen; no provider idle time; 5 patients waiting; 20% clinic overtime
12
Rocky Mountain INFORMS, March 17, 2011
14. Overbooking Utility Model
Maximize clinic “utility”
Trade-off
Patient access (number of patients seen)
Average patient waiting times
Expected clinic overtime
Note that provider productivity is implicit in
this model
14
Rocky Mountain INFORMS, March 17, 2011
15. Relative Benefits and Penalties
= Benefit of seeing additional patient
= Penalty for patient waiting
= Penalty for clinic overtime
The values of , , and don’t matter
Just their ratios or relative importance
15
Rocky Mountain INFORMS, March 17, 2011
16. Utility Function
Expected utility without overbooking
U SN
Expected utility with overbooking
U O A W O
Expected net utility with overbooking
U N U O U ( A SN ) W O
16
Rocky Mountain INFORMS, March 17, 2011
17. Utility Function Described
U N ( A SN ) W O
Utility Benefit of
Less Utility Benefit Patient
Less Less Clinic
Patients that
w/o Overbooking Penalty
Waiting Overtime Penalty
“Show”
Net Utility Benefit from Overbooking (could be negative)
17
Rocky Mountain INFORMS, March 17, 2011
18. Simulation Experiments
Five clinic size levels N
N = {10, 20, 30, 40, 50}
Ten show rates S
S = {100%, 90%, … , 10%}
Full factorial experiment
SN = 5 × 100 = 500 factor levels
10,000 replications per factor
500,000 observations
18
Rocky Mountain INFORMS, March 17, 2011
19. Regression Analysis
Results from simulation analyzed using
regression analysis
Regression equations obtained
Expected patient wait times
Expected clinic overtime
Expected provider productivity
All coefficients significant
R2 = 98%+
19
Rocky Mountain INFORMS, March 17, 2011
20. Sensitivity to Service Uncertainty
40
N50R90
30 N30R90
Average Net Utility
N50R50
20 N30R50
N10R90
10 N10R50
N10R10
0 N30R10
0.0 0.2 0.4 0.6 0.8 1.0 N50R10
-10
Service Time Variability
Average of net utility UN with overbooking as a function of
service time variability cs , with and (=1, =0.5, τ =1.2)
20
Rocky Mountain INFORMS, March 17, 2011
21. Conclusions
Overbooking is one solution for
appointment no-shows
Can significantly improve performance
Patient access (more patients seen)
Clinic utility
But with a cost
Increased patient waiting & clinic overtime
Good for some clinics, not for others
21
Rocky Mountain INFORMS, March 17, 2011
22. Directions for Future Work
Scheduling policies
Double booking
Wave scheduling
Optimal overbooking policies
Current overbooking policy is not “optimal”
Dynamic programming
Nonlinear waiting & overtime functions
Long waits much worse than short waits
22
Rocky Mountain INFORMS, March 17, 2011
24. Data Mining in Appointment
Scheduling
Michele Samorani
PhD Candidate
Leeds School of Business, University of Colorado at
Boulder
24
Rocky Mountain INFORMS, March 17, 2011
28. Using Data Mining to Schedule
Appointments
28
Rocky Mountain INFORMS, March 17, 2011
29. Overbooking – Shortcomings
Suppose service time = 30 minutes
1 1 0 1 0 1 0
Little waiting time
1 1
and no overtime
11:20
11:40
12:00
11:00
10:00
10:20
10:40
9:40
9:00
9:20
0 0 1 1 0 1 1 1
Some waiting time
1 11:20 and a high overtime
11:40
12:00
11:00
10:00
10:20
10:40
9:40
9:00
9:20
If we could predict which patients show up and which don’t, we could obtain
a more controllable schedule
29
Rocky Mountain INFORMS, March 17, 2011
30. The method
Every time a visit request arrives:
1)A classifier is used to predict if it shows or not (for each day)
2)The visit request is scheduled by solving a stochastic program through
column generation
Non‐controllable parameters Controllable parameters
•Service time •Number of slots K
•Revenue from seeing a patient •Scheduling horizon h
•Clinic overtime cost •Classification
•Waiting time cost performance:
– Sensitivity (sn)
– Specificity (sp)
How good we are at retrieving showing patients
How good we are at retrieving non‐showing patients 30
Rocky Mountain INFORMS, March 17, 2011
31. Productivity vs Punctuality
Productivity: number of patients seen. It is increased by:
Punctuality: 1/(overtime + waiting time). It is increased by:
31
Rocky Mountain INFORMS, March 17, 2011
32. Real world case: MHCD
Show rate Same day 1 day 2 days 3 days 4 days R
Low .74 .64 .65 .62 .61 .65
MHCD .87 .74 .75 .72 .71 .76
• Goal: Find the best policy for MHCD in terms of:
– Overbooking
– Open Access
– Data Mining
After playing for a few hours with the MHCD data set, I can
achieve any of the following classification performances:
sn = 0.9, sp = 0.5
sn = 0.7, sp = 0.7
sn = 0.6, sp = 0.8
32
Rocky Mountain INFORMS, March 17, 2011
33. Data Mining Open Access .
Policy DM OB OA
∗ ∗
(min) (min)
Overbooking
1 No No No 6.39 0.00 0.00 5.99 8 4
2 No No Yes 6.39 0.00 0.00 5.99 8 1
3 No Yes No 7.10 36.22 20.61 8.37 12 4
4 No Yes Yes 7.22 35.33 21.37 8.40 12 1
.6, .8 No No 6.82 0.00 0.00 6.44 8 5
5 .7, .7 No No 6.99 0.00 0.00 6.62 8 4
.9, .5 No No 7.36 0.00 0.00 7.00 8 5
.6, .8 No Yes 6.84 0.00 0.00 6.44 8 1
6 .7, .7 No Yes 6.83 0.00 0.00 6.43 8 1
.9, .5 No Yes 6.66 0.00 0.00 6.27 8 1
.6, .8 Yes No 7.24 21.11 14.96 7.78 12 3
7 .7, .7 Yes No 7.42 29.33 17.88 8.33 12 5
.9, .5 Yes No 7.58 40.78 23.56 9.03 12 2
.6, .8 Yes Yes 7.35 25.00 15.92 8.03 12 1
8 .7, .7 Yes Yes 7.44 28.44 18.51 8.28 12 1
.9, .5 Yes Yes 7.32 35.22 19.83 8.47 12 1
33
Rocky Mountain INFORMS, March 17, 2011
34. .
Policy DM OB OA
∗ ∗
(min) (min)
1 No No No 7.28 0.00 0.00 6.88 8 4
2 No No Yes 7.27 0.00 0.00 6.87 8 1
3 No Yes No 7.47 29.07 15.32 8.39 10 5
4 No Yes Yes 7.52 28.00 15.62 8.39 10 1
.6, .8 No No 7.49 0.00 0.00 7.11 8 5
5 .7, .7 No No 7.56 0.00 0.00 7.18 8 2
.9, .5 No No 7.85 0.00 0.00 7.47 8 2
.6, .8 No Yes 7.56 0.00 0.00 7.17 8 1
6 .7, .7 No Yes 7.59 0.00 0.00 7.19 8 1
.9, .5 No Yes 7.52 0.00 0.00 7.12 8 1
.6, .8 Yes No 7.60 20.73 13.26 8.14 10 2
7 .7, .7 Yes No 7.65 12.11 8.69 7.83 9 5
.9, .5 Yes No 7.86 15.22 9.81 8.18 9 2
.6, .8 Yes Yes 7.62 21.87 13.83 8.20 10 1
8 .7, .7 Yes Yes 7.64 24.87 14.53 8.36 10 1
.9, .5 Yes Yes 7.57 28.13 15.82 8.44 10 1
34
Rocky Mountain INFORMS, March 17, 2011
35. Conclusions
Data mining can improve appointment scheduling in the
presence of no-shows
If adopted in conjunction with overbooking, data mining can
either increase punctuality or productivity, depending on
sensitivity and specificity
In case of low show rate, the advantage obtained by using
overbooking is similar to the one obtained with data mining.
On the other hand, in case of high show rate, data mining is a
superior technique
Interestingly, if we can achieve a decent classification
performance, using open access is the worst choice
Thank you for your attention. Questions?
35
Rocky Mountain INFORMS, March 17, 2011
36. What about the scheduling horizon h?
h does not have any significant impact by itself:
But its interaction with sn and sp is significant:
36
Rocky Mountain INFORMS, March 17, 2011
38. Driving Clinical Quality
Improvement through Mental Health
Recovery Control Charts
INFORMS Annual Meeting 2009
San Diego, CA
October, 11th, 2009
CJ McKinney, MA*
Antonio Olmos, PhD
Linda Laganga, PhD
Mental Health Center of Denver
Denver, CO, USA
* - Corresponding Author
38
Rocky Mountain INFORMS, March 17, 2011
39. Literature
Olmos-Gallo, P.A. DeRoche, K.K. (2010, August). Monitoring Outcomes
in Mental Health Recovery: The Effect on Programs and Policies.
Advances in Mental Health (9)1, 8-16. http://amh.e-
contentmanagement.com/archives/vol/9/issue/1/ contact P. Antonio
Olmos for a copy of the publication
McKinney, C.J., Olmos-Gallo, P.A. McLean, C., LaGanga, L.R. (August
2010). Driving Clinical Quality Improvement through Mental Health
Recovery Control Charts. Presented at the 3rd Annual Mayo Clinic
Conference on Systems Engineering & Operations Research in Health
Care, Rochester, MN. Awarded First Place for Best Poster Presentation.
Clark, C.R., Olmos-Gallo, P.A. (2007). Performance Measurement: A
signature approach to outcomes measurement improves recovery.
National Council Magazine, 3, 26-28.
Glover, H. (2005). Recovery based service delivery: Are we ready to
transform the words into a paradigm shift? Australian e-Journal for the
Advancement of Mental Health, 4(3),
www.auseinet.com/journal/vol4iss3/glovereditorial.pdf (accessed 15 May
2009)
Montgomery, D. C. (2005) Introduction to Statistical Quality Control, Fifth
Edition. Hoboken, NJ: John Wiley and Sons, Inc.
Olmos-Gallo, P. A., DeRoche, K. K., McKinney, C. J., Starks, R., & Huff,
S. (2009). The Recovery Markers Inventory: Validation of an instrument
to measure factors associated with recovery from mental illness. Working
paper
39
Rocky Mountain INFORMS, March 17, 2011
40. The Heart of Recovery Measurement
40
Rocky Mountain INFORMS, March 17, 2011
41. Act Plan
Continuous
Improvement
Check Do
41
Rocky Mountain INFORMS, March 17, 2011
42. Quality Components in Mental Health Services
Quality Components Relationship to MH Services
How well are MH services working? Are
Performance consumers improving in their recovery?
How often do we see improvements in recovery?
Reliability How consistent are the outcomes across
consumers?
How long does the consumer retain the
Durability recovery-supportive skills and tools taught
through MH services?
How does the consumer perceive our ability to
Perceived Quality
support MH recovery? Community?
Conformance Are we meeting program fidelity standards?
to Standards
42
Rocky Mountain INFORMS, March 17, 2011
43. Quality Control in Mental Health
Allocate and reallocate clinical resources more
efficiently
Improve and maintain clinical program fidelity
Reduce length of treatment, while sustaining same
level of recovery and recovery supportive factors
Increase the number of consumers served, while
decreasing burden on case managers/therapists
Identify most effective programs based upon
consumer needs
43
Rocky Mountain INFORMS, March 17, 2011
44. Mental Health Recovery
Concept of Recovery has taken root
around the world
Working Definition (MHCD):
“A non-linear process of growth by which people move
from lower to higher levels of fulfillment in the areas of
hope, safety, level of symptom interference, social
networks, and activity.”
Federal Grant (SAMHSA) for Transformation to
Recovery-Oriented Mental Health Systems
For information on the Recovery Transformation
Summit, see
http://www.gmhcn.org/files/RRecovery_Newsletter_Fall2010.pdf
44
Rocky Mountain INFORMS, March 17, 2011
45. Mental Health Recovery Outcomes
MHCD has developed 3 consumer specific recovery
outcomes
Consumer Recovery Measure – (Consumer Perspective)
Hope, Safety, Activity, Level of Symptom Management,
Social Networks
Recovery Marker Inventory – (Clinician Perspective)
Housing, Employment, Education, Active Growth,
Participation, and Symptom Management
Recovery Needs Level – (Clinical Algorithm) Provides for
one of 5 levels of treatment based upon clinical criteria
The examples in this presentation will utilize the
Consumer Recovery Measure.
45
Rocky Mountain INFORMS, March 17, 2011
48. Relationship among Recovery Outcomes
(1) Recovery Marker
Inventory (RMI)
(Longitudinal data to support
clinical decision making)
To what degree is
RECOVERY
(4) Recovery happening for
Needs Level consumers at MHCD
(RNL) (Formative and summative
(Appropriate level of services) evaluation of recovery)
(2) Promoting Recovery (3) Consumer Recovery
in Organizations (PRO) Measure (CRM)
(Consumer’s perceptions of how well (Consumer’s perception of their
specific programs and staff are own recovery)
promoting recovery)
48
Rocky Mountain INFORMS, March 17, 2011
49. Consumer Recovery Measure v3.0
The CRM V3.0 includes the 15 items listed below:
2. Lately I feel like I’ve been making important contributions (active-growth)
4. I have hope for the future (hope)
5. I am reaching my goals (active growth)
7. I have this feeling things are going to be just fine (hope)
8. Recently my life has felt meaningful (hope)
9. Recently, I have been motivated to try new things (active-growth)
11. There are some people who cause me a lot of fear (safety)
12. I get a lot of support during the hard times (social network)
14. In most situations, I feel totally safe (safety)
15. My life is often disrupted by my symptoms (symptom interference)
16. Sometimes I’m afraid someone might hurt me (safety)
17. I have people in my life I can really count on (social network)
18. Life’s pressures lead me to lose control (symptom interference)
19. I have friends or family I really like (social network)
20. My symptoms interfere less and less with my life (symptom interference)
21. When my symptoms occur, I am able to manage them without falling
apart (symptom interference)
49
Rocky Mountain INFORMS, March 17, 2011
50. Quality Control Issues in Recovery
Multiple sources of variability
Measurement
Consumer
System
Changing environmental, treatment, and
consumer specific factors affect outcome
measurements.
Difficulty in detection of small changes due to
large variability within and among consumers
50
Rocky Mountain INFORMS, March 17, 2011
51. Multilevel Modeling and Recovery
Multilevel modeling allows for the partitioning of
variance among multiple levels of nesting, i.e.
measures within consumers within therapists
Allows for regression based correction of expected
outcomes for any unit at any level, i.e. conditional
estimates based upon consumer characteristics in
environment or treatment.
Can be used to simultaneously monitor multiple
aspects of the system from measurements to clinical
sites.
Based upon Mixed-Effects ANOVA design
51
Rocky Mountain INFORMS, March 17, 2011
52. Example of Multilevel modeling
concepts
Consumer Level Effect
Typical SLR Model System Level Effect
Intercept
Intercept = +
ACT Tx
Intake = +
Intercept
Mood = +
CRM + Disorder ACT Tx
=
Scores
Intercept
Intercept
= +
Time ACT Tx
= +
in Tx
Intercept
Mood = +
Disorder ACT Tx
Higher Level
Effects 52
Rocky Mountain INFORMS, March 17, 2011
53. Multilevel Regression Corrected Control
Charts
CUSUM for Consumers (between consumer
comparisons)
Allows for determination of a consumer’s
progress as compared to peers in same
treatment with environmental and
demographic similarities
53
Rocky Mountain INFORMS, March 17, 2011
54. Example MRC-CUSUM
Self Comparison
54
Rocky Mountain INFORMS, March 17, 2011
55. Example MRC-CUSUM
Peer Comparison
55
Rocky Mountain INFORMS, March 17, 2011
56. Utilization of MRC-CUSUM
Improved allocation of resources – by
allowing consumer comparison to peers
Identification of factors that may
promote/inhibit recovery
Provide feedback regarding progress and
relapse more quickly to clinicians
56
Rocky Mountain INFORMS, March 17, 2011
57. Multivariate Control Chart
Bivariate Control Chart for plotting of
regression parameters (intercept and slopes)
Corrections may be made based upon
environmental, treatment, and demographic
characteristics
57
Rocky Mountain INFORMS, March 17, 2011
58. I II
III
IV
58 58
Rocky Mountain INFORMS, March 17, 2011
59. Recovery Intercept
BELOW ABOVE
AVG. AVG.
Decreasing Increasing
Recovery Slope
I II
IV III
NOTE: ANY Outlier within a quadrant indicates it is farther away from the
average than would be expected under typical circumstances.
59
Rocky Mountain INFORMS, March 17, 2011
60. Utilization of Bivariate Control Chart
Identify “outlying” consumers to help determine
aspects of a program that promote self-perceived
recovery, and those aspects that may be a deterrent
to improvement in self-perceived recovery.
Allow for identification of consumers who may need
further resources or different treatment.
Allows for overview of consumer progress, where
comparisons over time may allow for evaluation of
process changes and overall consumer effect.
60
Rocky Mountain INFORMS, March 17, 2011
61. Summary of Benefits
Allow for more efficient allocation of treatment and
resources.
Identify program aspects that promote or deter
improvement in self-perceived recovery.
Identify consumer in need of additional treatment or
resources.
Allow for the identification of consumer and system
factors that affect or interact with consumer
outcomes and program effectiveness.
Being able to cater to differing needs of the wide
variety of consumers served.
Identification of Episodes of Care
61
Rocky Mountain INFORMS, March 17, 2011
62. Moving forward in recovery models to drive
quality improvement
Statistical Models
Information
Technology
Knowledge Building
& Dissemination:
Learning Collaboratives
Staff Involvement,Training
62
Rocky Mountain INFORMS, March 17, 2011
63. Future Directions to
Drive Recovery System Improvement
Identify clinically significant patterns
Expand to other recovery measures and aspects.
Coordinate with data mining to identify
relationships between services and recovery
outcomes
Automate quality control process
Integrate fully into clinical quality review processes
Develop accessible reporting and dashboard
systems for clinicians and managers
63
Rocky Mountain INFORMS, March 17, 2011
64. More information
If you would like to see more information
concerning MHCD’s research and work with
Recovery please visit:
http://www.outcomesmhcd.com/
http://www.reachingrecovery.org/
Or contact Christopher.McKinney@mhcd.org
64
Rocky Mountain INFORMS, March 17, 2011
65. Extra slides that were mentioned but not presented
on 3/17/11 due to time limitations
From Mayo Clinic Conference on
Operations Research & Systems Engineering in
Healthcare
Lean Options for Walk-In, Open Access, and Traditional Appointment
Scheduling in Outpatient Health Care Clinics
(LaGanga & Lawrence, 2009)
Includes further development to appointment scheduling models to
include metaheuristic optimization of overbooking levels
Comparison of traditional scheduling, open-access,
and walk-in policies
Lean process improvement to reduce no-shows and expand intake
capacity.
Condensed slide set. See http://www.outcomesmhcd.com/Pubs.htm
for complete, original presentation.
Driving Clinical Quality Improvement Through Mental Health Recovery
Control Charts (McKinney, Olmos, McLean, LaGanga, 2010)
Poster presentation
First Place Award 65
Rocky Mountain INFORMS, March 17, 2011
67. 1. Background on
Appointment Scheduling
67
Rocky Mountain INFORMS, March 17, 2011
68. Motivation
Healthcare Capacity
Funding restrictions
Demand exceeds supply
Serve more people with limited resources
Manufacturing Scheduling
Resource utilization
Maximize throughput
Healthcare Scheduling as the point of
access
Maximize appointment yield
68
Rocky Mountain INFORMS, March 17, 2011
69. 2. Lean Approaches
Rapid Improvement Capacity Expansion (RICE) Team
January, 2008
Article in press, Journal of Operations Management (2010).
Available at http://dx.doi.org/10.1016/j.jom.2010.12.005 69
Rocky Mountain INFORMS, March 17, 2011
70. Lean Approaches
Reducing Waste
Underutilization
Overtime
No-shows
Patient Wait time
Customer Service
Choice
Service Quality
Outcomes
70
Rocky Mountain INFORMS, March 17, 2011
71. Lean Process Improvement in Healthcare
Documented success in hospitals
ThedaCare, Wisconsin
Prairie Lakes, South Dakota
Virginia Mason, Seattle
University of Pittsburgh Medical Center
Denver Health Medical Center
Influences
Toyota Production System
Ritz Carleton
Disney
Hospitals to Outpatient
Clinics run by hospitals
Collaborating outpatient systems
71
Rocky Mountain INFORMS, March 17, 2011
72. Lean Process Improvement: One Year After
Rapid Improvement Capacity Expansion
RICE Results
Analysis of the1,726 intake appointments for the one year before and
the full year after the lean project
27% increase in service capacity
from 703 to 890 kept appointments) to intake new consumers
12% reduction in the no-show rate
from 14% to 2% no-show
Capacity increase of 187 additional people who
were able to access needed services, without increasing staff or other
expenses for these services
93 fewer no-shows for intake appointments during the first full
year of RICE improved operations.
72
Rocky Mountain INFORMS, March 17, 2011
73. Lean Process Improvement:
RICE Project System Transformation
Appointments Scheduled
and No-Show Rates
450 20%
400
Appointments
350 15%
300
250
10%
200
150
100 5%
50
0 0%
Mon Tue Wed Thu Fri Mon Tue Wed Thu Fri
Year Before Year After
Lean Improvement Lean Improvement Appointments
No-Show Rate
73
Rocky Mountain INFORMS, March 17, 2011
74. How was this shift accomplished?
Day of the week: shifted and added
Tuesdays and Thursdays
Welcome call the day before
Transportation and other information
Time lag eliminated
Orientation to Intake Assessment
Group intakes
Overbooking
Flexible capacity
74
Rocky Mountain INFORMS, March 17, 2011
75. Lean Scheduling Challenge
Choice versus Certainty
Variability versus Predictability
Sources of Uncertainty / Variability
No-shows
Service duration
Customer (patients’) Demand
Time is a significant factor
Airline booking models?
75
Rocky Mountain INFORMS, March 17, 2011
76. 3. Response to Overbooking
76
Rocky Mountain INFORMS, March 17, 2011
77. Sample Responses
Campus reporter’s visit to student health
center
“Not now and never will”
Patient waits 15 – 20 minutes
New administration, new interests
Morning News Radio
“Overbooking leading to increased patient
satisfaction? That just doesn’t make any sense!”
Public Radio Interviewer
Benefits of increased access at lower cost
77
Rocky Mountain INFORMS, March 17, 2011
78. Other Responses
Practitioners
Dentists
General medicine
Child advocacy
How should we overbook?
Other options
Lean Approaches
Open Access (Advanced Access)
Walk-ins
78
Rocky Mountain INFORMS, March 17, 2011
79. 4. Enhanced Appointment
Scheduling Model
20%
15%
Probability
10%
5%
0%
0 1 2 3 4 5 6 7 8 9 10 11 12
Number Waiting (k)
79
Rocky Mountain INFORMS, March 17, 2011
80. Objectives of Research
Optimize patient flow in health-care clinics
Traditionally scheduled (TS) clinic
Some patients do not “show” for scheduled
appointments
TS clinic wishes to increase scheduling flexibility
Some capacity allocated to “open access” (OA)
appointments, OR
Some capacity allocated to “walk-in” traffic
Balance needs of clinic, providers, and patients
80
Rocky Mountain INFORMS, March 17, 2011
81. Objectives of Research
Study impact of open access and
walk-in traffic
When is open access or walk-in traffic
beneficial?
What mix of TS, OA, and WI traffic is
best?
What are trade-offs of TS, OA, and WI
on clinic performance?
81
Rocky Mountain INFORMS, March 17, 2011
82. Relative Benefits and Penalties
= Benefit of seeing additional client
= Penalty for client waiting
= Penalty for clinic overtime
Numéraire of , , and doesn’t matter
Ratios (relative importance) are important
Allow linear, quadratic, and mixed (linear +
quadratic) costs
82
Rocky Mountain INFORMS, March 17, 2011
83. Linear & Quadratic Objectives
Linear Utility Function
N k
ˆ S A k i 1
N 1, k
Patient waiting Patient kwaiting N 1,k
U ˆ
jk
ˆ
Benefit from A j 1 k k i 1 k Clinic overtime
penalties during penalties during
patients served penalties
normal clinic ops clinic overtime
Quadratic Utility Function
N k
ˆ S A 2k 1 i 12
U ˆ
jk N 1, k k 2 N 1,k
ˆ
A j 1 k k i 1 k
83
Rocky Mountain INFORMS, March 17, 2011
84. Heuristic Solution Methodology
1. Gradient search
Increment/decrement appts scheduled in each slot
Choose the single change with greatest utility
Iterate until no further improvement found
2. Pairwise interchange
Exchange appts scheduled in all slot pairs
Choose the single swap with greatest utility
Iterate until no further improvement found
3. Iterate (1) and (2) while utility improves
4. Prior research: Optimality not guaranteed, but
almost always obtained
84
Rocky Mountain INFORMS, March 17, 2011
85. How does Open Access contribute to
leaner scheduling?
1. It provides a more reliable method of
overbooking.
2. It eliminates the uncertainty of demand for
same-day appointments.
3. It guarantees that patients will be seen when
they want.
4. It reduces uncertainty caused by no-shows.
5. It eliminates waste caused by unfilled
appointments.
85
Rocky Mountain INFORMS, March 17, 2011
86. How does Open Access contribute to
leaner scheduling?
1. It provides a more reliable method of
overbooking.
2. It eliminates the uncertainty of demand for
same-day appointments.
3. It guarantees that patients will be seen when
they want.
4. It reduces uncertainty caused by no-shows.
5. It eliminates waste caused by unfilled
appointments.
86
Rocky Mountain INFORMS, March 17, 2011
87. 5. Computational Results
10
10
9
9
Net Utility per Provider
8
Net Utility per Provider
8
7
7
6
6
5
5
4 Walk-ins
4 Walk-ins
3
3 Open Access
2 Open Access
2
1 -6.19
1 -6.19
0
0 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Open Access (OA) Traffic (% of capacity)
Open Access (OA) Traffic (% of capacity)
87
Rocky Mountain INFORMS, March 17, 2011
88. Computational Trials
44 sample problems solved
Session size N = 12
Appointment show rate = 70%
Number of providers P = {1, 2, 4, 8}
OA call-in rate = {0%, 10%, …100%} capacity
With P = 4 and N = 12, then = 24 is 50% of capacity
Walk-in rate = {0%, 10%, …100%} of capacity
With P = 4, then = 2 is 50% of capacity
Quadratic costs
Parameters =1.0, =1.0, =1.5
88
Rocky Mountain INFORMS, March 17, 2011
89. Patients Seen
12
12
Patients Seen per Provider
Patients Seen per Provider
Walk-ins
Walk-ins
Open Access
Open Access
11
11
2 Providers (P=2)
2 Providers (P=2)
10
10
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
OA or WI Traffic (% of capacity)
OA or WI Traffic (% of capacity)
N=12, P=1, =0.7, =1.0, =1.0, =1.0, =1.5
89
Rocky Mountain INFORMS, March 17, 2011
90. Patient Waiting Time
1.0
1.0
Expected Waiting Time / Patient
Expected Waiting Time / Patient
0.9 Walk-ins
0.9 Walk-ins
0.8 Open Access
0.8 Open Access
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
0.3
0.3
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
OA or WI Traffic (% of capacity)
OA or WI Traffic (% of capacity)
N=12, P=1, =0.7, =1.0, =1.0, =1.0, =1.5
90
Rocky Mountain INFORMS, March 17, 2011
91. Clinic Overtime
2.5
2.5
Expected Provider Overtime
Expected Provider Overtime
2.0
2.0 Walk-ins
(d time units)
Walk-ins
(d time units)
1.5 Open Access
1.5 Open Access
1.0
1.0
0.5
0.5
0.0
0.0
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
OA or WI Traffic (% of capacity)
OA or WI Traffic (% of capacity)
N=12, P=1, =0.7, =1.0, =1.0, =1.0, =1.5
91
Rocky Mountain INFORMS, March 17, 2011
92. Provider Utilization
90%
90%
Expected Provider Utilization
85%
Expected Provider Utilization
85%
80%
80%
75%
75%
70%
70% Walk-Ins
Walk-Ins
65% Open Acess
65% Open Acess
60%
60%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
OA or WI Traffic (% of capacity)
OA or WI Traffic (% of capacity)
N=12, P=1, =0.7, =1.0, =1.0, =1.0, =1.5
92
Rocky Mountain INFORMS, March 17, 2011
93. Net Utility
10
10
9
9
Net Utility per Provider
8
Net Utility per Provider
8
7
7
6
6
5
5
4 Walk-ins
4 Walk-ins
3
3 Open Access
Open Access
2
2
1 -6.19
1 -6.19
0
0
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Open Access (OA) Traffic (% of capacity)
Open Access (OA) Traffic (% of capacity)
N=12, P=1, =0.7, =1.0, =1.0, =1.0, =1.5
93
Rocky Mountain INFORMS, March 17, 2011
94. 6. Insights and Recommendations
94
Rocky Mountain INFORMS, March 17, 2011
95. Managerial Implications
TS appointments provide better clinic utility
than does WI traffic or OA call-ins
Any WI or OA traffic causes some decline in utility
An all-WI or all-OA clinic performs worse than any
clinic with some TS appointments
Even a relatively small percentage of scheduled
appointments can significantly improve clinic utility
Degree of improvement depends on number of
providers
A mix of TS appointments with some OA or WI
traffic does not greatly reduce clinic performance
(utility)
95
Rocky Mountain INFORMS, March 17, 2011
96. Insights from the Model
Loss of utility with WI traffic is due to the long
right-tail of Poisson distribution
Excessive patient waiting & clinic overtime
Loss of utility with OA traffic is due to uncertainty
about number of OA call-ins
TS appts reduce patient waiting and clinic
overtime
Binomial distribution has truncated right tail
Multiple providers improves clinic utility
Portfolio effect – variance reduction
96
Rocky Mountain INFORMS, March 17, 2011
98. Driving Clinical Quality Improvement Through Mental Health Recovery Control Charts
C.J. McKinney, Pablo A. Olmos, Cathie McLean, Linda R. LaGanga,
Division of Quality Systems, Mental Health Center of Denver, Denver, CO
Presented at the Mayo Clinic Conference on Systems Engineering & Operations Research in Health Care (August 2010), Rochester, MN. Awarded First Place for Best Poster Presentation.
INTRODUCTION RECOVERY ASSESSMENT continued QUALITY CONTROL CHARTS continued
Every community mental health center focuses on clinical quality. Benefits of effective service
delivery support quality through: Recovery Needs Level 3. The Utilization Review Process: When a consumer is “flagged” by the Change Chart they will be an
The Recovery Needs Level is a series of indicators that through an objective algorithm assigns the automatic candidate for a utilization management review. This review is done by other clinicians
• optimize resource allocation, consumer to an appropriate clinical service level. The RNL is completed by the clinician every six months reviewing a consumer’s medical record to determine if a gap in services has occurred and if other
• increase consistency in consumer outcomes, and as needed. The measure consist of 15 different dimensions such as the GAF, Residence, Case services should be considered. The recommendations from this review are forwarded to the program
• increase service fidelity, Management, Substance Abuse, and Service Engagement. manager for further review and implementation.
• decrease administrative load on clinicians, and
• increase access to consumer services.
Utilization Review Form
This poster presents our development of a set of reliable and valid mental health recovery Promoting Recovery in Organizations Qualitative Identification of
measures, which we combine for a multi‐perspective assessment of recovery progress, which The PRO survey is completed by the consumer, and consists of 7 sections covering all major service
anchors an objective clinical quality control system. positions at MHCD, i.e. front desk, nursing/medical, case management, and rehabilitation. This data is Service Outliers
collected annually through a random sampling of consumers. The survey summaries are then utilized to
determine how well the teams and system are promoting recovery ideals.
RECOVERY ASSESSMENT
MHCD consistently collects, reviews, and analyzes data across all consumers on four different
recovery‐oriented outcome measurement tools. The combined data from these assessments
provide multi‐perspective viewpoints for a more comprehensive picture of the consumer’s
recovery experience and what factors may be impacting their recovery. It also provides
supporting information to ensure the consumer is placed at a level of care that appropriately
reflects their needs.
Recovery Marker Inventory – Clinician Assessment
Assessments are recorded on seven factors associated with recovery: Employment,
Learning/Education, Activity/Growth Orientation, Symptom Interference, Participation in
Services, Housing, and Substance Use.
Documentation of this data provides the clinician with a longitudinal perspective – from both an
overall standpoint, as well as more specific recovery dimensions. These observations can then be
CONCLUSION & FUTURE DIRECTIONS
used to help guide clinical discussion with the consumer, and indicate treatment focus. Consistent with continuous quality improvement, integration of these tools
into the clinical workflow is a constantly evolving process. We feel the
QUALITY CONTROL CHARTS following are basic needs to meet, and opportunities for operational
The Recovery Outcome Tools have enabled us to develop a quality review system to monitor individual
enhancement:
consumer outcomes and recommend review in cases where the consumer may not be progressing as
expected. We are able to do this in three ways: • Education of Clinical staff, Executive Management, Consumers, and other stakeholders
as to the value of outcomes data collection and analysis and integration into the clinical
1.The Consumer Recovery Profile provides a snapshot of a person’s current mental health recovery
progress. It demonstrates through graphs and tables the current status of a consumer to aid in service
practice
planning. • Technological ability to “communicate” with the Electronic Medical Record ‐ the
Recovery Profile is connected to the Electronic Medical Record, so it can be easily
accessed by clinicians by bringing the information to them, without having to log in or
open other data storage sites
Consumer Recovery Measure – Consumer Assessment
• Integration into the daily clinical work flow – clinicians can review outcomes
With the Consumer Recovery Measure, the consumer rates agreement or disagreement with information with consumers during individual sessions, so as to make the information
statements regarding their current recovery experience. These responses gauge consumer more meaningful; it is employed as part of the Peer Review process; and can be used
perspective on five dimensions of recovery: Symptom Management, Sense of Safety, Sense of
Growth, Sense of Hope, and Social Activity. during six month case reviews
• Automation of Quality Review process – control charts “flag” concerning outcomes
Graphic representation of this data is shared with the consumer to initiate clinical discussion about
changes in these areas, what the consumer attributes the changes to, and possible relationships outliers and identify them for Utilization Management Review, so as to address and
between categories. This promotes insight, and empowers the consumer to share their story in a redirect treatment inefficiencies in a timely manner
new and different way.
• Exploration of “super performer” characteristics to identify benchmarks for
2. The Recovery Change Chart automatically identifies consumers needing further review by flagging those
with substantial change in their recovery outcomes. A flag occurs whenever a consumer deviates from teams/programs
their expected outcomes for an extended period of time or if the deviations are large. • Consumer Recovery Portal – consumers will have access to their outcomes data for
increased engagement in the recovery process
Self‐Comparing Control Chart Peer‐Comparing Control Chart
•Integrate physical and mental healthcare
•Maximize outcomes to improve human lives! mental
For more information about research or
health recovery at MHCD, please view conference
presentations on our website: 98
www.outcomesmhcd.com
Rocky Mountain INFORMS, March 17, 2011