1. Joint Confidence Level for
Small Projects
Yunjin, Kim, Alberto Ortega, and
Yolanda Cuevas
Jet Propulsion Laboratory
California Institute of Technology
Presented at the PM Challenge 2011Conference, February 9-10, 2011, Long
Beach, California
2. Agenda
• Fundamentals of Joint Confidence Level (JCL)
– Cost-schedule relationship
– Importance of the probability density function for schedule activities
– Simple examples
• NuSTAR example
– NuSTAR JCL process
– NuSTAR JCL results
– Lessons learned
• Use of JCL for subsystems
• Conclusions
02/09/2011 2
3. What is Joint Confidence Level (JCL)?
• Official definition of JCL (from JCL FAQ from NASA HQ)
– The probability that cost will be equal or less than the targeted cost and
schedule will be equal or less than the targeted schedule date.
– A process and product that helps inform management the likelihood of
a projects’ programmatic success.
– A process that combines a projects’ cost, schedule, and risk into a
complete picture.
• NASA policy for JCL (NPD 1000.5)
– Joint cost and schedule confidence levels are to be developed and
maintained for the life cycle cost and schedule associated with the
initial lifecycle baselines (such as project baselines at KDP-C).
02/09/2011 3
4. Joint Confidence Level (JCL) Process Used for
NuSTAR Mission
• Inputs
– Resource loaded schedule
• Summary project schedule
• Burn rate for each cost element
– Probability density functions for schedule activities
– Probability density functions for cost elements
• Monte Carlo simulation to produce a two dimensional (cost
and schedule) probability density function for cost/schedule
success P (c, s) where c cost, s schedule
cs
• Outputs C R
Probability of cost less than CR Pr(c CR , s) Pcs (c, s) dc
0
SR
Probability of schedule less than S R Pr(c, s SR ) Pcs (c, s ) ds
0
02/09/2011 4
5. Schedule
A Simple Example (1) uncertainty
• Consider a single schedule activity 100 days & $5k/working days
– Duration: 100 working days
• Schedule risk probability density function = uniform probability
over [-10 working days, +30 working days]
– Burn rate = $5k/ working day
700
650
Cost (in $k)
600 70% schedule = 118
working days
550
500 70% cost = $590k
450
400
80 90 100 110 120 130 140
Schedule Duration (in working days)
02/09/2011 5
6. A Simple Example (2) Cost Schedule
uncertainty uncertainty
• Consider a single schedule activity
100 days & $5k/working days
– Duration: 100 working days
• Schedule risk probability density function = uniform probability
over [-10 working days, +30 working days]
– Burn rate = $5k/ working day
• Cost probability density function = uniform probability over
[0%, +30%]
900
850
800 70% schedule = 117
Cost (in $k)
750 working days
700
650
70% cost = $676k
600
550
500
450
400
80 90 100 110 120 130 140
02/09/2011 6
Schedule Duration (in working days)
7. A Simple Example (3)
• Consider a slightly more complex case shown below.
– Total schedule: 220 working days
– Burn rate
• Task 1: $15k/day
• Task 2: $20k/day
• Task 3: $10k/day
• Task 4: $5k/day
– For all four schedule activities
• Schedule PDF = uniform over [-10 working days, +30 working days]
• Cost PDF = uniform over [0%, 30%]
100 working days
Task 1
Task 2 (100 working days) 20
Task 3 (80 working days) 40
Task 4 (90 working days) 30
02/09/2011 7
9. PDF Selection for JCL
• It is obvious that the most important information for JCL is the
probability density function for each schedule activity.
• Two probability density functions that we considered are
– Uniform PDF =
Minimum Maximum
– Triangle PDF = (
Minimum Maximum
• A truncated Gaussian PDF can also be used based on the central
limit theorem.
02/09/2011 9
10. Determination of Maximum and Minimum
Values of PDF
• To specify uniform PDF or triangular PDF, we have to specify
both the maximum and the minimum values.
• To be conservative, we can use the baseline schedule and
budget to specify the minimum value.
• The maximum value can be specified based on the project risk
list.
02/09/2011 10
11. NuSTAR Mission Overview
Salient Features
• PI-led (PI: Fiona Harrison, Caltech) SMEX mission
• NuSTAR will carry the first high-energy X-ray
focusing telescope
• NuSTAR partners include Caltech, JPL, GSFC,
Orbital, ATK, UCB, DTU, KSC, Columbia
University and ASI
• JPL managed project
• Category 3, Class D (enhanced) mission
• Launch readiness date: November 15, 2011 Goddard Space
Flight Center
Kennedy Space
Flight Center
• Launch date: February 3, 2012
• Science operations: 2 years
Science
• NuSTAR will open a new window on the Universe by making maps of the high-energy X-ray sky (6 keV to 79
keV ) that are more than 100 times deeper than from any previous mission
• Objective 1: Determine how massive black holes are distributed through the cosmos, and how they influence the
formation of galaxies like our own
• Objective 2: Understand how stars explode and forge the elements that compose the Earth
• Objective 3: Determine what powers the most extreme active black holes
02/09/2011 11
12. NuSTAR JCL Process (1)
• The NuSTAR JCL was completed in November 2009.
• From the project integrated master schedule (about 3500 lines), a
summary schedule (about 162 lines) was developed.
– This step is critical to the efficient implementation of JCL.
– The schedule includes the actual performance and costs incurred through
August 2009.
– The summary schedule must maintain the work flow and the schedule
network accurately.
– The summary schedule was reviewed several times to validate the
accuracy.
• The cost information was included in MS project based on the burn
rates of schedule activities.
• The schedule/cost probability density functions were determined by
reviewing the project risk list.
– These probability density functions were also reviewed with the system
managers.
02/09/2011 12
13. NuSTAR JCL Process (2)
• A “penalty” task was created in the schedule in the form of a
hammock task to capture the “marching army” costs
associated with supporting a launch past its planned date .
• Monte Carlo simulations were performed using @risk add-on
tool to Microsoft Project.
02/09/2011 13
17. NuSTAR JCL Results (4)
Current launch date of 8/15/11
is slightly less than 5%.
70% confidence of 11/9/11
projects close to a 3 month slip.
02/09/2011 17
19. Lessons Learned for Determining
Schedule/Cost PDF
• The PDFs used for the NuSTAR JCL were too optimistic for
heritage hardware.
• The PDFs did not include typical test failures and engineering
mistakes accurately.
• When we derived the PDFs, we should have considered the
historical data.
• Overall, the schedule PDFs were optimistic.
02/09/2011 19
20. JCL Concept Applied to Subsystem
Schedule/Cost Assessment (1)
• The JCL concept can be used to estimate the subsystem
delivery date and the cost.
• An example is shown below.
– This assessment was done in early September 2010.
– The acceptance test started in late September 2010.
– Schedule probability density functions are shown in the diagram.
– The cost probability density function is [0%, 30%] uniform. The burn
rate is $15k/day .
Acceptance test Integration Vibration test
(10 working days (10 working days (8 working days
PDF =[0,10] uniform) PDF =[0,10] uniform) PDF =[0,3] uniform)
Function check-out &
Preparation for shipping
alignment
(3 working days
(15 working days
PDF =[0,1] uniform)
PDF=[0,10] uniform)
02/09/2011 20
21. JCL Concept Applied to Subsystem
Schedule/Cost Assessment (2)
• Subsystem JCL in September 2010
1400
1300
1200
70% schedule = 65.8
Cost (in $k)
1100
working days
1000
900
70% cost = $1140.5k
800
45 50 55 60 65 70 75 80 85
Schedule Duration (in working days)
02/09/2011 21
22. JCL Concept Applied to Subsystem
Schedule/Cost Assessment (3)
• The subsystem JCL was repeated in October 2009 based on
the acceptance test progress.
– The baseline schedule and the schedule probability density functions
have been revised as shown in the diagram.
– The cost probability density function is [0%, 20%] uniform. The burn
rate is $15k/day .
Acceptance test Integration Vibration test
(25 working days (15 working days (8 working days
PDF =[0,5] uniform) PDF =[0,5] uniform) PDF =[0,3] uniform)
Function check-out &
Preparation for shipping
alignment
(3 working days
(15 working days
PDF =[0,1] uniform)
PDF=[0,15] uniform)
02/09/2011 22
23. JCL Concept Applied to Subsystem
Schedule/Cost Assessment (4)
• Subsystem JCL repeated in October 2010 based on the
acceptance test progress
1600
1500
1400
Cost (in $k)
70% schedule = 83.5
1300
working days
1200
70% cost = $1340.3k
1100
1000
60 65 70 75 80 85 90 95 100
Schedule Duration (in working days)
02/09/2011 23
24. JCL Concept Applied to Subsystem
Schedule/Cost Assessment (5)
• Comparison between the September JCL and the October JCL
September October
70% schedule = 65.8 working days 70% schedule = 83.5 working days
70% cost = $1140.5k 70% cost = $1340.3k
1400 1600
1300 1500
1200 1400
Cost (in $k)
Cost (in $k)
1100 1300
1000 1200
900 1100
800 1000
45 50 55 60 65 70 75 80 85 60 65 70 75 80 85 90 95 100
Schedule Duration (in working days) Schedule Duration (in working days)
02/09/2011 24
25. Conclusions
• The NuSTAR JCL demonstrated that JCL can be performed
efficiently using a summary schedule derived from the
project integrated master schedule.
• The most important step in JCL is driving the probability
density function for each schedule activity based on the
project risk list.
– In addition to the project risk list, historical data should be
considered if available.
• The NuSTAR JCL accurately predicted key schedule/cost
drivers.
• The JCL concept can be used to estimate the subsystem
delivery date and cost.
02/09/2011 25
