Tampa BSides - The No BS SOC (slides from April 6, 2024 talk)
Wheatcraft hill.lou terry
1. Developing Requirements for
Constellation’s Next Generation
Space Suit
Presented at PM Challenge 2010
Presenter's:
Terry Hill, NASA/JSC
Lou Wheatcraft,
Compliance Automation
February 2010
Used with Permission
2. Overview
Part 1: The Basics
Risk and Requirements
Requirement Validation
Part 2: Application Case Study
CxP Suit Requirement Development Process
Results
CxP Suit Continuing Requirement Management
Process
What we could have done better
National Aeronautics and Space Administration 2
3. Risk and Requirements
WHAT’S COMING NEXT
National Aeronautics and Space Administration 3
4. NASA OIG
“NASA must be vigilant in its process of establishing and
validating project requirements.”
“Program risks increase when NASA awards contracts
before developing a sound business case and clearly
defining requirements;”
Placing “the project at risk of significant cost overruns, schedule
delays, and performance shortfalls.”
“Effective risk management, safety, and mission
assurance controls are key to supporting robust and
reliable operations in the context of very challenging
launch and mission schedules.”
NASA’s Most Serious Management
and Performance
Challenges – Nov 2008
National Aeronautics and Space Administration 4
5. GAO
“The start of product development represents the point at which
program managers make a commitment to provide a product that
will perform as required and be delivered on time and within
estimated costs.”
“Programs are more likely to succeed if program managers are able
to achieve a match between user needs, which eventually become
requirements, and resources (technology, design and production
knowledge, money, and time) at the start of product development.”
“Conversely, if they do not match requirements with resources, cost
overruns and schedule delays are likely to occur, reducing the
organization’s buying power in other areas.”
GAO-03-598
National Aeronautics and Space Administration 5
6. Effect Of Requirements Definition
Investment On Program Costs
200
OMV
180
GRO 78
160 GALL Why are you running so fast
TDRSS
when you don’t know where
Actual – Target Cost
140
CEN
IRAS HST you are going?
120
Target Cost
100 GOES I-M
TETH German proverb
MARS
LAND 76
MAG
80
ACT STS LAND 78
ERB 77 COBE
60
ERB 80
40 SEASAT
UARS GRO 82 HEA
VOYAGER
EUVE/EP ISEE
20
DE
SMM ULYSSES IUE
PION/VEN
5 10 15 20 25 30
Requirements Definition and Preliminary Design
Target Total Cost
National Aeronautics and Space Administration 6
7. Cost to fix requirement defects
1,000-
1000
70
60-
50-
40- 40 40
30- 30
20-
15
10
10-
6
1 3
0-
Requirements Design Coding Development Acceptance Operations
Phase Phase Phase Testing Testing
National Aeronautics and Space Administration 7
8. Importance of Best Requirement Practices on
Project Success
“The companies using best requirements practices will estimate a
project at $3 million and better than half the time will spend $3
million on that project. Including all failures, scope creep, and
mistakes across the entire portfolio of projects, this group will spend,
on average, $3.63 million per project.”
“The companies using poor requirements practices will estimate a
project at $3 million and will be on budget less than 20% of the time.
50% of time, the overrun on the project both in time and budget will
be massive. Across the entire portfolio of successes and failures,
this company with poor requirements practices will (on average) pay
$5.87 million per project.”
2008 Study by Keith Ellis, IAG Consulting of 100
companies with projects in excess of $250,000
9. Setting Yourself Up for Failure
Project success is “improbable” for 68% of the companies Ellis
studied
“Projects might succeed – but not by design. Based on the
competencies present, these companies are statistically unlikely to
have a successful project.”
While these companies indicated they recognized that requirements
are important to project success, they still failed to take effective
actions to insure a good set of requirements.
By doing so, they tripled their chances of project failure
2008 Study by Keith Ellis, IAG Consulting of 100
companies with projects in excess of $250,000
10. What Needs to be Done
“Organizations understand conceptually that
requirements are important, but do not internalize this
understanding and change their behavior as a result.
The most successful of companies do not view
requirements as a document which either existed or
didn’t at the beginning of a project, they view it as a
process of requirements discovery.
Only companies that focus on both the process and the
deliverables are consistently successful at changing
project success rates.”
2008 Study by Keith Ellis, IAG Consulting of 100
companies with projects in excess of $250,000
11. No Surprises
People who write bad requirements
should not be surprised
when they get bad products…
but they always are.
Ivy Hooks
National Aeronautics and Space Administration 11
12. Words of Wisdom
“Putting forth the same effort, or
using the same approach, then
expecting different results
is...insanity”
- Charles Bolden,
NASA Administrator, July 2009
National Aeronautics and Space Administration 12
13. A Winning Product
Delivers what’s needed
Within budget
Within schedule
With desired quality
Risk: Anything that can
prevent you from delivering
a winning product!
National Aeronautics and Space Administration 13
14. Requirement Validation
WHAT’S COMING NEXT
National Aeronautics and Space Administration 14
15. Requirement Validation vs. Verification
Requirement validation Verification confirms that
confirms the the designed and
completeness and built product meets the
correctness of the requirements
requirements
Done after design and build
Starts with first requirement
and continues through life
cycle
Requirement validation
makes sure you are Verification makes sure
building the right thing. you built it right.
National Aeronautics and Space Administration 15
16. Requirement Validation
Helps ensure our requirements are:
Needed, verifiable, achievable
Clear, concise
Correct, consistent, and complete
Two types
Continuous
Discrete
National Aeronautics and Space Administration 16
17. Continuous Validation Process
Requirement Writer:
Write requirement and attributes
Check requirements against standards
Submit to gatekeeper for review in “chunks”
Gatekeeper (person or inspection points)
One or more experts
Reviews requirements against standards
Accepts defect-free requirements for input to
database or document
Return requirements to author if defects found
National Aeronautics and Space Administration 17
18. Who Does Requirement Validation?
Writers Managers
Everyone is
accountable
Developers Reviewers
National Aeronautics and Space Administration 18
19. Continuous Validation Process
Continuous validation holds everyone
responsible
Requires standards and checklists
Requires training
Management has to enforce discipline and
accountability
Benefits
Stops the creation of BIG bad documents
Most effective way to realize process improvement
Reduces time for milestone reviews
Prevents lost time due to rework
National Aeronautics and Space Administration 19
20. Discrete Validation Process
Key milestone that requires Effective IF
time and resources The right people are involved
Formal process The products are ready for review
Complete document The participants know what to do
Involves a wide range of Management ensures
stakeholders compliance
Requires standards and
feedback mechanisms
Requires training
Management has to ensure
responsiveness
SRR results in a requirement
baseline
National Aeronautics and Space Administration 20
21. System Requirement Review (SRR)
Scope
Requirements
Design
MCR Manufacture
or Code
Verification
SRR
Operations
SDR
Upgrade/
PDR Maintain
CDR
TRR
MCR: Mission Concept Review
Baseline requirements SRR: System Requirements Review
Assess feasibility SDR: System Definition Review
PDR: Preliminary Design Review
Set expectations CDR: Critical Design Review
TRR: Test Readiness Review
National Aeronautics and Space Administration 21
22. CxP Suit Element Requirement
Development Process
WHAT’S COMING NEXT
National Aeronautics and Space Administration 22
23. Background: Case Study
Shuttle / ISS Extra Mobility Unit (EMU)
Shuttle was designed to be a shirt sleeve environment
not requiring space suits of any kind.
EMU developed after Shuttle was designed to address a
late in design possible failure scenario with the Shuttle –
risk that payload bad doors would not close and would
have to do it manually.
Suit dimensions were driven by Shuttle hatch sizes.
Many, many requirements and later functionality of the suit were
not addressed or defined early in the process.
The Shuttle EMU was later augmented and recertified to
work at and around the ISS to perform station assembly
and repair.
National Aeronautics and Space Administration 23
24. Shuttle/ISS EMU Case Study –
Design and Development Cycles
Pre-declared Cert.
Initial Cert.
Re-Certification
and Ops Con &
Requirements
evolved
National Aeronautics and Space Administration 24
25. Background: EMU Case Study
Lessons Learned
Any little design change will take you into some
level of re-certification.
Always larger and more expensive and takes longer
than ever planned.
The better the ConOps is thought through in the
beginning allows the writing of better and more
comprehensive requirements eliminating many
re-certification cycles later and thus saving Life
Cycle dollars.
National Aeronautics and Space Administration 25
26. Background: CxP Suit Element
Requirement Development Challenge
The Task
The team was asked to develop a CxP Initial
Capability (and Lunar requirements when common
hardware would be used) requirement set for the
Space Suit Element in time to support the CxP Suit
Element SRR and for release with the RFP for a
prime contractor in mid-fall.
Schedule
Very Short Schedule – 3 Months from initiation of
requirements generation to Major Project Milestone
review and 5 months to baseline set of functional
requirements.
National Aeronautics and Space Administration 26
27. Background: CxP Suit Element
Requirement Development Challenge
The Philosophy
Learn from past projects mistakes in how and when
requirement are written
Clean sheet approach to developing a space suit and
writing the requirements
Exercise the text book methodology of Systems
Engineering and Requirement generation in a NASA
project
Produce quality requirements that are verifiable in a
cost effective manner that address the functionality
defined in the CxP EVA System Operations Concept
and EVA Systems Architecture documents.
National Aeronautics and Space Administration 27
28. Background: CxP Suit Requirement
Development Schedule
Planned Schedule for 2007:
May 31-Jun 5 – Requirement Training and Kick-off
Jun 1-22 – Suit Element Requirements Generation Activities
June 25-29 – Suit Element SRR Doc(s) review
July 2-6 – Suit Element SRR Doc(s) update & SRR final prep.
July 10 – Suit & Vehicle Interface Elements SRR Kick-off
July 10-20 – Suit/VI Element SRR Doc review & RID submittal
July 22-Aug. 7 – Suit/VI Element SRR Panels and Boards.
Aug. 9-Oct 20 – Close SRR Actions and update ERD
Oct. 23 – Suit ERD for Baselining at EVA PCB.
Oct. 29 – Suit ERD rev. A draft submitted to EVA CM for pre-blackout CSSS
Tech. Library drop for Prime contract RFP release
* Final outcome, Element SRR slipped one week to ensure
quality of products where ready for review. Review
revealed products were ready and of the appropriate
fidelity by EVA project management.
National Aeronautics and Space Administration 28
29. Background: CxP Suit Requirement Development
Approach: Ground rules
Co-located the team off-site in a conference Requirements meet the criteria for good
room facility to enable a concentrated effort requirements in the Checklist for Good
– this reduces the risk of day-to-day Requirements,
distractions which is a risk to product quality
Rationale for each requirement, no copying
and schedule.
parent requirement with a noun change,
Provided task training in requirement
All requirements verifiable, clear, concise
development and writing processes – this
and if it can be interpreted in more than one
reduces the risk to quality.
way, it is not ready for acceptance.
Reduce risk of requirement development by
contracting out the training and using
consultation which provided standard
training as provided to CxP, and an
independent “fresh set of eyes” on how
requirements could be interpreted and
implemented – this reduces the risk to
quality.
Used CRADLE tool to develop requirements
prior to baseline. Post baseline, the
requirements were managed out of
CRADLE, but draft revisions were handled
outside of CRADLE until change approval.
National Aeronautics and Space Administration 29
30. Putting Requirement Risk in the Proper
Perspective
Not to put too much pressure on you….
The Requirements Document is probably the single most influential
piece of paper that we have control over in the entire Constellation
Program.
This is our chance to make sure that we are asking for what we
really want. Let’s get it right.
This is a big, fat, hairy deal. If we don’t get this right, folks 20 years
from now will be shaking their heads and saying, “What were those
yahoos thinking?”
I’ll be around and don’t want to go to that meeting.
CxP EVA Suit PGS Team Requirement Kickoff
Meeting 5/2007
National Aeronautics and Space Administration 30
32. Subsystem Teams
Responsible for the drafting & modifying requirements
Four independent teams
Suit Element-level (integrated across subsystems)
Subsystems
Pressure Garment, Crew Survival, Power-Comm & Avionics, Life
Support, (Ground Support Equipment team was after the initial
requirement generation effort)
Each team consisted of:
Functional Lead
NASA Subject Matter Experts (SMEs)
“Core” Scrub team
Teams given training during kick off meeting
Briefed in the Goal, Schedule and Process
Checklist for Good Requirements
National Aeronautics and Space Administration 32
33. Scrub Team
“Core” Scrub Team:
SE&I representative
Compliance Automation representative – independent “goodness
assessment”, and guidance in requirement development.
Tech Writer
CRADLE Operator
Review requirements from the Subsystem Teams
Edit & “clean” Requirements based on Checklist for Good Requirements
If they could “fix” requirement & verify the intent was understood, would do
so & pass onto the CRADLE Board
If intent not understood and the requirement needed more work,
representatives from appropriate Requirement Team were called in for
clarification.
Kept list of common defects and briefed Requirement Teams daily
National Aeronautics and Space Administration 33
34. CRADLE Board
Had three functions:
Determined if requirements where technically traceable and
appropriate
Determined if requirements where technically appropriate and
the correct mission phases applied
And determine any outstanding issues were resolved as the
CRADLE Board was used as a dry-run for final approval.
Base Team:
SE&I representative
Subsystem representative or subject matter expert
Secondary Team:
Project Management
Subsystem Leads
Focus Meetings a.k.a. Offline
Resolves outstanding issues
Membership as required
National Aeronautics and Space Administration 34
35. Approval Board
Function: Team Consensus/Final Approval for requirement to be
included in ERD
Base Team:
SE&I representative
Project Management
Subsystem Leads
CRADLE operation
Subject matter expert
If discussions proceeded for longer than 15 minutes, the
requirement and discussion were sent to a Focus Meeting for
resolution. The idea was that if a requirement was not clear enough
to convey the intent and importance in 15 minutes of discussion,
then it was not written correctly.
Focus Meetings a.k.a. Offline
Resolves outstanding issues
Membership as required
National Aeronautics and Space Administration 35
36. Results
WHAT’S COMING NEXT
National Aeronautics and Space Administration 36
37. Results in Project Reviews
For the Suit ERD SRR, a ratio of 0.38 Review Item Descriptions
(RIDs) were received per requirement.
In comparison, the parent document had a 2.94 RID/requirement ratio at
its SRR.
Potential bidders for the “… the most comprehensive and of the highest
development of the Suit Element
stated that the ERD was: quality they ever remember seeing.”
“I can't say enough about how amazed I am by
The JSC Engineering Directorate this set of requirements documents. As far as I
Crew and Thermal Systems
Division Chief was also very know, no other Cx project has allocated and
impressed with the quality of the decomposed anywhere near to this level of
Suit ERD, saying: depth. You are the first. I have also never seen
anything like these from previous programs.”
National Aeronautics and Space Administration 37
38. CxP Suit Continuing
Requirement Management Process
WHAT’S COMING NEXT
National Aeronautics and Space Administration 38
39. Post-baseline Requirement
Development and Validation
As a result of the extremely compressed requirement development
schedule, there remained several areas that needed more work and
issues that needed to be resolved prior to preliminary design taking
place.
These open areas included:
resolving TBDs/TBRs in the requirement set,
finishing the verification requirements,
defining the internal interfaces and maturing applicable interface requirements
and adding Ground Support Equipment (GSE) requirements to the ERD.
Therefore there was requirement development and maturation
process needed to continue this process.
National Aeronautics and Space Administration 39
41. Requirement Review Process Post-
ERD Baseline: SEIWG
The SEIWG (Systems Engineering and Integration Working Group)
Meets once a week to discuss new and existing requirements and/or
requirements issues, such as TBDs/TBRs, action items, allocations, etc.,
Serves as the SE&I pre-coordination for the Suit Control Board.
SEIWG Procedure
Step 1:
All potential changes must be submitted via a change request form.
Step 2:
Scheduled SEIWG meeting to present/discuss change and any final modification
of the draft FROM/TO language.
National Aeronautics and Space Administration 41
42. Suit Control Board
Functions:
Detailed review of proposed FROM/TO language to
go into the ERD.
A minimum of a 2 business day review by board members
prior to the Suit Control Board meeting.
Functions as the official forum of stakeholder review,
concurrence or rejection
Functions as the final approval gate for base-lined
requirements.
Note: Since Baselineing of ERD the SCB closed
approximately 180 TBD/Rs prior to levying the
document on the prime contractor.
43. Wrap up
WHAT’S COMING NEXT
National Aeronautics and Space Administration 43
44. What we could have done better
Challenging schedule resulted in a lot of open work post baselining
of ERD.
Large number of TBDs/TBRs
Interfaces identification and definition incomplete
Some of the external interfaces just didn’t exist due to sister projects were
evolving in parallel and at times without the same rigor demonstrated in the
Suit Element effort.
Traceability incomplete
Verification requirements
GSE requirements
Difficult to keep requirements at the right level
Some requirements may not have been needed
Some requirements reflected or assumed a design where NASA was
clear there was only one desirable functional solution.
National Aeronautics and Space Administration 44
45. Parting Thoughts
Address Requirement Risk at the beginning of
your project
Develop a formal requirement development
process that includes continuous requirement
validation
Include continuous requirement validation into
your requirement management process
Train your team
Enforce the process
Allocate the time and resources needed to do
the job right – the first time!!
National Aeronautics and Space Administration 45
47. Terry Hill
NASA/JSC Constellation Space Suit System Engineering Project Manager
Terry Hill is NASA’s Johnson Space Center’s Engineering Project Manager and deputy CxP EVA Suit
Lead for the CxP Suit Element, responsible for the development of the functional, performance, and
quality requirements and preliminary design of NASA’s next generation space suit system.
Terry has a BS in Aerospace Engineering and a MS in Guidance, Navigation & Control Theory with a
minor in Orbital Mechanics and Mathematics from the University of Texas at Austin.
He began his career at NASA while working on his masters thesis
project in developing banks of simplified Kalman filters integrated
into an artificial neural network to obtain an optimal state solution for
precision landing on Mars.
While at NASA ,Terry has worked on projects and programs
spanning verification of ISS navigation software, Shuttle Design
Test Objectives (DTO) and back room mission support, X-38 Crew
Return Vehicle navigation algorithm development, Space Launch
Initiative technology development, Orbital Space Plane project
office ISS-Prime integration, STS-107 Return to Flight Tile Repair
capability development, to Constellation Program Space Suit
System leadership.
In leading the CxP Suit Element engineering team, Terry has
facilitated the development of system requirements for space suit
development and a clean-sheet design approach that has widely
recognized within and outside of NASA.
National Aeronautics and Space Administration 47
48. Lou Wheatcraft
Compliance Automation – Senior Consultant Trainer
Lou Wheatcraft is a senior consultant/instructor for Compliance Automation, who has over
40 years experience in the aerospace industry, including 22 years in the United States Air
Force. Lou has taught over 120 seminars in requirement development and management
for NASA’s APPEL Program and industry over the past nine years. He has worked with,
and provided intact team training and consultation to multiple NASA project teams at many
of the NASA Centers.
Lou has had articles published in the International Council of Systems Engineering
(INCOSE) INSIGHT magazine and in DoD’s magazine, CrossTalk. Lou has made
presentations at NASA’s PM Challenge, INCOSE’s International Symposium, and at the
local Project Management Institute (PMI) Chapter Meetings.
Lou has a BS degree in Electrical Engineering, an MA degree in Computer Information
Systems, an MS degree in Environmental Management, and has completed the course
work for an MS degree in Studies of the Future.
Lou is a member of INCOSE, co-chair of the INCOSE Requirements Working Group, a
member of PMI, the Software Engineering Institute, the World Futures Society, and the
National Honor Society of Pi Alpha Alpha. Lou is the recipient of NASA’s Silver Snoopy
Award and Public Service Medal and was nominated for the Rotary Stellar Award for his
significant contributions to the Nation’s Space Program.
National Aeronautics and Space Administration 48