1. Facilitating
NPR7150.2 Compliance and
CMMI L2/L3 audit
NASA PM Challenge 2012
Michel Izygon, Scott Hetherington, Svetlana Hanson

2. NASA & Process
Increased Process Focus
Strong Call for
Process Institutionalization
2

3. Process Evolution
Chief Engineer increase awareness and consistency across the Agency and
advance the practice of engineering …The engineering of NASA systems requires
a systematic and disciplined set of processes … for the design
development, operation, maintenance, and closeout of systems throughout the
lifecycle of the programs and projects.
ISO 9001
CMMI
NPR7150.2
ITIL You
IEEE 12207 are
Six Sigma ... here
Best Practices
Process Improvement
Institutionalization
COMPLIANCE IS MANDATORY
Resistance is Futile
Software Productivity Consortium “The Quagmire”
2000
3
www.software.org

4. SDA Roots - Flight Software Problem
• How can we make it easier for engineers to follow the rigorous
process with minimal overhead?
• Software Process Automation
• Survey of workflow tools (2002) found very limited support for
dealing with unanticipated events
– Need a workflow tool to automate the flight software process but
also allow for the following process exceptions:
• Go back in the process to redo a task
• Go back in the process to redo an entire group of tasks
• To jump ahead
• To Tailor the process
• SBIR Phase I: Jan 04 – July 04
• SBIR Phase II: Nov 04 – Nov 06 Implemented EA-WI-025
• SBIR Phase III: Sep 07 – Mar 09 – SDA for MOD & Engineering
• ER6 funded improvements: Apr 09 – Sept 11
• SARP Funded Infusion: 2009-2011
4

5. SDA Concepts
• Flexible Process Customization – Multiple Levels
– Project, program, organization
– Workflow Engine + Rules – key to FLEXIBILITY
• Framework to model/automate any software process
– Currently implement NASA/JSC Engineering Flight Software and
MOD Ground Software
– Can be customized to any other NASA process
• Easy to capture and share NASA “Best Practices”
– Lessons learned, Process Asset Library (PAL), Best toolset
• Facilitate software process execution by supporting:
– Developers: worklist, guidelines, instructions, templates,
examples
• “Everything is at your fingertips”
– Teams: distribution of tasks, collaboration tools, notification
– Managers: visibility, reports, metrics
5

6. SDA Core Features
• Flexibility
– Designed to support any process
• Select existing one
• Create new one
• Robust workflow engine
– Process clarity
– Workflow Logic
• Tool Integration
– Currently: Version control, MS Project, Calendar, Message
Board, Wiki,…
• Rich Instrumentation
– Real time status of projects
– Dashboard
– Reporting
• Mapping to any policy or regulations
– NPR 7150.2A
– CMMI
6

7. Evolution of SDA
TPA 1.0 TPA 2.0
2003 2006
Project Project
Reports Activity/task Portlet
Dashboard Summary
…
News Calendar
Web Interface
Business Layer
TieFlow Process
TieFlow Process
Automation Engine V1 Rules Portal
Automation Engine V2 Engine V1 Framework
Database Repository
Software Process Automation (SDA)– SBIR Phase II and III
NASA Infusion Awards – SDA 2007-2012
SDA used by Tietronix on most software projects
CCB Automation – 600 Users – 3 NASA Sites
DCA – Budget App – Widely used at JSC
7

8. Tietronix Process Architecture 3.0
TPA 3.0
2008
Project
Project Activity/task
Reports Summary Notify
Dashboards
News Calendar
SOA/Web
Services
Business Layer
TieFlow Process
Automation Engine
Portal
MS Project Rules Engine
Graphical Workflow Framework
CM tools Editor
SVN/CVS
TieTracker 3rd Party
Tool Repository Services
Integration Process
PAL Artifacts CM Rules
Harness Library
8

9. NASA Class A – Software Process
Example Process
EA-WI-025 – GFE Flight SW/Firmware
230 Activities
SDA Guides/Leads team through the process
9

10. Iterative Process
10

11. SDA - Process Management
Any SW Process Processes Enactment Tool
Document Templates
Examples & References
Individual ToDo List
Task related Instructions
Each Activity
Each Team Member
11

12. SDA - Project Management
Multiple Project Task Assignments
Dashboard View
Reports
Project Status & Health
Audit Trail, etc.
12

13. SDA – Perform Work
13

14. NASA-JSC benefits from SDA
• Supports multiple processes and lifecycles for Mission
Operations and Engineering Directorates:
‒ MOD SMP processes based on the NPR 7150 class, safety criticality,
software type,…
‒ Waterfall, iterative and agile lifecycles
‒ EA-WI025: GFE Flight Project Software and Firmware Development
• Provides a wizard to assist team members and new
project owners in selecting Process adapted to project
characteristics
‒ Reduces time for decision making
• Support SCAMPI Appraisals by providing a streamlined
way to gather all required information
‒ reduced the cost of man hours necessary in the preparation of SCAMPI B
and Appraisals significantly
14

15. NASA-JSC benefits from SDA (cont’d)
• Spacecraft Software Engineering Branch/ER6 (SSEB) is currently
integrating all software projects into SDA
• Assist the projects in meeting all applicable project planning,
monitoring, and control requirements imposed by SMPs
• Provide greater visibility into projects
• Help in training novice developers
• Make it easier to coordinate large, complex, geographically
dispersed project
• Promote higher quality/less defects at each stage through fill-the-
blank templates
• Earlier notice of issues, problems and schedule deviations
15

16. CMMI Support
• Facilitate CMMI Implementation and Audit
– Provides painless enforcement of CMMI compliant process
– Organization wide consistent process execution
– Traceability of all work
• Reduced Effort to Achieve CMMI compliance
– Accelerate the compliance schedule
• Generation & Auto capture of CMMI specific metrics
• Reports – Detailing CMMI compliance
• Supports Continuous Process Improvement
– Project History and Metrics stored in DB
– Enable comparison with other projects
– Share Best Practices with other projects/departments
– Easily customize existing processes
16

17. Latest Features Additions
• Iterative Processes: • Process Editor
– Multiple iterations and decomposition of tasks – Creation and modification of processes
• Relationships between projects : – Browsing of SDA processes
– Parent-Child relationships – Publication of processes to SDA
– Sharing of documents between related – Simulation of processes in the Process
projects Editor
– Parent project have read-only access to child • PAL (Process Asset Library)
project
– Project observers
Improvements:
• read-only access to the project – PAL portlet outside of project scope
– User-requested addition and deletion of
• External project artifacts documents to and from the PAL
– Documents external to SDA may be added as – PAL Administrators control PAL content
project artifacts
– Definition of categories and association
• Multiple process standards with documents
– Processes mapped to multiple process • Usability and bug fixes
standards
– Compliance reports available for multiple
process standards
• MS Project integration improvements
– Import/export of non-SDA tasks
17

18. Latest Features Additions
• Features: • Infrastructure:
– Additional PAL Capabilities – e-Auth support for authentication
– Improve Process Editor – Upgrades of SDA third-party software
• Process Library for Editor – Section 508 Accessibility Compliance
– Formal document review mechanism – SDA Lite
• Document states
• Document relationships • Tools:
• Subscription/Notification for document changes – BugFlow improvements
– Additional Reporting – Tool content integration
– Backload project data for existing – Relate RIDs, Bugs, and Action Items
projects migrating to SDA
• Processes:
– Process Selection Wizard improvements
– WI-023, WI-035 (Engineering)
– Editable activity instructions and
– Scrum / Agile
background using Wiki
– Improve/add MOD Processes
– Improve integration with MS-Project
– Improve Project relationships
18

19. SDP Lite (ER WI-035)
19

20. SDP Lite: simplified process for
small projects
• Implement Design Elements
• Conduct Formal Code Inspection
• Perform Unit Testing
Completion of a single activity requires
performance of multiple tasks
20

21. SDP1 version 2 (ER WI-025)
Instructions for 3.4.4.1 “Implement Design Elements” activity:
The Development Team implements the project’s products, based on the requirements
and design. The implementation should include:
1. Implementing all of software safety design features and methods.
2. Following all applicable standards and recommendations defined in the Project Plan.
3. Following the organization polices and standards.
If the activities are broken down it is easier to
assign, easier to close and easier to monitor
compliance and project progress.
21

22. SDA Benefits for Project
• Ease of scaling from large to small and growing from
small to large are the key.
• Having different templates provides flexibility to serve
current needs of the organization
• Provides a jump start in your project:
– it has project organization that comes with the process,
– predefined set of activities,
– templates,
– CM and document repositories,
– web based
– easy to collaborate from different locations.
• The ability to keep all project artifacts in one place.
22

23. SDA tracks compliance
1.7.9 CMMI Mapping
This process activity:
- 2 Practice Areas (CM, PP)
- 5 Specific Practices
23

24. CMMI Compliance Report
For Process Step 1.7.9:
“Complete Project Plan”
CMMI
Artifacts/Objective Evidence
+ embedded history + comments
For SP 1.1 – Identify Configuration Items
- six (6) artifacts containing evidence:
Direct Evidence
& possibly Indirect Evidence
24

25. SCAMPI Audit Process observations
from SEI Method Description Document – SCAMPI A
1.4 – Obtain & analyze initial objective evidence SDA use
1.4.1 – Prepare participants
1.4.2 – Administer Instruments
1.4.3 – Obtain initial objective evidence Large Productivity Gain
1.4.4 – Inventory objective evidence
1.5 – Prepare for Collection of Evidence
1.5.1 - Perform readiness review
1.5.2 – Prepare Data Collection Plan Organizationally Simplified
1.5.3 – Replan Data Collection (if needed) Productivity Gain
2.1 – Examine Objective Evidence
2.1.1 – examine OE from instruments
2.1.2 – examine OE from Presentations
2.1.3 – examine OE from Docs Operationally Simplified
2.1.4 – examine OE from Interviews Productivity Gain
2.2 – Verify and Validate Objective Evidence
2.2.1 – Verify Objective Evidence Simplified
2.2.2 – Characterize Implementation of Model Practices
2.2.3 – Validate Practice Implementation
2.3 – Document Objective Evidence Partially Automated
2.3.1 – take/review/tag notes Productivity Gain
2.3.2 – Record presence/absence of OE
2.3.3 – Document Practice Implementation
2.3.4 – Review and Update the data collection plan
2.4 – Generate Appraisal Results
25

26. 26

27. Compliance Report Sample
No Not Partially Largely Fully
Standard Requirements Details
Mapping Implemented Implemented Implemented Implemented
SWE-040 Access to software
[Compliance : 0%]
products
SWE-041 Open source
[Compliance : 0%]
notification
SWE-042 Electronic access to
Source code
SWE-027 COTS, GOTS,
[Compliance : 100%]
MOTS, etc.
SWE-034 Acceptance Criteria [Compliance : 50%]
Partially
Legend: Fully Implemented Largely Implemented Not Implemented No Mapping N/A
Implemented
27

28. Process Institutionalization
From SEI - CMMI
Generic Goals Generic Practices
GG1: Achieve GP 1.1: Perform Base Practices
Specific Goals
Institutionalization implies that
GG2: GP 2.1: Establish an Organizational Policy
Institutionalize a GP 2.2: Plan the Process the process is ingrained in the
Managed Process GP 2.3: Provide Resources way the work is performed and
GP 2.4: Assign Responsibility there is commitment and
GP 2.5: Train People
GP 2.6: Manage Configurations
consistency to
GP 2.7: Identify and Involve Relevant Stakeholders performing the process. An
GP 2.8: Monitor and Control the Process institutionalized process is
GP 2.9: Objectively Evaluate Adherence more likely to be retained
GP 2.10: Review Status with Higher Level
Management during times of stress
GG3: Institutionalize GP 3.1: Establish a Defined Process
a Defined Process GP 3.2: Collect Improvement Information
GG4: Institutionalize GP 4.1: Establish Quantitative Objectives for the
a Quantitatively Process
Managed Process GP 4.2: Stabilize Subprocess Performance
GG5: Institutionalize GP 5.1: Ensure Continuous Process Improvement
an Optimizing GP 5.2: Correct Root Causes of Problems
Process
Institutionalization a cornerstone of CMMI
28

29. How to Achieve Institutionalization
• Organization wide Culture Adjustment
– Visible, Persistent & Genuine Display of Management buy-in
– Key people with strong Conceptual & Operational Process Skills
– Commitment to a process-centric lifestyle
– Continuous Improvement Expected – Everyone is involved
– Process Mentality Woven into Organizational Fabric – Survives all
Storms
• Organization wide Support
– Tools and Infrastructure supporting the methods, practices and
procedures
– Strong multi-level training, references, examples, and general support
– Internal Marketing – Process Models, Info/Results sharing, Community
– Diligent Feedback, Monitoring and Improvement Response
• Infusion into multiple Centers/Projects
– SARP funded Infusion Projects
29

30. SARP Infusion of SDA into the
Orion Software Oversight Process
• Process workflow development
– Technical Review Process in place
• Supports major review events
• Compliant with NASA standards
– Orion Software Development Oversight (SDO) Process assessed
• Some process elements have been identified
• Rapid customization will be required by a software System Manager (SM)
• Orion Tool Chain integration assessment
– Results were mixed; integration challenges remain
– Complete tool suite integration will be required to ensure SDA effectiveness
– All engineering information sources must be integrated
• SDA Oversight Process assessment
– Evaluated SM/SFM behaviors against SDA oversight capabilities
– Identified key integration concepts
30

31. Oversight Process Enactment
• Oversight Process Flows Implemented for Tech Reviews
• Based on NPR 7150.2 requirements and Oversight Review process.
31

32. Infusion Study Conclusions
• Software Development Oversight (SDO) process enactment via SDA
will rely on a process “building block” approach rather than on the
tailoring of existing processes.
– Stable SDA oversight processes will exist, e.g. major milestone reviews
– Majority of the insight/oversight work however is very fluid
– Emphasis is on applying scarce resources & skills to pre-empt issues
– SM’s will construct short-duration, high-intensity studies and sub-projects
– SDA “Process Editor” capability is now available; supports agile processes
• Tool and information integration is a key to SDA institutionalization
– Objective is “process transparency”
• SDA serves the needed tools and information according to the task at hand
• Push the process to the background
– Web-based tools can be integrated easily using I-Frames
– Integration of server-based or “installation” resident tools will be a challenge
– Access to engineering data located on other internal/external networks will
require additional investigation

33. SDA Use
• 9 NASA Projects – Engineering Directorate
– ARI
– LIDS Software
– Morpheus
– JEOD
– SAFER
– SPIP (software process improvement project)
• 5 NASA Projects – Mission Operations Directorate (MOD)
– PLATO
– Timeliner – Several Bundles
– RegenSafing
– ConFRM
– CWSyncR8toR9ConfigTest
• Non-NASA Projects – Medical Device Industry
– AlfaVue Server
– Micro Transponder – FDA Medical Device process
– DocControl (document management system)
– AlfaThermo Tablet
– All Tietronix internal projects
33

34. SDA Observations & Results
• SDA Concept well received
– Managers especially enthusiastic on automatic status capture
• Lots of feedback
– Usability
– Enhancement Suggestions
– Eerily upbeat
• Most Believe they are more Productive
– With good opportunity for improvement
• Management, Engineers – same side of fence
– Unusual lack of resistance
• Everyone a Process Critic, Knowledgeable Citizen
– Process awareness & understanding higher across the board
– Parallels a trend in the Business Intelligence (BI) market
• Next Generation BI products allow everyone to be an analyst
34

35. Value to NASA – observations
Compliance Governance Compliance Governance
Without process technology using process technology - SDA
(more) Manual coordination & effort:
• Executing project processes ….. correctly
• Determining & locating Objective Evidence
SDA tool :
• Handling Objective Evidence
Coordinating & recording the interactions of people,
• PIID creation & maintenance
information and process execution in a project context.
• More human interviews vs. audit trail & history
• guiding project/process work per SOP
• Figuring who, what, when, where
• on demand reports verifying compliance
• Collecting metrics
• Monitoring the process
• Improving the process
35

36. SDA Roadmap & Status
Roadmap Status
• Recent Additions • Core Lifecycle Processes
– Process Editor – End users – Waterfall, Iterative & Agile
– Tailored for projects
• Near Term
• Process Wizard (for MOD)
– Multiple Standards Mappings
– 3rd party upgrades – User Q & A
– Improved Process Mgt. – Lifecycle?
– NPR Class – A, B, …, H?
• Future
– Safety? …
– Globalization – time zones, languages – Auto generate/tailor Process
– Auto – Artifact Semantic Analysis
– Tool Integration Harness • Deployments
– Requirements Level Traceability – 19 NASA Projects
– 3rd party integrations – 10 Internal Tietronix Projects
• Application Lifecycle Management – 4 Medical Projects
– Other Technologies – EA, BI • Infusion
• Enterprise Architecture, Business Intelligence
– SARP Infusion to Multiple NASA
• Processes Centers
– SCAMPI, Process Improvement
36