08448380779 Call Girls In Greater Kailash - I Women Seeking Men
How DOORS Helps JPL Get to Mars & Beyond
1. How DOORS Helps JPL Get to
Mars and Beyond
Best Practices in Metrics,
V&V and Traceability
Margaret H. Smith
Jim M. Grimes
Ross M. Jones
Trisha Jansma
Jet Propulsion Laboratory
California Institute of Technology
2. Outline
Overview of DOORS Use at JPL
Topic 1 Metrics
Topic 2 Verification and Validation (V&V)
Topic 3 Traceability
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 2
3. JPL use of DOORS at JPL
Statistics
– 800+ users
– 600+ users have taken our in-house DOORS training.
– 50 DOORS licenses
– 50 projects have used DOORS, 22 are active
– 15 years using DOORS, preceded by 8 years using in-
house TRACER tool.
– 5 years of a standard process with in-house training
– 2 System Analysts for daily project support
– 1 Full-time Equivalent (FTE) Systems Engineer for
defining the method, teaching and project
consultation
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 3
4. DOORS Provenance – According to JPL
Source code and User Guide were QSS Telelogic IBM Rational
distributed from NASA COSMIC site from DOORS DOORS DOORS
1990 to ~1995. Developed by the Cassini
Project and used by JPL and ESA (1996) (2000) (2008)
JPL JPL TRACER DOORS
TRACER mothballed adopted at
(1988) (1995) JPL(1996)
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 4
5. Sample Project Sizes
installed
Project # objects # links # Modules TTT website
size (GB)
DSN 98995 5709 125 1.9 Large legacy infrastructure project
SMAP 17278 13311 91 1.9 Medium complexity earth orbiting mission
GRAIL 36031 30650 123 2.7 Smaller complexity lunar mission (JPL managed)
SES_JUNO 84844 43526 227 3.6 Large complexity deep space mission (JPL managed)
MSL 25641 5808 106 1.2 Flagship mission
# Modules Measurements were generated by
the JPL Trace Tree Tool (TTT)
0 50 100 150 200 250
MSL
SES_JUNO
# links
GRAIL # objects
# Modules
SMAP
DSN
0 20 40 60 80 100 120
# Links, Objects (Thousands)
Note: Number of objects includes all objects in formal modules: table cells,
headers, regular objects with or without text or OLE content. The number of
requirements is about one-third the number of objects.
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 5
6. Why Have a Standard Process?
• DOORS is a general purpose tool and there are many possible ways
to use it.
• For the first 15 years we used DOORS, each new project reinvented
the wheel.
– Big projects could afford to invent their own method and have internal support.
• We learned a lot from what the big projects did wrong and did right.
– We now have more small projects and fewer big projects.
• Small projects cannot afford to reinvent. Need a ready-made solution.
• The most common mistake: trying to run the old document-centric
process in DOORS and not taking advantage of the tool’s
capabilities.
• Projects were making mistakes, blaming DOORS, and delaying
import of requirements into tool as late as possible.
– Managing requirements in spreadsheets and not getting the benefits of early
sharing and collaboration.
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 6
7. What Has and Hasn’t Been Standardized?
Has
• Ownership, allocation and negotiation mechanism
• Attributes and their definitions – for requirements and V&V planning/tracking
• Views – requirements traceability, allocation status, requirements to V&V traceability
• High-level requirements template, boilerplate material
• Scripts – Requirement Maturity Metrics, and V&V Burndown and Statistics, and numerous housekeeping
scripts used by DOORS System Analysts (SAs).
• Tools – Trace Tree Tool for graphically displaying individual requirements traceability and project
requirements flow-down.
• Practices – the JPL Systems Engineering practices that culminate in requirements.
• Licensing and Support – the JPL institution purchases DOORS licenses for use by all projects and funds two
Systems Analysts (support).
• Training/Deployment – Engineers who are involved in requirements work attend a JPL developed course
that teaches basic DOORS skills and the JPL standard method for development requirements and tracking
verification in DOORS.
Hasn’t
• Requirements levels and element decomposition
• Detailed requirements templates
• DOORS collaboration method with partners and subcontractors
We owe the success of the JPL Institutional Requirements and V&V Tracking process and training
program to Richard Stoller (retired), and Karen Boyle (who sadly passed away in February, 2011).
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 7
8. Topic 1 Metrics
• Maturity Definition:
+ V&V Method/Approach defined
+ Rationale defined
+ Linked to parent(s)
+ Links exist to this requirement from Satisfiers (including V&V activities)
+ Satisfier (lower level owner) accepts allocated requirement
– TBD, TBR, TBCs
• Requirement Maturity Metric script:
– Implemented in DOORS Extension Language (DXL)
– Generates per module statistics across all modules in a Project, one module per
row
– Produces .csv file of statistics.
– Project can customize output to suit their Project Management/reporting needs.
– Provides a quick summary of requirement progress and facilitates comparison
between modules.
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 8
9. Example
Unknown Allocations
capability? Gold undecided or under
plating? negotiation
3
219
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 9
10. What Has Changed Today?
Daily email to Project’s requirements owner
Assessing Requirements activity and churn
Module name
Requirement number
Type of change
Requirements are marked Modified if any
attribute, with the setting Affect change dates
turned on, has been changed since the last time
this report was generated.
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 10
11. Topic 2
Verification
Planning and Tracking
Other benefits:
• JPL Projects have • Lower tooling cost–
use existing
been recording
institutionally
Verification supported tool and
information in support
DOORS for the last
10 years. • Enables useful views of
requirements, verification
• Initial benefit -- requirements planning & status, and
changes were immediately visible verification events.
to engineers responsible for • Projects can use as-is or customize a
verifying requirements because generic DXL script to generate
verification activity artifacts and verification statistics and graphics.
verification events were now
linked directly to requirements • Saves projects money through reuse of
tooling and method.
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 11
12. Verification Process
Project
Verification
Planning
Get DOORS Verification Verification Verification
Training / Activity Activity Status
consulting Planning Execution Reporting
Verification Execute VA
Activity Groups procedure, analyze Generate V&V
Requirements defined results and determine burndown charts
Development status and statistics and
present at
Reqmts to VA Record status of management
mapping the VA (procedure) meetings
captured run and create
Develop V&V
method, venue problem/failure
and approach reports (if needed)
Schedule VA Take corrective
activity action (if
completion (need necessary)
by) date
Cognizant person
approves VA
execution result
Plans and procedures
developed outside
DOORS and hyper- Cognizant person
linked from the DOORS approves that the
VA requirement has
been verified
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 12
13. Verification Elements
Definitions
VI – Verification Item, is anything that needs to be verified. All requirements
become Vis when they are being verified.
VA – Verification Activity. The testing activities described in a single Test
Plan, Procedure or Analysis. A group of associated VAs are a VA Group (VAG) Test Data Directory
1
VE – Verification Event. A run or execution of a VA.
1
0..1 Test
Test Plan Activity Report (AR)
1
1..M Procedure
0..1 1
In Other Tools 0..M
In DOORS
1 1 1
1..M 1
Verification Item (VI) Verification Activity (VA) Verification Event (VE)
1..M 1..M
VE data elements originate in the AR
Verification Activity and are pushed to VE to complete
the status picture.
Group (VAG)
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 13
14. Verification Attributes
Test Data Directory
1
Test Plan 0..1 Test Procedure 1
• version 1..M • version Activity Report (AR)
0..1 0..M
1 1
VA 1
VI or ITL item • activity name or activity 1
•object identifier (ID) group name
•statement of requirement • VA priority VE
•rationale • VA status • VA id (a reference)
•…. 1..M • VA owner 0..M • …. data imported from AR
•method 1..M • venue • url of Activity Report
1
•verification approach • date scheduled (for • url of test data directory
•audit status completion)
•audit completed on (date) • date completed
•audited by • plans and procedures
• PFRs
Requirements
V&V Activity Planning V&V Activity Execution
Development
V&V Status Reporting
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 14
15. Project Module Configurations
VIM – Verification Item (Requirements) Module
VAM – Verification Activity Module
VEM – Verification Event Module
VIM VAM
Multiple VAMs reflecting the
VIM VIM VAM VAM VEM Project’s requirements
structure or V&V organization.
VIM VIM VIM VAM VAM VAM
VIM
A single VAM for the Project’s
complete requirement tree
VIM VIM VAM VEM • partition VAM to give owners
granular write access
VIM VIM VIM
VIM
Combined VAM and VEM for
projects that only need
VIM VIM VAM / VEM information on the latest
run/execution.
5/24/2012
VIM VIM VIM Caltech – Jet Propulsion Laboratory M. Smith - 15
16. Verification Item (VI) Burndown
250
Current date
Not Confirmed
Number of Items to be Verified
200
Actual
(auto status)
A pick list allows selection of
150
any group of VIMs for this
chart.
100 Scheduled
Not Scheduled
50
0
4/1/2008
1/8/2008
2/5/2008
3/4/2008
7/8/2008
8/5/2008
9/2/2008
12/9/2008
1/6/2009
1/22/2008
2/19/2008
3/18/2008
4/15/2008
4/29/2008
5/13/2008
5/27/2008
6/10/2008
6/24/2008
7/22/2008
8/19/2008
9/16/2008
9/30/2008
10/14/2008
10/28/2008
11/11/2008
11/25/2008
12/23/2008
Count of VIs (modules included selected from pick list):
Scheduled
Scheduled = total number of VIs remaining to be completed per schedule
Not Completed = total number of VIs remaining to be completed actually
Actual Interval End Date
Not Confirmed = total number of VIs remaining to be Confirmed-Final
Not Confirmed
Not Scheduled = total number of VIs remaining to be completed but not scheduled
5/24/2012 Caltech – Jet Propulsion Laboratory R. Stoller, K. Boyle - 16
17. Verification Activity Burndown
16
Actual
14
Number of Activities to be Verified
12
Scheduled
10
8 Not Scheduled
6
A pick list allows selection of
4 any group of VAs for this
chart.
2
0
Scheduled Scheduled - Count of notof VAs remaining to be completed per schedule
= total number completed linked VAs scheduled to be completed
Actual Not Completed - Count ofof VAs remaining to be completed actually
= total number not completed linked VAs remaining to be
completed (Actual)
Not Scheduled = total number of VAsnot scheduled completed and are not
Not Scheduled - completed-NF or that are not
scheduled
5/24/2012 Caltech – Jet Propulsion Laboratory R. Stoller, K. Boyle - 17
18. Verification Item Statistics "Bad" if > 0
"Neutral"
"Good"
from V&V Burndown Script that generates data to Excel "Normal"
Count of VIs in VIMs (shall statements only)
A: B: C: D: E: F: G: H: I: J: K: L M: N:
VIs VIs VIs VIs Not VIs VIs Not VIs Not VIs Not VIs Not V&V Engr
VIM Linked to Linked to Linked to VIs Auto VIs Conf Linked to Linked to Linked to Auto Conf Verif and Responsible
NAME Total VIs VAM VAGs VAs Verif Verif VAM Reject a VA Verif Verif Late for this VIM
VIM1 625 600 15 575 200 150 25 10 50 425 450 5 John
Jones
VIM2 500 500 10 485 100 75 0 5 15 400 etc 5 Susie
Jones
VIM3 1000 1000 0 1000 0 0 0 0 0 1000 etc 0 Mary
Smith
… … … etc etc etc etc etc etc etc etc etc etc
VIM n 200 200 5 190 25 5 0 5 10 175 etc 0
Total 2325 2300 30 2250 325 230 25 20 75 1975 etc 10
Relationships: B=C+H; I is not included in D; E=C-I-D; J=B-E=H+I+D; K=B-F; L=B-G
M = VIs not auto verified AND scheduled completion date is missed or has not been scheduled or is not
linked. Any object containing the word “shall” enclosed in quotes (“shall”) is not included in any statistic.
5/24/2012 Caltech – Jet Propulsion Laboratory R. Stoller, K. Boyle - 18
19. Verification Activity Statistics "Bad" if > 0
from V&V Burndown Script that generates data to Excel "Neutral"
"Good"
"Normal"
# of VI Are VAs Fully ** Is there a VA Are there any Schedule Days
VA Parent Linked to VIs VA Sched- Plan &/or a Suspicious VI Remaining (if not V&V Engineer /
NAME* Links ? VA Status uled? Procedure? Links? completed) Owner of VA
VA-x 25 Yes Blank 08/10/09 No Yes 10 Jack Jones
VA-y 20 Yes Completed-NF 07/26/09 Yes No -5 Mary Jones
VA-z 100 Yes Completed No Yes No Susie Que
… 350 … Blank No No No Not Sched Tom Smith
VA-n 55 Yes Blank 08/20/09 Yes Yes 20 Susie Smith
* VAs are organized in VAM under VAGs and sub-header VAGs, but tabulated separately here.
** Identifies if there is an http entry in the Plan and Procedure attribute.
5/24/2012 Caltech – Jet Propulsion Laboratory R. Stoller, K. Boyle -19
20. Topic 3 DOORS Trace Tree Tool (T3)
JPL has parent-to-child requirements relationships where a parent has 1 to tens of
children, and links between levels can be many-to-many. We have trouble seeing the
forest for the trees and navigating requirements flow within DOORS.
• Primary Motivations
– Provide JPL users with an alternative graphical presentation of
requirement flow (links)
– Allows navigation of requirements based on hierarchy or flow
– Enables seamless navigation between modules following flow-
down/links
– Provides easy access via Web-based, platform independent interface
• Secondary Motivation - evolved with implementation
– Provide an indication of timeliness of requirements flow-down
– Provide an efficient way to checkout verification planning
• History
– Conceived and developed by a JPL Software Systems Engineer (James
Grimes) in 2006
– Enthusiastically used by 8 JPL Flight Projects
5/24/2012 Caltech – Jet Propulsion Laboratory J. Grimes - 20
21. Trace Tree Tool View – Example Index Page
ProjX
ProjX
ProjX
ProjX
ProjX
•••
5/24/2012 Caltech – Jet Propulsion Laboratory J. Grimes - 21
22. Project Map
Number of links/flowed-down requirements appear on edges
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 22
27. Typical DOORS T3 User Scenarios
• Quickly obtain the contents of a requirement with a known ID
– Main page Module page Requirement number
• Scan a particular Project module for a specific requirement
– Main page Module page, then
• Scroll up and down at web browser speeds
• Use browser’s find capability (not as flexible as DOORS, but faster)
• Trace requirements flow-down starting at any requirement
– Click on boxes in graph going up- or downstream in requirements flow
– Go from there to text, attributes
• Obtain immediate visual indication of timeliness of flow-down
– E.g., has upstream requirement X been updated more recently than my
requirement? If yes, then it’s red. Actual date is shown on X’s page.
– Is flow-down complete? Does the requirement have a parent?
• Provide supportive evidence during Project Reviews
– Quickly show existence of flow-down
– Quickly show V&V attribute, implementation status (if in attribute)
5/24/2012 Caltech – Jet Propulsion Laboratory J. Grimes - 27
28. DOORS Trace Tree Tool (T3)
High-Level Process/Execution Flow
5/24/2012 Caltech – Jet Propulsion Laboratory T. Jansma - 28
29. Development Tools Used
• DXL – DOORS eXtension Language, a DOORS proprietary C-like, object-
oriented language available within the DOORS client
• Perl – a scripting language widely available on UNIX, Mac OS X, Linux,
Windows
• Graphviz – open source Graph Visualization software developed at AT&T
that includes the Dot language and dot program for drawing directed
graphs
• XML and Perl/C-based XML parsing software – open source software
• Java for processing images
• Standard UNIX utilities such as date, rm, ls, rsync, etc.
• SVN – Subversion open-source version control tool
5/24/2012 Caltech – Jet Propulsion Laboratory J. Grimes - 29
30. JPL’s Direction with DOORS
• Systems Engineers on JPL Projects are starting to infuse Model Based
Systems Engineer / SysML.
– Modeling includes architecture, behavior and resources.
– Many pockets of discipline specific modeling tools exist in the Spacecraft
subsystem organizations
– Vision is to use a central, standards based repository to interconnect and
exchange model data.
– Requirements will still be important but there is now an opportunity to
augment requirements with more extensive and formal designs.
• Small and technology development Projects may need a lighter weight
DOORS-based process.
Aimed at
• JPL is working on more partnership projects. Module exchange needs to be influencing,
negotiated and piloted early. sharing
Lessons
• We are piloting a Requirements Architecture, table-top review that will Learned and
occur before Project and System requirements are developed. emerging best
practices.
• We have begun teaching just-in-time DOORS and process classes directly
to and with Flight Projects.
5/24/2012 Caltech – Jet Propulsion Laboratory M. Smith - 30