1. NSC Audits & Assessments Workshop - September 9 - 10, 2009
National Aeronautics and Space Administration
Safety & Mission Assurance (SMA) Audits: Here to Help you
February 2011 PM Challenge
Cheevon (Mi-Mi) B. Lau
Director, NSC Audits and Assessments Office (AAO)
cheevon.b.lau@nasa.gov
(440) 962 – 3093

2. PM Challenge – 2011
2
Agenda

3. PM Challenge – 2011
Background: S&MA Audits
 OSMA/NSC AAO is responsible for conducting the
following:
– Institutional/Facility/Operational (IFO) Safety Audit:
– Independent verification that institutions, facilities, and operations at each Center,
Component Facility, and JPL are compliance with applicable Agency’s SMA
requirements as well as Federal, State, and local OSHA requirements
– Requirements Flow-down and SMA Engineering Design Audits and Assessments
(REDAA)*:
– Independent verification of SMA requirements flow-down to Centers, Component
Facilities, and JPL; Programs/Projects down to NASA contracts
– Independent evaluation of SMA requirements implemented on Programs/Projects for
system safety, reliability/maintainability, risk analysis and management
– Quality Audit, Assessment, and Review (QAAR)*:
– Independent verification that Center, Component Facility, JPL, and
Programs/Projects down to the suppliers are compliant with SMA quality
requirements
– Independent Assessments as requested
*REDAA and QAAR activities began in FY10 (result of restructuring the scope of the audit program);
Replaced the Institutional Programmatic Support (IPS) and Programmatic Audit and Review (PA&R)
3

4. Background: S&MA Audits (continue)
PM Challenge – 2011
 Before 2009,
OSMA/NSC AAO focus was on conducting audits and returning to
the Center every 2-3 years
 Starting in 2009,
OSMA/NSC AAO focus is to conduct audits
–and- to help the Center prior to and after the audits
4

5. 2009/2010 Improvements and Collaboration
PM Challenge – 2011
•2 year audit schedule: online
Increased collaboration with stakeholders
•Audit process defined/documented: online
COMMUNICATIONS •Customer feedback
•Annual Audit report: online
•Annual AAO Workshop: online
and improved audit processes
KNOWLEDGE TRANSFER •Audit data: online
ACROSS NASA •Best practices: online
•Audit cards: consistency, quality, etc.
•Restructured/renamed IPS and PA&R to have a
EVOLUTION: PROCESS clear focus and to line up with Program/Project
Lifecycle
IMPROVEMENT •Corrective Action Plan (CAP): content, feedback
•Real-time audit tool
•Agency Systemic Issues
ANALYZE AUDIT DATA – RISK •Action Plan: update quarterly
•Risk Factors
REDUCTION •Correlating mishaps to audit findings
•Repeat audit findings
5

6. AAO Organization: Personnel
PM Challenge – 2011
Cheevon (Mi-Mi) Lau
Audits & Assessments
Director
(NC)
Paul Boldon
REDAA * Brad Kaustinen
Robert Cross
Manager COTR
Analyst, Auditor
(located at HQ) (located at NSC)
(located at JSC)
Brian Jackson
QAAR* Michael Milbert
Manager Analyst, Auditor
(located at NSC) (located at KSC)
Arthur R. Lee
IFO Safety
Audit Manager
(located at HQ)
Note: For audits, Center and OSMA personnel comprise the team
* Replaced Institutional Programmatic Support (IPS) and Programmatic Audits and Review (PA&R)
6

7. PM Challenge – 2011
Yearly AAO Workshop (September timeframe)
 Key Focus:
– Identify the Potential Systemic Issues (noncompliances for a
specific requirement that occur across 3 or more Centers)
– Address potential solutions and develop action plans
• Session is co-chaired by OSMA requirements owner and
OSMA/NSC Audit Manager
– Discuss Best Practices tied to potential Agency systemic
issues
– Recommend list of Agency systemic issues to bring forward to
an appropriate Headquarters Forum
• Present to Headquarters in February and October
 In 2010, 130 Attendees from:
– All NASA Centers, including JPL
– Headquarters (OSMA, Procurement, OCHMO, OCE, OICMS,
OSI)
– NESC, NSC
7

8. PM Challenge – 2011
IFO Safety Audit: Analysis of the
Noncompliances
8

9. PM Challenge – 2011
IFO Safety Audit Findings: Analysis
 Reviewed audit data from FY2007 - FY2010
- 2010
- 2009
- 2008
- 2007
 Total number of noncompliances reviewed: 1,349
 Total number of noncompliances related to Potential Systemic Issues (PSI): 930 (69%)
 Majority of the IFO Safety Audit’s PSI were in Electrical Safety
9

10. PM Challenge – 2011
QAAR: Analysis of the
Noncompliances
10

11. PM Challenge – 2011
QAAR Audit Findings: Analysis
 Reviewed audit data from FY2007 - FY2010
- 2010
- 2009
- 2008
- 2007
 Total number of noncompliances reviewed: 244
 Total number of noncompliances related to PSI: 182 (75%)
 Majority of the QAAR’s PSI were in Inspection/Work Records
* NOTE: Previous audit data was removed from calculation
11

12. PM Challenge – 2011
REDAA: Analysis of the
Noncompliances
12

13. PM Challenge – 2011
REDAA (IPS Design) Audit Findings: Analysis
 Reviewed IPS design audit data from FY2007 - FY2009
- 2009
- 2008
- 2007
 Total number of design noncompliances reviewed: 15
 Total number of design noncompliances related to PSI: 6 (40%)
 Majority of the PSI in design was related to Probabilistic Risk Assessment (PRA)
13

14. 2009/2010 Potential Agency Systemic Issues and
PM Challenge – 2011
Action Items
2009 2010
Potential Agency Systemic Issues 44 21
Closed 7 2
Work in Progress 34 19
Not Addressed 3 0
Proposed for Elevation to 0 0
Headquarters Forum (action
required)**
Action Items* 56 25
Closed 30 0
Work in Progress 25 25
Not Addressed 1 0
*For each issue, an action plan was developed that has at least 1 action
item
**In October 2010, the Pressure Vessel Systems and Software
Assurance/Engineering concerns were addressed at the Headquarters’
forum as “Information Only” 14

15. PM Challenge – 2011
Backup Information
15

16. PM Challenge – 2011
Online Audit Information
 Integrated Audit Schedule available at http://secureworkgroups.grc.nasa.gov/ifo-
sa?go=333384, http://secureworkgroups.grc.nasa.gov/redaa?go=424238, and
http://secureworkgroups.grc.nasa.gov/qaar?go=434005
 Integrated Audit Schedule also available at
– OSMA homepage http://www.hq.nasa.gov/office/codeq
– Agency Calendar http://aci.nssc.nasa.gov/index.cfm?event=monthView
 Milestone Templates defining the audit process
http://secureworkgroups.grc.nasa.gov/ifo-sa?go=344812 and
http://secureworkgroups.grc.nasa.gov/qaar?go=440698
 All Centers (designated personnel) have access to their own and other Centers’
audit data as well as best practices (except JPL)
http://secureworkgroups.grc.nasa.gov/ifo-sa?go=z51072 ,
http://secureworkgroups.grc.nasa.gov/ips-audit?go=z65494,
http://secureworkgroups.grc.nasa.gov/redaa?go=z67204 and
http://secureworkgroups.grc.nasa.gov/qaar?go=z58387
 Annual Audit Report – audit and analysis results
http://secureworkgroups.grc.nasa.gov/nsc-audit-assmt-ows?go=393554
 Annual AAO Workshop – potential Agency systemic issues, work towards
compliance http://secureworkgroups.grc.nasa.gov/nsc-audit-assmt-
ows?go=z56696
16

17. Audit Trends: Electrical Safety
PM Challenge – 2011
(Based on Total Number of Noncompliances)
Percent/Number of Total Noncompliances by
IFO Safety Audit Functional Area Percent/Number of Total Noncompliances by IFO Safety
Audit Electrical Safety Functional Area
70.0% 61.1% / 824 35.0%
32.9%/271
60.0%
50.0% 30.0%
40.0%
30.0% 25.0%
16% / 216
20.0% 18.7%/154
7.56% / 102 7.2%/97 5.63%/76 20.0%
10.0% 2.52%/34 15.4%/127
0.0%
15.0%
Electrical PVS Facility LDE Mishap Fire 11.7% /96
Safety System Reporting Protection 10.2%/84
10.0%
Safety
5.3%/44
5.0% 2.1%/17
1.1%/9 .8%/7 0.6%/50.6%/50.6%/5
0.0%
Percent/Number of Total Noncompliances by IFO
Safety Audit Electrical Safety Functional Area with
Potential Systemic Issues across Centers by Clause
35.0% 32.9%/271
30.0%
25.0%
18.7%/154
20.0%
15.4%/127
15.0% 11.7%/96
10.0%
10.2%/84 ANSI – American National Standards Institute
5.3%/44 NEC – National Electrical Code
5.0% 2.1%/17 1.1%/9 NESC – National Electric Safety Code
0.0% IEEE – Institute of Electrical and Electronics Engineers
NFPA – National Fire Protection Agency
OSHA – Occupational Safety and Health Administration
17

18. Audit Trends: Electrical Safety
PM Challenge – 2011
Potential Systemic Issues
Potential Systemic Issues: Details by Clause/Paragraph
60
52
Number of Noncompliances
50 46
40
40 37 35 37 37
30 26
24
22 22
20
20 17 16
14 14 14 13
11 12 12
10 11 9
7 6 6 8 8 8
10 5 5 4 6 5 5 4 6
3 3 3 5 4 4 4 3 5 5
0
Percent/Number of Total Noncompliances by IFO Safety Audit Electrical Safety Functional
Area with Agency Potential Systemic Issues by Clause
35.0% Clause/Paragraph
32.9%/271
30.0%
.147(c)(5) Protective materials and hardware .147(f)(1) Testing or positioning 7.4.6 Certification Requirements 29 CFR 1910.147
25.0%
.147(c)(6) Periodic inspection .303 Installation and use 7.3 Plan/Implement-Safety Training Program 29 CFR 1910
.147(c)(4) Energy control procedure .269 18.7%/154
20.0% Safety of power gen, trans, and dist 3.6.1 Electrical Safety
15.4%/127 NEC
.147(c)(7) Training and communication .146 Employee protection-confined spaces 3.8.2 Hazardous Operations
15.0% 11.7%/96 3.17.4 Confined Spaces
.147(f)(3) Group lockout or tagout .335 Personal protective equipment NPR 8715.3
10.2%/84
.147(c)(3) Full employee protection 10.0% Flexible cords and cables
.305 8.3.1 Facility safety
5.3%/44 NFPA 70E
.147(e)(3) Lockout or tagout devices removal 5.0% Use of Equipment
.334 130 Electrical work hazards
2.1%/17
1.1%/9 NESC&IEEE
.147(b) Definitions to section .137 Electrical Protective Equipment 400 General requirements- electrical installs
0.0%
.147(f)(2) Outside personnel (contractors, etc.) .332 Training requirements 110 General requirements-electrical safety ASTM F496
.147(a)(1) Servicing and maintenance 110 Requirements for electrical installations 120 Electrically safe work
ANSI
.147(a)(3) Purpose 408 Switch boards and panel boards 110A Necessary description for safety signs
.147 Control of hazardous energy 400 Flexible cords and cables 180B11 Safety sign application
.147(f)(4) Shift or personnel changes 314 Electrical devices 146B Safety signs for battery room
.147(c)(2) Lockout/tagout 406 Receptacles, cords and plugs .445 Protective grounds
.147(a)(2) Application 300 Wiring Methods and Materials 7.1Dimensions
.147(d)(6) Verification of isolation 348 Flexible metal conduit .1-.5 Obeyance of Safety Signage
18