DSPy a system for AI to Write Prompts and Do Fine Tuning
Jeff davis
1. A System for Innovation
Jeffrey R. Davis, M.D.
Director, Space Life Sciences
Elizabeth E. Richard
Senior Strategist, Wyle
Project Management Challenge
February 9-10, 2010
Exploring Space, Enhancing Life
Used with Permission
2. Presentation Overview
• Background
• The System for Innovation
– Human System Risk Management
– Collaborative Innovation
• Human System Risk Management Process
– Standards to deliverables
– Master list of human system risks
– Risk Mitigation Analysis Tool
– Human System Risk Board
• Driving Innovation through Collaboration
– SLSD Strategic Plan
– Why collaborate?
– Critical success factors, best practices, and challenges
• Future Work
Exploring Space, Enhancing Life
2
3. System for Innovation
• The Space Life Sciences Directorate’s
(SLSD) system for innovation involves two
key components:
– 1) an evidence-based risk management system that
continuously evaluates all human system risks across
current and future operations, and identifies gaps in the
research, technology, operations and service portfolios, and
– 2) a strategic system to drive innovation through strategic
alliances and collaboration to optimize SLSD research,
technology, operations, and service portfolios.
• SLSD’s change process was stimulated by internal
and external changes to the environment
Exploring Space, Enhancing Life
3
4. Change Drivers
• 2001, Institute of Medicine Safe Passage
published
• 2003, CAIB Report issued
• 2004, Vision for Space Exploration released
• 2005, Exploration Systems Architecture Study
• 2005, Institute of Medicine Report regarding the
Bioastronautics Roadmap
• 2006 NASA Strategic Plan
2/19/2010 Exploring Space, Enhancing Life
4
5. Institute of Medicine Recommendations
• From Safe Passage: Astronaut Care for
Exploration Missions
– Develop and use an occupational health
model for the collection and analysis of
astronaut health data, giving priority to the
creation and maintenance of a safe work
environment
Exploring Space, Enhancing Life
5
6. CAIB Report Recommendations
• Establish an independent Technical
Authority, with responsibilities that include:
– Develop and maintain technical standards
– Sole waiver-granting authority for all technical
standards
– Conduct integrated hazard analyses
– Review and approve research deliverables
Exploring Space, Enhancing Life
6
7. Institute of Medicine Recommendations
• Review of Bioastronautics Roadmap
– Incorporate an evidence-based risk assessment
and communication process into the risk
identification and reduction approach
– Risks and mitigation strategies should be
represented separately for assessment and
comparison
Exploring Space, Enhancing Life
7
8. Human System Risk Management
• Human System Risk Management
Process
– Standards to deliverables
– Master list of human system risks
– Risk Mitigation Analysis Tool
– Human System Risk Board
Exploring Space, Enhancing Life
8
9. Standards to Deliverables Process
• Standards to deliverables first developed 2005-
2006: first response to environmental changes
• Standards - implement health and medical policy
– The first step in risk mitigation is establishing standards
to define an acceptable level of risk
– Standards are based on the best available information
and evaluated against the space flight environment
– Operational experience is assessed to inform the
standards
– Research projects are defined to fill the gaps in
knowledge
• Deliverables
– Desirable solutions to research and/or technology
questions that mitigate human system risks
Exploring Space, Enhancing Life
9
10. Standards to Deliverables Process
2005-2006
Vision for Exploration
NASA Strategic Plan
Human Exploration Human Rating
Strategy Requirements
Health and Med Policy
Procedural Rqts
Human System Exploration
Standards Requirements
Research
Requirements
Deliverables
•Ops Experience
•Countermeasures
•Knowledge
2014 2015 2025 2030
Crewed CEV Human Lunar Human Expl Human
Exploration Mars Vicinity Exploration of
Transportation to LEO Mars
2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 .... 2025 ... 2030
10
11. Risk Management:
An Evidence Based Approach
• A risk management system was needed to implement
the standard to deliverables process
– A master list of 90 human system risks was identified
– A rigorous evidence-based system to mitigate the
approximately 90 human system risks for space is tracked and
monitored through a Risk Mitigation Analysis Tool (RMAT)
– A flight activities control board and human system risk board
manage all standards development, risks, and mitigation
approaches to the 90 human system risks
– When solution to mitigate risks are not readily available from
the research and technology portfolios, gaps are identified that
must be closed for successful risk mitigation
– Gaps are evaluated for closure through innovative,
collaborative approaches
Exploring Space, Enhancing Life
11
12. Risk Management:
An Evidence Based Approach
• Need to regularly collect/evaluate the accumulated
data and evidence from space flight operations and
space and ground-based research
• Use evidence base to make recommendations to
create/alter health and medical human system
standards and/or requirements
• The Human Research Program discipline reviews
began the review of the best available evidence in
2006-2007
• Evidence review is now weekly in the Human System
Risk Board (HSRB began April 2008)
Exploring Space, Enhancing Life
12
13. Developed Master List of
Human System Risks
• Collected all risks currently being worked in SLSD
organizations
• Re-worded to reflect proper risk statement: clear
statement of condition and consequence
• Categorized by threat
– Hypo, hyper gravity environments
– Closed and hostile environment
– Remote deployment
• Modified by missions
– Length
– Vehicle configuration
– Distance from Earth
Exploring Space, Enhancing Life
13
14. Example from Master List Applicable Responsible
ID Disposition RISK STATEMENT Mission Timeframe Program
Risk of Inability to Adequately Treat an Ill or Injured
Crew Member
Given that there is limited mass, volume, power, and
crew training time which therefore limits the amount of
equipment and procedures available to treat medical Lunar Outpost
7 Risk problems, there is a poss Mars During Mission HRP
Risk of Compromised EVA Performance and Crew
Health Due to Inadequate EVA Suit Systems
Given the possibility that EVA Suit Systems are
inadequate, the crew may be unable to perform mission
objectives.
Lunar Outpost
8 Risk Given the possibility that EVA Suit Systems are inad Mars During Mission ESPO/HRP
Risk of Error Due to Inadequate Information
Given the condition of lack of adequate information CEV to ISS
available to the crew, there is a possiblity that operator Lunar Sortie
error could occur on critical tasks resulting in failure to Lunar Outpost
11 Risk perform assigned tasks within required Mars During Mission HRP
Risk of Behavioral and Psychiatric Conditions
Given that behavioral issues are inevitable, there is a ISS
possiblity that psychiatric conditions will occur on long Lunar Outpost
12 Risk duration space flight. Mars During Mission HRP
Given that the Intra-Vehicular Charged Particle
Directional Spectrophotometer has Exceeded its
Expected Lifetime, thus limiting our ability to monitor
the radiation environment, there is a probability that
34 Risk crews will exceed career radiation exposure limit ISS
14
15. Risk Mitigation Analysis Tool (RMAT)
• Collects the current status of work in developing and
addressing compliance with standards for identified risks
and/or risk factors
• Assesses the need for mitigation to stay within the standard
for a particular mission
• Facilitates documentation and base lining of the risk
management approach - allows us to assess gaps and
provide traceability of our work
• The tool flows from a standard to a deliverable for a
particular mission
• Progress is assessed at the Human System Risk Board
Exploring Space, Enhancing Life
15
16. Radiation Carcinogenesis: Risk Management Analysis Tool
Architectures
CEV ISS ISS Moon Moon Mars
(CEV to ISS) (6 Months) (1 Year) (<14 days) (Lunar Habitat)
Has the risk factor been Y Y Y Y Y Y
verified? (Y/N)
Probability of the adverse *PRA calculations: *PRA calculations: *PRA calculations: *PRA calculations:
outcome (without current Females 0.7%, Females 1.3%, Females 1.5%, Females 5.0%,
or proposed mitigation) Males 0.6% Males 1.2% Males 0.8% Males 4.5%
Uncertainty associated 0.3-1.9% 0.5- 4.0% 0.5- 5.0% 1.5-18%
with outcome 0.2-1.5% 0.4- 3.0% 0.5- 4.0% 1.5-15%
1) individual, acute 1) individual, acute 1) individual, acute 1) individual, acute 1) individual, acute
Impact of the Adverse health health health health health
Outcome 2) mission impact 2) mission impact 2) mission impact 2) mission impact 2) mission impact
3) program impact 3) program impact 3) program impact
1) shielding 1) shielding 1) shielding 1) shielding 1) shielding 1) shielding
Mitigation, Current and/or 2) monitoring to 2) monitoring to 2) monitoring to 2) monitoring to 2) monitoring to 2) monitoring to reduce
Proposed reduce SPE reduce SPE threat reduce SPE threat reduce SPE threat reduce SPE threat SPE threat
threat 3) maximize 3) maximize surface 3) maximize surface
surface time to use time to use planetary time to use planetary
planetary shielding shielding shielding
Probability of the Adverse
Outcome (with
current/proposed
mitigation)
1) vehicle 1) cost impacts 1) cost impacts 1) vehicle design 1) vehicle design 1) vehicle design
Cost/Benefit Trades design impacts 2) mission planning 2) mission planning impacts impacts impacts
(Including risks of 2) cost impacts impacts impacts 2) cost impacts 2) cost impacts 2) cost impacts
3) mission 3) mission planning 3) mission planning 3) mission planning
mitigation)
planning
1) optimize modeling 1) optimize 1) optimize modeling 1) optimize modeling
Current Work technique modeling technique technique technique
2) refine confidence 2) refine confidence 2) refine confidence 2) refine confidence
interval interval interval interval
Describe appropriate Describe mitigation Revision of the Revision of the standard
Future Work mitigation for lunar for Mars mission; standard to conform to conform with ALARA
habitat mission transit, & surface with ALARA
operations
16
17. Human System Risk Board
• Established at JSC – April 2008
• Working group develops and brings to the HSRB
– RMAT
– Presentation of evidence base
• HSRB chaired by JSC Chief Medical Officer (CMO)
– Manager, Human Research program is a voting member
– Divisions, technical specialties, operations and research
– Ames, Glenn, KSC, Langley and KSC participate
– Stops, starts and modifies research
– Identifies gaps and assigns actions for research and technology
development
– Recommends updates to standards and transition to operations of
new findings
Exploring Space, Enhancing Life
17
18. Human System Risk Board
• Future work
– ISS international partners invited Jan 2010 (International
Space Medicine Summit III recommendation)
• Will participate quarterly
– Develop RMATs for all 90 human system risks
– Recommend future work for all gaps to a development
control board to try new collaborative approaches
– Risks can be modified to mission requirements
• Suborbital to Mars
– An example of organizational innovation
• Operations and research integrated
• Best available evidence continuously reviewed
• Gaps are evaluated for closure through innovative
approaches
Exploring Space, Enhancing Life
18
19. Driving Innovation Through
Collaboration
• Driving Innovation through Collaboration
– SLSD Strategic Plan
– Why collaborate?
– Critical success factors, best practices, and
challenges
Exploring Space, Enhancing Life
19
20. Driving Innovation Through
Collaboration
SLSD Strategic Plan 2007
• Four strategic goals:
• Manage balanced internal/external portfolio
• Drive health innovations
• Drive human system integration innovations
• Education and outreach
• Key strategies for implementation
• Alliances (supported by benchmark report)
• Risk management and research/ops integration
• Open collaboration/innovation
• Support commercial space for LEO
Exploring Space, Enhancing Life
20
21. Why Collaborate?
• Collaborative innovation and alliances are used to:
– Innovate quickly and cost effectively
– Improve quality of innovation
– Address technical challenges in a rapidly changing
environment
– Enhance portfolios by supplementing internal core
capabilities with external capabilities
– Remain competitive (for profit or funding)
– Transfer knowledge
– Share risks
– Grow and develop market
– Leverage brand name
“Alliances are crucial to driving growth and innovation”
Exploring Space, Enhancing Life
22. How Alliances and Collaborations
Drive Innovation
• Transfer of compatible competencies
• Joint research and development
• Access to new technologies
• Accelerated research
• Knowledge acquisition
• Variability resulting in novel ideas
• New applications for existing technologies
• Shared investment/cost efficiency
• Facilities sharing resulting in enhanced innovation
“Can we get there faster with a partner than by
ourselves?”
Exploring Space, Enhancing Life
23. Enabling Collaborative Innovation
• Critical Success Factors and Best Practices
– Demonstrate senior management commitment and
support
– Identify and manage cultural issues
– Embed strategy into the overall strategic planning process
– Adopt a structured approach for implementation
– Establish a support structure to enable and facilitate
alliances and collaborative innovation
– Enhance communications
• Internally: culture, skills, awareness, flexibility
• Externally: improving ways to connect to outsiders
– Institute a reward system aligned with strategy
Exploring Space, Enhancing Life
24. Optimizing Innovation Requires a
Deliberate, Structured Approach
• Create a unifying vision and clearly defined goals and
objectives
• Evaluate strategy and objectives annually
• Conduct gap analysis to determine areas of greatest need
• Conduct core competencies assessment to determine the
greatest opportunities for partnerships
• Clearly articulate and communicate gaps/innovation needs
• Assess and map gaps/needs to optimal collaboration mode
• Establish processes to scout for ideas and scan/access the
external environment for emerging technologies and
collaborations
• Pursue open innovation models and/or strategic alliances
based on overall assessment
Exploring Space, Enhancing Life
24
25. Challenges
• Hurdles to creating a collaborative environment and
fostering innovation
– Conflicts between sourcing internally v. externally
• “Not Invented Here” syndrome
• Risk/change averse culture
• Limited interaction with potential resources/problem solvers
– Limited external network (serendipitous
partnerships/opportunities)
– Bureaucratic barriers to collaboration (policies, regulations,
security)
– No overarching process to analyze gaps at
strategic/organizational level
– Misaligned job expectations and/or performance evaluations
Exploring Space, Enhancing Life
26. Results to Date
• Strategy Execution and Implementation Office 2008
• Human System Risk Board April 2008
– Multiple centers participate (Ames, Glenn, KSC,
Langley, HQ)
– Expanding to international and academic partners 2010
– Evidence driven, manages all 90 human system risks
• Rice Business Plan Competition for Earth/Space Innovation
prizes began 2008
• Ongoing student projects with Dr. Karim Lakhani, Harvard
Business School (HBS)
– MBA Project: Leveraging Collaborative Innovation for SLSD
April 2009
– Collaboration/innovation course winter 2010
Exploring Space, Enhancing Life
26
27. Results to Date (cont.)
• Alliances Benchmark Report July 2009
– Implementing recommendations and exploring partnerships with GE
and Philips based on best practices
• Portfolio mapping July 2009 – Dr. Gary Pisano (HBS)
– Determine best collaborative strategy to address portfolio gaps
• Open Innovation Service Providers/Competition September 2009
– TopCoder (ongoing, 3500 lines of code to date)
– InnoCentive (training Nov 18-20, launch Dec 15)
– Yet2.com (training Dec 2-3, launch Jan 2010)
• Strategic communications/education on collaborative innovation
processes and benefits (ongoing)
– SLSD Innovation Lecture Series
– Lunch with Director
– Tagline: Exploring Space, Enhancing Life
– Weekly Newsletter
Exploring Space, Enhancing Life
27
28. Forward work
• Develop and execute a deliberate system for pursuing life
sciences alliances and collaborative innovation opportunities
• Restructure Boards Jan 2010
• Current board architecture manages 90 human system risks
• Gaps are identified and novel approaches considered through
development control board
• Assures knowledge management and lessons learned (flow chart
draft in backup)
• Continue to enhance communication
– Internally: Website 1Q 2010, Quarterly Focus Groups
– Externally: Website 1Q 2010: human system and innovation
needs/gaps, contact info re: partnering
• Continue to evaluate collaborative innovation models/new
ways of doing business
– GE – reverse innovation: smaller, lower cost technical solutions
for space flight
Exploring Space, Enhancing Life
28
30. IDENTIFY ANALYZE
Identify Human Analyze Human
Health & Health & Validated Risk SFHS
Master
Performance
List
Performance Acceptable Standards
Risks by Mission Risks by Mission Level
(V1, V2,)
Evidence
Base Analysis
Exploration Invalidated Risk Validated Risk
Data Collection: or Risk Factors Unacceptable
Performance, Medical, Mission
needing research Mitigation
Research, Terrestrial Operations
Concepts
Research Requirements
Research Data Research Reviews (Quantify/Substantiate)
Research
Requirements
(CM/Technology
Development)
Cx Vehicle
Requirements
Mission
Med Ops Data Countermeasure
Implementation
Env. Data
Medical Operational Program
Monitoring Requirements
(MORD, Flt Rules)
TRACK/ Environmental
CONTROL Monitoring PLAN
30
31. Space Flight History, Research, Lessons learned
Knowledge Capture
Knowledge Capture
3 Threats: μG, remote deployment, closed environment
Health and Medical Technical Authority
Standards
SLSD Ops HRP
(STS, ISS, Cx, Commercial Space) Integrated Research Plans
90 Human System Risks
Pre-board Working Group
FACB HSRB
↑ Continuous Process
↑ CPI/I Solution? Solution? Improvement Innovation
YES NO NO YES
↓ N/DI ↓ Novel/Disruptive
Gaps Innovation
SEIO (Mapping ; Novel Approaches)
Development Control Board
Open Innovation
Prizes
Commercial Collaborations