The document summarizes the work of a team called Sea++ over 10 weeks. In the first 3 weeks, they interviewed cyber engineers to understand barriers to testing ship navigation systems safely. They developed an initial MVP of a collaboration tool for commanding officers and engineers to plan cyber tests. In weeks 4-7, they sought to understand why integrated cyber testing does not occur on ships post-acquisition. They identified issues such as lack of tools, documentation, and cultural barriers. In weeks 8-10, they focused on what commanding officers need to know about cyber threats and developed a way to convey essential mission impact information to them. Going forward, they see opportunities to expand their work to other maritime sectors.

1. Ruben Krueger
BS CS
Software
Scott Pratz
MS EE
Strategy
Meilinda Sun
BS CS
Software
Ross Ewald
BS CS
Policy
Sea++
Original Problem:
Test engineers cannot
safety or adequately cyber
test ship nav systems.
99 interviews
Current Problem:
COs must understand
impacts of cyber
vulnerabilities to
missions.
Sponsor: MITRE (in collaboration with the Office of Naval Research)

4. Weeks 1-3: Understanding problem scope
Customer Discovery Objective: Interview cyber
engineers to learn why cyber testing is unsafe
and inadequate.

5. - MITRE Systems Engineer, Former USAF
“[Operational leadership] is often the
barrier to cyber testing, as they worry about
the safety of the crew and integrity of the
asset.”

6. Mission Model Canvas™ Week 1
Key Partners
NavSea Philadelphia
Cyber Command
Private Cybersecurity Firms
Shipbuilders
Commanding officers
Key Activities:
Understanding hardware
testing procedures
Developing evaluation criteria
Gaining trust and gauging
expectations of ship captains
Value Proposition
Development of physical
and electronic security
testing procedures.
Streamline testing
procedures to minimize
operational downtime
Target individual systems
for most useful testing
applications
Incremental aggression
capabilities to maximize
vulnerability detection.
Beneficiaries
Test Engineers
Commanding Officers
Key Resources
Safety parameters of cyber-
physical systems
Integrated navigation system
models/block diagrams
User Design Input
Buy-In / Support
Operational commanders for
conducting testing
NavSea/MITRE (Navy) and
C4IT (CG) to conduct testing
and evaluate effectiveness of
methodology
Deployment
Testing of methodology on
sample test plans
Full electronic software
deployment to customers via
web-app
Mission Budget / Costs
Fixed:
Software design & engineering
Testing and reporting costs
Variable:
Expansion of hardware testing capabilities
Travel
Mission Achievement
More aggressive testing taking place onboard ships
Adoption and continued use metrics of the methodology by engineers
Increased vulnerability detection on ships
User retention during testing lifecycle

7. Develop testing procedures
Target individual systems
Cyber Engineers
Commanding Officers
Mission Model Canvas™ Week 1
Value Proposition Beneficiaries
Streamline testing procedures
Incremental aggression capability

8. Initial MVP - Collaboration Tool Between COs and Engineers
Ship’s Personnel CONCERNS
Cyber Engineers/Testers TESTING RQMNTS
Sea++
Platform
Testbed Creation:
Environmental/
Cyber Monitoring
Requirements
Integrated
Navigation
System
Data
Identification of System Vulnerabilities
s
Hypothesis: Better
testing planning is needed
to improve safety.
MVP: Platform to engage
COs and engineers in the
planning of cyber tests
and to create a
corresponding testbed.

9. “There is no routine entity that tests cyber
security on ships.”
- Member of National Security Council, Navy Officer

10. What we had been doing:
“Can you please direct us to the group that does integrated testing?”
NAVSEA
SPAWAR
NAVAIR
OFFICE OF
NAVAL
RESEARCH
FLEET
CYBER

11. Weeks 4-7: Why isn’t anyone testing ships?
Customer Discovery Objective: Understand why
post-acquisitions integrated cyber testing does
not occur.

12. Nonstop Pivoting, Beneficiary Confusion
R&D
High-value assets may
be damaged
Why doesn’t post-acquisitions cyber testing occur?
Pentesters
Post-test remediation
is time-consuming
and inadequate.
Test Engineers
Inadequate
documentation for a
thorough test
Test Engineers
Integrated system of the
ship is too complex to
understand
Government
Officials
No tools to
understand the
impact of cyber
Commanding
Officers
Organizational issues,
cultural issues

13. Nonstop Pivoting, Beneficiary Confusion
R&D
High-value assets may
be damaged
Why doesn’t post-acquisitions cyber testing occur?
Pentesters
Post-test remediation
is time-consuming
and inadequate.
Test Engineers
Inadequate
documentation for a
thorough test
Test Engineers
Integrated system of the
ship is too complex to
understand
Government
Officials
No tools to
understand the
impact of cyber
Commanding
Officers
Organizational issues,
cultural issues

14. How can we make post-acquisitions cyber testing
happen in the Navy?
Commanding
Officers
3
Establish Office to Manage
Integrated Testing
Chief of Naval
Operations1
2
Naval Systems
Commands /
Fleet Cyber

15. How can we make post-acquisitions cyber testing
happen in the Navy?
Commanding
Officers
3
Establish Office to Manage
Integrated Testing
Chief of Naval
Operations1
2
Naval Systems
Commands /
Fleet Cyber
Empower Commanding
Officers to Demand Fix

16. Weeks 8-10: What does a commanding officer
need to know about cyber threats?
Commanding officers will address cyber threats
once they understand them.
Customer Discovery Objective: What does a
commanding officer need to know about cyber?

17. “To many commanders, a cyber threat is not
as real as a missile attack.”
- National Security Council Member, Naval Officer

18. Make cyber as real as traditional threatConcrete
Only relevant info about mission impactEssential
Clear path to remediationActionable
Communicating Cyber Impacts to Commanders

19. “I understand everything in terms of mission
impact ... I just want sh*t to work.”
- Navy Commanding Officer

20. Make cyber as real as traditional threatConcrete
Only relevant info about mission impactEssential
Clear path to remediationActionable
Communicating Cyber Impacts to Commanders

21. “It’s depressing when cyber people tell
us that everything is vulnerable. My
focus is not on assessing vulnerabilities
but on fixing them.”
- Naval Commanding Officer

22. Make cyber as real as traditional threatConcrete
Only relevant info about mission impactEssential
Clear path to remediationActionable
Communicating Cyber Impacts to Commanders

23. Make cyber as real as traditional threatConcrete
Only relevant info about mission impactEssential
Clear path to remediationActionable
Communicating Cyber Impacts to Commanders

24. Current Mission Model Canvas™
Key Partners
Equipment
Manufacturers
Naval Engineers
MITRE
Fleet Cyber
Chief of Naval
Operations
NavSea
Key Activities:
Development of software
and visualization
integration
Contract for acquisitions
Modifications for private
sector
Value Proposition
Understand how cyber
threats degrade missions
Understand how to
address cyber threats
Beneficiaries
Ship Commanders
Key Resources
Access to private sector
simulations
Buy-In / Support
Chief of Naval
Operations
Office of Naval Research
Ship commanders - Fleet
Cyber
Deployment
Contract for MVP written
and submitted to
acquisitions
Mission Budget / Costs
Development of MVP (materials)
Travel
Installation costs for MVP
Mission Achievement
Increased clarity of cyber-vulnerabilities leading to better decision-
making and increased testing

25. Understand how to address
cyber threats
Naval Operators
(Commanding Officers)
Current Mission Model Canvas™
Value Proposition Beneficiaries
Understand how cyber threats
degrade missions

26. Filling the gap: Convey information to commanding
officers
Convey
Information to
Fleet Cyber
(Navy Cyber
Operational View)
Present IT Tools
(VRAM, Nessus, etc.)
Convey information to
commanding officers

27. Filling the gap: Convey information to commanding
officers
Convey
Information to
Fleet Cyber
(Navy Cyber
Operational View)
Our MVP
Present IT Tools
(VRAM, Nessus, etc.)
Additional Tools/Capabilities

28. https://missioncritical.app

29. https://missioncritical.app

30. “This is exactly what we needed.”
- Former Program Manager,
Army Cyber Command

31. Maritime Commercial Industry
$500 BILLION
SAM
Maritime Government
$194 BILLION
SOM
Naval Ships and Subs
$63 BILLION
Commercial Factories
$155 BILLION
TAM
Size of Opportunity
Department of Defense
$716 BILLION
s

32. Moving Forward
Interest from:
- Former program manager at Army Cyber Command
- Ship commanding officers
- Lockheed Martin cyber security personnel
All four of us plan on continuing with Sea++

33. Acknowledgments
Our work would not have been possible without our sponsors at MITRE (Suresh
Damodaran, Matt Mickelson, and Alex Schlichting) and numerous other supporters,
including the teaching team, TAs (especially Will Papper), H4D military liaisons, and our
mentor Daniel Bardenstein. Additionally, a special thanks to individuals at the following
organizations:

34. Questions?

35. APPENDICES

36. High-value assets may be damaged
R&D
Inadequate documentation
for a thorough test
Test Engineers
No tools to understand
the impact of cyber
Commanding Officers
Post-test remediation is time-
consuming and inadequate.
Pentesters
Integrated system of the ship
is too complex to understand
Test Engineers
Organizational issues,
cultural issues
Government Officials

37. Pivot
Testing of all
systems (i.e.,
ship-wide
testing)
Our previous focus

38. Pivot
Many of the same problems (not damaging high value assets,
organizational issues)
Testing of
systems
(electrical,
navigational,
etc.)
Testing of all
systems (i.e.,
ship-wide
testing)in a
“red team”
style

39. Pivot
Testing of all systems cannot be solved without understanding
how to test individual systems
Testing of
systems
(electrical,
navigational,
etc.)
Testing of all
systems (i.e.,
ship-wide
testing)

41. Problem summary

42. MVP - “Black Box”
● During testing, replace high-risk component with MVP
● Using an AI simulation of data sent and received by the
component, simulate the behavior of the component
● Virtually replicate the behavior of the system without
risking damage to it
● Mitigate risk by removing at-risk and expensive
components from testing

43. Customer Discovery
Hypothesis: The USS Secure is inadequate for some
kinds of cyber testing.
Result: Some systems, such as an engineering control
system, would probably not be able to be adequately
tested in a purely simulated environment (the USS
Secure). Our sponsor - "Lower fidelity (constructive
or virtual) simulations do not usually come with
adequate attack surface."

44. Customer Discovery
Hypothesis: A machine learning model can accurately
simulate the responses of a physical component (such
as an engine) under a variety of conditions.
Result: Confirmed, if we can get the necessary data
for the model.

45. Customer Discovery
Hypothesis: The device does not currently exist.
Result: Confirmed; the Navy is currently working on
building out USS Secure, a virtual testing environment
for ships, but a physical proxy to replace components
does not exist.

46. MVP
Who
What
Where
When
Why

47. Buy-In/Support
Supporters Advocates Saboteurs
Private
Sector
Commercial Shipping
Companies
Industrial Control System
Operators
Integrated System
Testers
Component-Level
testing Companies
Public
Sector
Fleet Cyber Command
NavSea
Crews/Captains
Time/Budget
Opponents
(Engineering
Testing, Weapons
Testing, etc.)

48. Previous Mission Model Canvas™
-Format for tracking
the testing process
-Determining
organizational
constraints
- US Cyber
Command
-Fleet Cyber
-Chief of Naval
Operations
Saboteur - Naval
Support
Organizations
- compile cyber
vulnerabilities into one
process for sake of
efficiency and security
-Ensure that red-team
cyber testing occurs
consistently and safely
on ships
-improved security of
naval vessels
- conflicts of interest
between proposed and
existing budgets and
schedules for
cybertesting
- Increased vulnerability detection on ships
- Consistent, comprehensive cyber testing on naval
vessels
-creation of cyber-
testing agency within
Fleet Cyber
- Testing and reporting costs
- Travel
-Ships to test on
-Trust of ship
commanders
-Safety parameters
of cyber-physical
systems
Beneficiaries
Mission AchievementMission Budget/Costs
Buy-In/Support
Deployment
Value
Proposition
Key Activities
Key Resources
Key Partners
-NavSea
-SPAWAR
- Fleet Cyber
Command
-Chief of Naval
Operations

49. CAPT Jim Passarelli: Value Proposition
Canvas™
Products
& Services
Black box that
simulates a
physical
component
Remove equipment from
testing entirely
Customer Jobs
-Complete missions
-Maintain health
and safety of
crew and ship-Testing which interferes with
mission scheduling
-Testing that damages any part
of the ship
9th Intelligence Squadron
Gains
Pains
Gain
Creators
Pain
Relievers
Increased operational readiness
Fast installation and removal
of tools to ensure that they
are implemented.
Fast removal decreases
system downtime and
restoration period

50. Test Engineer Mr. Gene Lockhart: Value
Proposition Canvas™
Products
& Services
Black box that
simulates a
physical
component
- Free up time by easy
installation and interface-
ability with multiple types of
industrial physical systems
Customer Jobs
Find cyber
vulnerabilities in the
electrical system
- High risk of Damaging
system components
- Tme spent mitigating safety
concerns of physical equipment
9th Intelligence Squadron
Gains
Pains
Gain
Creators
Pain
Relievers
-More secure integrated ship
systems, better testing available
- Removes risk of damaging
equipment by entirely
removing equipment from
the systems testing

51. Research Engineer Suresh Damodaran: Valu
Proposition Canvas™
Products
& Services
Black box that
simulates a
physical
component
Removes physical equipment
from the system entirely to
decrease machinery damage
risk
Customer Jobs
Find cyber
vulnerabilities in the
electrical systemDamaging components in
sub-systems.
Decreasing thoroughness of
testing to accommodate risk
to damage equipment
9th Intelligence Squadron
Gains
Pains
Gain
Creators
Pain
Relievers
- Faster, more accurate testing of
components that will be installed
on ships
Does not compromise
accuracy of results due to
accurate modelling of
physical systems.

52. Grow
EXPAND customer base
through the Coast
Guard, private sector
shipping (tankers,
cruise ships), and
private sector industry
(industrial farming,
utility companies)
OBTAIN REFERRAL via
proof of successful
military adoption to
commercial industry
Keep
MAINTAIN interest by
refining and
integrating the product
to meet new needs
IMPROVE product by
adding more complex
features that improve
the capabilities of the
product
Get
ACQUIRE customers via
successful testing,
approach potential
users and market value
of our solution
ACTIVATE customers with
pushing naval offices
to adopt the cyber
testing device, train
users in test device
usage to ensure
adoption

53. Week 1 Mission Model Canvas
- NavSea
Philadelphia
- Cyber
Command
- Private
cybersecurity
firms
- Shipbuilders
- MITRE
- Development of
physical and
electronic security
testing procedures.
- Streamline testing
procedures to
minimize operational
downtime
- Target individual
systems for most
useful testing
applications
- Incremental
aggression
capabilities to
maximize
vulnerability
detection.
- More aggressive testing taking place onboard ships
- Adoption and continued use metrics of the
methodology by engineers
- Increased vulnerability detection on ships
- User retention during testing lifecycle
- Safety parameters
of cyber-physical
systems
- Integrated
navigation system
models/block
diagrams
- User Design Input
Beneficiaries
Mission AchievementMission Budget/Costs
Buy-In/Support
Deployment
Value
Proposition
Key Activities
Key Resources
Key Partners
- Operational
commanders for
conducting testing
- NavSea/MITRE
(Navy) and C4IT
(CG) to conduct
testing and evaluate
effectiveness
Primary: safety of
ship crewmen, safety
of large assets
Primary: Providing
defense contractors
with methodology for
determining security
of navigational
systems
Secondary: Decrease
in potential for
international
incidents
Fixed:
- Software design & engineering
- Testing and reporting costs
Variable:
- Expansion of hardware testing capabilities
- Hardware testing
procedures
understanding
- Developing
evaluation criteria
-Gaining trust and
gauging expectations
of ship captains
- Testing of
methodology on
sample test plans
- Full electronic
software deployment
to customers via
web-app

54. Week 2 Mission Model Canvas
-Understanding
hardware testing
procedures
- Understanding
evaluation criteria
-NavSea Philadelphia
-Cyber Command
-Private
cybersecurity firms
-Shipbuilders
-MITRE
-Operational
commanders
- Test engineers
- Defense
contractors
- Streamline testing
procedures to minimize
operational downtime
- Yield best ID of
vulnerabilities for
correction
- Development of
physical and electronic
security testing
procedures
- Target individual
systems for most useful
testing applications
- Incremental
aggression capabilities
to maximize
vulnerability detection.
- More aggressive testing taking place onboard ships
- Adoption and continued use metrics of the
methodology by engineers
- Increased vulnerability detection on ships
- Using methodology on
ships
- Full electronic
software deployment to
customers via web-app
- Software design & engineering
- Testing and reporting costs
-Ships to test on
-Trust of ship
commanders
-Safety parameters
of cyber-physical
systems
Beneficiaries
Mission AchievementMission Budget/Costs
Buy-In/Support
Deployment
Value
Proposition
Key Activities
Key Resources
Key Partners
- Operational
commanders
- NavSea/MITRE
(Navy) and C4IT (CG)
to conduct testing
and evaluate
effectiveness of
methodology

55. -Determining how to
streamline the
restoration process
-Determining
necessity of
adversarial
assessment
-NavSea Philadelphia
-Cyber Command
-Private
cybersecurity firms
-MITRE
-Operational
commanders
- Test engineers
- Streamline testing
restoration procedures
to minimize operational
downtime
- Yield best ID of
vulnerabilities for
correction without
risking damage to ship
- Development of
physical and electronic
security testing
restoration procedures
- Target individual
systems for most useful
testing applications
- Incremental
aggression capabilities
to maximize
- More ships undergoing aggressive testing procedures
- Adoption and continued use metrics of the
methodology by engineers
- Increased vulnerability detection on ships
- Using methodology on
ships
- Full electronic
software deployment to
customers via web-app
- Software design & engineering
- Testing and reporting costs
- Travel to naval bases
-Ships to test on
-Trust of ship
commanders
-Safety parameters
of cyber-physical
systems
Beneficiaries
Mission AchievementMission Budget/Costs
Buy-In/Support
Deployment
Value
Proposition
Key Activities
Key Resources
Key Partners
- Operational
commanders
- NavSea/MITRE
(Navy) and C4IT (CG)
to conduct testing
and evaluate
effectiveness of
methodology
Week 3 Mission Model Canvas

56. Week 4 Mission Model Canvas
-Format for tracking
the testing process
and constructing
relevant information
upon completion
-Determining how to
streamline the
restoration process
-NavSea
- US Cyber
Command
-Private cybersecurity
firms
-MITRE
-Operational
commanders
-Test engineers
- Ensure that red-team
cybertesting occurs
consistently and safely
on ships Streamline
cybertesting tracking
process restoration
procedures to minimize
operational downtime
- Yield comprehensive
ID of test parameters
and results without
risking damage to ship
- Development of tool
to aid in tracking of
testing and restoration
physical and electronic
security testing
restoration procedures
- Reduced restoration time
- Increased vulnerability detection on ships
-consistent, comprehensive cybertesting on naval vessels
- Full electronic
software deployment to
customers via web-app
-creation of cyber-
testing agency within
Fleet Cyber
- Software design & engineering
- Testing and reporting costs
- Travel Installation of app
-Ships to test on
-Trust of ship
commanders
-Safety parameters
of cyber-physical
systems
Beneficiaries
Mission AchievementMission Budget/Costs
Buy-In/Support
Deployment
Value
Proposition
Key Activities
Key Resources
Key Partners
- Operational
commanders
- NavSea / MITRE
- Fleet Cyber
Command

57. Week 5 Mission Model Canvas
-Format for tracking
the testing process
and constructing
relevant information
upon completion
-Determining how to
streamline the
restoration process
-NavSea
- US Cyber
Command
-Private cybersecurity
firms
-MITRE
-Operational
commanders
-Test engineers
- Ensure that red-team
cybertesting occurs
consistently and safely
on ships Streamline
cybertesting tracking
process restoration
procedures to minimize
operational downtime
- Yield comprehensive
ID of test parameters
and results without
risking damage to ship
- Development of tool
to aid in tracking of
testing and restoration
physical and electronic
security testing
restoration procedures
- Reduced restoration time
- Increased vulnerability detection on ships
-consistent, comprehensive cybertesting on naval vessels
- Full electronic
software deployment to
customers via web-app
-creation of cyber-
testing agency within
Fleet Cyber
- Software design & engineering
- Testing and reporting costs
- Travel Installation of app
-Ships to test on
-Trust of ship
commanders
-Safety parameters
of cyber-physical
systems
Beneficiaries
Mission AchievementMission Budget/Costs
Buy-In/Support
Deployment
Value
Proposition
Key Activities
Key Resources
Key Partners
- Operational
commanders
- NavSea / MITRE
- Fleet Cyber
Command

58. Week 6 Mission Model Canvas
-Fleet Cyber
-NavSea
-Risk of damaging
high value assets
reduced
-Ability to run wider
battery of tests faster
and more securely
- Removal of
Equipment from
testing plans
-Near perfect simulation of physical component with the
black box
-More comprehensive testing and remediation of cyber-
physical system
-Training data for AI
model
-Microcontrollers
with clock speeds
fast enough to
respond to PLC’s
Beneficiaries
Mission AchievementMission Budget/Costs
Buy-In/Support
Deployment
Value
Proposition
Key Activities
Key Resources
Key Partners
- Chief of Naval
Operations
-Office of Naval
Research
-Ship commanders
- Fleet Cybercommand
-Ship commanders
-Test engineers
- Research Engineers
-Development of MVP (materials)
-Travel
-Installation costs for MVP
-Build physical
prototype
-Develop machine-
learning technology
- Contract for MVP
written and
submitted to
acquisitions

59. Week 7 Mission Model Canvas
-Fleet Cyber
-NavSea
-Risk of damaging
high value assets
reduced and
increased clarity on
impact of results
-Ability to run wider
battery of tests faster
and more securely
- Removal of
Equipment from
testing plans
-loss of monopoly
over diagnostics of
their product
-Simulation of physical component with black box
- Integration of existing private sector technologies into
testing process and increased clarity of results
- Metrics are impossible to know from security issues
-More comprehensive testing and remediation of cyber-
physical system
-Training data for AI
model access to
private sector
simulations
-Microcontrollers
with clock speeds
fast enough to
respond to PLC’s
Beneficiaries
Mission AchievementMission Budget/Costs
Buy-In/Support
Deployment
Value
Proposition
Key Activities
Key Resources
Key Partners
- Chief of Naval
Operations
-Office of Naval
Research
-Ship commanders
- Fleet Cybercommand
-Ship commanders
-Test engineers
- Research Engineers
-Saboteur:
equipment
manufacturers
-Development of MVP (materials)
-Travel
-Installation costs for MVP
-Build physical
prototype and
reporting tool
-Develop machine-
learning technology
-Contract for MVP
written and
submitted to
acquisitions

60. Week 8 Mission Model Canvas
-Equipment
Manufacturers
-Naval Engineers
-MITRE
-Fleet Cyber
-Chief of Naval
Operations
-Risk of damaging
high value assets
reduced and
increased clarity on
impact of results
-Ability to run wider
battery of tests and
convey importance of
vulnerabilities to COs
- Removal of
Equipment from
testing plans
-loss of monopoly
over diagnostics of
their product
- Integration of existing private sector technologies into
testing process and increased clarity of results
-Test engineers implement as primary tracking tool for
testing; COs utilize as reference point for cyber-threats
-access to private
sector simulations
-Microcontrollers
with clock speeds
fast enough to
respond to PLC’s
Beneficiaries
Mission AchievementMission Budget/Costs
Buy-In/Support
Deployment
Value
Proposition
Key Activities
Key Resources
Key Partners
- Chief of Naval
Operations
-Office of Naval
Research
-Ship commanders
- Fleet Cyber
-Ship commanders
-Test engineers
- Research Engineers
-Saboteur:
equipment
manufacturers
-Development of MVP (materials)
-Travel
-Installation costs for MVP
-development of
software and
visualization
integration
-contract for
acquisitions
-modifications for
private sector
-Contract for MVP
written and
submitted to
acquisitions

61. Week 9 Mission Model Canvas