This document discusses complexity challenges in integrating systems and organizations. It begins by looking at whether systems engineering needs an overhaul due to increasing system size, complexity, and globalization of engineering teams. It then examines complexity from an outside perspective, discussing different definitions of complexity from fields like information theory, computation, dynamics, and decision making. The document ends by discussing how multi-disciplinary research shows how complexity affects team behavior in socio-technical systems.

1. Complexity Challenges in the Integration
of Systems and Organizations
Does Systems Engineering need an Overhaul?
NASA PM Challenge 2012
1

2. Agenda
Complexity Challenges in the Integration of
Systems and Organizations
• Does Systems Engineering Need an Overhaul?
• Looking at Complexity from the Outside In
• Complexity & Teams
• Dialogue
2

3. Does Systems Engineering Need
an Overhaul?
Michael C. Lightfoot
NASA Langley Research Center, Hampton, VA
PM Challenge 2012, Orlando, FL February 22, 2012
3

4. Systems Engineering is Being Placed Under
the Microscope
There is a growing number of engineering communities who are asking
tough questions about the current practice of Systems Engineering.
Tough Questions:
 Why do the current SE processes, if rigorously applied, not guarantee
us safe, effective, robust systems delivered on time and within budget?
 What is it about our methods, processes and tools that seems to fail in
newsworthy fashion when we attempt to design and build large-scale
systems.
 Has our SE system somehow evolved to become a system that defies
our control?
Why the tough questions now?
4

5. Systems Engineering Trends
System Size and Complexity has increased:
One Example*: F-16, 15 Subsystems, 103 Interfaces
F-35, 130 Subsystems, 105 Interfaces
Organizations:
• Size increase (100’s to 1000’s),
• most likely global teams,
• different cultures w/ different incentives
• multiple companies,
• many reporting structures,
• sometimes competing incentives
Subsystems that were once modular in design are now
irreducibly entwined (tightly coupled)
Many systems are one of a kind (NASA) or limited quantity
productions
* Data courtesy of United Technologies Research Center:
https://www.fbo.gov/download/9cb/9cb78f01aa9db1fe92e093e786bc6733/Abstraction_Based_Complexity_Management_Final_Report_Dist_A.pdf
5

6. Increases in Aerospace Systems Complexity
Paul Eremenko, DARPA, META Program
6

7. Large- Scale Complex Engineered Systems
7

8. Characteristics of Large-Scale Complex
Engineered Systems
 Increased Engineering Complexity
 Highly-coupled interfaces, many of which are only discovered during integration &
testing or system operation.
 Design Cycles are Longer and More Complicated
 Significant Cost and Risk
 Extremely high political and monetary risk
 Low tolerance for failures or degraded performance
 Public fear of catastrophic failure is high
 Limited opportunities to experiment (trial and error)
 Very Large, Dispersed Engineering Organizations
 Yet organizations are expected to function synergistically
 Coordination and data exchanges are greater in frequency and volume of data.
 Unlike the early days of SE, no one Chief SE is able to keep the entire system view in
his/her head.
8

9. Classes of Engineered Systems
(Relative Comparisons, Not Rigorous Definitions)
Simple System:  Consist of few parts,
 Small number of interfaces
 Interactions well understood & well controlled,
 Typically used as building blocks for more sophisticated parts & components
Complicated  Consist of many parts, components, subsystems
System:  Moderate to large number of interfaces
 Interactions/reactions understood for controlled cases
 Vigilant control required to properly construct
 V & V is the basis to accept/reject bad parts, components, subsystems
 Global system behavior is mostly predictable; Part decomposition & analysis
leads to reasonable global property predictions
Complex  Can possess extreme numbers of parts, components, subsystems
System:  Extreme numbers of interfaces- sometimes impossible to identify
 Interactions understood for limited number of highly controlled cases but
mostly unknown due to dynamic adaptations
 Vigilant control often exercised but system sensitivity is nonlinear &
dependent on initial conditions (path dependent).
 Current analysis tools are poor predictors of system behavior
 Complete system V & V not possible.
 Global system behavior can be emergent (reductionist approaches fail)
9

10. Complicated System Example
Star Caliber Patek Phillipe mechanical watch.
 We understand:
 how it is constructed,
 the required tolerances,
 the order of assembly.
 Each component works in unison
to accomplish a global function: keep
time precisely.
 We can take a reductionist path to
define the smallest required parts
and can further write equations of
motion to predict the performance
and functionality of the watch.
10

11. Complex Systems
Through a Complexity Science Lens
• Dynamical systems
Dynamical/non-linear
Highly-coupled
• System response is non-linear & sensitive to initial conditions
• Consist of many parts, components, or subsystems (agents) that interact
Adaptive
with each other & the environment
Can be Self-
organizing
• They learn & adapt their behaviors to survive
If the adaptation strategy is good they continue to exist
If the strategy is bad or non-existent they cease to exist
Global behaviors
happen without a • They can move from an ordered to disordered state
centralized controller unpredictably, and can be self-organizing
Reductionist • No centralized controller
approaches do not
describe global
behaviors
• Knowledge of the inner workings of each agent typically shed no
information about the global behavior/response of the system
11

12. Examples of Complex Systems
Dynamical/non-linear • Ant colonies
• Rain forests
Highly-coupled
• Communities where you live
• U.S. Power Grid
• The World Wide Web
Adaptive
• The Stock Marker
• Propagation of infectious diseases
Can be Self-
organizing • The Global Economy (financial system collapse 2008)
• The Occupy ?? Protest Groups
• Multinational corporations
Global behaviors
happen without a • The NASA employees and contractors who supported the
centralized controller Constellation Program
The various engineering organizations that developed specific flight
Reductionist hardware for Pad Abort Activities
approaches do not The NASA PM and SE groups that supported Constellation
describe global
behaviors
Complex Systems can be Technical(Engineered),
Biological, Social or some combination
12

13. Domains of Complexity
Social Technical
Complex Engineering
Organizations
Socio- Technical
Creating Complex
Engineered Systems
13

14. Why is a Complexity Science Framework
Important to the SE Community?
 Current SE processes consists of experientially-based guidance.
 Although this guidance is tailorable, it is not deterministic.
 There currently is no theory, nor “science of system engineering” that
enables us to predict the efficacy, resilience or robustness of the systems
we produce.
 Our gut tells us that organizations impact the products we create but
we have no analytical tools to express the relationship between the two.
 A complexity science framework encourages us to question the
existence of dynamical relationships where we formerly assumed no or
linear relationships existed. This includes interactions between social
systems and technological systems.
 Many of the basic tenets/tools of complexity science are quite familiar
to engineers that work in dynamical systems (chaos, non-linear behavior,
neural networks, genetic algorithms, graphical modeling & simulation
tools tools, etc.)
14

15. What are the building blocks needed to grow a
competency in Complex Engineered Systems?
??????
Listen, Share and Solve Explore, Understand,
problems across Integrate social systems
disciplines & use new complexity into our decision
tools in novel ways. making & SE processes
Holistic Systems Uncertainty-Based Statistical Thinking &
Thinking Modeling and Simulation Probabilistic Uncertainty
[ embrace non-linearity ] Tools and Techniques Analysis
15

16. Potential Domain Infusions
Science of Trans-disciplinary
Socio-Technical Engineering
Systems Science
Social Technical
Complex Engineering
Organizations
Socio- Technical
Creating Complex
Engineered Systems
Engineering of Systems Engineering
Activities
16

17. NSF/NASA Workshop on Design of
Large-scale Complex Engineered
Systems
February 7-8, 2012
Arlington, Virginia
Organizers:
Steven McKnight, NSF Vicki Crisp, NASA
Christina L. Bloebaum, NSF Anna-Maria McGowan, NASA
George Hazelrigg, NSF Michael Lightfoot, NASA
Paul Collopy, University of Alabama, Huntsville
17

18. Workshop Overview
Objective:
• Examine the challenges unique to large-scale complex engineered
systems
• Examine how we can better prepare for a future of growing system
complexity?
Four Topic Areas Explored:
1. New approaches to system complexity by framing it through a
‘complexity science’ lens.
2. Current developments in design science and how might they help
us in designing within the SE process.
3. Awareness of what is known in organization science and how the
engineered product is a function of the organization.
4. How decision science can provide a more rigorous approach to
decision making in large-scale project teams.
18

19. Who Attended the NSF/NASA Workshop
on The Design of Large-Scale Complex
Engineered Systems?
• A total of ~115 people in attendance
• Government:
NSF, NASA, DoD (ODASD, AFRL, AFOSR, ONR, NRL, ARL), V-DOT
• Academia (25):
 University of Illinois at Urbana-Champaign, University of Minnesota, George Mason University, University of Maryland, Northwestern University,
University at Buffalo – SUNY, Purdue University, Schulich School of Business, York University, North Carolina State University, Georgia Institute of
Technology, Pennsylvania State University, Texas A&M University, Oregon State University, Stevens Institute of Technology, Johns Hopkins
University, University of Virginia, University of Michigan, University of Florida, Brigham Young University, Massachusetts Institute of Technology,
 Iowa State University, Stanford University, George Washington University, Mills College
• Industry & Others:
Lockheed Martin, Boeing, MITRE, SpaceWorks, Global Project Design, Google, NAE and
others
• Disciplines Represented:
Engineering, Social Science, Cognitive Science, Organization Science,
Anthropology and Economics
19

20. My Workshop Takeaways
• Systems Engineering as practiced is laden with human decision making which
could be enhanced by the understanding & practice of decision science
• SE needs to embrace nonlinearity and embrace a future where the systems we
build will not be fully testable (within the current practice of V&V).
• In order to better design & build large-scale complex engineered systems of the
future we need to 1st build better relationships between:
– Complexity Science Researchers
– Engineering Design Science Researchers
– Organizational Science Researchers
– Systems Engineers (PM+SE)
– Optimization Researchers
– S & T Leaders within Government Agencies
• Government participant agreed to form a Community of Practice to exploit unique
strengths that NSF, NASA, and DoD can bring to the challenge of large-scale
complex engineered systems.
20

21. Agenda
Complexity Challenges in the Integration of
Systems and Organizations
Does Systems Engineering Need an Overhaul?
• Looking at Complexity from the Outside In
• Complexity & Teams
• Dialogue
21

22. Looking at Complexity from the Outside In
a fresh look including outside our current processes
Ed Rogan
NASA PM Challenge
February 22, 2012
www.gpdesign.com | info@gpdesign.com

23. What is
Complexity  Complexity means different things in
different technical and professional
contexts
 We encounter most of them in practice
 A common language accessible to
non-experts would be useful
Slide 23
Global Project Design © 2012 www.gpdesign.com

24.  Complexity as length of a bit string
Compressible
Bit
(Kolmogorov, 1965)
Strings
 What is the shortest computer program that
will output a given bit string?
 Simple: 0101010101010101…
 Slightly more complex:
3.14159265358979323846….
 A definition of randomness: a string that
cannot be compressed to any shorter
Slide 24 computer program
Global Project Design © 2012 www.gpdesign.com

25.  Complexity as time required to find a solution
Complexity
of
to a computational problem (e.g. factoring a
Computation large composite number, scheduling, routing)
(Cook, 1971)
 If we can verify an answer quickly (time
bounded by a polynomial function of the input
length), can we also find an answer quickly?
 Probably not in all cases.
 “P = NP?”. $1 million prize remains unclaimed
for solving.
Slide 25
Global Project Design © 2012 www.gpdesign.com

26.  Paradox: how can unpredictable behavior
Complex
Dynamics
result from the laws of classical physics?
 Examples: celestial mechanics (Poincare, 1895), fluid dynamics
(Lorenz, 1963)
 Nonlinear terms amplify small differences in boundary or initial
conditions
 Solutions to compressible Navier-Stokes equations exhibit qualitative
changes in behavior with changes in a parameter (e.g. Reynolds
number, Mach number) – bifurcation, strange attractors, and chaos.
Slide 26
Global Project Design © 2012 www.gpdesign.com

27.  Large, highly interconnected networked
Other
systems with
systems
complex  Hybrid (discrete-continuous or digital-analog)
dynamics systems
 Example: brains.
 80 - 100 billion digital-analog/analog-digital converters
 Up to 10,000 inputs to a single converter
 Emergent behaviors: decision-making, attention
Slide 27
Global Project Design © 2012 www.gpdesign.com

28. Example: stock markets
Complex
Interactions  Assume decision makers are rational. All available
in information about the value of a stock is reflected in its price.
Decision- How can stock markets crash?
Making
 Decision-makers are not always rational (noise traders,
prospect theory, risk of arbitrage).
 Sometimes, buyers and sellers in the stock market choose
not to reveal all of the information that they know about the
value of stocks.
 Or, decision-makers as a group may have more knowledge
than they have as individuals (muddy children puzzle). An
event may make this information common knowledge.
 When previously hidden information becomes common
knowledge, behavior of many (rational) buyers and sellers
Slide 28 can (and does) change quickly.
Global Project Design © 2012 www.gpdesign.com

29.  Software (Kolmogorov and P = NP?)
Summary:
What  Digital-analog interconversion (hardware-software
Complexities interfaces)
do we face in
Engineering  Nonlinearity (fluids and structures)
Systems?  Large systems with many interfaces, dependencies, or
couplings
 Human decision makers
 Sharing or exchange of knowledge and information
 Risk and uncertainty
Complex systems have elements we haven’t
considered in the past
Slide 29
Global Project Design © 2012 www.gpdesign.com

30. Agenda
Complexity Challenges in the Integration of
Systems and Organizations
Does Systems Engineering Need an Overhaul?
Looking at Complexity from the Outside In
• Complexity & Teams
• Dialogue
30

31. Complexity & Teams
What multi-disciplinary research shows
about behavior in socio-technical systems
Bryan Moser
NASA PM Challenge
February 22, 2012
www.gpdesign.com | info@gpdesign.com
Slide 31

32. Who is GPD?
• Technology Leaders from Complex Global Industries
• U. Tokyo: Graduate School of Frontier Sciences
The Design of Global Projects
• Rapidly prototype and adjust plans
• Predict coordination activity
• Drive attention to interactions of value
GPD’s Methods & Experience
• Visual Modeling of integrated socio-technical architecture
• Behavior based simulation including global factors
• 15 years of case experience in industry: 3/ month globally
GPD’s Partnership Agenda
• Measures of Coordination by “Humans in the Loop”
• Leverage observation and massive sensing
• Practicability of new techniques
Slide 32
Global Project Design © 2012 www.gpdesign.com

33. Models of “organization” have shifted from
Shifting centrally controlled mechanical systems to
Models of dynamic organisms with distributed,
Organization adaptive, and behavior based subsystems.
 planning and forecasting, organizing,
commanding, coordinating, and controlling
(Fayol, 1916)
 structure, hierarchy, authority, roles
(Weber, 1924)
 as systems with boundaries, goals,
incentives, behaviors (Simon, 1962)
 differentiation, formalization, complexity,
centralization, span of control, rules,
procedures… (Burton, 1995 and others)
Slide 33
Global Project Design © 2012 www.gpdesign.com

34. Work as a
Product Development viewed as a
Socio- “socio-technical” system if we include
Technical “Humans in the Loop”
System
 People do work, process information,
and interact as part of an organization
 Individuals allocate attention based on
behaviors within limited capacity
 Organizations with architecture exhibit
emergent behavior (e.g. exception
handling, quality…)
Slide 34
Global Project Design © 2012 www.gpdesign.com

35. Engineering
 What if an IT system allowed
in Socio-
̵ teams with access to all information?
technical
systems ̵ all processes clearly written?
̵ workflow software to support tasks?
 If requirements, work packages, and
dependencies are clearly written and
assigned, is performance guaranteed?
What about human performance during
complex work isn’t addressed above?
Slide 35
Global Project Design © 2012 www.gpdesign.com

36. Coordination
 Coordination is the activity to manage
dependencies.
 What portion of your weekly effort is
spent coordinating?
 What happens to items in your inbox
when it overflows?
Manufacturing has shown for decades
that managing human attention is a key:
if we over-automate, quality drops
Slide 36
Global Project Design © 2012 www.gpdesign.com

37. Three activities to realize a
Architecture subsystem. Three teams.
&
Coordination  Independent activities.
Where will coordination
occur?
 Dependent activities.
Where coordination?
 Changed pattern of roles
and dependence, yet
scope and resources
unchanged.
Where Coordination?
The demand and supply of coordination activity are driven
by the integrated architecture of the project.
Slide 37
Global Project Design © 2012 www.gpdesign.com

38. Architecture  Teams have structure.
& Quality What coordination is this?
What impact on performance?
 “Exception Handling”
 Dependent activities.
Why does capacity of Team_1
now matter?
And in this case?
 What if:
Teams in different time zones?
Teams speak different native
languages?
…
 Organization attributes matter. They can be
observed, measured, and their impacts predicted.
 If we do not explicitly predict, we assume that
Slide 38 teams behave according to (our) past experience.
Global Project Design © 2012 www.gpdesign.com

39. Case
 Can one predict coordination and its
Example impact on a program’s likely duration,
cost, and risk?
 If we can see impacts of complexity
ahead of time, what might we do to:
̵ Reduce scope?
̵ Re-organize the system?
̵ Re-system the organization?
Slide 39
Global Project Design © 2012 www.gpdesign.com

40. Coordination • Coordination, Low Utilization,
is Predicted & Rework Predicted
in a Complex • Not only the amount of
Industrial coordination, but when, where,
Case and WHY it is demanded
40
Global Project Design © 2012 www.gpdesign.com

41. Resources: Teams Size, Location, Calendar, Abilities
Architectural choices within a
Architecture: Dependence, Roles, OBS, WBS & PBS
Scenarios: complex program are a lever
for better performance Externals: Targets, Start, Supplier Delivery…
Basis of
Scope: Activities, Direct Work, Complexity…
Project Model 30
Change
25
20
15
10
# of scenarios
5
0
WKSP 1 day 1 WKSP 1 day 2 WKSP 1 day 3 WKSP 2 WKSP 3 WKSP 4
Workshop Session
Global Project Design © 2012 www.gpdesign.com

42. Interfaces
 Does an interface allow reduced or no
interaction? In what horizon?
 Is an interface a call to interact?
Should a team pay more attention to
the interface?
 What happens when an interface
breaks?
Our teams need to be engaged, to
interact and respect that which we yet
don’t know or will discover
Slide 42
Global Project Design © 2012 www.gpdesign.com

43.  Normally work cultures evolve over time to
Design of align behaviors, promote learning, and control
Projects in a risk.
Complex
̵ Through a stable career one knew which interactions
Environment mattered, and others were “on the same page.”
̵ Today coordination and risk arise in unexpected
places
 Attention to coordination is not “soft”. These
are real attributes of time, cost, and quality
 Cases in complex global industrial programs
confirm observed behaviors and sufficient
predictability
 Design of a project architecture can weigh
team capacities and strengths, coordination,
and flexibility
Slide 43
Global Project Design © 2012 www.gpdesign.com

44. Agenda
Complexity Challenges in the Integration of
Systems and Organizations
Does Systems Engineering Need an Overhaul?
Looking at Complexity from the Outside In
Complexity & Teams
• Dialogue
44

45. Wrap Up
1. Organizations and the technical
systems we engineer – are more and
more complex
2. SE as framed today is strained and
needs to respond
3. Changes are needed which include
observation, analysis, and integration
of socio-technical dimensions
Slide 45
Global Project Design © 2012 www.gpdesign.com

46. Dialogue
 Does this hypothesis resonate with
your experiences and practices?
 Do you agree that SE needs to evolve
 What tough questions should we be
asking and researching about the SE
process?
 What are best practice examples in
current programs which recognize of
these real world behaviors? How was
the SE process adapted?
Slide 46
Global Project Design © 2012 www.gpdesign.com