Balancing Up-Front Planning and Emergence

1. Enterprise Architecture Balancing Upfront Planning and Emergence

2. Published Description of this Talk “How do we handle the balance between the big up front plan, the ivory tower EA team, with the need to inspect and adapt and respond to the emerging needs of our customers? Can you identify the challenges with this description?

3. Survey Question #1What stakeholder groups are represented in the audience? End-Users The Business (sales, marketing, key management, etc) Customers Domain Experts (analyst, architect, systems engineer, etc) Developers (designer, coder, tester, etc)

4. Survey Question #2How many of you consider yourself an “agilist”? Agile Development Waterfall Development Iterative Development Iterative and Incremental Development Parallel Test Development Acceptance Test Driven Development Measure of Success Conformance to Plan Constant Flow of Business Value Process Handoffs/Sequential Teamwork/Parallel Culture Command and Control Leadership and Collaboration Design Big Design Up Front Continuous QA Big Test on Backend Continuous Tool Support Highly Specific Fully Integrated Lean and Efficient Value Streams

5. Which is Better? Predictive? Adaptive?

6. Which is Better? Trick Question Obviously Both Are Required Balance Is Required Balance is the Real Trick! Predictive Adaptive

7. Balance Is The Trick “Any development effort should be a balance between anticipation (planning based on what we know) and adaptation (responding to what we learn over time).” Jim Highsmith, Embracing Change, 17th March 2011

8. How Do We Find The Right Balance? “It Depends” “It’s Situationally Specific” “It’s All About The Context” Shucks! I Want It To Be Easy! Don’t you hate those answers?

9. How Do We Make It “Easier”? Focus On The Goal! Huh? What’s the Goal?

10. What Is Our Goal? Every Business Is Exactly The Same Every Business Has Exactly The Same Goal Every Business Is In Business… TO MAKE MONEY!

11. So, How Much Should We Plan?(How do we find the right balance?) Plan as much as necessary to receive a positive ROI on the planning investment We Plan Only To Maximize Value Delivery

12. Balance Is The Trick “If one has strong discipline without agility, the result is bureaucracy and stagnation. Agility without discipline is the unencumbered enthusiasm of a start-up company before it has to turn a profit.” Balancing Agility and Discipline, Barry Boehm and Richard Turner

13. Survey Question #3On A Scale of 1 – 5, How Important Is Planning? Planning Is The Purest Form Of Evil … … … Planning Is The True Secret Sauce of Development Another Trick Question. Have You Been Paying Attention? 

14. Agenda Introduction What is Architecture? What is an Architect? What is Emergence? How Can We Foster Emergence? How To Maximize Value Delivery Conclusion and Q&A

15. Survey Question #4How large are the companies represented by this audience? < 500 Associates 501 – 1,000 Associates 1,001 – 5,000 Associates 5,001 – 25,000 Associates > 25,000 Associates

16. Definition of Architecture “Architecture is the fundamental organization of a system embodied in its components, their relationships to each other, and to the environment, and the principles guiding its design and evolution.” [IEEE 1471] IEEE Computer Society, IEEE Recommended Practice for Architectural Description of Software-Intensive Systems: IEEE Std 1472000. 2000.

17. Definition of Architecture “The software architecture of a program or computing system is the structure or structures of the system, which comprise software elements, the externally visible properties of those elements, and the relationships among them.” Bass, Len; Clements, Paul; & Kazman, Rick. Software Architecture in Practice, Second Edition. Boston, MA: Addison-Wesley, 2003.

19. Behavior as specified in collaborations among those elements

20. Composition of these structural and behavioral elements into larger subsystems

21. Architectural style that guides this organizationG. Booch, P. Krutchen, K. Bittner and R. Reitman. The Rational Unified Process — An Introduction. 1999. Definition derived from Mary Shaw’s definition presented in 1995 at the First International Workshop on Architectures for Software Systems.

22. Definition Of Architecture "Architecture is about the important stuff. Whatever that is.“ Who Needs An Architect?, Martin Fowler http://martinfowler.com/ieeeSoftware/whoNeedsArchitect.pdf Architecture is the stuff that's hard to change later. And there should be as little of that stuff as possible. Martin Fowler via Neil Ford Evolutionary architecture and emergent design: Evolutionary architecture http://public.dhe.ibm.com/software/dw/java/j-eaed10-pdf.pdf

23. Setting The Record Straight Architecture Has A Tremendous Impact On Results Few properties have as much impact on an organization’s success as does architecture. There is ALWAYS an Architecture Architecture is an inherent property of your organization and your software. You may not understand it. You may not communicate it. You may not have planned it. But it’s there! Architecture <> Documentation Architecture is that inherent property of the system – not the artifacts that describe it. Architecture <> Infrastructure / Hardware Architecture encompasses the myriad of perspectives and concerns that make up a solution - and enable its production. That includes hardware, software, operations, support, organization, etc. These Concepts Apply To Both “Enterprise” and “Application” For purposes of today’s discussion – how does emergence apply to architecture and planning – there is no difference between “Enterprise Architecture” and “Application Architecture”

24. Agenda Introduction What is Architecture? What is an Architect? What is Emergence? How Can We Foster Emergence? How To Maximize Value Delivery Conclusion and Q&A

25. Why Do We Do Architecture? To capture the stakeholder perspectives that affect design To embrace change and to reduce the cost of solving problems To create a shared vision across the team and stakeholders To smooth the decision-making process - Lean Architecture, Jim Coplien and Gertrud Bjørnvig Focus On The Goal!

26. What Is An Architect? “Leads the development of the system's software architecture, which includes promoting and creating support for the key technical decisions that constrain the overall design and implementation for the project.” - Rational Unified Process

27. What Is An Architect? “An architect is a business-minded person who best understands how to leverage technology to maximize profitability.” - Paul Preiss, IASA (paraphrasing)

28. What Is An Architect? “A true software architect is one who is a domain expert, who knows how to apply the domain expertise to the design of a particular system, and who materially participates in implementation.” - Lean Architecture, Jim Coplien and Gertrud Bjørnvig

29. Competencies By Role A Perfect 10! 

30. Architect Focus Over The Life of a Project Discovery Invention Implementation Focus The Architect is accountable from “Concept to Cash” Project Lifecycle IBM – Brian Selic

31. Survey Question #5True of False: My Organization Has Individuals Specifically Assigned To The Architect Role.

32. Why Do We Do Architecture? “Architects” are not required, but these outcomes must be satisfied! To capture the stakeholder perspectives that affect design To embrace change and to reduce the cost of solving problems To create a shared vision across the team and stakeholders To smooth the decision-making process Lean Architecture, Jim Coplien and Gertrud Bjørnvig

33. Agenda Introduction What is Architecture? What is an Architect? What is Emergence? How Can We Foster Emergence? How To Maximize Value Delivery Conclusion and Q&A

34. Survey Question #6True or False? Emergence is a principle that encourages us to defer decisions to the “Last Responsible Moment” and avoid “Big Up Front Planning” or “Big Up Front Design”

36. Shorter Definition Of Emergence Collective Intelligence

37. Stop The Insanity Emergence is notabout the timing of decisions. Emergence is about harnessing the incredible intelligence that lies dormant in our organizations – trapped in antiquated and outdated philosophies of management. Emergence is about enabling and empowering the people to achieve a greater purpose.

38. Agenda Introduction What is Architecture? What is an Architect? What is Emergence? How Can We Foster Emergence? How To Maximize Value Delivery Conclusion and Q&A

39. Simple Principles Of Emergence The source of emergence is the interaction among agents who mutually affect each other. Attend to relationships characterized by mutuality among people, among teams and among companies in order for novelty to emerge. Small change can lead to large effects. Seek to lead change through many small experiments, which search the landscape of possibilities. Emergence is certain, but there is no certainty of what it will be. Create conditions for constructive emergence rather than to plan a strategic goal in detail. Evolve solutions, don’t design them. Greater diversity of agents in a system leads to richer emergent patterns. Seek a diversity of people, cultures, expertise, ages, personalities, gender so that when they interact in teams, creativity has the potential of being enhanced. - Complexity: life at the edge of chaos, Roger Lewin Simple to Describe. Hard to Accept!

40. How Do We Foster Emergence? Accept that there is in fact a “Complex Adaptive System” Provide leadership in the design of the system Manage the system – not the people Quit thinking linearly  Passionately apply a “new management model”

41. A New Model For Management Management Needs Changing New model leveraging “Complexity Theory” Acknowledge that organizations are really networks 6 Important Practices (“views”) Energize People Empower Teams Align Constraints Develop Competence Grow Structure Improve Everything - Management 3.0, JurgenApello

49. Agenda Introduction What is Architecture? What is an Architect? What is Emergence? How Can We Foster Emergence? How To Maximize Value Delivery Conclusion and Q&A

50. Recipe For Value Delivery Create The Opportunity Design The “System” Nurture Emergence Apply A New Management Model Execution Loop Do Just Enough Planning Create The Context Establish The Guard Rails Lubricate Execution Get On With It Sense and Respond Repeat

51. What Planning Do We Need To Do? Plan the right things at the right time Just enough to “accomplish the goal” by: Establishing Clear Context Energizes and Empowering The People Typically Focus On These Outcomes: Capture the stakeholder perspectives that affect the priority aspects of the design Embrace change and to reduce the cost of solving problems Create a shared vision across the team and stakeholders Smooth the decision-making process

52. Emergence and Decisions Making decisions sets the context required to enable emergence There are often multiple “scales” involved Decision cycles are different at different scales Every scale can operate in an emergent fashion Strategy/Vision/Mission -> Enterprise -> Application Vertical feedback loops are required

53. How Much is “Enough”? “In these systems agents residing on one scale start producing behavior that lies one scale above them.” - Emergence, Steven Johnson

54. Establishing Context Context for Energizing and Empowering Context for Safety Context for Efficiency

55. Context For Energizing / Empowering A compelling description of the goals Sense of urgency and purpose Alignment with vision, mission, and strategy

56. Empowerment “If you want to build a ship, don't drum up the men to gather wood, divide the work and give orders. Instead, teach them to yearn for the vast and endless sea.” - Antoine de Saint Exupéry

57. Emergence and Uncertainty Emergence involves some degree of uncertainty Empowerment introduces independence Does the Heisenberg Uncertainty Principle play?

60. Emergence and Boundaries Boundaries Establish Freedom Is This contrary to popular belief? Freedom is Required For Speed and Agility

61. Context for Efficiency Not all decisions are created equal Some things are harder to change than others Some decisions are more “core” or “fundamental” and get the ball rolling Don’t forget “strategic” vs “tactical” Evolutionary architecture and emergent design: Evolutionary architecture http://public.dhe.ibm.com/software/dw/java/j-eaed10-pdf.pdf

62. Agenda Introduction What is Architecture? What is an Architect? What is Emergence? How Can We Foster Emergence? How To Maximize Value Delivery Conclusion and Q&A

64. The World Is A Fractal “Once you start thinking of strategy as an emergent phenomenon, you realize that we have often attacked the wrong end of the problem. Strategists and senior executives have too often worked on "the strategy," rather than on the preconditions that could give rise to strategy innovation. In essence, they've been trying to design complex, multicell organisms, rather than trying to understand and create the conditions from which such organisms will emerge.” - “Strategy Innovation and the Quest for Value, Gary Hamel (http://www.strategos.com/articles/questforvalue.htm)

65. Q&A

66. Agenda Introduction What is Architecture? What is an Architect? What is Emergence? How Can We Foster Emergence? How To Maximize Value Delivery Conclusion and Q&A Appendix (Gartner On Emergent Architecture)

67. Gartners “New” Emergent EA “Enterprise architects must adopt a new style of enterprise architecture (EA) to respond to the growing variety and complexity in markets, economies, nations, networks and companies, according to Gartner, Inc. Analysts advised companies to adopt ‘emergent architecture’, also known as middle-out EA and light EA, and set out definitions of the new approach.” Key Characteristics of the Emergent Approach “Summarised as ‘architect the lines, not the boxes’, which means managing the connections between different parts of the business rather than the actual parts of the business themselves.” “Models all relationships as interactions via some set of interfaces, which can be completely informal and manual – for example, sending handwritten invitations to a party via postal letters - to highly formal and automated, such as credit-card transactions across the Visa network.” Bruce Robertson, research vice president at Gartner Gartner Press Release, Egham, UK, August 11, 2009 http://www.gartner.com/it/page.jsp?id=1124112

68. Gartner Emergent EA Principles#1 Non-Deterministic Non-deterministic - In the past, enterprise architects applied centralized decision-making to design outcomes. Using emergent architecture, they instead must decentralize decision-making to enable innovation. I've heard about this "deterministic" EA practice. And I've also heard about unicorns. Every effective EA practice I've seen recognized its role as one of leadership - and context. One of the biggest drivers in business today is Agility - the need to respond rapidly to changing needs and opportunities. By definition, then, we are operating in a climate where the future is not pre-determined or predicted. As such, at some scale, the specific outcomes are obviously non-deterministic. But there's a huge risk of this property being abused (see Agile Is Not "Make It Up As You Go"). As a whole, any organization with an EA practice absolutely has some destination in mind... some target. It's our job to create context and to provide leadership that helps the organization translate that target into actionable goals... and to adapt its way to success. http://blog.softwarearchitecture.com/2009/09/effective-enterprise-architecture.html

69. Gartner Emergent EA Principles#2 Autonomous Actors Autonomous actors - Enterprise architects can no longer control all aspects of architecture as they once did. They must now recognise the broader business ecosystem and devolve control to constituents. Again, the idea that an Enterprise Architect could ever "control all aspects of architecture" is a farce. The power of an organization always lies with those that are serving the organizations clients - the business units. Our role in EA is to serve those people on the front line and empower them to better meet the needs of their customers - and at the same time advance the organization as a whole towards its targets. This is a role of leadership - not control - and I wrote about it in Leadership - The Secret Sauce. Emphasizing the value of "collective intelligence," that post reminds us that we can "achieve outrageous levels of performance by harnessing the intellect and energy of the people." This also came up in Nurture The Freaks where we contemplated these words from Gary Hamel, "Going forward, no company will be able to afford to waste a single iota of human imagination and intellectual power." http://blog.softwarearchitecture.com/2009/09/effective-enterprise-architecture.html

70. Gartner Emergent EA Principles#3 Rule-Bound Actors Rule-bound actors - Where in the past enterprise architects provided detailed design specifications for all aspects of the EA, they must now define a minimal set of rules and enable choice. It's a reasonably well accepted principle that an EA practice should never make a decision (or set a constraint) that could be left to the business unit or development team. In fact, enabling choice and encouraging participation are important vehicles for gaining buy-in and goodwill (see Governance Without Goodwill Is Dead). This is yet another reminder of how we need to establish context by creating guard rails that keep the organization out of the ditch. http://blog.softwarearchitecture.com/2009/09/effective-enterprise-architecture.html

71. Gartner Emergent EA Principles#4 Goal-Oriented Actors Goal-oriented actors - Previously, the only goals that mattered were the corporate goals but this has now shifted to each constituent acting in their own best interests. With the guard rails in place, responsibility for driving rests on the individual drivers, each in their own vehicle with both hands on the wheel. http://blog.softwarearchitecture.com/2009/09/effective-enterprise-architecture.html

72. Gartner Emergent EA Principles#5 Local Influences Local Influences - Actors are influenced by local interactions and limited information. Feedback within their sphere of communication alters the behaviour of individuals. No individual actor has data about all of an emergent system. EA must increasingly coordinate. There is a massive amount of information flowing through the modern organization, and the majority of it originates and circulates right on the front line where the dynamic nature of today's agile organization demands a high degree of "in the heat of battle" decision-making. This suggests EA add value by encouraging a broad community that is willing and able to actively contribute to and consume a real-time, high-bandwidth stream of communication. It's not our job to assimilate it all and make decisions. Instead, as individual consumers at the trough of the information stream, we help drive the information out to those who need it the most. I don't really like the use of the word "coordinate" here. We're only coordinating in the indirect sense. Perhaps this role is some combination of the Connector, Maven, and Salesman roles described by Malcolm Gladwell in The Tipping Point. http://blog.softwarearchitecture.com/2009/09/effective-enterprise-architecture.html

73. Gartner Emergent EA Principles#6 Dynamic or Adaptive Systems Dynamic or Adaptive Systems - The system (the individual actors as well as the environment) changes over time. EA must design emergent systems [that] sense and respond to changes in their environment. This is one of the most important functions of the Enterprise Architecture discipline in a modern organization. We have a responsibility to bring a "systems thinking" perspective to the table and influence the design of flexible and adaptive systems - systems that have the ability to learn from and respond to their experience. When talking about systems, here, it's critical that we deliberately design this adaptive nature into our products AND our organizations. I'm happy to see that Gartner is beginning to recognize organizations as a type of "complex adaptive system". http://blog.softwarearchitecture.com/2009/09/effective-enterprise-architecture.html

74. Gartner Emergent EA Principles#7 Resource-Constrained Environment Resource-Constrained Environment - An environment of abundance does not enable emergence; rather, the scarcity of resources drives emergence. Most of us are operating under the influence of unprecedented economic conditions, and these times demand that we become more creative. In fact, creative isn't really the right word. To respond to the reality of the corporate climate of today (and tomorrow), we're going to need to be "clever" - adroit, nimble, resourceful, and mentally quick. The organizations that thrive in the future will be those that respond today by building a sustainable system of Agile capabilities that maximize the contribution of every associate. http://blog.softwarearchitecture.com/2009/09/effective-enterprise-architecture.html