Getting Some Respect - How to Measure and Communicate Your EA Success

Getting Some Respect:
How to Measure and
Communicate Your EA Success
David Baker
Michael Janiszewski
Diamond Management and Technology Consultants
david.baker@diamondconsultants.com,
michael.janiszewski@diamondconsultants.com

October 26, 2007

Introductions

David Baker Mike Janiszewski
Partner and chief architect for Manager for Diamond
Diamond Management & Management & Technology
Technology Consultants Consultants

We both have extensive experience practicing Enterprise Architecture
as well as developing Enterprise Architecture organizations,
processes, and measurement techniques

Page 1

Introduction – Ice Breaker

Split up into groups and discuss your expectations for this
workshop
 Before you begin the discussion, assign a person who will take notes and
share the highlights of the discussion with rest of the team
 Upon completion of the assignment, share your group’s thoughts

Intro – 5 minutes
Exercise – 10 minutes
Debrief – 15 minutes

Page 2

Introduction – Ice Breaker

Take the next 10 minutes to discuss these three questions…

1. What do you hope to learn from this workshop?

2. Why are EA metrics important to your organization?

3. What emerging trends are shaping the future of EA measurement?

Page 3

Share your group’s discussion

Page 4

By the end of this course, you should be able to…

 Explain the importance and classification of EA metrics
 Include appropriate resources in your EA metrics program
 Analyze EA measurements to improve your EA program
 Integrate EA metrics into your organization’s IT lifecycle
 Identify training and management issues associated with launching
an EA program
 Identify organizational problems that may limit your EA metrics
program
 Design your own EA metrics deliverables and reporting templates

Page 5

Agenda

8:00 AM Introduction
8:30 AM What Problem Are We Solving?
9:00 AM Context for Measuring EA
9:30 AM Team Exercise
10:00AM Break
10:15 AM EA Metrics – What to Measure
11:00 AM Organizational Impacts
11:30 AM Lunch
12:00 PM Metrics Process – How to Use Metrics
12:30 PM Deliverables and Templates (Case Examples)
1:15 PM Team Exercise
1:45 PM Break
2:00 PM Metrics Frameworks
2:30 PM Using Tools
2:45 PM Wrap-up
Page 6

What Problem Are We Solving?
We know EA is valuable, but our stakeholders often do not

 Business
 “I just can’t see the benefit of our EA spend. Do we really
need this EA thing around?”
 IT Managers
 “We know doing EA is a best practice, but we can’t really
quantify how good we are at it. You [the business] can’t
measure it that way”
 Technical Staff
 “Our EA is so conceptual, we don’t know if we’re really doing
what you want us to do with it”

Page 7

The reality is…

 EA offers significant benefits
 Architecture and IT are cost organizations
 EA groups don’t have significant budgets
 We need to demonstrate the EA value associated with our business
impact
 If we can’t quantify our value, we shouldn’t be in business
 Metrics are a key part of marketing the value of EA

Page 8

Discussion: How do you market your EA

 What do you measure?
 EA operations?
 EA compliance?
 Business value?
 How do you communicate those measures?

Page 9

We can market EA by focusing on results

IT
Business
Managers
wants to know
want to know

EA Metrics
Is my strategy being realized? Is my IT portfolio aligned with
Is my EA delivering business the business?
value? Is IT meeting expectations
efficiently?
What risks am I taking?

Technical
Staff
wants to know

Am I making efficient use of EA assets?
How well am I aligning with our EA?
What things should I NOT be developing?
Page 10

Why are EA metrics important?

 Metrics quantify what we do in a useful way
 Metrics allow stakeholders to make informed decisions
 Manage change
 Set targets and challenges
 Manage risk
 Celebrate what goes well
 Understand and address what does not
 The capability to measure, track, and evaluate drives continual
improvement

Page 11

EA metric capture and reporting is difficult for a variety of
reasons

 Scope – The Enterprise
 Dynamic - EA is a moving target
 Skills - Lack of development within the discipline
 Vague - Success and failure are sometimes difficult to define
 Measurability - Actionable EA deliverables are not always
discrete/easily measurable
 Priorities - Stakeholders value other disciplines to assess
performance (e.g., program management, development practices)

Page 12

Agenda

10:00AM Break
11:30 AM Lunch
1:45 PM Break
2:30 PM Using Tools
2:45 PM Wrap-up
Page 13

Context for Measuring EA
EA Operates within the context of the IT Lifecycle…

Business Release Project
IT Strategic Planning Business
Strategic Execution
Planning (Portfolio
(SDLC) Operations
Planning Mgmt)

Prioritize the allocation of
Develop projects that
IT resources to achieve
Use enterprise and business unit support businesses’
business strategy, in Run the business
direction and goals to drive IT plans annual and strategic
alignment with enterprise
plans
architecture

Portfolio 1 Project
Blueprints
Project
Enterprise Portfolio 2
Filter

Blueprints Blueprints Project

Project
Portfolio 3
Blueprints Project

Multi-Year Plan Budget Cycle Project Cycle Continuous

Page 14

…with EA involvement at every stage

Strategic Execution
Planning (Portfolio
(SDLC) Operations
Planning Mgmt)

 Strategic Architecture  Application Portfolio  EA Oversight
 Innovation Management  Architecture Reviews
 Blueprinting  Application Portfolio  Architecture Issue
 Standards Definition Rationalization Management
and Management  Inventory Refresh  Issue Management
 Reference Architecture  Annual Budgeting Cycle  Corrective Action Plan
 Impact Analysis  Escalation Management
 Risk Analysis  Exception Management
 QA of IT Financials  Architecture Support
 RFP Process Support  Short Term Issue
 Content Resolution
 Vendor Support  Long Term Engagement
 Assessment and  Needs Assessment
Selection
Page 15

EA activities during strategic planning

Business
IT Strategic
Strategic
Planning
Planning

Objective  Define Enterprise Architecture guidelines

 Innovation
Associated  Architecture Planning
Processes  Standards Definition and Management
 Reference Architecture

 Domain architects responsible for Architecture Planning
Participants  Enterprise architects responsible for other Strategy processes
 Business owners and SMEs

Things to  Guidelines (tech standards, blueprints, reference architecture)
measure used in other Arch. functions, (e.g. Project Execution)

Page 16

EA activities during release planning

Business Release
IT Strategic Planning
Strategic
Planning (Portfolio
Planning Mgmt)

 Review the application portfolio
Objective  Facilitate financial review of the application portfolio
 Review content of RFPs, RFIs; assist in vendor evaluation

 Application Inventory  Impact Analysis  Content
Associated Management  Risk Analysis  Vendor Support
Processes  Inventory Refresh  QA of IT Financials  Assessment and
Selection

 Domain architects responsible for these portfolio activities
Participants
 Enterprise architects provide oversight and enterprise view

 Updated application  Dependencies  RFP, RFI
Things to inventory  Tech, bus. & delivery  RFP/proposal
measure  Inventory assessments risks assessment/scoring
 Updated costs
Page 17

EA activities during project execution and operations

Business Release
IT Strategic Project Business
Strategic Planning Execution
Planning (Portfolio (SDLC) Operations
Planning Mgmt)

 Monitor adherence to standards, blueprints, reference architectures
Objective  Manage architecture-related issue resolution/escalation
 Provide support for requests for architecture assistance

 Issue Management  Issue Management  Short Term Issue
Associated and Gate Reviews  Corrective Action Plan Resolution
Processes  Vendor Assessment  Escalation Mgmt  Long Term Support
Oversight  Exception Mgmt  Needs Assessment

 Domain architects responsible for these delivery activities
Participants
 Enterprise architects provide oversight

 CAPs, issues, exceptions, deferrals  Resolution of CAP
Things to generated during reviews  Project support
measure

Page 18

Discussion: How many of these processes do you have in
place?

 How mature is your EA practice?
 Is there a relationship between the maturity of your EA practice and
your ability to measure it’s impact?

Page 19

Agenda

10:00AM Break
11:30 AM Lunch
1:45 PM Break
2:30 PM Using Tools
2:45 PM Wrap-up
Page 20

Team Exercise 1
Brainstorm EA metrics

Brainstorm EA metrics by stakeholders
 Break into teams, teams will brainstorm metrics in different topic areas
 Before you begin the discussion, assign a person who will take notes and
share the highlights of the discussion with rest of the team
 Upon completion of the assignment, debrief rest of the team about list of EA
metrics required by various stakeholders

Intro – 5 minutes
Debrief – 15 minutes

Page 21

Team Exercise 1
Brainstorm EA metrics

Topics for discussion
IT
Business
Managers
wants to know
want to know
EA Metrics
Topic 1 - Brainstorm a list of EA Topic 2 - Brainstorm a list of
metrics necessary for Business EA metrics necessary for IT
Managers

Technical
Staff
wants to know

Topic 3 - Brainstorm a list of EA
metrics required by Technical Staff

Page 22

Agenda

10:00AM Break
11:30 AM Lunch
1:45 PM Break
2:30 PM Using Tools
2:45 PM Wrap-up
Page 23

Agenda

10:00AM Break
11:30 AM Lunch
1:45 PM Break
2:30 PM Using Tools
2:45 PM Wrap-up
Page 24

EA Metrics – What to Measure
EA not only takes place at different points in the Lifecycle, it also
takes place at different levels of detail
Scope:
Enterprise The Enterprise

Business Business Business Business
One (or more) Domain Domain Domain Domain
lines of (line of (line of (line of (line of
business business) business) business) business)

One (or more)
individual
projects

Page 25

It is helpful to group EA metrics into a small number of
categories

 Measures benefits delivered as result of architecture
Value Creation
processes

IT/Business  Measures how aligned a technical project is with respect to
Alignment the underlying business strategic requirements

 Measures architectural risks inherent in project activities
Risk Management
(e.g. new platform risks, sunsetting, upgrades)

Operational  Measures compliance accuracy with respect to enterprise
Effectiveness architecture
 Measures effectiveness of architecture processes such as
governance

Page 26

Measuring EA is different from program management

Enterprise Architecture Program Management
 Cross-LOB and cross-program  Cross-project and project
measures measures
 On the path to the future state  On-time, on-budget
 Alignment focused  Execution focused
 Blueprint earned value  Project earned value
 Business operations  System operations
 e.g. compliance with SLAs  e.g. system uptime

Page 27

Supporting that difference requires a linked architecture…

Business Architecture
Business Business
Strategy Operations

Desired Business Capabilities

Blueprint &
Solution Architecture
Service Model
Roadmap
Information Application Interface
Model Model Model

Infrastructure Model

Technology Architecture

Data App Development Execution Operations Network Security
Models Models Models Models Models Models Models

Page 28

…with sufficient detail to allow the relationships to be measured

Business Architecture

Mission
Linked Organization
Vision
Stakeholders Supported
business Goals
Business Functions business
metrics Objectives
Metrics functions

Business Services

Am I building
Solution Architecture the right
Reuse things
Data / Application / Common Services things?
that are of
interest to the
enterprise
Master
Applications Interfaces
Data

Infrastructure Model

Page 29

Value creation metrics

Metric Definition Calculation
 Metric that provides quantitative view of  IT initiatives must have assigned
advancement towards a blueprint. Metric business capabilities
Blueprint Value
is created by tracing back from initiative -  Calculate project advancement based
Measurement > capability -> objective -> corresponding upon blueprint's earned value
performance indicator (yearly target) measurements

 Estimated architecture complexity* for
each initiative
 Comparison of the top 10 architecturally  Obtain top 10 architecturally complex
Architecture complex projects versus the top 10 high projects
Coverage business priority projects  Obtain top 10 projects with the highest
 Also applies to Linkage metrics category business priority
 Count the number of projects that
intersect
% of Innovation
 # of ideas explored that ended up in
Ideas  Captures the success rate of ideas
production divided by the total # of ideas
Converted into explored by architecture explored
Production
* Discussed during “project typing” in “Metrics Process – How to Use Metrics”
Page 30

Value creation metrics (continued)

Metric Analysis
 The business should ensure that IT is making continual progress on delivery of the
defined business capabilities
Blueprint Value
 IT should be able to explain any acceleration / deceleration in delivery
Measurement
 The business should be able to clearly see how infrastructure projects
(investments) support desired business capabilities

 There is no right or wrong answer here, but it is valuable to show how many of the
Architecture
business priorities depend upon solutions to architecturally complex problems. This
Coverage demonstration is valuable to the EA community.

# of Innovation
 A high percentage is good, but does too high mean there’s not enough innovation
Ideas Converted (no risk being taken)?
into Production

Page 31

Additional value creation metrics (continued)

 % of Blueprints Defined
 # of Shared Technology Services Identified
 # of Shared Technology Services Delivered
 # of Reference Implementations Developed
 Savings from reuse of Standard Patterns
 TCO Savings from Application Retirement

Page 32

IT/Business alignment metrics

Spend amount  Cost of all IT projects associated with  IT initiatives must have assigned
per business delivery of a business goal. Can be at the business goals
goal enterprise or LOB level.  Sum the initiative costs for each goal

Number of
projects  IT initiatives must have assigned
involved with  Can be at the enterprise and LOB level business goals
each of the  Count the projects under each initiative
business goals

 Identifies the traceability between  IT initiatives must be traceable back to
Change
business strategy, objectives and business capabilities and capabilities
Agenda strategic technology investments. traceable back to objectives and goals

Page 33

IT/Business alignment metrics (continued)

Metric Analysis
Spend amount  Look for cases where the spend is not commensurate with the business priority
per business  High priority business goals with low spend (good thing)
goal  Low priority business goals with high spend (bad thing)

Number of
projects  Helps identify high risk business goals
involved with  High number of projects may reflect complex implementation and therefore
each of the high risk
business goals

 Can be used to conduct impact analysis of project delivery on advancing the
business strategy (and vice versa)
Change Agenda  What is the business impact of NOT spending on an IT project?
 What is the IT impact of changing a business strategy?

Page 34

Risk management metrics

 Calculate percentage of lifecycle risks
 Risks associated with project
attributed to each piece of the
Technology architectural component maturity.
technology lifecycle (e.g. emerging
Lifecycle Risk Measured in relation to product
technology, mature outside your
technology lifecycle
company, obsolete technology)
 An architecture risk is a technical  Accumulate risks as they are identified.
characteristic of a solution that has a Classify both their likelihood (high,
Architecture potential negative impact on that medium, low) and severity (severe,
risk count solution’s chances of a success (e.g. moderate, low)
vendor risk, new technology risk, security
risk, supportability risk, . . . )  Segment by architecture domain

 Calculate ratio of high probability and
Severe severe risks to total architectural risks
 Measures the number of architecture
Architecture risks across technical domains per project.
Risk Ratio
 Segment by architectural domain

Page 35

Risk management metrics (continued)

Metric Analysis
 Provides insight into technology lifecycle root causes for projects and articulates
where the enterprise is with respect to technology early adopter, late adopter, etc.
Technology
This information can be used to ensure that projects use the right strategy for
Lifecycle Risk implementing the EA vision (e.g. due to high importance of integration efforts, such
projects should use DB2 (mature) as opposed to IDMS (sunsetting))

Architecture  Can help identify common architecture elements that are at risk.
risk count  Can help identify weaknesses in architecture skills (architecture domain disciplines)

Severe  Provides insight into the overall severity of architectural risks encountered by
Architecture projects. Segmentation by architectural domain allows a starting point to take
Risk Ratio action to remedy at-risk projects (and at-risk reference architectures)

Page 36

Operational effectiveness metrics - compliance

 Percentage of project using / reusing  % of projects using standard products
Compliance standards, services, reference
architectures  % reuse of standard services, patterns
 Calculate number of projects that
EA Compliance  Measures the percentage of projects receive governance signoff on first try
Efficiency with architecture signoff on first try divided by total number of reviewed
projects
 Measures the percentage of projects  Calculate number of reviewed projects
EA Compliance fully compliant with standards by found to be fully compliant with EA
Ratio business domain and/or technology standards divided by total number of
architecture domain reviewed projects
 Measures the frequency of
Exception/  Calculate the total number of
exceptions/deferrals in information,
Deferrals exceptions/deferrals per architecture
application, technical and business
frequency domain
architecture as well as standards
 Number of reference material revision
Reference  Calculate total number of suggestions
suggestions submitted by business
Recommendation against a given reference material (e.g.
domain and/or technology architecture
Revision Count business domain blueprint)
domain
Page 37

Operational effectiveness metrics - compliance (continued)

Metric Analysis
 A low percentage could indicate an incomplete set of guidelines, or poorly
Compliance communicated guidelines, or just the wrong set of guidelines
 A high percentage shoud be correlated with better program management metrics
EA Compliance  Provides insight into accuracy of projects' delivery in compliance with your
Efficiency company’s EA standards

EA Compliance  Allows the business to understand the level of EA compliance within its project
Ratio portfolio. Provides confidence level in delivered architecture.
 Too many exceptions and deferrals may indicate an unachievable architecture
Exception/ vision or possibly an unachievable project schedule
Deferral
frequency  Exceptions and deferrals are and important and valuable tool for IT governance,
yet deferrals should be tracked to ultimate remediation

Reference
 Provides insight into use of reference materials. Also provides insight into
Recommendation
reference material needed by the enterprise.
Revision Count

Page 38

Additional operational effectiveness (compliance) metrics

 Number of exceptions (by level)
 Reference Recommendation Revision Approval Ratio
 Exception/Deferrals frequency by Blueprints across domains
 % of Projects Rejected
 % of Applications that are in Obsolescence Stage

Page 39

Operational effectiveness metrics - governance

Oversight  Measures the number of architecture  Calculate the total number of oversights
Frequency oversights conducted per week  Find the average over a week

 Measures the amount of time / level of  Calculate the number of days and
Oversight
difficulty to complete the architecture sessions required to complete signoff
Efficiency
reviews for a project per project per gate (also average)

Issues,
exceptions,  Totals and averages for the various  Track, count, and average totals for
deferrals, dispositions of architecture issues each
corrective action accumulated during reviews  Provide totals per project, per gate
plans
 Week-to-week comparison of key
 A complete view of architecture health metrics (issues, CAPs, exceptions,
Trending includes the total counts as well as their deferrals)
week-over-week trend
 Indicate week-over-week trend

Page 40

Operational effectiveness metrics – governance (continued)

Metric Analysis
Oversight
 Ensures governance processes are occurring according to targets
Frequency

Oversight  Ensures that governance processes remain lightweight and do not impact project
Efficiency delivery schedules
Issues,
exceptions,  Low numbers indicate maturity level of project compliance
deferrals,  Compare these historical numbers with problems during development and
corrective action operations to identify value of architecture compliance
plans

 Trend combined with phase can warn of impending problems
Trending
 Upward trend can signal the need to deploy senior architects

Page 41

Additional operational effectiveness (governance) metrics

 Issue Closure Efficiency
 Oversight Capacity Sufficiency

Page 42

Agenda

10:00AM Break
11:30 AM Lunch
1:45 PM Break
2:30 PM Using Tools
2:45 PM Wrap-up
Page 43

Organizational Impacts
Discussion: Architecture Organization

 Do you identify different types of architects?
 Or is everyone becoming an architect?
 How are your architects organized?
 Highly distributed?
 Centralized?
 Virtual?
 Who develops standards and reference architectures?
 If you do blueprints, who does them?
 What role does architecture play in project delivery?

Page 44

Enterprise architecture is delivered via several different types of
architects
Type Role
 Oversee the blueprinting process to translate enterprise and business unit
strategies into the technology roadmap
Enterprise Architect
 Ensure interoperability across domains
 Ensure compliance with architectural blueprints, roadmaps, and standards
 Translate business unit strategy into functional domain (such as Sales) blueprints
 Provide deep business and technology knowledge
Domain Architect
 Guide project architects to ensure project compliance with enterprise and domain
architectural blueprints
 Provide architectural expertise to lead the solution delivery projects
Project Architect  Responsible for overall system architecture and integration with other domains
and systems
 “Internal” architecture
 Requirements analysis, detailed software design
Systems Engineer
 Process planning and control
 Verification, validation
Software Developer  Code, unit testing

Page 45

Architects are social, community oriented creatures, found in
groups
CEO

Operations CIO

Business Unit Business Unit Business Unit Enterprise Solutions
Infrastructure
A B C Architecture Delivery

Business Project Infra
Architect Architect architect

Domain Domain Domain Application Systems
Architect Architect Architect Architect Engineer

Information Software
Architect Developers

Enterprise Architecture Virtual Community
Diagram assumes a large
organization, same principles
apply to smaller organizations
but structure will be different

Page 46

The various architects engage at different levels to deliver
projects
Business Environment
Emerging Technologies (regulation, compliance,
market factors)

Enterprise Architecture Solutions Delivery
Business Strategy
Business Architects IT Strategy
Application Architecture Program Management Office

Information Architecture Blueprints Application Development Services
(Enterprise &
Infrastructure Architecture Domain) Quality Engineering Services

Services Services Services Services Services Services
Release Planning

Sample Application Domain
Domain Architects Domain Manager

Architecture
Community Project Manager
(updates, exceptions)

Project Architects Systems Engineers

Software Developers

Page 47

EA requires the attention of a small group focused on
Architecture project management

 The Enterprise Architecture Group should be a small, high powered
group of Enterprise Architects
 One per architecture discipline
 Domain architects may also be in this organization (or in the
business)
 The Enterprise Architecture Group should also contain a small
number of project management resources
 Oversees the operation of the virtual community and its
working groups
 Manages and tracks oversight and issue management
 This is the group that collects and reports the EA metrics

Page 48

Transformation to / creation of a centralized enterprise
architecture group requires:

 Identify the true architects
 Be sure to look in the business groups as well
 Software engineers / programmers don’t always make the bets
architects
 Establish career path for architects
 Application -> project -> domain -> enterprise
 Assess architect skills to identify gaps
 Define standard architecture processes so that standard metrics can
be measured and reported across the enterprise
 Innovation, blueprinting, portfolio assessment, governance,
issue management, etc
Page 49

Training and change management

 Establish a small enterprise architecture group, initial focus on
establishing the virtual community and defining the architecture
processes (including metric definition)
 Provide adequate training and learning opportunities to teach the
standard architecture processes to both architect and non-architect
resources
 Implement these processes one domain at a time, gaining
experience as you go
 Start with the operational metrics (governance and compliance)
 Add risk management metrics as your processes mature
 Then add the alignment and business value metrics
Page 50

Training metrics

% completion of  Establish a curriculum
Architecture  Measures progress towards creation
Training and delivery of architecture training  Use curriculum project plan to estimate
Curriculum completion

 Skills Assessments
 Identification of sub-par skills among
# of Skills Gaps
the architects and specific plans to  Count of skills gaps
addressed
address each
 Plans to address each gap

% of Architects  To understand the level of training  # of architects trained divided by total
trained activity occurring in EA number of architects

Page 51

Training metrics (continued)

Metric Analysis
x% completion of
Architecture  A high percentage indicates that the Architecture training curriculum is complete
Training and available for use
Curriculum
# of Skills Gaps  Ensure any identified skills gap is being addressed either via training or on job
addressed experience

% of Project
 Low is bad, high is good
Architects trained

Page 52

Agenda

10:00AM Break
11:30 AM Lunch
1:45 PM Break
2:30 PM Using Tools
2:45 PM Wrap-up
Page 53

Agenda

10:00AM Break
11:30 AM Lunch
1:45 PM Break
2:30 PM Using Tools
2:45 PM Wrap-up
Page 54

Metrics Process – How to Use Metrics
Establish the reporting process

 Establish the Architecture program management office
 Establish and communicate targets
 Track trends
 Integrate metrics with
 Business & IT Strategy planning (feedback loop)
 Release Planning
 Project Execution and Operations

Page 55

Remember the context within which EA takes place

Strategic Execution
Planning (Portfolio
(SDLC) Operations
Planning Mgmt)

 Innovation Management  Architecture Reviews
 Blueprinting  Application Portfolio  Architecture Issue
 Standards Definition Rationalization Management
and Management  Inventory Refresh  Issue Management
 Reference Architecture  Annual Budgeting Cycle  Corrective Action Plan
 Impact Analysis  Escalation Management
 Risk Analysis  Exception Management
 QA of IT Financials  Architecture Support
 RFP Process Support  Short Term Issue
 Content Resolution
 Vendor Support  Long Term Engagement
 Assessment and  Needs Assessment
Selection
Page 56

Responsibility for collecting and using metrics varies by
lifecycle stage
Strategic Execution
Planning (Portfolio
(SDLC) Operations
Planning Mgmt)

 Enterprise Architects Management  Domain Architects
 Domain Architects  Domain Architects (primary)
 Enterprise Architects
 Architecture Issue
 Annual Budgeting Cycle Management
 Domain Architects  Domain Architects
 Enterprise Architects
 Architecture Support
 RFP Process Support  Domain Architects
 Domain Architects

Page 57

Follow a goal -> objective -> metric framework to establish
targets
Strategic Business Architecture
 This approach should be built into
MISSION the your business architecture
 Generate domain specific
business metrics for
KEY DRIVERS & GUIDING PRINCIPLES

VISION measuring value and
alignment
 For each objective, set
progress goals by establishing
GOAL GOAL GOAL GOAL
yearly targets
 Approach can also be applied to
BUSINESS
OBJECTIVE
METRICS generating targets for the risk and
operational metric categories
CAPABILITIES

Page 58

Example capabilities form a financial management system

Capability Linked Goal

 Maintain common processes centrally  Improve financial performance
across the Department to support
centralized administration and
standardization
 Support ad hoc data access across all  Improve operating efficiency of financial
LOBs. This will provide a simplified, management and procurement functions
single source for report information in a
timely fashion
 Generate performance and  Consistently comply with federal,
accountability reports, within OMB accounting, and system standards
specified timeline
 Capability for drill-down to transaction  Improve financial performance
level information

Page 59

Using metrics in the Business & IT Strategy phases

Generate future state
alignment & value
Business Alignment Business Future State metrics
• Confirm business strategy Definition
and goals • Define key business
• Refine understanding of capabilities to enable the
requirements driven by business strategy
recently defined business • Define future state business
strategy vision and models

Apply current state Roadmap
• Develop and prioritize
alignment and value metrics projects and required
investments
• Define implementation plan
Apply current state risk & and analyze economics
• Define risk mitigation strategy
operational metrics
Technology Technology Future
Assessment State Definition
• Review current architecture • Define key technology
and plans capabilities to enable the
• Document current technology business strategy
environment and pain points • Define future state IT vision
• Confirm enterprise and models
Apply future state
architecture strategy and
initiatives
alignment, value and risk
metrics

Page 60

Using metrics in the release planning phase

Blueprint Initiate Release Budget & Risk Assessment Update Release Transition Solution Design
Phase Plan Plan Project Initiation
Roadmap

Initiative’s Apply risk
Budget
metrics

Initiative
Risks Approved &
Initial Updated Funded
Sequenced Initiative Release Project
Release Release
Initiatives Assessment Schedule Charter
Plan Plan
Initiative
Depend-
encies
Apply compliance
Apply alignment Initiative metrics
Resource
metrics Reqs
Initiative/
Domain - wide Initiative-focused Domain-wide Portfolio-focused Project
focused

Page 61

Metrics can be used to determine a project’s complexity rating
Low Medium High
Attribute Key Considerations

Technology &  Are there significant exceptions to existing technology
Information standards and reference architectures?
No standard More than one
 Are products to be sunset used?
Standards exception exception

 Does the project proposal match overall architecture
Architecture planning? Is an implementation out of sequence with
Planning proposed roadmap? No standard
exception
Significant
deviations
Compliance  Are applications to be sunset being significantly enhanced?
 Is data redundancy being introduced?

Cross-Domain  Are there significant impacts to applications outside of the
business domains?
Impacts  Are there significant impacts to shared technology services?
No impact Significant
impacts
 Do the impacts require new model of integration?

New  Are new technologies going to be introduced into the
environment?
Technologies  How mature is the new technology?
No new
technologies
Significant new
technologies

Service Level  Are the service levels going to be significantly changed (e.g.
number of users, performances, availability levels)?
Changes No service Significant service
level changes level changes

 Are we enabling new / modifying existing business
Business capabilities? No impact to Significant impact
Impact  Are new user types being introduced? business to business
 Are there compliance issues?

Page 62

Project typing drives the level of governance during the project
execution phase

Category Level of EA Metric Reporting Characteristics

Maximum: • Highest # of Gate reviews
Type 3 • “High” metrics indicators
Enterprise & Domain
Projects • Enterprise and Domain Architects
Architect Involvement
involved

• Moderate # of Gate Reviews
Moderate: • “Medium” metrics indicators
Type 2 Domain level business • Domain Architect involved,
Projects value, blueprint alignment, Enterprise Architect involvement as
EA compliance needed

• Fewest # of Gate Reviews
Basic:
Type 1 • “Low” metrics indicators
Operational metrics such as Issue,
All Projects • Driven by Project Architect with
Exception, Deferral Mgmt
oversight by Domain Architects

Page 63

Agenda

10:00AM Break
11:30 AM Lunch
1:45 PM Break
2:30 PM Using Tools
2:45 PM Wrap-up
Page 64

Deliverables and Templates (Case Examples)
Case #1 – Healthcare Payer

 The Company
 ~$20bn multi-line healthcare insurance company employing over 27,000 people with over
$19bn in revenue managing over 10 million medical members in all 50 states
 The Challenge
 Lack of integration across business units presents fragmented view of constituents
 Second highest cost structure amongst direct competitors
 Slow and expensive product development inhibits ability to anticipate or lead in the market
 Corporate vision and strategy in place without an execution plan
 The EA program
 Phase I: Establish a program level architecture governance & design board to ensure the
3-Year program adheres to architecture goals & principles
 Phase II: Establish an architecture governance process & metrics/measurement model for
the entire I.T. organization

Page 65

Sample Report: Value creation report

Purpose: To provide visibility into the alignment of the actual release state with original target
release goals
Description: The Value Creation Report illustrates the each domain’s alignment with functional
and technical business capabilities. This report takes into account the impact of
exceptions and deferrals on the intended release state of each domain and exposes
any gaps that have formed between target and actual end-states.
Frequency: Monthly/Quarterly
Audience: CTO, Business Owners

Empty circle - Not
addressed in release
Half circle - Core
functionality met, high
exception score
3/2 circle - Core
functionality met, low
exception score
Full Circle - On target with
goal
Page 66

Sample Report: Architecture health status report

Purpose: To provide a detailed illustration of the architectural health of each domain.
Description: The architecture health status report is a synthesis of key metric data provided by
the architecture dashboard and textual highlights of key issues and milestones.
Frequency: Bi-weekly
Audience: CTO, Program Directors

Page 67

Sample Report: EA governance operational effectiveness report

Purpose: To illustrate project compliance with blueprints and standards and to measure
reference material effectiveness
Description: The Enterprise Operational Effectiveness Report will measure project compliance
with domain architecture standards, enterprise blueprints, and EA standards. It will
also measure the usage and effectiveness of EAG reference material.
Frequency: Bi-weekly
Audience: CTO, EA Core Team, Domain Architects, Program Leads, project Architects

Page 68

Process example: Weekly project architecture health reporting
EA Manager
Request Architecture
health status
overview
Enterprise
Architecture
PMO EA Admin
Consolidate all
Create health
TSMs health status
report overview
information

Domain admin
Submit
Determine project aggregate
Domain Lead health status health status
report

Project Lead

Project Team
Project Architect
1
Submit Weekly
project health
status report

Legend

Triggering End Org Role
Process Gate Package Process Break
Event Result

Page 69

Case #2 – Pharmaceutical Company

 The Company
 A leading pharmaceutical company, with over 100,000 employees and
$19 billion in reported net income for 2006
 The Challenge
 Having launched an aggressive enterprise-wide modernization program,
the company was looking for a way to ensure the program delivered the
promised capabilities and economic improvements
 The EA program
 Improve enterprise architecture service delivery through new organization
and governance structures and strong alignment with business value
 Provide the mechanisms to more effectively link architecture to concerns
of the business as well as to the updated software Delivery Model
 Create meaningful and objective measures which can be used to assess
the success of architectural efforts in business terms

Page 70

Sample Report – Local Portfolio Architecture Health

Page 71

Sample Report – Business Unit Portfolio Architecture Health

Page 72

Sample Report – EA Operational Effectiveness Dashboard

Page 73

Sample Report – Business Alignment and Value Creation

Page 74

Agenda

10:00AM Break
11:30 AM Lunch
1:45 PM Break
2:30 PM Using Tools
2:45 PM Wrap-up
Page 75

Team Exercise 2
What will you do differently when you get back to the office?

Consider metrics that you can begin to implement in your
organization
 This is an individual exercise using the worksheets in your handouts
 Identify one or two metrics in each category that would benefit your
organization
 Jot down some notes about what you might do to get a metrics project
underway
 The instructors will be around to help you think about what you can do

Intro – 10 minutes

Page 76

Team Exercise 2
EA Metrics Worksheet (1 of 3)

Operational Effectiveness (Governance) What will do you do?
_________________________________________________
 Oversight Frequency
_________________________________________________
 Oversight Efficiency
 Issues, Exceptions, Deferrals, CAPs _________________________________________________
 Trending _________________________________________________
 Issue Closure Efficiency
_________________________________________________
 Oversight Capacity Sufficiency
 __________________________________ _________________________________________________

IT/Business Alignment Metrics What will do you do?
_________________________________________________
 Spend per Business Goal
_________________________________________________
 Number of Projects per Business Goal
 Change Agenda Metrics _________________________________________________
 __________________________________ _________________________________________________
 __________________________________
_________________________________________________

Page 77

Team Exercise 2

Operational Effectiveness (Compliance) What will do you do?
_________________________________________________
 Compliance
_________________________________________________
 EA Compliance Efficiency
 EA Compliance Ratio _________________________________________________

 Exception/ Deferrals frequency _________________________________________________
 Reference Recommendation Revision Count
_________________________________________________
 Number of exceptions (by level)
 Reference Revision Approval Ratio _________________________________________________
 Exception/Deferrals frequency by Blueprints _________________________________________________
 % of Projects Rejected
_________________________________________________
 % of Applications in Obsolescence Stage
 __________________________________ _________________________________________________

Risk Management Metrics What will do you do?

 Technology Lifecycle Risk _________________________________________________
 Architecture risk count _________________________________________________
 Severe Architecture Risk Ratio
_________________________________________________
Page 78

Team Exercise 2

Value Creation Metrics What will do you do?
_________________________________________________
 Blueprint value
_________________________________________________
 Architecture Coverage
 % of Innovation Idea converted _________________________________________________

 Shared Technology Services _________________________________________________
 % of Blueprints Defined
_________________________________________________
 # of Shared Technology Services Identified
 # of Shared Technology Services Delivered _________________________________________________
 # of Reference Implementations Developed _________________________________________________
 Savings from reuse of Standard Patterns
_________________________________________________
 TCO Savings from Application Retirement
 __________________________________ _________________________________________________

Page 79

Agenda

10:00AM Break
11:30 AM Lunch
1:45 PM Break
2:30 PM Using Tools
2:45 PM Wrap-up
Page 80

Agenda

10:00AM Break
11:30 AM Lunch
1:45 PM Break
2:30 PM Using Tools
2:45 PM Wrap-up
Page 81

Metrics Frameworks
Federal Enterprise Architecture – Performance Reference Model

Source: “How to Use the Performance Reference Model”, Version 1

Page 82

Metrics Frameworks
The PRM structure is designed to clearly articulate the cause
and effect relationship between inputs, outputs and outcomes


Page 83

Metrics Frameworks
Key intersections of the PRM and other management processes

Budget Assess progress and inform subsequent budget decisions

GPRA Progress towards relevant PRM Measurement Indicators informs
evaluations to inform subsequent strategic plans

Program Progress towards Measurement Indicators can facilitate cross-agency
Assessment evaluation of PART programs within a BRM Sub-function. Higher-
Rating Tool (PART) scoring programs can share best practices with lower-scoring
programs with similar missions.

IT Capital Planning Progress towards Measurement Indicators inform updated Exhibit
& Investment 300s (business cases) and Agency Post-implementation Reviews
Control (CPIC)
Enterprise Progress toward indicators can determine changes needed in
Architecture Migration Plans


Page 84

Metrics Frameworks
Example: Using PRM measures to track business case (Exhibit
300) progress


Page 85

Metrics Frameworks
COBIT – The overall framework is a form of IT lifecycle

Source: “CobiT 4.0”, www.itasca.org
Page 86

Metrics Frameworks
COBIT – Interrelationship of components

Page 87

Metrics Frameworks
COBIT – IT Metrics and indicators

Page 88

Agenda

10:00AM Break
11:30 AM Lunch
1:45 PM Break
2:30 PM Using Tools
2:45 PM Wrap-up
Page 89

An integrated EA toolset strategy is critical to EA success

Enterprise Architecture Tools

Strategic Planning EA Governance Support Project Implementation
Tools Tools Tools
Strategic planning tools such Governance support tools Project Implementation tools
as Metrics/Dashboard tools such as scheduling tools enable the sharing of
enables visualization of ties allows the EA PMO to important knowledge across
between the business, operate and function the enterprise leading to
vision, goals, and objectives. effectively and efficiently. fewer redundancies and
This feedback will help Such tools can optimize increased visibility and
escalate visibility of projects performance of the EA PMO clarity. Issue/ exception/
to upper management deferral tracking and
isolating key issues and management, approval
common inadequacies tracking are some examples

Architecture Deliverables Support
Support tools such as repository tools help provide a centralized store that can help the enterprise gain access to,
learn and implement standards and blueprints. EA modeling tools can also help deliver information in a reusable
and consistent manner

Page 90

Tools must operate across governance levels and easily scale to
address enterprise-wide number of projects

Architecture Strategic Planning EA Governance Project Implementation Tools
Deliverables Tools Support Tools
Support
LEVEL 3

EA Governance
Dashboard
EA Modeling
Tools
Level 2 & 3 EA
LEVEL 2 Issue, Exceptions &
Deferral
Tracking and
Document Reporting DB
Management
Tools
EA Repository
Tools Decision CAP
Log Mgmt
LEVEL 1
Level 1 EA
Issue, Exceptions &
Scheduling Deferral
Status Reporting
Tools Tracking and
Reporting DB

Page 91

Agenda

10:00AM Break
11:30 AM Lunch
1:45 PM Break
2:30 PM Using Tools
2:45 PM Wrap-up
Page 92

Wrap-up
Key success factors for an EA metrics program

 Get your architects working in unison via common architecture
processes
 Innovation, blueprinting, portfolio management, governance, issue
management
 Engage your architects in all phases of the IT Lifecycle
 Business & IT Strategy
 Release Planning (Budgeting/Funding)
 Project Execution
 Operations
 Use a virtual community to report EA progress
 Business value, Alignment, Risk, Compliance/Governance
 Establish an EA program management office to aggregate and
report EA progress and value
Page 93

Thank you and good luck with your metrics efforts!

Page 94

Getting Some Respect - How to Measure and Communicate Your EA Success

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Getting Some Respect - How to Measure and Communicate Your EA Success

Similar to Getting Some Respect - How to Measure and Communicate Your EA Success (20)

Recently uploaded

Recently uploaded (20)

Getting Some Respect - How to Measure and Communicate Your EA Success