Agile Project Estimates Emerging Requirements

Agile development methodologies emerged in response to
failings of traditional software development to reliably
produce results in a predictable timeframe.
Criticism of the Requirements Engineering (RE) practice in
traditional organizations is prominent among the
motivations spurring the Agile movement.
A study of 8,380 IT projects reveals two main reasons
projects fail: (1) lack of user input and (2) incomplete or
changing requirements.
When using an Agile Software Development Method
(ASDM) users describe the tasks they must be able to
accomplish with the system, but they can’t identify all the
specific functional and non-functional requirements
needed to implemented to let them accomplish those
tasks.
In traditional projects, the response is to a better job of
requirements engineering. In Commercial ASDM, the
response is continuous, on-site, customer collaboration
with the developers. As work progresses, additional
requirements will emerge and will be rapidly addressed by
the customer and developer.
In Federal IT acquisition, this may be problematic due to
organizational issues, remote user communities, and other
structural issues.
Estimating the needed work in the presence of these
issues is difficult in traditional project. When ASDM are
used these estimates become more problematic, due the
emerging nature of the requirements.
1
7/29/17
Thomas J. Coonce / Glen B. Alleman
DHS Estimating and Reporting Agile Projects

ALL PROJECTS OPERATE IN THE PRESENCE OF UNCERTAINTY
Uncertainty comes in two forms:
§ Aleatory Uncertainty ‒ this is the naturally occurring
statistical variances in the underlying processes,
environment, and technology.
§ Epistemic Uncertainty ‒ this a probabilistic event there
will occur something that is currently uncertain.
Both these uncertainties create risk:
§ Irreducible risk from aleatory uncertainty – which can
only be managed with Margin.
§ Reducible risk from epistemic uncertainty ‒ which can
be managed with risk buy down activities, funded on
the Performance Measurement Baseline.
Risk Management is How Adults Manage Projects
– Tim Lister
Be an adult and mange the project as a risk manager.
7/29/17
Thomas J. Coonce / Glen B. Alleman, July 2017 2

The acquisition or development of Software Intensive
System of Systems (SISoS) in the current paradigm is made
both complex and complicated by external influencers.
§ Technological change is accelerating, possible faster
than the requirements can be stabilized.
§ Commonality of systems is now the norm, with possibly
unknown interaction behaviors.
§ Need to have a Global understandings of how systems
work is now the norm.
§ Information is a commodity and requires management
on its own.
§ Collaboration outside our comfort zones is required
§ Economies of scale drive new technology and rapid
change, reducing the ability to build purpose built
products or services.
§ Commercial Off The Shelf everything with reducing
control over the behavior of these products.
§ We never saw it coming is now the norm.
7/29/17
Thomas J. Coonce / Glen B. Alleman 3

Why Agility Matters?
We tend to take the fact that agility is important as a
given, when the reality is that not everyone in the
business world has reached the same conclusion. Thus,
it’s important sometimes to take a step back and
examine why agility actually matters, so that when we’re
faced with people who aren’t as convinced as we are, we
have salient points that we can raise to help them
understand the value that agility brings with it. Here are a
few important things to remember when thinking about
why agility is important in our jobs
† The Clever PM, July 20, 2017
http://www.cleverpm.com/2017/07/20/why-does-agility-
matter/
7/29/17

While the 6 steps above are Critical Success Factors for
increasing the Probability of Program Success (PoPS),
there are four immutable truths in the developemnt of
software using agile inside the FAR procurement
framework.
1. On software intensive system of systems, capabilities
may be fixed, but technical and operational
requirements emerge.
2. As the program evolves, changes will occur. This is a
desired condition for Agile development.
3. With emerging requirements, bounded time and cost
will constrain the ability to respond to these emerging
requirements.
4. Estimating in the presence of uncertainty and
emergence means continuous updating that estimate
in response to these changes.
7/29/17

EMBRACING AGILE MEANS
Applying these six critical success factors to increase the
probability of program success
1. Start with the description of what DONE looks like in
the Concept of Operations, but be prepared for
emerging requirements to change the description.
2. Build the Integrated Master Plan and identify the top
level risks from the Government’s point of view before
issuing the RFP, and update that plan when new
requirements appear.
3. Award a contract based on the assessment of the
vendors Probability of meetings the Measures of
Effectiveness and Performance, on the needed
completion data for the needed cost.
4. Assess the PMB at the IBR and require an IMP, a
technical plan, a high integrity WBS and WBS
Dictionary, and a Program Management Plan. Confirm
these are being followed and update then when
change occurs.
5. Use these documents as Program Performance
Management processes in the presence of uncertainty
and emergence.
6. Measure Physical Percent Complete against the Risk
Adjusted PMB on a monthly basis with possible a Mid-
Month Flash Report. Never measure progress as the
passage of time and consumption of money ‒ working
products on the planned date for the planned cost.
Take corrective actions to Keep the Program Green
7/29/17

In the federal acquisition domain, the Performance
Measurement Baseline (PMB) and Contract Budget Base (CBB)
are the anchor documents for the contractual work.
These Traditional documents describe what is being delivered
and what to budget is for each deliverable.
In the Agile acquisition domain, definitized requirements are
not available, nor are they encouraged.
What needs to be on baseline is the set of Capabilities the
project will produce when it is done.
Capabilities describe a solution independent requirements for
the needed system. Above the LINE, capabilities are baselined
for the contractual compliance. Below the LINE, the
requirements can emerge as the project progresses.
As the Capabilities are fulfilled by emerging Requirements in
the form of Features and Stories, the user of the system can
assess the maturity of those Capabilities using Physical Percent
Complete Measures of Effectiveness and Measures of
Performance.
This idealized view of systems development, must be managed
with traditional Change Control to avoid the Death March
projects where both the Capabilities and emerging
requirements have open ended solutions.
This issue is addressed in Agile Software Development with a
Capabilities Based Planning process [1] in the same way
Requirements Management is used on traditional development
projects.
This means continuous determination of what capabilities are
needed at what time, what is the funding for those capabilities,
what technical requirements are needed to fulfil those
capabilities, and how to balance these needed capabilities
across the user’s needs, within the constraints of the business,
technical, and operational environment.
[1] “Capabilities Engineering – An Analysis of Perspectives, INCOSE
International Symposium, Volume 21, Issue 1, June 2011, Pages: 712–727.
7
7/29/17

GOVERNMENT PROGRAMS ARE NOT LIKE COMMERCIAL PROGRAMS
The budget for Government programs belongs to a
sovereign.
Spending that budget is subject to FAR clauses.
The Performance Measurement Baseline (PMB), with it’s
time phased cost to produce the contractual outcomes is
the core document in Government Software development.
Integrating Agile with FAR procurement is straight forward
and at the same time difficult.
The Agile Manifesto assumes small, self-organizing teams
and low formality contracts.
Data and the resulting Information is the basis of all
decision making in the presence of uncertainty.
Software development and acquisition projects are highly
uncertain.
We need an integrated set of data from which to extract
information to make informed decisions in the presence of
uncertainty.
Here’s the structure of this data, starting with the RFP and
moving to Program Execution.
Both horizontal and Vertical integration of the data is
needed. Without this integration there is no integrity is
the information produced by the data.
Vertically we go from RFP Capabilities down to the Work
Packages that implement to Features of the software that
implement those Capabilities.
Horizontally we go from RFP to program execution.
7/29/17

There are 10 steps in estimating, developing, managing,
and delivering a software program using Agile methods.
This talk focuses on developing the ❶ Engineering
Estimate and the use of that estimate in the measurement
of progress to plan.
The other 9 steps are needed of course, but are the
subject of another talk.
This effort needs to produce a credible estimate for the
needed Capabilities.
We can start this effort by decomposing Capabilities those
into Features and starting developing the software.
These Capabilities are described in the Product Roadmap
and Release Plan.
The Features are described in the Product Backlog.
The Stories that implement the Features are described in
the Sprint Backlog.
All these containers (backlogs, roadmaps, and plans) of
information provide input to the estimating process – up
and down the chain of deliverables, for Capability
estimates at the highest level to Tasks at the lowest levels.
The collection of this data is the foundation for Reference
Classes.
This is the starting point for estimating Agile development.
7/29/17

§ ConOps ‒ describes the characteristics of the proposed
system from the viewpoint of an stakeholders who will
use that system. It communicates the quantitative
(Performance) and qualitative (Effectiveness) system
characteristics.
Our new warehouse management system will
improve response time, reduce waste, and increase
efficiency and effectiveness of our capital assets
and staff
§ Capabilities ‒ a high-level solution behavior providing
the ability to accomplish some business function or fulfill
a mission.
We need the ability to process 100K transactions a
day for our national-wide warehouse inventory
tracking system
§ Feature ‒ a distinct element of functionality that fulfill a
stakeholder need.
We need to add a new supplier to the warehouse
inventory control system
§ Story ‒ a user oriented description of enough
information so developers can have a reasonable
understanding of how they would need to implement.
As a warehouse manager, I want to track
movement of shippable goods in real time, so I can
schedule fork truck and loading dock personnel
§ Task – a small activity that is required to get the Story
done.
Create database entries to capture the availability
of fork truck drivers by specific shift and geographic
location
7/29/17

Estimating in general is hard work.
Estimating on Agile projects, where requirements are
changing, users are not clear on the details of the needed
software is even harder.
But estimates are needed, no matter what domain we’re
in.
In the FAR acquisition domain, estimating is not only a
critical success factor, but it is a contractual obligation.
When we don’t have Reference Classes of Feature, we
have to start building our estimating database using
Function Points as defined in IFPUG guidance.
7/29/17

The basis of success with Reference Class Forecasting is to
build a distributional information database instead of a
singular information estimate.
Singular information is what we know about an individual
case ‒ a Point Estimate.
Distributional information, by contrast, consists
of knowledge about the distribution of outcomes in similar
situations.
As the database of comparable cases grows, the predictive
accuracy of new product features improves.
This effort starts with the develop of categorical tags and
applying them both to completed and new backlog items.
Product feature tags and process feature tags can be
assigned to capture the ways that a given feature involves
complexity or dependencies on external factors or actors.
Product feature tags focus on what the planned feature
does (e.g. “has a front-end user interface” or “connects ≥2
databases”) while process feature tags focus on what the
product development team does to develop the feature.
Process feature tags are derived from asking the following
questions during estimation:
§ Does external dependency on other teams exist?
§ Do we depend on code that does not rely on the same
standards (e.g. Ruby on Rails, APIs)? Is research needed
to complete a problem?
§ Do we have to write tests?
§ Are there interdependencies within the code base?
§ Are we touching new territory in some other way?
7/29/17

13
Developing the Engineering Estimate is the starting point
for managing the Agile Software Development program.
This estimate is required to respond to the RFP..
This is different from the commercial software
development domain.
In our FAR domain, we need to provide a Performance
Measurement Baseline on the proposal. The award of the
contract is based on the cost, schedule, and technical
performance proposal ‒ long before the writing of code.
Here’s the steps to develop the Engineering Estimate.
7/29/17

A major source of risk in project management is
inaccurate forecasts of project costs, demand, and other
impacts.
Forecasts of cost, demand, and other impacts of planned
projects have remained constantly and remarkably
inaccurate for decades. No improvement in forecasting
accuracy seems to have taken place, despite all claims of
improved forecasting models, better data, etc.
Reference Class Forecasting was first described by
Kahneman and Tversky in 1979 to compensate for
cognitive bias they found in their work on decision
making.
They found that many errors of judgment are shared by
experts and laypeople alike. They found that errors remain
compelling even when one is fully aware of their nature.
Reference Class Forecasting has three steps:
1. Identify a relevant reference class of past, similar
projects. The class must be broad enough to be
statistically meaningful but narrow enough to be truly
comparable with the specific project.
2. Establish a probability distribution for the selected
reference class. This requires access to credible,
empirical data for a sufficient number of projects
within the reference class to make statistically
meaningful conclusions.
3. Compare the specific project with the reference class
distribution, in order to establish the most likely
outcome for the specific project.
7/29/17

Since Story Points are Ordinal numbers – relative
measures, with no calibrated value ‒ we need a measure
that is Ordinal.
The SRDR uses Stories and Story Points, which are not
calibrated and do not have the same meaning for effort or
complexity as hours and dollars do, outside of a single
development team, let alone across multiple contractors.
Hours and Dollars are Cardinal.
But estimating in hours or dollars is difficult, when
definitized requirements are not present
Function Point Counting can be a way to estimate agile
development efforts, and still have a cardinal measure of
cost and schedule.
There is research and experience reports on connecting
agile estimates to Function Points.
7/29/17

For all contract work in the Federal Government, some set
of capabilities and supporting Features needed to be
stated in the Request for Proposal.
The ConOps and the Statement of Work are good spots to
capture these capabilities.
In the Agile paradigm, these can be decomposed to
Features
7/29/17

These 10 steps are needed to produce a credible estimate
on any program ‒ agile or traditional.
The steps don’t actually produce the estimate, they are
preparations for producing the estimate.
These steps are typically missing when we see poor
estimates or estimates that are not credible.
With this information in place, we can start the estimating
process.
7/29/17

Here’s an example of a worksheet for capturing estimates
on a agile project in the Federal Space.
Since the FAR requires us knowing how much it will cost
in dollars ‒ contract budget base ‒ and how long it will
take ‒ period of performance ‒ in hours, the estimates
submitted by the contractor to the government has to be
in those units of measure.
Story Points and Function Points are not units of measure
in the contract award award documents.
So we have to produce this Engineering Estimate in those
units.
But, those units can be derived from Functional Points,
Feature estimates, and reference classes forecasts.
This is the current challenge in deploying agile
development in the federal space ‒ the units of measure
of agile from the commercial world, don't match up with
the federal (FAR) procurement world.
There is no simple solution, but a transition from the
current situation to one where agile can provide
information for the dollar/hour estimates can be
developed, starting with Reference Class Forecasting
7/29/17

The critical success factor for all projects ‒ agile or
traditional ‒ is to measure progress as Actual Physical
Percent Complete against the Planned Physical Percent
Complete at the Planned Time.
Here’s an example from an Agile tool (rally) where the
planned percent complete is defined in hours for the Tasks
that implement each Story.
The stories implement the Feature, and the Feature may
(in this example) span multiple Sprints (three here).
In agile new work can be discovered after the Sprint starts.
This emergent work is part of the agile processes and in
encouraged.
The key here is, when the new work is discovered the TO
DO field is increased. This indicated more work is
identified and the current Physical Percent Complete goes
DOWN.
In the initial planned effort, 100 hours were assigned ot he
Feature. During the execution of the Sprint 25 more hours
of work were discovered. That is recorded in the system
and used to calculate Physical Percent Complete.
When the TO DO equals ZERO, the User Story is done.
When all the User Stories have their TO DO ZERO, the
Sprint is complete, assuming the Sprint did not come to an
end before that.
The formula for Physical Percent Complete (P%C) is
P%C = (Planned Work – TO DO) / Planned Work
If the Planned Work goes up the P%C goes down
7/29/17

While Earned Value Management may appear
burdensome and overly complex, it is actually very simple.
§ What is the Planned Budget for the development of a
product or service at a specific point in time?
§ What is our planned Physical Percent Complete at that
point in time?
§ What is our actual Physical Percent Complete at that
point in time?
With those measures, we can determine how far behind
we are in schedule, and how far off we are of our budget.
With those measures, we can determine what it will cost
in the end and when we’ll actually be done, compared to
when we planned to be done.
Earned Value is actually a misnomer ‒ it should be called
Earned Budge.
We budget X$, did we earned that Budget at the planned
time?
7/29/17

With a Budget Spread for the development and support
staff and any other direct costs, the measures of Physical
Percent Complete can be used to calculate the earning of
value in the Earned Value Management sense.
This is not business Value, but it is the basis of computing
the Estimate to Complete and Estimate at Completion.
Normal Agile development does not provide these
measures are in units meaningful to the decision makers.
These units can only be Dollars and Hours.
This is the primary benefit of using Earned Value
Management on Agile programs.
7/29/17

Managing Software Development programs is difficult. It’s
not the same as managing bending metal into money
programs.
The deliverables are intangible in the sense you cannot
count them, weigh then, turn wrench's to assemble them.
But there are tangible outcomes of software development
programs ‒ working software that provides a Feature that
enables a Capabilities to accomplish a business function or
fulfill a Mission need.
These Features and Capabilities can be assessed through
Measures of Effectiveness and Measures of Performance.
What is difference is we may be able know up front know
needed Capabilities, Measures of Effectiveness and
Performance, but now know the technical requirements
that will deliver those.
This is where Agile development comes in.
But the Principles of Program Performance Management
based on assessing Physical Percent Complete against the
MOE, MOP, TPM, and KPPs is the same in traditional and
Agile programs.
With the use of Physical Percent Complete for the
Features in the Work Packages in the PMB, program
performance measures can produce credible EAC, ETC,
and ECD using standard methods.
It getting the Physical Percent Complete that is the Critical
Success Factor.
7/29/17

7/29/17

The Product Roadmap is the starting point for Estimating.
This roadmap described the Capabilities needed by the
customer in exchange for the Funding in the contract.
This document is part of the traditional acquisition process
– ConOps, Integrated Master Plan and Agile.
A critical success factor is simple
The Customer needs to know what Done Looks Like in
terms of business Capabilities needed to accomplish the
Mission
Capabilities Planning, is planning under uncertainty, to
provide capabilities suitable for a wide range of modern-
day challenges and circumstances while working within an
economic framework that necessitates choice.
Without this understanding, we’re not only not going to be
successful at estimating, we’re not going to reach Done in
any way that supports project success.
In Agile, Features and their Stories and Tasks, implement
the needed Capabilities
Stories are the critical success factor. These stories say the
following:
§ Who is the user?
§ Why are they here?
§ What’s an example of this person?
§ Why do they want to do this thing?
§ What’s the benefit of them doing it?
§ How will we know it is working?
7/29/17

In FAR procurement, the budget for the program is
established at contract award. This could be a Not to
Exceed, and IDIQ, a CP Award Fee, and a variety of other
funding vehicles.
But the budget is on baseline. This budget is consumed in
an Agile software program by the development staff and
their management.
In Agile development – Scrum, XP, DSDM, and others – the
spend profile is essentially flat. No Curve, no spikes, no
ramp up and ramp down – flat.
This is the ideal situation to measure program
performance by measuring value produced in exchange for
budget.
Physical Percent Complete, defined for the Effectiveness
and Performance Measures, starting with the ConOps and
moved to the Product Roadmap, can produce the estimate
to Complete, Estimate at Completion, and Estimated
Completion Data in units of measure meaningful to the
decision makers. These Units are Dollars and Hours.
The units of measure in agile – Story Points – have no
meaning for those decision makers.
7/29/17

With the contract awarded and the PMB established and
verified at the Integrated Baseline Review (IBR), the
program can starts to be executed.
Success on Agile programs starts with limiting the planning
horizon.
Rolling Wave planning is one way to do this. Release
Planning is another, but Release Planning has a tendency
to lock in the Features too soon and commit to the
content of the Release.
Rolling Waves are easier to make changes to, since they
are just containers for the work, not the definition of the
work.
That definition tales place in the Product Roadmap,
Product Backlog, and Sprint Backlog. The rolling wave is
just an artificial boundaries to perform the work, then stop
and assess the progress for the work performed
7/29/17

In the presence of uncertainty – both aleatory and
epistemic – requirements evolve as a natural process.
In agile we encourage requirements evolution, since we
are discovering the needs of the customer at the lower
levels of detail. This idea is part of the culture of Agile.
But there are other drivers for the use of Agile.
The risk produced by the uncertainty, the evolution of the
system from it’s use.
7/29/17

7/29/17

§ Benchmarking ‒ the functional size provided is assumed
to be the size of the software delivered at the end of
the project, regardless if the software development or
management method was Agile or not. It is reasonable
for Agile projects to provide the Functional Size
Measurement of functionalities delivered at project
end.
§ Upfront estimation ‒ program require allocating
budget prior to starting, based on its cost estimation.
The first challenge in an Agile context is to obtain this
estimation for a defined project scope. At the early
stage of an Agile project, scope can be defined as a list
of needed Capabilities, with their functional
requirements behaviors (Features and Stories).
§ Iteration Planning and Project Re-Estimation ‒ At Sprint
Planning, with functional size estimated, it is
appropriate to perform a Functional Size Measurement
on Product Backlog items for the current Sprint.
§ Process Improvement Monitoring ‒ continuous process
improvement through “retrospective” is performed at
the end of each iteration.
7/29/17

7/29/17

Success for Agile projects starts with the RFP/RFQ.
That’s where the ground rules for the agile acquisition are
laid down for the suppliers.
In the PWS the supplier needs to describe the Software
Development Method and how that method implements
the 12 Principles of Agile
A Product Roadmap is the foundation of success for Agile
projects. It states what Features will be contained in what
releases and when those releases will be ready for use by
the Government. The releases can be Capability Releases
or Cadence Releases.
A Performance Management Plan and Quality Assurance
Surveillance Plan (QASP) are the basis of assessing
programmatic performance of the program. This is where
the Measures of Effectiveness, Measures of Performance,
and Key Performance Parameters are collected.
These are connected to the Released Capabilities with
there Features and form the basis of Physical Percent
Complete.
7/29/17

7/29/17

Starting with IT systems subject to OMB A-11 Part 7 and
moving on the FAR 34/DFARS 234 systems, agile
development processes are integrated with Earned Value
Management in the following way.
Even without the flow down for EVM, managing the
development of software using Agile, inside a FAR 15
contract, can be done with this structure
§ Define the needed capabilities
§ Produce a Product Roadmap and Release Plan
§ Assign budget for the staff developing the software at
the Sprint level
§ Decompose the Capabilities into Features, then into
Stories, and assign the development work to Sprint.
§ Perform the work in the Sprint
§ Measure Physical Percent Complete at the Sprint level
§ Roll this Physical Percent Complete to the Features and
Capabilities and calculate ETC, EAC, and ECD from those
measures.
7/29/17

Agile Project Estimates Emerging Requirements

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a Agile Project Estimates Emerging Requirements

Similar a Agile Project Estimates Emerging Requirements (20)

Más de Glen Alleman

Más de Glen Alleman (20)

Último

Último (20)

Agile Project Estimates Emerging Requirements