Joseph Flahiff, CEO of Whitewater, did a recent Agile QA audit for the Office of the CIO for the State of Washington.

1. “Your systems are currently optimized to deliver the results you are now getting.
If you want different results you have to change the system. To expect anything
else is by definition, insanity.” ~ Joseph Flahiff
June 16, 2015
Michael Cockrill, CIO
Washington State Office of the CIO
Dear Michael,
Attached you will find the WA Business Hub Project’s QA report for May. This is only the second
report sent. From now on they will be coming on a monthly basis. Since this approach to QA is a
work in process you may see some categories changing, being added, or being removed.
It will appear that the overall numbers have gone down but it really is an effect of the number of
categories changing.
Thank you again for this exciting and innovative project, I am proud to be a part of it.
Sincerely,
Joseph Flahiff,
CEO
Whitewater Projects, Inc.
206.276.1386
joseph@whitewaterprojects.com

2. WA Business Hub Project
Agile Project QA
Prepared for: Michael Cockrill, CIO
Prepared by: Joseph Flahiff, CEO
Description
An independent assessment of the agile practices on the WA Business Hub project. This
report includes an executive summary and detailed analysis of the project’s agile project
delivery practices.

3. QA for WA Business Hub Project: WA OCIO
We want you to be successful
Executive Summary
Project Assessment
This project is a follow on to the research project held in 2014 to address Senate Bill 5718 which found
that, “regulatory agencies were directed through an executive order in 2006 to develop a one stop
business portal, but that a one stop business portal has not been developed.”
The project is early in the startup and is being run with the Scrum framework. The work began in
2014. The project budget is $737,114
Findings Summary
As stated in the last report in January, they know what product they are developing but they do not
have a well-defined and prioritized/ordered backlog. This results in some confusion and a lot of
mixed priorities.
 The culture of the team is its greatest strength.
I am optimistic about the chances for success of this project based upon the culture, teamwork and
their receptivity to suggestions.
A few items will be critical to this success. These items include:
 Reducing the interruptions from other work
 Obtaining feedback from actual customers
 Establishing a good backlog of stories that can be used to forecast the completion of the
project

4. How to read the assessment
The agile coach/independent QA observed the team and provided interactive coaching to help the
team improve their agile practice. The assessment includes an evaluation of the team in 4 areas.
The 4 Aspects of Agility
Every organization is a complex system that can be described with 4 key aspects:
 Business processes
 Technical practices
 Organizational Culture
 Leadership Approach
Business Processes - Business Processes are the set of continuously evolving and improving; processes,
tools, beliefs and policies that support the business. Agile/nimble business systems enable managers and
leadership to make quick decisions and they support the prioritization of work that enables the delivery of
customer and business value.
Agile/nimble business processes support the technical practices and are enabled by the leadership. They are the
systems that enable teams, business leaders, analysts and managers to easily work together in tight knit groups
to deliver customer value.
Technical Practices - Technical Practices are the practical tools and techniques used by teams,
management, and leadership to deliver customer value quickly and incrementally.
Flexible Technical Practices do not merely make exceptions for the possibility of change, they are based upon
the certainty of change and are designed to leverage it for the benefit of the customer and the business.
Organizational Culture – Organizational Culture is a set of organizational beliefs and behaviors. In
organizations that supports the ability to adapt to changes as they come. A high value is placed on being
transparent and realistic, even about the ‘bad news’ these cultures focus on delivery of one thing, innovation
and customer value above all. These organizations have fully engaged teams that self-organize with motivated
team members who collaborate in real-time. A mindset of continuous improvement is key to the long-term
success.
Leadership Approach- The leaders’ job is twofold. The first job of the leader is to create psychological
safety for the teams, empowering them to innovate. Second, leaders need to provide clear strategic intent,
guidance on WHAT is to be done helping the team focus on the value that they are delivering to the customer.

5. Measurement
The following scale was used to evaluate the team. These results were shared with the team and they
agreed with the assessment.
Scale
 0 indicates does not exist
 1: indicates exists but not understood
 2: indicates exists and understood but not improving
 3: indicates is in a kaizen state
 N/A Not Applicable to this context
Risk Ranges:
 Low Risk = Average between 3 and 2.1
 Moderate Risk = Average between 2 and 1.1
 High Risk = Average between 1 and 0
Project Summary
Jan-2015 May 2015 June-2015
Technical
Practices
High Risk
Medium
1.25
Business
Processes
High Risk
Medium
1.4
Organizational
Culture
Low Risk
Medium
1.81
Leadership
Approach
Low Risk
Low
2.42

6. Observations
Technical Practice
Technical Practice Observation Recommendations Response
Pairing 2 – Some swarming and some
pairing.
Team will continue to push to
reduce work streams.
Continuous Integration/
Continuous Delivery
2 – Vendors are using GIT Hub. Pairing is occurring within
every recent sprint and
swarming was exceptional
recently.
Loosely coupling systems 2 – They do it but are not really
improving it.
New QA plan and definition of
done will require continuous
integration. This process was
just documented and reviewed
with team.
Refactoring 1 – some is being done but not
consistently
I believe this is more of a
symptom at this time than a
cause. It is difficult to refactor
when the team is not focused
on one project.
Plan for Business Licensing
Guide is to decouple it into its
own app, separate from the
site.
Emergent Design 1 – It is happening but not at a
conscious level
Agreed. Quite a bit of
refactoring happens because
of the separate work streams

7. and it is not currently
purposeful.
Test First Development 0 – Not really doing this yet.
Technically don’t have the skills
in PHP
More attention could be
placed upon developing
acceptance criteria for stories.
Agreed. Team unfortunately
sacrificed user engagement the
past few months to gain dev
cycles. It wasn’t an easy
choice.
Collective Code Ownership 3 – Still improving. Ryan has
integrated well with this
culture.
+1 Goal is to include user
community after v1.
Coding Standards 2 – developed standard on
Accessibility, compliance, API,
etc… given to the vendors and
following internally,
There is not clear
understanding across the team
about this issue. Having a brief
discussion would help.
Somewhat disagree. Current
state is we faltered in this due
to terminating work with
primary vendor, but it was in
place and awareness of it
exists.

8. Business Processes
Business Process Observation Recommendations Response
Backlog 1 – Partial backlog exists. But it is
not sufficient to plan the work
Improvements have been
made but the team is so busy
delivering that it has not
backed off to plan.
Agreed. We have developed a
complete backlog for features
like search and BLG—an
improvement--but not for the
entire site when stories are
created typically prior to the
sprint.
Will continue to press on this
important issue.
Velocity/Throughput tracking 1 – They are tracking it but Do
not appear to be doing
anything with it. Commitment
is consistently higher than
delivery
Agreed. To further use velocity
as a predictive measure, we
need a backlog to predict.
Currently, It is used solely used
to measure yesterday’s
weather to estimate sprint
commitment.
Stakeholder Demo 3 – Their demo from what I
saw was good.
No additional emphasis need
be put on this topic until the
team can focus and is not
randomized.
Thanks!

9. Customer Demo 0 – There appeared to be one
planned but it did not come
through.
No customer demos yet. Agreed. Unfortunately, we cut
the planned customer demo
work due to capacity issues.
Vertical slicing of customer
value for stories
2 – they are working on it.
Stories still do not appear small
enough.
No additional emphasis need
be put on this topic until the
team can focus and is not
randomized.
Noted.
Stories meet INVEST criteria 2 – Do not really understand
Value for the stories at this
point and stories are not very
negotiable
Ben needs to better articulate
value to the customers and
stories should be focused on
the why and some of the what
but not the how.
Agreed. Value is the piece
missing from the cards. Plan is
to introduce hypothesis driven
development after V1 launch to
supply foundation for
measuring value.
Daily Standup 3 – they are improving on the
practice
Thanks!
Story to express intent 3 – Kudos. (Told but not
validated)
Thanks!
Small Release Cycles of
product
2 – Released search,
autocomplete is released,
contact suggestion, others to
Business.wa.gov every sprint
Agreed. Team really wants to
do more of this.
Incremental development and
release of product
1 – Product hasn’t been
released yet. But they are
doing incremental
development
Identify the MVP, the stories to
complete it and estimate the
date.
Somewhat disagree with “1”
score. We have released
incrementally several features
for search, especially.

10. Iterative improvement of
product
1-Getting input from the search
vendor and curating content
based on the feedback.
Ibid Somewhat disagree with a “1”
score. We did a lot of curation
and tweaking of content to
improve search results for
users and tracked those
significantly improved results
via user behavior.
Will publish these for broader
awareness.
Do team members know that
they are working on the most
important thing every day
1 – Stories are almost all the
same size. (1-2-3 points) limit
of 3 in motion. Stack ranked
with an interrupt buffer.
Create an experiment of your
own to define what work is top
priority and work on it.
Agreed. Will emphasize during
sprint planning.
Clearly articulated and
understood vision for the
overall effort
2 - Agreed.
Vendor Management 1 – Out of control. Agreed.
We came up with an as yet
unimplemented idea for a
Vendor Playbook, as well as a
yet unfilled role of Vendor
Manager to clearly identify the
responsibility. These two
actions are on our backlog.

11. Is value measured? 0- Agreed. It is not.
Is the project delivering more
value than it is costing to run?
0-
Does the team use a team
developed definition of done?
0 - Don’t a DOD but not
everyone knows what it is.
Take 20 minutes and define a
DOD.
Completed. We now have a
definition of done.
Does the team, on a regular
basis, reflect on how they are
working together and make
improvements
3- Consistently holding them
and identify 1-2 actions to
improve.
Clearly identify the experiments
and make them visible in the
room.
Use the 5 part sprint retro
format.
Agreed.
Will try the recommended
format in upcoming sprints.
Organizational Culture
Culture Trait Observed Recommendations Response
Commitment to the
team
3 – Really committed and passionate.
They are not excited about the
Vendors though.
Keep it up but do not over commit.
Your health and your family are
more important than the work. See:
Sustainable Pace below
Thanks, they are super committed.
Agreed that sustainable pace needs to
happen.
Interdependence 1 – It happens but not consistently and
not by design. “random happy
accidents”
I suggest that the team try to work
in pairs more but this really isn’t a
high priority. The team is
randomized too much to make
interdependence actually pay off.
Agreed. Team is progressing to more and
more pair coding work. Additionally, we
have built into our definition of done the
requirement of paired code review prior to a
card being marked “done.”

12. Sustainable Pace 1 – Several people related that they
were working on the weekend or
evenings, it was said in the context
that they are committed to the team.
Encourage NOT working on the
weekends but pushing back on
management to create realistic
working expectations.
Agreed. Will continue to try and give air
cover in this regard. This team needs to
focus.
Focus (One thing at a
time)
1 – This is a constant struggle. People
are randomized and pulled off for
other work. Plus the time spent
managing vendors.
This is an issue mostly for leadership
(Michael D. and Ben) to address.
This is the single most important
issue for the organization. If they
can reduce the amount of
randomization from leadership, they
will improve their productivity and
can focus on some of the other
areas.
Agreed. We will work harder on focus.
Common Goal 2 – They do this well. They all know
the common goal.
No additional emphasis need be put
on this topic until the team can
focus and is not randomized.
Agreed.
Positive Failure 2 – In their team improvements they
are safe to fail. Produce piple-line
work (e.g. can we deliver BLG) it is not
ok to fail on.
Vendors are a 0 we don’t fail and
cannot fail with them because of the
vendor management process.
No additional emphasis need be put
on this topic until the team can
focus and is not randomized.
Agreed. Team has progressed on
embracing an experimental, curious attitude
towards failure and leveraging it to improve.
Restating vendor management issue is that
contract(s) have constrained vendor choice
too much, procurement reduced time to

13. deliver, ownership of delivery was unclear,
and requirements missed.
New role implemented to oversee this.
Collaboration 1 - No additional emphasis need be put
on this topic until the team can
focus and is not randomized.
Agreed. We will work on increasing focus.
Flow of Value or
Utilization Focus
1 – The focus on vendors is
detrimental to focusing on value, it is
mostly focusing on utilization.
Additionally the randomization by
other people and departments is a
utilization not value focus.
No additional emphasis need be put
on this topic until the team can
focus and is not randomized.
Agreed. We will work on increasing focus.
Psychological Safety 3 Thanks!
Team spirit - esprit de
corps (Happiness)
3 Thanks!
Is there clarity about
responsibilities/
accountabilities and
corollary authority
(Roles/whole
team/etc)
2- No additional emphasis need be put
on this topic until the team can
focus and is not randomized.
Agreed.

14. Leadership Approach
Leadership Trait Observed Recommendations Response
Servant Leadership 2 1:1 coaching for the leaders of
the organization is
encouraged. They could even
coach each other.
Agreed. This is a great idea.
We’re exploring the notion of
coaching as a service.
Continuous involvement 2 This is impacted by the leaders
(PO) being randomized, as
much if not more than others
on the team.
Agreed.
Individual leadership is
encouraged
3 Holacracy has helped in this Thanks!
Encouraging Decision Making
at the level of the information
2 They are working on this. Agreed.
Does the leadership foster
psychological safety
3 The team seems to support the
holacracy approach.
Agreed. Team seems
confident and trusting of each
other, and have made great
gains in this area.
Does the leadership provide
lean/agile coaching for the
team
3 Though external training,
consultants and coaches.
Thanks! We brought in Joe
Justice and look for other

15. opportunities in this area as
well.
Does the leadership focus on
communication of strategic
intent rather than details and
micromanagement
2 No additional emphasis need
be put on this topic until the
team can focus and is not
randomized.
Agreed.