1. Agile Project Success and Failure
The Story of the FBI Sentinel program"
&"
Tools for your Agile Tool box "
“US Digital Services Playbook” & “Tech-Fars”"
"
3 December 2015"
www.TomFriend.com "

2. Tom Friend PMI-ACP, CSP, FLEX, PSM, CSM, AHF
• Agile-Scrum Trainer - Coach "
• 12 + Years Agile Scrum Experience "
• 25 + Year IT Application Development"
– GAO Agile Advisory Work Group"
– Carnegie Mellon Agile Government Group"
– DOD CIO Software Assurance Working Group"
– Scrum Alliance Veteran Outreach Representative "
"
• US Naval Aviator & Air Force “Galaxy” Pilot"
• Squadron Commander"
• Air War College / Honors Graduate"
• Bronze Star, Combat Action Badge "

3. Learning Objectives"
1. Why is this story of value?"
2. Stress Systematic Failure"
3. Tell the FBI story of Failure and
Success of Agile."
4. Tools for Agile in Federal and DOD
programs."

4. WHY?
The Old Way is not working!"
• BDUF programs are train wrecks."
• Long Cycle can’t react to change."
• Base assumptions are not valid. "
Cost!!!! Time-Effort-Opportunity-Money "

5. Overview of the cost of Federal Failures"
• Federal IT is 78 Billion USD"
• 90% of large IT projects fail "
• Over half were miss expectations."
• 41% failed completely. 32 Billion Waste!"

6. How much is a Billion Dollars?"
• A Billion is a “thousand millions”. "
• A billion seconds ago was 1959."
• A billion minutes ago, Jesus was alive."
• A billion hours ago was the Stone Age."
• A billion dollars ago was 9 hours of Federal
spending."

7. 1 Billion In $100 dollar Bills"

8. Original Source of FBI Story"
• Brian Wernham"
• Director at The Association for Project Management "
• Delivery of £100m+ IT-enabled programs"
• Business Case development at Director Level +"
"
brian.wernham@gmail.com

9. Brian's Research Focus Area"
• Brian’s research focused on critical, main-stream
Federal projects. "
• Projects that had used various methods. "
• Based on the FBI case study, here is the question
that we will address together today: "
– What evidence is there that Agile is
suitable for large-scale projects?"

10. Viewpoints on Project Approaches"
• Normal argument is Waterfall or Agile"
• The Federal Waterfall approach will tend towards what
Kent Beck called ‘Big Design Up Front’ (BDUF)"
– Due to BDUF being the starting point for waterfall"
• Brian stresses that we should aim for Enough Design Up-
Front (EDUF), not BDUF."

11. What is Big Design Up Front (BDUF)?"
• ‘The Simpsons’ Homer the average
American, is the perfect person to
design a new car."
• The car turns out to be totally unusable
and too expensive to produce!"
• Normal output from BDUF"
– BIG"
– DESIGN"
– UP "
– FRONT"

12. Remembering failed DOD
BDUF programs"
We repeat what we forget."

13. Kinetic Energy Interceptor (KEI)"
$1,300,000,000"

14. Airborne Laser (ABL)"
$5,000,000,000"

15. XM2001 Crusader"
$2,000,000,000"

16. Expeditionary Fighting Vehicle (EFV)"
$3,300,000,000"

17. RAH66 Comanche"
$6,900,000,000 "

18. Future Combat System (FCS)"
$19,000,000,000 "

19. Brian’s Case Study :
A Story of Agile Success At the FBI
"
• The following FBI projects detail a fascinating case study of project
methodologies and how Agile leadership delivered where traditional
approaches failed."
• Three reasons this cases study is relevant"
– Size: This was a large project - hundreds of millions of dollars:
Agile Scales."
– Prior Failure: This Agile project succeeded where two previous
waterfall projects using the same technology had failed."
– Visibility: This was a high-profile and mission-critical project - in
government. Precisely the environment and scale at which Agile is
often misconstrued as being inappropriate."

20. Case study : History (part 1)"
• 2001 FBI Agent Robert
Hanssen Arrested"
• 1.4 Million in payments "
• 20 separate occasions "
"

21. Edward Snowden former CIA Employee
"
• 2013 Document discloser"
• 50,000 to 200,000 documents
leaked."
• Job was to integrate data and store
and protect it from Chinese hackers."
• Made $200,000 a year."
• Caused billions of dollars in damage."
© 2013, cPrime Inc. All Rights Reserved"

22. Case study : History (part 2)"
• Three months later, in May 2001,
the Oklahoma City bomber was
about to be executed. "
• 700 documents had not been
disclosed to the defense. "
• The FBI had forgot to send and
lost evidence. "
• Stay of execution was granted the
FBI came under severe criticism"

23. FBI looks for a Solution"
• Root Cause old computer system and outdated manual processes "
• A secure and reliable set of systems and processes were required. "
• The FBI sets up a project to build a new Virtual Case File system (VCF)."
– One $400 contract "
– Brand new system as no market alternatives were deemed sufficient"
• (SAIC) won the bid and created a classic waterfall project."
– 200 person team spent 6 months creating the requirements"
– A grand design upfront"
– Go live at once - a classic big-bang."

24. Results of First Attempt over 4 years"
• Fours CIO’s, one per year "
• 2002 No Delivery"
• 2003 No Delivery"
• 2004 No Delivery"
• 2005 No Delivery"
• Four years of no results!"

25. Scope Change 9/11"
• The 9/11 attacks increased political pressure for better
homeland security and data sharing between agencies."
• Responding to this pressure, the FBI made promises
they could not keep. "
• The FBI received an additional $78m of funding for
compliance and also promised to chop off another 6
months from the schedule."
• Summary:"
– Waterfall-style Big Bang rollout"
– $400 Million "
– $78 Million Additional Funding"
– 300 person team for Requirements + 6 months = 600 pages of
listed requirements"

26. Results of First Attempt ="
• The 600 pages of requirements were now the subject
of over 400 documented ‘change requests’, and over
700,000 lines of program code had been being
written and re-written time and time again. "
• 700,000 lines of code were cut, then worked and
reworked again and again… "
• The size of the requirements document went on to
nearly double to 1,200 pages as the development
work tried to keep pace with the changes to this Big
Design Up Front. "

27. Final Disposition Failure"
• Program Cancelled"
– A 318 page post-mortem report
concluded that the contractor had:
“badly bungled the project – it
should be abandoned … the
software is incomplete, inadequate
and (incompetently) designed -
essentially unusable under real
world conditions.”"
– Randolph Hite, GAO, said: "When
you do a program like this, you
need to apply a level of rigor and
discipline that's very high.”"

28. The need is still there…."
• That year, yet another FBI
agent, Leandro Aragoncillo,
is arrested. "
"
• The ancient FBI system had failed to spot his
suspicious behavior as he fished through the
existing case-management systems for over 9
months."
• Obviously the need for a new system
remained."

29. When you fail, blame the vendor and try
again with more money! "
• Lockheed Martin wins the new ‘Sentinel’
system project 3 years 2006 to 2009. The
total project budget was $425m."
– $305m was budgeted for Lockheed
Martin"
– $120m was allocated for the FBI to run
a massive program office to carry out
detailed and prescriptive oversight of
the work."
• That's one quarter of the budget being
spent on planning and control of the
contractor!"

30. The Sentinel project Second Try"
• The new project run as Waterfall"
– $60 million spent just creating
a web-based front-end to the
already broken system in-place"
• Some agents called it
“lipstick on a pig”."
• Due to its incompleteness,
they stopped using it."

31. Took their Eye off the ball"
• Overly optimistic reporting"
• 2010, the stakeholders
rejected the system, even
though it compliant with the
original specifications. "
• The dream of implementing
electronic information sharing
system was shattered."

32. Time to get serious - Remove impediments"
• Congressional inquiry"
• Project internalized"
• The FBI CIO takes ownership"
• Agile is adopted as the project framework"
– Design is broken into 670 user stories"
– Self-organizing teams"
– 45 staff (not 300 as previous)"
– Product Owner prioritized the work"
– Two week sprints"
– Demo every sprint"

33. Outcome, Rubber Meets Road"
• After a few sprints, it became possible to forecast
the rough timescales and start to plan the dates
for incremental business change and adoption of
releases of the new software. "
• System delivered using only half of the budget. "
• Agents used the system on real cases. In the first
quarter of its use, over 13,000 agents progressed
over 600 cases, meeting or exceeding all
expected targets. "
• The old mainframe system was turned off. "

34. Outcome in dollars and cents"
• The three-year Agile project delivered
the requested system and
improvements. "
• A success after 10 years of failure
and $600 million wasted on the two
previous aborted 'Waterfall' attempts. "
• Total cost of only $99 million. "

35. “Lessons Learned” from this?"
• Much of the Enterprise Agile literature out there was based
on a leap of faith and has not necessarily been evidence-
based."
• Organizational Leaders need credible, evidence-based
argument that puts the business case for Agile to top
management."
• Consider approaching change in a different way:"
– You can help your organizations think differently about
how objectives are agreed up front on projects. "
– You can influence your colleagues in procurement, and
those that carry out project audits, and assurance work
to adapt and harness an Agile - not waterfall, approach."

36. Lessons learned were brought to focus. "
• Patient Protection and
Affordable Care Act
(PPACA)"
• Heatcare.gov changed the
conversation. "
• HealthCare.gov went live in
2013,"

37. US Federal Agile Agency Leaders were
called upon for their Lessons Learned "
• FBI Federal Bureau of Investigation"
• USCIS US Customs and Immigration
Service."
• Veterans Administration"
• USPTO Patent and Trademark Office"

38. Directives"
• U.S. Digital Services
Playbook (Website)"
• The TechFAR Handbook
(GitHub Collaboration)"
• Leveraging Best Practices
to Help Ensure Successful
Major Acquisitions (GAO,
Feb 2014)"

39. Digital Services Playbook"

40. Purpose of the Playbook"
1. Establishing standards to bring the
government’s digital services in line with the
best private sector services;"
2. Identifying common technology patterns that
will help us scale services effectively;"
3. Collaborating with agencies to identify and
address gaps in their capacity to design,
develop, deploy and operate excellent
citizen-facing services; and"
4. Providing accountability to ensure agencies
see results."

41. Tech-Far Handbook GitHub"

43. In the end, what we need is Agile leadership"
• We need proof that Agile leadership will bring success. "
• We need people to lead their projects, not manage them. "
• 9 Agile Leadership Behaviors"
– Satisfy the customer"
– Harness Change"
– Be very incremental"
– Create trust through ‘high touch’ leadership"
– Encourage face to face conversations"
– Set targets and reword progress"
– Pursue simplicity, not complexity"
– Give team space they need to excel "

44. Retrospective"
1. Why is this story of value?"
2. Stress Systematic Failure"
3. Tell the story of Failure and
Success of Agile."
4. Tools for Agile in Federal and DOD
programs."

45. "
"
QUESTIONS?"
"
Tom Friend"
"
www.TomFriend.com "
"
"
"
"
"

46. Sources"
• The Agile approach saves the FBI Sentinel Project…"
– Brian Wernham FBCS FAPM"
• SOFTWARE DEVELOPMENT; Effective Practices
and Federal Challenges in Applying Agile Methods"
– United States Government Accountability Office"
• David A. Powner, (202) 512-9286 or
pownerd@gao.gov "
• Dr. Nabajyoti Barkakati, (202) 512-4499 or
barkakatin@gao.gov "