James grenning test is not about finding bugs

www.renaissancesoftware.net
james@renaissancesoftware.net
Copyright ©2008-2009 James W. Grenning
All Rights Reserved. . Photos from James Grenning’s Personal Collection or Wiki-Commons
Test is NOT About
Finding Bugs!
James W Grenning
Renaissance Software Consulting, Co.
twitter: jwgrenning
1
1Monday, October 18, 2010
All Rights Reserved.
2
www.agilemanifesto.org
3
4
More Functionality in Software
6 years
5
More and More Software
6

7
8
9
10
11
12

13
14
15
16
2000’s
Now vs. Then
17
SW
SW
HW
1960’s
SW
HW
The Nature of Software
• Software is fragile
– The nature of discrete systems
• Any change can break just about anything
• Test and forget model leads to big surprises and
problems
18

19
Lot’s of Software means
Lot’s of Problems
• 10-50 defects per 1000 Lines of Code (KLOC)
• How many bugs in
20
21
Voyager
22
23
Mars Climate (burns in) Orbiter
24

25
1990 - AT&T #4ESS
26
27
28
29
Did you say something?
What the heck!?! Where are we?
30

Defect Found During Simulation
31
32
33
34
Don’t Test to Find Defects
Test to Prevent Defects
• Automate
• Test as you go
• Re-test everything
• Code is guilty until proven innocent
35
Manual Test Is Unsustainable
36

Et = f(Ed)
Et is the effort to test a new feature, and is a function of the
effort to develop the feature.
Ed is the effort to develop a new feature
Assume a constant linear relationship
Et = f(Ed) = KEd
37
Conservative Estimate of Developer
to Tester Ratio
38
0
5
10
15
1 2 3 4 5 6 7 8 9 10
NumberofPeople
Iteration
New functionality manual test effort
New functionality development effort
It’s not that Easy
Software is Fragile
• 25% of defects are introduced while changing
existing code (study by Grady)
– The new code works
– Some existing code is unintentionally broken
39
Etn = f(Ed) + C Et(n)
Etn is the effort to fully test a product at iteration N
Because any change can break previously working
software, we must retest.
Etn is a function of the effort to develop the feature
plus some fraction of the effort to test all previous
iterations.
40
i = 0
n
x
Unsustainable Growth with any
Coefficient
41
0
5
10
15
1 2 3 4 5 6 7 8 9 10
PeopleNeeded
Iteration
Assume regression test effort is 50%
original manual test effort
Total manual test effort is
unsustainable
Ever Growing Need
Flat Budget
UntestedCodeGap
So, Maybe we should only test at the
end of the development cycle
42

FBI Case-File System
43
Cost of BDUF gone bad
44
Should We Wait Until the End to Test?
Requirements
Code
Design
Test
Time
NovemberSeptemberJulyMayMarch
Test and Fix
Test and Fix
Test and Fix
Test and Fix
Test and Fix
Test and Fix
Test and Fix
Test and Fix
Test and Fix
Test and Fix
Test and Fix
Test and Fix
Test and Fix
Test and Fix
Test and Fix
Test and Fix
Test and Fix
45
Test
This Work Flow is Designed to Allow
Defects
46
Development
Test
Defects
Development and Test Work Together
Preventing Defects
47
Development
Test
Testing is not a Phase
• Testing starts on day one
• Tests provide the
specification of what is to
be developed
• QA/System Test moves
upstream.
48
HELP!

Edsger Dijkstra
Those who want really reliable software will
discover that they must find means of avoiding
the majority of bugs to start with, and as a
result, the programming process will become
cheaper. If you want more effective
programmers, you will discover that they
should not waste their time debugging, they
should not introduce the bugs to start with.
49
Can we Realize Dijkstra’s Dream and
Prevent Defects with
Test Driven Development?
50
Zune 30G
• On December 31, 2008 this
function went into an infinite loop.
• Ruining New Years Eve Parties
around the world.
51
The buggy function:
It accepts the number of days since January 1, 1980 and
figures out: day, date month and year.
52
Zune 30G Bug
BOOL ConvertDays(UINT32 days, SYSTEMTIME* lpTime)
53
This Test Could Have Prevented the
Zune Bug
TEST(Rtc, check20081231)
{
days = daysSince1980(2008, 366);
CHECK(ConvertDays(days, &time));
assertDate(WED, 31, 12, 2008);
}
What Tests do we Need?
• One Test Would Have Prevented Zune Bug
• If we knew where the bugs are, we could just write
those tests (or fix the bug).
• Bugs can be anywhere, so we have to write tests for
everything.
54

Uncle Bob’s Three Rules of TDD
1.You are not allowed to write any production code
unless it is to make a failing unit test pass.
2.You are not allowed to write any more of a unit test
than is sufficient to fail; and compilation failures are
failures.
3.You are not allowed to write any more production
code than is sufficient to pass the one failing unit test.
source: Robert C. Martin, blog.objectmentor.com
55
TDD State Machine
56
Write the test
Deal with the
linker
Deal with the
Compiler
Compilation error
Link error
New test failsMake it work
Refactor
(Make it right)
All tests pass
All tests pass
No more tests
Choose a test
Start
Compilation error
Link error
DONE!
Programming error
Compiles Clean
The Physics of Debug Later
Programming (DLP)
• As Td increases, Tfind increases dramatically
• Tfix is usually short, but can increase with Td
57
Bug discoveryMistake made
(bug injection)
Bug found Bug fixed
Td Tfind T fix
Time
The Physics of Test Driven
Development
• When Td approaches zero, Tfind approaches zero
• In many cases, bugs are not around long enough to be considered
bugs.
• See: http://www.renaissancesoftware.net/blog/archives/16
58
Mistake
discovery
Mistake
made
Root cause
found
Mistake fixed
T d Tfind T fix
Time
TDD
Progress in the Right Direction
• See: http://www.renaissancesoftware.net/blog/archives/33
59
Code
without
new feature
Code with
new untested
feature
Code with
new tested
feature
Write all production code at once
Debug
Test Driven Development
Write the test
Deal with the
linker
Deal with the
Compiler
Compilation error
Link error
New test failsMake it work
Refactor
(Make it right)
All tests pass
All tests pass
No more tests
Choose a test
Start
Compilation error
Link error
DONE!
Programming error
Compiles Clean
How Can we Keep the Cost or Retest
Sustainable? -- Automate
60
0
5
10
15
1 2 3 4 5 6 7 8 9 10
PeopleNeeded
Iteration
Assume automation test effort is 100%
more than original manual test effort
Relatively Flat and Sustainable

How Can we Keep the Cost or Retest
Sustainable? -- Automate
61
0
5
10
15
1 2 3 4 5 6 7 8 9 10
PeopleNeeded
Iteration
Relatively Flat and Sustainable
Ever Growing Need
Flat Budget
UntestedCodeGap
But TDD won’t work for us!
We have special problems!
• Fire alarm
• Home automation
• Banking
• Insurance
• Robotics
• Medical electronics
• Printing
• Gambling machines
• Telecom
• Police radio
• Cell phone
• On-line gaming
• Security software
• Mesh networks
• Test equipment
• Medical records
62
Yes, You are Special and Unique
• But TDD can work in many environments
63
But I’ve seen it work.
Everywhere someone lets it.
• Fire alarm
• Home automation
• Banking
• Insurance
• Robotics
• Medical electronics
• Printing
• Gambling machines
• Telecom
• Police radio
• Cell phone
• On-line gaming
• Security software
• Mesh networks
• Test equipment
• Medical records
64
But I’m special
I have ____________ Dependencies
65
Fill in the blank*
*Hardware, OS, Database, Regulatory, Real-time, User, Network...
Unmanaged Hardware Dependency Lead
to Manual Testing
66
The Net
Core Software

But I Have Hardware Dependencies
• Break the
dependencies
67
<<interface>>
Time Service
+ getDay()
+ getTimeOfDay()
+ setPeriodicAlarm()
Light
Scheduler
Test
Light
Scheduler
+addSchedule()
+removeSchedule()
+wakeUp()
<<interface>>
Light Controller
+ on(id)
+ off(id)
Light Controller
Spy
Fake
Time Service
<<implements>> <<implements>>
Design Must Accommodate Test
TEST(LightScheduler, TurnOnEverydayNotTimeYet)
{
LightScheduler_TurnOn(3, TimeService::EVERYDAY, 510);
whenItBecomes(TimeService::MONDAY, 509);
expectLight(3, LIGHT_UNCHANGED);
}
TEST(LightScheduler, TurnOnEveryday)
{
LightScheduler_TurnOn(3, TimeService::EVERYDAY, 510);
whenItBecomes(TimeService::MONDAY, 510);
expectLight(3, LIGHT_ON);
}
68
Managed Dependencies Enables
Automated Testing
69
Copyright (c) 2009 James Grenning
Core Software
Test scripts
Test
Agent
Test
Controller
There are Different kinds of
Automated Tests
• Unit Test
– Feedback to the developer that the code does what is expected
– Written using a Unit Test Harness (e.g. unity, CppUTest, …)
• Story Tests - Executable use cases - Our focus
– Feedback to the Customer that the code meets the requirements
– Used at many levels
–Component
–Groups of integrated components
–System
– Written in a domain specific language (e.g. FitNesse) [FITNESSE]
• Integration, System, Load Tests...
70
Unit tests are critical
• A house made from solid bricks can stand a long time
• TDD gives solid building blocks
• Unit tests tell programmers that code works like they
think it works
71
System Level Tests Cannot be
Thorough
• 1000 (or more) tests are needed to test this simple
system
72
10
states
10
states
10
states
5 interactions
5 interactions
5 interactions
Tests

TDD unit tests can be thorough
• As few as 30 unit tests and 15 integration test when
tested as units
73
10
states
10
states
10
states
5 interactions
5 interactions
5 interactions
Tests Tests
Higher Level Tests are Needed Too
Could Test Help Specify what is needed with Executable
Use cases
See http://renaissancesoftware.net/papers.html
For Executable Use Cases Paper
74
Use-Case Template
• Name:
• Goal:
• Preconditions:
• Success End Condition:
• Failed End Condition:
• Primary Actor:
• Trigger:
• MAIN SUCCESS SCENARIO
• EXTENSIONS
Source Alistair Cockburn
http://alistair.cockburn.us/Basic+use+case+template
75
Use-Case Example
76
Information Description
Name Schedule light control
Goal Allow system users to schedule lights to turn on, off, or
dim
Preconditions System has controllable lights attached
Success End Condition The scheduled light has been controlled at the scheduled
time
Failed End Condition The scheduled light has not been controlled at the
scheduled time
Primary Actor Home owner
Trigger Scheduled time is reached
Main Success Scenario 1.The home owner schedules a light to turn on at a
specific time on a specific day
2.The scheduler wakes up at the right time of the right day
3.The light scheduled for this minute is turned on
Use-Case Example
Continued
77
Information Description
Extensions/Variations 1a. Homeowner can schedule the light to turn on
1b. Homeowner can schedule the light to turn off
1c. Homeowner can schedule the light to set to a dim
level
1d. Homeowner can specify weekend schedule
1e. Homeowner can specify weekday schedule
2a - Scheduler does nothing when it wakes up and there
are no scheduled controls.
3a - Light is turned on when on is scheduled
3b - Light is turned off when off is scheduled
3c - Light is set to a specified level when dim is scheduled
Executable Use-Case with FitNesse
78
Items in tables are tell to FitNesse how to
interact with the system under test
Initialization
Cleanup

Passing Test
79
Failing Test
80
A Suite of Tests While all Test Pass
81
When the Story Test is Ready Before
the Development Finishes
82
When Things are not Working
83
FitNesse Test Architecture
84
<<interface>>
Time Service
+ GetTime()
+ SetPeriodicAlarm()
Light
Scheduler
Fixture
Light
Scheduler
+ ScheduleTurnOn()
+ RemoveSchedule()
+wakeUp()
<<interface>>
Light Controller
+ On(id)
+ Off(id)
Light Controller
Spy
Fake
Time Service
<<implements>> <<implements>>
FitNesse
Wiki
Server
CSlim
Library
<<socket>>FitNesse Test Pages

Test is NOT About
Finding Bugs!
• TDD to prevent bugs
• Automate repetitious tests
– to keep bugs out
– to keep the cost of re-test low
• It’s not easy, until you learn how to do it
– then it’s still not always easy, but worth it
85
Don’t let your Product Appear on a
Future Famous Failures Presentation
86
Thank You!
87
Available now in Beta
http://pragprog.com/titles/jgade/
In print early 2011
88

James grenning test is not about finding bugs

Recomendados

Recomendados

Más contenido relacionado

Similar a James grenning test is not about finding bugs

Similar a James grenning test is not about finding bugs (16)

Más de Odd-e

Más de Odd-e (20)

James grenning test is not about finding bugs