The Pathologies of Failed Test Automation Projects

W3
Test Automation
5/1/2013 11:30:00 AM

The Pathologies of Failed Test
Automation Projects
Presented by:
Michael Stahl
Intel

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Michael Stahl
Michael Stahl is a software validation architect at Intel, working with a team that validates Intel's graphics
hardware drivers. In this role, Michael defines testing strategies and work methodologies for test teams,
and tests part of the product himself - the work he enjoys most. As a twenty-two year veteran and senior
engineer, he has been witness to what works - and what doesn’t - in test strategies, communications, and
automation. An avid teacher, Michael enjoys sharing his observations with others. Contact Michael at
michael.stahl@intel.com and review previous conference presentations on his website testprincipia.com.

The Pathologies of failed
Test Automation Projects
Michael Stahl, Intel
Apr 2013

Based, in part, on work done with Alon Linetzki, Best Testing (www.best-testing.com)

2

On the Menu…
•
•
•

Automation failure patterns
What can we do?
Summary

Based, in part, on work done with Alon Linetzki, Best Testing (www.best-testing.com)
© Michael Stahl, 2013, all rights reserved

Disclaimers







Names and brands referenced herein may be claimed as the property of
third parties
The views expressed in this presentation are solely my own, and do not in
any manner represent the views of my employer
Information in this presentation is provided “AS IS” without any warranties
or representations of any kind

A common automation story…
4



Once upon a time…
I can automate
my work!...

Mr. Otto Mate

Cool!
Can you make it…

Otto’s Team


5

Piece of
cake!
Perfect!
Can you add…
Mr. Otto Mate

What a wonderful
world!...

Otto’s Team
Mr. Mann a. Ger

6

Arggghhh!
Fix Fix Fix
Otto! The
last release
fails!...

Otto’s Team

Mr. Otto Mate
… and mates

Sir!
I need time!
I need help!

Makes sense…

Mr. Mann a. Ger

7

Mr. Otto Mate
… and mates

… and mates

… and mates

8

Let’s
REDESIGN!!!

Mr. Oto Mate
… and mates

#$&@***!!!
… and mates

Mr. Mann a. Ger

… and mates

9

A pattern emerges…


Pattern #1: Mushrooming


A pattern…
11

Single User / Developer

Simple Tool

Stage 1 – Small & Local

A pattern…
12

Multiple Users / Single Developer

Enhanced Tool

Stage 2 – Generalization

A pattern…
13

Multiple Users &
Developers

Complicated Tool

Stage 3 – Staffing

A pattern…
14

Multiple Users &
Developers

Test Case
Management

Stage 4 – Non-core features

A pattern…
15

Stage 5 – Overload

Arghhh!

Pattern #2:
The Competition
16


Pattern #2 – The Competition (ver. 1)
17

Team A

Team B


18

Team A

Team B

Team C

!!!

Team D

19

Team A

Team B

!!!

Team C

!!! …

…

!!!

…

Team D

Pattern #3: The Night Run Fallacy


Pattern #3 – The Night Run Fallacy
21


22

Night Time
Test Time

23

Add a snapshot – or a movie snippet – of PAVE

Corporate Truism:
It’s easier to get budget for machines than for more testers

24

Add a snapshot – or a movie snippet – of PAVE


25

Test Automation Truism:
Machines create work for more testers

The Tragedy of the Commons
multiple individuals…
will ultimately deplete a shared limited resource…
even when it is not in their long-term interest
Garrett Hardin, Science, 1968


Pattern #4:
Going for the Numbers


28


Pattern #5 – Going for the Numbers
29


30

Robustness is Invisible


Pattern #5:
The Magician Stahl, 2013, all rights reserved
Apprentice Syndrome
© Michael

Pattern# 5 – The Magician Apprentice
32


Recap: The Patterns
Mushrooming
3
3

The Competition
The Night time Fallacy
The Magician Apprentice


34

So...

What can we do?


35

Are the patterns…

Unavoidable ?


?

36

Counter measures


37

Mushrooming


Alert Signals & Counter Measures
38

How to use this information?
 GPS
 Locate

the stage you are at
 Get directions for the way out


Map
 Start

right and avoid
the wrong turns


Alert Signals
39


Alert Signals
40


Alert Signals
41

Failing the project

Failing the project

Failing the project

Failing the project

Failing the project


Alert Signals: Stage 1
Small, local, feature-centered

42







Single feature test tool?
The creator is the user?
“Skunk works”?
Key words:



“Tool”
“Utility”


Counter Measures: Stage 1
Small, local, feature-centered

43

Ensure the following… and relax:
 Code control
 Documentation





User Manual (Usage line…)
“Green” in the code
High level design


Generalization

44










Additional features?
Multiple users?
Automation web site?
>25% of the tester’s time?
Key words:
“Use by other testers”
“Common Libraries”

Generalization

45

“The hardest part of building a software system is deciding
precisely what to build... No other part of the work so cripples
the resulting system if done wrong. No other part is more
difficult to rectify later”
- Fred P. Brooks (author of “The Mythical Man-Month”)


Generalization

46






Stage 1 measures
Strategy
Architecture
Lightweight PM





Version control
Scope control
Bugs & Requests database

Institutionalization and staffing

47












Requests for additional heads?
Automation F2F?
Tool-related delays in execution?
Key words:
“Tool Owner”; “Automation team”
“Framework”; “Infrastructure”
“Roll back”
“Bug fix release”

48



49

Stage 1, 2 counter measures
Management level decision time













Programming language
Framework focus

Code and release management
Acceptance tests
Skillset development
Metrics


50

Metrics suggestions
 Automation framework quality









Number of false fails
Framework’s test results; Bug trends

ROI
Number of runs
Invested effort by type (new,
maintenance, rewrite)
Number of bugs found by Automation (?)

Change of focus: Technology  Management

51









Generic features?
Test log wading?
False fails?
Key words:
“test suite / cycle generation”
“robustness enhancement”
“setup issues”


52

Stage 1, 2, 3 counter measures
Build VS Buy
Re-architect







Core / Non-Core
Solid infrastructure

Influence upstream





Testability hooks
Design for Test (automation)




Build VS Buy?

(hint: Buy)
See: http://www.stickyminds.com/s.asp?F=S17601_COL_2

Stage 4 – Non-core features


54

Build (VS Buy) if…










Competitive edge
Existing expertize
Core competency
Cheaper; Faster
Good use of resources
Acceptable risk
Long term support

Main source: Allen Eskelin
http://www.informit.com/articles/article.aspx?p=21775


Maintenance overload; Re-design

55

Maintenance & logistics overload?
Limitations overplay?
Loss of credibility?
Stage 1 initiatives?
Key words













“Did it fail in manual test?”
“Architecture limitation”
“refactoring”; “redesign”
“…I can write a small program…”


56

Loss of credibility



57

Options:







Continue… 
Give up problematic areas
Partial return to Stage 1
“We value Robustness over New
Features”
Prepare for re-design – with a new
map...

How to Use this information?
58



GPS




Locate the stage you are at
Get directions for the way
out

Be alert for
Alerts

Implement
Counter
Measures

Identify your
stage

Analyze your
situation


How to Use this information?
59



Map


Start right and avoid
the wrong turns

Plan your trip

Implement
Counter
Measures

Be alert for
Alerts

Analyze your
situation


60

The Competition


61




Salvageable up to stage 2
Stage 3 and up:




Merge the teams
EOL both tools
Start a 3rd


62



Plugin Architecture


63



Fix the main problem
Accept, encourage Stage 1 stuff…



Maintaining vigil:





Avoid ver.1 pattern


64

The Night Run Fallacy


65

6
5






6
5

Never forget test time, test efficiency
Balance test skills VS automation skills
Allocate machines or machine time

66



67



Change how you count “Done”


68

The Magician Apprentice Syndrome


69

http://www.youtube.com/watch?v=zlnz1rSJj7Y


70

Automation should not necessarily mimic
humans…
… it should get the job done


71

http://www.youtube.com/watch?v=qDVYIT85ntg


72






Holistic approach VS “test after test”
Re-think, re-strategize, re-evaluate
before automating
Identify Actions, Keywords (KDT)


73

Summary


The Patterns
Mushrooming
7
4

The Competition
The Night time Fallacy
The Magician Apprentice


Summary
75

The patterns are pervasive
 Almost inevitable
 Awareness is key (engineers; managers)
 Driven by organizational and human nature
 The solution is only partially technical



Thank You!

Questions time…

michael.stahl@intel.com

www.testprincipia.com


The Pathologies of Failed Test Automation Projects

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (7)

Más de TechWell

Más de TechWell (20)

Último

Último (20)

The Pathologies of Failed Test Automation Projects