Testers are often evaluated by metrics that don’t really quantify the value of their work. Metrics such as tests planned, tests executed, coverage achieved, and defects reported all measure effort rather than results. Since people generally want to meet metrics goals, measurements that focus on activity rather than effectiveness often encourage unintended behaviors. Since the true value of testers lies in their ability to analyze and communicate risks and impacts, we must change the focus of metrics from numbers to insights. Scott Barber shares what stakeholders are really looking for when they request specific metrics, how the metrics they request frequently fail, and how to help your organization create metrics that do provide real insight. Discover the tools you need to explain what can be measured, what those measurements mean, and how to combine measurements into metrics that tell insightful stories about your testing.
2. Scott Barber
SmartBear
Chief performance evangelist for SmartBear Scott Barber is a respected leader in the
advancement of software testing practices, an industry activist, and load testing
celebrity of sorts. Scott authored several books―Performance Testing Guidance for
Web Applications, Beautiful Testing, How to Reduce the Cost of Testing, and Web
Load Testing for Dummies―and more than 100 articles and blog posts.
Founder/president of PerfTestPlus, Scott co-founded the WOPR, served as director of
the AST and CMG, and is a founding member of ISST. His industry writing, speaking,
and activism focus on improving the effectiveness and business alignment of software
development practices. Learn more about Scott Barber.
3. 2/25/2014
1
How Metrics Programs
Can Destroy Your Soul
Created for:
How Metrics Programs
Can Destroy Your Soul
By:
Scott Barber
Product Owner – Load Testing Tools
SmartBear Software, Inc.
4. 2/25/2014
2
Product Owner – Load Testing Tools
SmartBear Software, Inc.
scott barber@smartbear comscott.barber@smartbear.com
@sbarber
Co‐Founder: Workshop On Performance and Reliability
www.performance‐workshop.org
Co‐Author:Author: Contributing Author:
Books: www.perftestplus.com/pubs
About me: about.me/scott.barber
Have valid needs to:
• Assess & compare performance & quality of:
di id l
Managers & Execs
• Individuals
• Teams
• Projects
• Products
• Compute costs
• Identify risks & trends
5. 2/25/2014
3
Commonly satisfy those needs via:
• Instinct
• Not Scalable
Managers & Execs
• Not Scalable
• Non‐transferable
• Qualitative Metrics
• “Group Gut”
• Inconsistent
i i i• Quantitative Metrics
• Scalable
• Transferable
Commonly satisfy those needs via:
• Instinct
• Not Scalable
Managers & Execs
• Tied to individuals
C ’ b i d• Not Scalable
• Non‐transferable
• Qualitative Metrics
• “Group Gut”
• Inconsistent
i i i
• Can’t be trained
• Predicated on trust
(not data)
• Virtually impossible to
defend
• Generally reserved for
“hands on” managers
• Quantitative Metrics
• Scalable
• Transferable
hands‐on managers
with one project or
team
6. 2/25/2014
4
Commonly satisfy those needs via:
• Instinct
• Not Scalable
Managers & Execs
• Very difficult to do
ll• Not Scalable
• Non‐transferable
• Qualitative Metrics
• “Group Gut”
• Inconsistent
i i i
well
• Expensive
• Time consuming
• Not good for detailed
comparisons
• Not good for
estimation
• Quantitative Metrics
• Scalable
• Transferable
estimation
• Few managers/execs
trained in using them
Commonly satisfy those needs via:
• Instinct
• Not Scalable
Managers & Execs
• Scale well
P id i li i &• Not Scalable
• Non‐transferable
• Qualitative Metrics
• “Group Gut”
• Inconsistent
i i i
• Provide simplicity &
consistency
• Easy to use
• Good for detailed
comparisons
• Good for estimation
• Good for trends
• Quantitative Metrics
• Scalable
• Transferable
• Good for trends
• Most managers/Execs
trained
• Validated in many
areas over long
periods of time
7. 2/25/2014
5
Are based on measurements that are:
• Verifiable
St d di d
Effective Metrics
• Standardized
Are explicitly linked to goals
• Correlated/causal
• Compliance
F t d b l tFavor trends over absolutes
Evolve
Are not used in isolation
“Most metrics are created, collected, and reported to satisfy a
leader’s request. The leader’s role is to supply clarity and direction
by providing the proper questions. Middle management’s role is to
th ti M t i ff f idi th
What is a Metric?
answer the questions. Metrics offer a means of providing the
answers so that all involved can have faith in them.
Unfortunately, leaders often don’t know exactly what they want.
Chances are you have played the Guessing Game with a leader,
where the data you provided wasn’t what he needed, so he asked
for different data, figuring he would know the right data when he
saw it Despite repeated failures you continued to chase data as ifsaw it. Despite repeated failures, you continued to chase data as if
all the effort invested in collecting the wrong data would eventually
prove worth your perseverance.”
From: “Do-It-Yourself Metrics”
by Martin Klubeck, Michael Langthorne, and Don Padgett
http://www.educause.edu/ero/article/do-it-yourself-metrics
9. 2/25/2014
7
Quantitative Metrics:
• Obscure Details
ll & i
Far From Perfect
• Roll‐ups & summaries
• Focus on “big picture”
• Risk Measurement Dysfunction
• People tend to optimize metrics
• Metrics optimization invites “bad stuff”
• Are “invitations to conversations”
• Don’t tell the whole story in isolation
• Can be wildly misleading
… and…
Common quantitativeCommon quantitative
management metrics
models are not a good fit
for software development
