The document discusses testing dashboards and how they can be used to provide meaningful information to guide testing projects and convey information to management. It provides examples of effective dashboards, things to consider when designing dashboards like the types of information and metrics to include, and methods for creating dashboards. The document emphasizes that dashboards should have a few key metrics displayed clearly and be designed in a way that is easy to understand and build.
2. Bio - Randall W. Rice
•
Over 40 years experience in building and testing
information systems in a variety of industries and
technical environments
•
Certified Software Quality Analyst
•
Certified Software Tester
•
ASTQB Certified Tester – Foundation level,
Advanced level (Full)
•
Director on the American Software Testing
Qualification Board (ASTQB)
•
Chairperson, 1995 - 2000 QAI’s annual software
testing conference
•
Co-author with William E. Perry, Surviving the Top
Ten Challenges of Software Testing
•
Principal Consultant and Trainer, Rice Consulting
Services, Inc.
3. Becoming an Information Provider
•
Traditionally, testers have been
“problem finders”.
•
The more positive and value-added
view is to be an information
provider.
4. What’s In This For You?
• An affordable and easy way to:
•
•
•
•
•
•
•
Build your credibility
Promote a positive view of testing
Help guide your testing projects
Be helpful to project management
Add value to projects
Start improving processes and methods
Be seen as more than a commodity
5. The Main Objective of Testing
Dashboards
•
To provide simple, meaningful and reliable information in
one place to help guide the testing effort and convey that
information to our clients.
6. Where Are You?
a) We have a great set of dashboards
that help us make decisions
b) We have one or two dashboards that
help us make decisions
c) We have one dashboard that helps
us make decisions sometimes
d) We would like to have a dashboard
but don't know where to start
e) What's a dashboard?
7. Results From an SQE Webinar
Survey
5%
25%
We have a great set of dashboards
that helps us make decisions
We have a few dashboards that
helps us make decisions
We have one dashboard that we
use sometimes to make decisions
We would like to have a dashboard
What's a dashboard?
22%
22%
26%
8. A Testing Project is Somewhat
Like Driving a Car
•
You need to know your:
•
•
•
•
•
•
•
Destination
Current location
Orientation (Direction)
Trip progress
Speed
Resource levels (gas, oil, etc.)
Engine operation (temp, charge, etc.)
9. The Goal
•
Arrive at the desired destination safely
•
Stay on the road
•
Make good progress
•
Don’t get lost
•
Don’t run out of fuel
•
Only one driver at a time
10. Key Components
• An effective testing strategy
• Defines the test objectives, scope
and approach early in the project.
• A workable test plan
• Defines scope, resources,
schedules, risks, contingencies, etc.
• A dashboard
• Monitors defect levels, test progress,
resource levels.
12. The Testing Dashboard
• Dashboards are not new
• They have been a common topic in articles and at
conferences for several years.
• At the same time, testers often struggle
with how to convey accurate and timely
information to management.
• So…let’s explore dashboards and look at
some examples.
• Then, we’ll look at the issues behind test
measurement and reporting.
13. Example: Your Car’s
Dashboard
• Car dashboards tell you
current:
• Trip progress (speedometer
and odometer)
• Resources (fuel)
• Car status (temp, oil
pressure, charging, engine
performance)
• Warnings (seat belt, open
doors)
14. Dashboard Principles
• The number of indicators are
relative to the vehicle type
•
•
•
•
Car vs. Tractor-trailer
Car vs. Small plane
Small plane vs. F22A Fighter
F22 Fighter vs. 747 Jet
• The information is almost
instantaneous.
• The information is conveyed by
sensors.
15. The Basis of Dashboards –
Metrics and Measurements
•
Your dashboards will contain a few meaningful and
appropriate metrics and measurements.
•
The key is to pick the right ones and show them in clear
ways.
16. What Makes a Good Metric?
• Simple
• Can be easily measured and understood
• Can be automated
• So we don’t have to take readings manually
• Also, people don’t get the chance to manipulate the
numbers
• Meaningful
• We can gain useful information to make decisions
17. What is a Testing Dashboard?
• A testing dashboard, just
like a car’s dashboard, is a
set of indicators that show
the current status of
testing.
• Dashboards can be seen
from various perspectives:
• Project
• Testing status
• Ongoing system maintenance
18. Why Have a Testing
Dashboard?
• For fast and easy reporting test results to
management
• To have all of your testing information in one
place
• To help guide the testing effort
• To help make good decisions
• To build project learning
• Better estimates in the future
• To build the credibility and visibility of testing
19. What is Required for a
Dashboard?
• Accurate and meaningful
measurements and metrics
• Plus…a clear understanding of what
the metrics mean.
• A culture of trust and openness
• Non-intrusive ways to measure
• Ideally, the measures should come
from activities already being tracked.
• Defect tracking systems
• Project management software
20. What is Required for a
Dashboard? (2)
• A way to display the
information in ways that are:
•
•
•
•
Understandable
Easy to Build and Maintain
Accessible
Integrated with tools
• Issue tracking
• Test management
• Project management
21. What is Shown on a Typical
Testing Dashboard?
•
Test Coverage
•
•
•
•
•
Test Status
•
•
•
•
•
Requirements
Functional
Test case
Code
•
• Technical
• Business
• Project
•
• Based on test goals and
objectives
• Blockages
Defects
•
•
•
•
Testing
Defect resolution
Readiness for deployment
Pass/Fail
Progress
Risk
•
Categories
Trends
Detection Percentage
Resolution Status
Testware
• Completion %
• Automation %
•
Resources
22. What Should You Show?
• Ask your customers!
•
•
•
•
What information do they value?
How do they need it shown?
When do they need it?
How timely must it be?
• However, just like in obtaining user
requirements, people often don’t
know what they want or need until
they actually see it.
• Start with a prototype
23. Things to Consider When
Designing Dashboard Items
• Purpose of the Dashboard
• Chart or Graphic Types
• Colors
• Positions
• Brightness
• Orientation
• Sizes
• Shapes
24. The Types of
Information
• Progress
• Meters are good
• Stages of effort
• Bar charts are good
• Benchmarks
• Bar charts, Line charts
• Meters (except to show historical comparisons)
• Trends
• Line charts, Bar charts
• Levels
• Dials
33. Some Ways to Structure
Testing Dashboards
•
•
•
•
Overall Software Quality Levels
Software Test Center Performance
Project Quality Levels
Test Effectiveness
• Overall
• By phase
• By type
• Testing Projects
• Progress
• Defect levels
• Functional quality levels
34. Expanding the View
•
Project Dashboards
• Have the same characteristics, but more points of
measurement.
• Contain testing measures.
• Guide the entire project, not just testing.
38. What Would it Mean…
•
To your project managers to have access to this type of
information at any point in time?
•
To the senior management in your company to see overall
software quality information?
•
To your career to be seen as the keeper of this kind of
information?
39. Words of Warning
• Too many items on a
dashboard can be distracting
and confusing.
• Unless you are flying a plane!
• Metrics can be abused.
• If people don’t understand human
behavior, more harm than good
can result.
• Stuff happens.
• Things not shown on your
dashboard can derail your test.
40. Words of Warning (2)
• With dashboards, everyone can see the same
information at the same time.
• This may be an issue if you don’t want to show
someone information until you have had a chance to
see it first.
• Manual input to the dashboard gets
overwhelming.
• You want to automate the data capture as much as
possible.
• Dashboards may be too general for some
managers.
41. Keeping the Process Working
• The dashboard tells you
about vehicle (process)
malfunctions.
• In testing, the process is the
engine.
• The process might not be
documented.
• How you perform the process
determines whether or not
you reach the intended
destination.
42. Other Concerns
•
Where do I find the time to create dashboards?
•
• This is not an extra task, it's part of the job of test
management!
Do I need tools?
• No. While there are some nice dashboard tools, all you need is
a spreadsheet application.
43. Final Thoughts
• A key purpose of testing is to
provide meaningful information
to management to make
informed decisions.
• This is a positive value-added
view of testing.
• Dashboards are one more tool
to help you guide your testing
project, but they don’t drive the
car!
44. Final Thoughts (2)
• Good dashboards have:
• Good design
• Current information
• The right metrics for your situation
• They should reflect the job at hand
• They should be understandable
• Meaning and value for the readers
• Interpretation and guidance
• Annotations are helpful
45. Resources
• http://www.perceptualedge.com
• Steven Few’s Website
• http://dashboardspy.com
• Dashboards by Example
• http://www.enterprise-dashboard.com
• Datapig Technologies
• http://datapigtechnologies.com
• A great source for Xcelsius videos
• Book - How to Lie With Statistics
• By Darrell Huff
46. Contact Information
Randall W. Rice, CTAL
Rice Consulting Services, Inc.
P.O. Box 892003
Oklahoma City, OK 73170
Ph: 405-691-8075
Fax: 405-691-1441
Web site: www.riceconsulting.com
e-mail: rrice@riceconsulting.com
Notas del editor
The main objective of the testing dashboard is to provide simple, meaningful and reliable information in one place to help guide the testing effort and convey that information to our clients.
In many ways a testing project is like driving a car. You need to know where you’re going, where you are, and the direction you’re headed. As the test continues, you need to know your progress, how fast you are going, if you have the time and people needed to sustain the effort, and how your process (engine) is working.
Our goal is to reach our desired destination safely. In the context of testing, that means we achieve our objectives without harming anything (for example, not destroying live data, crashing the live system, etc.)
To do this, we need to maintain focus and stay on the road. This means that we keep our goals and objectives in mind so we don’t get distracted.
Also, we don’t want to get lost. This means we follow our map to the intended destination. In the context of testing, this means we follow the test plan. However, things change and sometimes our plans must change as well. So, we must also be flexible.
We don’t want to run out of fuel. For our project, we need to make sure we have the resources to continue. This typically amounts to having enough time and people.
Finally, the testing effort needs a leader. Ideally, just one leader at a time.
In this discussion, there are three effective techniques we will discuss:
An effective testing strategy – this defines the test objectives, scope and approach early in the project. The great thing about a testing strategy is that it doesn’t take very long to create and it adds a lot of value to the project. The test strategy is a great vehicle for getting the entire team involved in testing. The strategy describes what the test is intended to accomplish and what makes it unique.
A workable test plan – this defines the scope, resources, schedules, risk, contingencies, anything else that is needed to describe how the test should be performed. Basically a test plan is a project plan for the test.
A dashboard – this monitors key information from the test in one place. We will examine these in much more detail in this session.
The concept of the dashboard is nothing new. At the end of this presentation, I will list some additional resources about dashboards.
Even with all this information, testers and test team leaders still struggle with effective ways to convey information to management and clients.
At the risk of covering something you may have already seen, let’s look at some examples of dashboards. Perhaps you may discover a new way to apply dashboards in your situation. Then we will look at some of the issues about test measurement and reporting.
A testing dashboard, just like a car’s dashboard, is a set of indicators that show the current status of the test.
You can see dashboards from various perspectives such as project dashboards, testing dashboards and dashboards that are created to monitor ongoing system maintenance.
There is no standard definition for what should be in a dashboard or even how it should look. That’s why I’m going to show several examples and approaches in the following slides.
This slide shows some of the major reasons for having a testing dashboard.
In order to have a dashboard, you must have a way to obtain accurate and meaningful measurements and metrics. You also need a culture of trust and openness or else the measurements and metrics will be at risk of manipulation. As the old saying goes, “you get what you measure.” That’s not always a good thing.
Also, the measurements should be non-intrusive. That means that people should not have to take special effort to measure and record them. Ideally, that they can come from activities already being tracked. An example would be defect information derived from defect tracking systems.
In order to have a dashboard, you must have a way to obtain accurate and meaningful measurements and metrics. You also need a culture of trust and openness or else the measurements and metrics will be at risk of manipulation. As the old saying goes, “you get what you measure.” That’s not always a good thing.
Also, the measurements should be non-intrusive. That means that people should not have to take special effort to measure and record them. Ideally, that they can come from activities already being tracked. An example would be defect information derived from defect tracking systems.
Once again, there is no standard or consensus as to what should be on a testing dashboard. This slide shows some of the typical items found on testing dashboards. You may think of others you would like to add.
Dashboards are also used at the project level. They have the same characteristics as a testing dashboard, but include different measurements and metrics.
This is a sample project dashboard from the U.S. Navy. It covers areas such as progress, change, quality, and risk. Not all dashboards are this elaborate!
It’s best to keep the dashboard simple. Too many items can be distracting and confusing – unless of course you’re flying a plane! The analogy here is that a more complex situation may require a more complex dashboard.
As we have mentioned, metrics can be abused. This especially happens when the metrics start to shape people’s behavior.
And finally, don’t forget that the unexpected happens. Things that are not shown on your dashboard can derail your test. This includes things such as reorganizations, scope increases, and changes dictated by management.
Dashboards also contain warning indicators to let you know when things are working right. In this context, the engine is the process.
It’s really great when testing can be seen as a value-added part of the project, instead of a “problem finding” part of the project.
