This webinar will demonstrate how to fully automate testing and test case design using Grid Tools' Agile Designer software. The software can automatically generate the smallest number of test cases needed to maximize coverage from requirements. It can then push these test cases out as automated tests, removing bottlenecks caused by manually creating, executing, and maintaining tests. Attendees will learn how to automatically generate optimized test cases and scripts from models of their system under test and keep tests up to date as requirements change.
Small is Beautiful- Fully Automate your Test Case Design
1. WEBINAR
Small is Beautiful – How to Fully Automate your Testing and Test Case Design
Huw Price, Managing Director – Grid Tools
2. • Automatically generate the smallest number of test cases from requirements for
maximum coverage
• Push test cases out as automated tests
• Remove bottlenecks caused by the manual creation, execution and maintenance
of test cases
This Agile Designer™ webinar will demonstrate how you
can:
4. What’s Supposed to Happen
TesterBusiness AnalystUser Developer
The end user knows
what they want
The analyst specifies
what that is.
The programmer writes
the code
The tester tests the
program
5. What’s going wrong?
• Requirements are stored
disparately & In different formats
• They are typically incomplete and
ambiguous
Requirements Code Test Cases Test Data Environments
The developer requires
clarification & defects
enter the code
• Test cases are manually created and
executed
• Provide just 10-20% coverage and
defects are detected late
• Cannot respond to change requests
• Never get deleted
• Contains sensitive information, and
time is spent securing it
• Provides poor coverage, and
testers spend half their time finding
or making data
• Is large, and 20% of the
SDLC is spent finding data
• Is unavailable ‘upstream’
Change Request
• Are shared with other teams
or are not complete yet
• Are legacy or expensive
mainframe systems
Hardware & Data
Dependencies • Automation is used late
• Scripts are manually
developed
• It cannot respond to change
• Tests the same functionality
Automation
6. Grid Tools - Products
• Automation Scripts
• User Stories
• Optimized Test Cases
• Backlog and Complexity
• Test Data
• Requirements and Coverage
• Data Coverage
• Find and Allocate Test Data
• Subset and Mask Data
• Synthetic Data
• Virtualized Messages
• Test Data on Demand
• Selenium
• QTP
• IBM Rational
• Ranorex
• SilkTest
• Microsoft UI Automation
• CA
• Parasoft
Javelin Automation
Dataviz
Javelin Data Orchestration
8. Test case design
• Currently manual
• No formal processes.
• “Exploratory Testing” the norm: poor requirements lead to poor overall testing,
with testers having to fill in the gaps
• No linkage to test data – process is manual, painstaking and very time-consuming.
• No flexibility for change requests: a critical weakness in an agile environment.
Changes take longer than the original requirement!
9. What is Testing Coverage?
The Business Thinks it’s:
• Code Covered
• Number of Test Covered – Tests Run
• Percentage of use cases
• All Paired Combinations
It is actually:
• Designing Sufficient Tests To VERIFY That The Design And Code Correctly Implement The Requirements
• Did you get the right answer for the right reason - Two or more defects may sometimes cancel each other
out - Observability
10. Different Coverage Techniques
Combinatorial – All Pairs – Constrained All Pairs
• Does not support Expected Results - You have to work it out for each combination which is very time
consuming
• Combinatorial is something that ‘accidentally’ increases functional coverage (and only to a point)
• Combinatorial does not give you any solid metrics on how good your testing actually is – you can only infer
that x out of y combinations have been covered, but it has no bearing in actual functional logic.
• You end up with lots of false errors that have to be checked by hand
11. Copyright Bender RBT Inc. 2009 11RBT24261
Cause-Effect Graphing
Observable Events and Path Sensitizing
• Assume C and F are not
observable events.
• Assume A is stuck at FALSE.
• Enter as a test case A(T), B(T),
D(T), E(T).
• Results should be C(T), F(T)
and G(T).
T
T
T
T
T
T
T
Different Coverage Techniques
12. Different Coverage Techniques
Optimized Flow Chart Modelling
• Advanced Graph Optimization
• Most projects already have a flow chart
• Similar results to Cause and effect
• Supports Looping
• Supports Constraints
• Not applicable for non sequential based logic
14. The path explorer tool and optimization algorithms
Create Perfect Test Cases
• Generate the smallest number of test cases with
maximum coverage
• Test more functionality in fewer tests
• Measure test coverage and know that every
requirement has been tested
The cost and complexity tool in Agile Designer™
16. Model Based Testing – Step by Step
CUSTOMER NUMBER: 123456
CUSTOMER NAME: HUW PRICE
CUSTOMER COUNTRY: UK
VALIDATION
PROCESS
VALIDATION
PROCESSDATA UPDATED
18. Model Based Testing – Process Coverage
ALL POSSIBLE
ALL PAIRS
ALL EDGES & PATH HINTS
19. Model Based Testing
Most testing is – Frankly:
• Random
• Unstructured
• Repetitive
• Not thorough enough
• Can’t be measured
• Can’t keep up
• Too slow
Model based testing, lets you define what
is supposed to happen and then test that.
Model based testing is:
• Accurate
• Structured
• Thorough
• Measurable
• Can keep up with change
• Allows you to measure Risk
• Lets you automate very quickly
20. Automated tests: manual generation
• Automated testing frameworks are heavily scripted
• Script generation is usually done manually
• Alternative solution: use scriptless automation frameworks – But your back to
Manual test case design
• Optimal solution: from automated test case generation, through automated
script generation, to automated execution. Automating the automation.
21. Active Automation
•If you could create better manual tests faster,
would you?
•If you could create automation scripts as quickly as
manual test cases, would you?
22. Active Automation
• Scripts Are Created Automatically
• Testing is Model Based
• Use a component library of covered tests
• Test Data is created or found as part of the Automation
Active Automation
23. Active Automation
Record an Automation Script
Import it into Agile Designer
Design the logic
Auto create your automation robots
If something changes, auto create new robots!
Cl Cl
Cl
Cl
N
O
H
N
N
N
H
O
Cl
Cl
Cl Cl
25. Agile Designer – Data driven or Flow Driven
LOGIC
DATA DRIVEN
AUTOMATION
AUTOMATION HARNESS
DATA 1
DATA 2
DATA 3
DATA 4
FLOW DRIVEN
AUTOMATION 1 – DATA 1
AUTOMATION 2 – DATA 2
AUTOMATION 3 – DATA 3
AUTOMATION 4 – DATA 4
AUTOMATION 5 – DATA 5
You can drive using a static flow
and feed it with test data and
expected results
You can drive by auto
generating brand new test
scripts for each path
Or a combination
27. Shift left the effort of testing
Business Analyst
User
Requirements
Programmer
Tester
Tester
Test Cases
& Test Scripts
Program
• Concentrate effort into the requirements
gathering stage
• User and BA map requirements to
unambiguous, active flowcharts
• Testers can generate automated test scripts
and test cases directly from the requirements
• If a user makes a change to the requirements,
the tests can be automatically updated
28. Active Automation
•If you could create better manual tests faster,
would you?
•If you could create automation scripts as quickly as
manual test cases, would you?
•Yes please!
29. Q & A
If you have any further questions please contact us on
agile-sales@agile-designer.com
30. For more information on what Agile Designer™ can do for
you, please contact:
www.agile-designer.com
agile-sales@agile-designer.com
or by phone:
UK: +44 (0)1865 884 600 or US: +1 888 603 1213
THANK YOU FOR WATCHING
Notas del editor
Even with what looks like a very simple requirement there are a large number of potential use cases. In this example the complexity comes from the reset button.
Point One: generate the perfect set of test cases directly from the requirements
From the requirements flowchart, Agile Designer will identify every possible path through the system. Where it gets clever is the path optimization. This uses “deep, dark maths” to identify the smallest number of paths needed to fully test a systems functionality – i.e., to fully test the requirements now that they are fully defined. Users can choose from multiple algorithms, to automatically generate the smallest number of test cases to cover: all possible paths; all edges (arrows in/out of the blocks); all nodes; all in/out edges; all pairs.
Automatically generating test cases removes testing bottlenecks:
Manually writing test cases and test scripts is slow and error-prone (i.e., it provides poor coverage)
For example: 6 hours to produce 11 test cases with 16% coverage (internal)
Testing currently takes up around 47% of the SDLC.
Path explorer –
Look at all possible paths through the functional logic of the flowchart
Identify the smallest number of test cases needed to test maximum functionality
Store/create your test cases (use cases)
Store these test cases – push them out to automation engines, ALM/QC etc.
In the path explorer: go to your test cases, and click ‘export special’. Select which folder you want to save them in. Update and add new test cases.
Point two: maximum coverage means you can test more for less
Agile Designer helps reduce testing time and costs by identifying the smallest number of tests needed. This means that you can systematically improve coverage, knowing that all requirements have been tested.
This is in contrast to industry standard, where much testing is redundant, and much functionality goes untested:
Over-testing of certain functions by 40 times
Typically only 10-20% coverage – negative/unhappy paths go untested
Up to 30% of testing time is wasted on duplicate, invalid or redundant tests
Examples of optimization:
A financial services company created 11 test cases in 6 hours with 16% requirement coverage
Agile Designer automatically created 17 test cases in 2 hours with 100% coverage
Another project relied on 3 test cases which provided just 5% coverage, this resulted in bugs making it into production which is expensive to fix
Agile Designer generated 12 test cases with 100% coverage in 30 minutes
In one project, the possible number of cases identified was 326; Agile Designer identified that only 17 were needed for 100% coverage
Point Three:
The same algorithms used to identify the test cases can be used assess how much functional coverage the stored test cases provide.
The notion of perfect test cases is based on the concept of coverage, path modelling and risk-based testing.
From these test cases…The secret of Agile Designer is that by doing one thing you create multiple outputs, for example by designing the perfect set of test cases you are creating a static test of the requirements document. You can push out
Story boards
Complexity analysis – how long should this take?
Virtualization end points
Test Data
Automation scripts
AND even push actions, use cases and test cases to story boards
From a flowchart, you therefore have all the qualitative information needed to work through the development lifecycle, up to the execution of automation scripts. Once you have stored your paths, you simply have to push them back to TMX. From there, you can use them in your automation engine of choice, automatically testing an application iteratively.
Typically the BA writes the requirement along with a few diagrams. There are usually lots of words. These have to be interpreted by the programmer and then by the tester. These words (based on each users local domain knowledge and location) are often ambiguous and result in the wrong thing being developed or tested.
Even harder are change requests, the amount of work for a simple change request is massive as all existing test cases have to be verified and changed again, and test scripts have to be rewritten.
With Agile Designer the hard work is concentrated in the requirements gathering stage – all the hard work is concentrated here.