Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Dynamic Testing
1. DYNAMIC TESTING
Hoang V. Nguyen
startnewday85@gmail.com
startnewday85.blogspot.com
Dept. of Computer Science – FIT - HUA
Tester Training Course
2. User Acceptance
requirements Testing
System
System Testing
requirements
Integration
Global Design
Testing
Detailed Design Unit Testing
implementation
Tester Training Course 2011
5. Identify Test conditions
Specify Test cases
Specify Test Procedures
• is also referred as test script
Tester Training Course 2011
6. Testing may be performed with varying
degrees of formality
• Test Design Specification(IEEE 829)/ Test Condition
Specification
• Test Case Specification
• Test Procedure Specification/ Test Script
• Manual
• Automatic
Tester Training Course 2011
8. Provides a quantitative measure of the
quality of the testing that has been done
By measuring what has been achieved
Provides a way of estimating how much
more testing needs to be done.
Can be applied to any systematic approach
Tester Training Course 2011
10. Test requirements/ test inventory/ test
objectives
Designed based on test basis
Traceability
Must be prioritized
Tester Training Course 2011
11. Test Data
• inputs,
• pre-conditions,
• expected results,
• post-conditions
Exhaustive Testing is impractical
• must select a subset of all possible test cases
Tester Training Course 2011
14. a procedure for selecting test cases
a way deriving good test cases
a way objectively measuring a test effort
maximize possibility faults found, minimize
time and resources
Tester Training Course 2011
15. Different people: similar probability find faults
• gain some independence of thought
Effective testing: find more faults
• focus attention on specific types of fault
• know you're testing the right thing
Efficient testing: find faults with less effort
• avoid duplication
• systematic techniques are measurable
Tester Training Course 2011
16. Systematic approach
• Specification-based or black-box
• Structure-based or white-box
Experience based approach
• Error Guessing
• Exploratory testing
Tester Training Course 2011
17. Specification-based approach
• based on specifications and models of what the
system should do
• do not care how do
Some techniques
• Equivalence partitioning
• Boundary value analysis
• Decision table testing
• State transition testing
• Use case testing
Tester Training Course 2011
19. divide the test case space into
areas(partitions)
• inputs
• outputs
• conditions
In an area, if one element works
correctly, all will work correctly
One from each partition better than
all from one
Tester Training Course 2011
20. Name From 18
Age to 50
Invalid valid Invalid
18 50
==> Test cases: 10, 20, 60, abc
Tester Training Course 2011
21. Invalid valid Invalid
18 50
Faults tend to lurk near boundaries
Good place to look for faults
Test values on both sides of boundaries
Tester Training Course 2011
22. From 3 to 50
characters
Name From 18 to
50
Age
Tester Training Course 2011
23. Number of characters
Invalid valid Invalid
3 50
Valid characters A-Z, a-z,
Invalid
space
Test Condition Valid Partition Invalid Valid Boundary Invalid
Partition Boundary
Name Field 3-50 chars <3 chars 3 chars 2 chars
>50 chars 50 chars 51 chars
valid chars invalid chars
Tester Training Course 2011
24. Invalid valid Invalid
18 50
Test Condition Valid Partition Invalid Partition Valid Boundary Invalid
Boundary
Age Field 18-50 <18 18 17
>50 50 51
Non numeric
Tester Training Course 2011
26. Question
In a Examination a candidate has to score minimum 24
marks in order to clear the exam. The maximum that
he can score is 40 marks. Identify the Valid
Equivalance values if the student clears exam.
22, 23, 26
21, 39, 40
29, 30, 31
0, 15, 22
Tester Training Course 2011
27. Question
Boundary value testing
Is the same as equivalence partitioning tests
Test boundary conditions on, below and above the
edges of input and output equivalence classes
Tests combinations of input circumstances
Is used in white box testing strategy
Tester Training Course 2011
28. Question
Equivalence testing divides the input
domain into classes of data from which test
cases can be derived to reduce the total
number of test cases that must be
developed.
True
False
Tester Training Course 2011
29. Question
An input field takes the year of birth
between 1900 and 2004. The boundary
values for testing this field are
0, 1900, 2004, 2005
1900, 2004
1989, 1900, 2004, 2005
1989, 1900, 1901, 2003, 2004, 2005
Tester Training Course 2011
30. Some software app, business rules can
be very complex
Decision table a good method to
describe complex business rules
Business Rule Business Rule Business Rule Business Rule
1 2 3 4
Condition 1 T T T F
Condition 2 T F F T
Condition 3 T _ T T
Action 1 Y N N N
Action 2 N N Y N
Tester Training Course 2011
31. Given the following decision table: Which of the
following test cases and expected results is VALID?
Rule 1 Rule 2 Rule 3 Rule 4
Age <21 yrs 21-29 yrs 30-50 yrs >50 yrs
Insurance class A A or B B, C or D C or D
Premium 100 90 70 70
Excess 2,500 2,500 500 1000
23 year old in insurance class A Premium is 0 and excess is,500.
51 year old in insurance class C Premium is 0 and excess is 00.
31 year old in insurance class B Premium is 0 and excess is ,500.
43 year old in insurance class C Premium is 0 and excess is ,000.
Tester Training Course 2011
32. A supermarket has a loyalty scheme that is offered to all
customers. Loyalty cardholders enjoy the benefits of either
additional discounts on all purchases (rule 3) or the
acquisition of loyalty points (rule 4), which can be converted
into vouchers for the supermarket or to equivalent points in
schemes run by partners. Customers without a loyalty card
receive an additional discount only if they spend more than
£100 on any one visit to the store (rule 2), otherwise only
the special offers offered to all customers apply (rule 1).
Tester Training Course 2011
33. Rule 1 Rule 2 Rule 3 Rule 4
Customer without loyalty card T T F F
Customer with loyalty card F F T T
Extra discount selected _ _ T F
Spend > £100 F T _ _
No discount T F F F
Extra discount F T T F
Loyalty points F F F T
Create at least one test case for each rule
• Conditions specify inputs
• Actions specify outputs
Tester Training Course 2011
34. based on state transition diagram
Example: ATM State Transition
NOT OK
Start 2nd try
3rd try
1st try
NOT OK
Access
wait Eat
to
for PIN card
Account
Tester Training Course 2011
35. Four different levels of coverage
• State Coverage
• Event Coverage
• Path coverage
• Transition coverage
Tester Training Course 2011
36. State Coverage
Generally this is a weak level of test coverage
2 NOT OK
1
Start 2nd try
3rd try
1st try
NOT OK
Access
wait Eat
to
for PIN card
Account
Tester Training Course 2011
37. Event Coverage
also is a weak level of coverage
2 NOT OK
1
Start 2nd try
3rd try
1st try
NOT OK
Access
wait Eat
to
for PIN card
Account
Tester Training Course 2011
38. Path Coverage
• is the strongest level of coverage
• but may not be feasible
• if the state transition diagram has loops, then the
number of possible paths may be infinite
Tester Training Course 2011
39. Transition Coverage
• good level of coverage with out generating large
numbers of tests
• this level is generally the one recommended
NOT OK
Start 2nd try
3rd try
1st try
NOT OK
Access
wait Eat
to
for PIN card
Account
Tester Training Course 2011
41. To exercise a complete process flow
At least one test case for the main
success scenario
At least one test case for each extension
Tester Training Course 2011
42. Test oracle
Partial oracle
Tester Training Course 2011
43. Based on “What”, with out “How”
Test cases can be designed:
• Input/output variables with their ranges and
boundaries
• Business rules
• System behaviors
• System’s functionalities from start to finish
Tester Training Course 2011
44. Question
Purpose of test design technique is
Identifying test conditions only, not Identifying test
cases
Identifying test conditions and Identifying test cases
Not Identifying test conditions, Identifying test
cases only
Identifying test conditions or Identifying test cases
Tester Training Course 2011
45. Question
’X’ has given a data on a person age, which
should be between 1 to 99. Using BVA which is
the appropriate one
0, 1, 2, 99
1, 99, 100, 98
0, 1, 99, 100
-1, 0, 1, 99
Tester Training Course 2011
46. Given the following state diagram: Which of the
following series of state transitions? S1 S0 S1 S2 S0
D, A, B
A, B, C, D
D, A, B, C
A, B, C
Tester Training Course 2011
47. Question
Which type of test design techniques does the
following statement best describe a procedure to
derive test cases based on the specification of a
component?
White Box Techniques
Black Box Techniques
Glass Box Techniques
Experience Based Techniques
Tester Training Course 2011
48. Given the following state diagram: Which of the following
series of state transitions contains an invalid transition
which may be indicate a fault in the system design?
Login Browse Basket Checkout Basket Checkout Pay Logout.
Login Browse Basket Checkout Pay Logout.
Login Browse Basket Checkout Basket Logout.
Login Browse Basket Browse Basket Checkout Pay Logout.
Tester Training Course 2011
49. Question
Which of the following is MOST characteristic of
specification based(black-box)
Test cases can be easily automated
Test cases are independent of each other
Test cases are derived systematically from models of
the system
Test cases are derived systematically from the
delivered code
Tester Training Course 2011
50. Question
Use cases can be performed to test
Performance testing
Business scenarios
Static testing
Unit testing
Tester Training Course 2011
51. Question
Which is NOT a test oracle
The existing system(For a benchmark)
Indinidual’s knowlege
The code
User manual
Tester Training Course 2011
52. Question
The Test Cases Derived from use cases
Are most useful in uncovering defects in the process
flows during real world use of the system
Are most useful in uncovering defects in the process
flows during the testing use of the system
Are most useful in covering the defects in the
process flows during real world use of the system
Are most useful in covering the defects at the
Integration Level
Tester Training Course 2011
53. Structure-based approach
• based on structures of system or component
• also called glass-box techniques
What we may be interested in structures
• Component level: program structures
• Integration level: the way components interact with
others
• System level: how user will interact with the system
Tester Training Course 2011
56. Design test cases based on what
statements which should be exercised
Statement coverage
• percentage of executable statements exercised
number of statements exercise
• =
total number of statements
• is normally measured by a software tool
• typical ad hoc testing achieves 60 – 75%
Tester Training Course 2011
57. 1 Program Coverage Example
2 A, X: Integer
3 Begin
4 Read A
5 Read X
6 If A > 1 AND X = 2 Then
7 X = X/A
8 Endif
9 If A = 2 OR X = 2 Then
10 X = X + 1
11 Endif
12 End
The number of executable statements: 6
Test case 1: A=1, X=2; 4/6 or 67%
Test case 1: A=4, X=2; 6/6 or 100%
Tester Training Course 2011
58. 1 Program BestInterest
2 Interest, Base Rate, Balance: Real
3 Begin
4 Base Rate = 0.035
5 Interest = Base Rate
6 Read (Balance)
7 If Balance > 1000 Then
8 Interest = Interest + 0.005
9 If Balance < 10000 Then
10 Interest = Interest + 0.005
11 Else
12 Interest = Interest + 0.010
13 Endif
14 Endif
15 Balance = Balance × (1 + Interest)
16 End
What test cases to achieve 100 percent
statement coverage
Tester Training Course 2011
59. Design test cases based on what decision
outcomes which should be exercised
same as Branch Testing
Decision coverage
• percentage of decision outcomes exercised
number of decision outcome exercised
• =
total number of decision outcomes
• is normally measured by a software tool
• typical ad hoc testing achieves 40 – 60%
Tester Training Course 2011
60. 1 Program BestInterest
2 Interest, Base Rate, Balance: Real
3 Begin
4 Base Rate = 0.035
5 Interest = Base Rate
6 Read (Balance)
7 If Balance > 1000 Then
8 Interest = Interest + 0.005
9 If Balance < 10000 Then
10 Interest = Interest + 0.005
11 Else
12 Interest = Interest + 0.010
13 Endif
14 Endif
15 Balance = Balance × (1 + Interest)
16 End
What test cases to achieve 100 percent decision
coverage
Tester Training Course 2011
61. 1 float dc(int a,b,c,d,h)
2 Begin
3 float r
4 If a=0 Then return 0
5 Endif
6 int f=0;
7 If (a=b) or ((c=d) and check(h))
Then
8 f=f+1;
9 Endif
10 f=f+1;
11 return 1/f
12 End
What test cases to achieve 100 percent decision
coverage
Tester Training Course 2011
62. Design test cases based on Boolean sub-
expression (BsE)
Condition coverage
• percentage of Boolean sub-expression outcomes
exercised
number of BsE outcome exercised
• =
total number of BsE outcomes
• There are no industry standard objectives for
condition coverage
Tester Training Course 2011
63. Example
7 If (a=b) or ((c=d) and check(h)) Then
Predicate True False
a=b Test case 1 Test case 2
dc(1,1,x,y,z) dc(1,2,x,y,z)
c=d Test case 3 Test case 2
dc(1,2,2,2,z) dc(1,2,1,2,z)
check(h) Test case 3 Test case 4
dc(1,2, 2,2,z) dc(1,2, 2,2, z’)
Tester Training Course 2011
64. Design test cases based on what paths
which should be exercised
Path coverage
• percentage of paths exercised
number of paths exercised
• =
total number of paths
Tester Training Course 2011
66. 1 2 3 4 5 6 7 8 ….
for as many times as it
is possible to go round
?
the loop (this can be
unlimited, i.e. infinite)
Tester Training Course 2011
67. Question
If the pseudo code below were a programming language,
how many tests are required to achieve 100% statement
coverage?
1 If x=3 then
1
2 Display_messageX;
3 If y=2 then
2
4 Display_messageY;
5 Else
3
6 Display_messageZ;
7 EndIf
4
8 Else
9 Display_messageZ;
10 EndIf
Tester Training Course 2011
68. Question
If the pseudo code below were a programming language,
how many tests are required to achieve 100% decision
coverage?
1 Program AgeCheck
2 CandidateAge: Integer;
3 Begin
4 Read(CandidateAge) 1
5 If CandidateAge < 18 Then
6 Print (“Candidate is too young”) 2
7 Else
8 If CandidateAge > 30 Then
9 Print (“Candidate is too old”) 3
10 Else
11 Print(“Candidate may join Club 18–30”)
12 Endif 4
13 Endif
14 End
Tester Training Course 2011
69. is an important measure in systematic
approach
But
• Coverage techniques measure only one dimension
• measures only what has written
In practice, are always multi-dimension and
have something which have not been written
Tester Training Course 2011
70. Question
Which of the following is NOT true
of test coverage criteria?
Test coverage criteria can be measured in terms of
items exercised by a test suite
A measure of test coverage criteria is the percentage
of faults found
A measure of test coverage criteria is the percentage
of user requirements covered
Test coverage criteria are often used when
specifying test completion criteria
Tester Training Course 2011
71. Unscripted testing
Experience based approach
Some techniques
• Error guessing
• Exploratory testing
A testing approach that is only rigorous,
thorough and systematic is in complete
Tester Training Course 2011
72. no rules, no script
situations in which the software may
not be able to cope are identified based
on experience(feel)
Consider:
• past failures
• brain storming
• intuition
• “what is the craziest thing we can do”
Tester Training Course 2011
73. always worth including
after systematic techniques have been
used
can find some faults that systematic
techniques can miss
a ‘mopping up’ approach
supplements systematic techniques
Tester Training Course 2011
74. Details will be focused in Agile testing lectures
Tester Training Course 2011
75. Question
A test design technique is
A process for determining expected outputs
A process for selecting test cases
A way to measure the quality of software
A way to measure in a test plan what has to be done
Tester Training Course 2011
77. Question
Error guessing
A supplements formal test design techniques
Can only be used in component, integration and
system testing
Is only performed in user acceptance testing
Is not repeatable and should not be used
Tester Training Course 2011
78. Question
Error guessing is
An appropriate way of deriving system tests
Only used if good requirements are not available
Only used when good requirements are available
The most appropriate way of deriving system tests
Tester Training Course 2011
79. Question
Random testing
A black-box testing technique
Program is tested randomly sampling the input
Both A and B
None of the above
Tester Training Course 2011