This presentation has been shared at the #21 Agile Vietnam monthly event.

2. Agenda
• Background and Motivation
• qEstimation Analysis
– Test Size Estimation (Test Case Point Analysis)
– Test Effort Estimation
• Conclusion
2

3. Background
• Software estimation
– process of determining the cost, time, staff, and other related
attributes of software projects, often before work is performed
• Estimation is important for the success or failure of software
projects
• Methods and Metrics
– Source Lines of Code (SLOC)
– Function Points
– Use Case Points
– Story Points
– COCOMO
– Expert Judgment
3

4. Motivation
• Testing accounts for up to 50% of project effort [1]
• Current problems
– estimates are done for the whole project rather than testing
specific
– lack of reliable methods designed for estimating size and effort
of software testing
– vague definitions of testing productivity
• due to the lack of a size measure for software testing
• Our aim
– To introduce a method for estimating the size of testing
activities
– To discuss methods to estimate testing effort using this size
measure
– To introduce a simple toolkit for this estimation process
4

5. Agenda
• Background and Motivation
• qEstimation Analysis
– Test Size Estimation (Test Case Point Analysis)
– Test Effort Estimation
• Conclusion
5

6. qEstimation Analysis’ Principles
• Size reflects the mass and complexity of each test cycle
of a testing project
• Test case’s complexity is based on
– Number of checkpoints
– Complexity of test setup or precondition
– Complexity of test data
• Test Case Point (TCP) is used as size unit
– representing the size of the most simple test case
• Calibration or model refinement is key to estimating
effort
– calibration based on feedback from different cycles within
project or of similar projects
• Focusing on independent testing (V & V)
6

7. qEstimation Analysis’ Process
Estimate size and effort of different test cycles of a same project:
[Test Cycle i]
Count TCPs Estimate
Test Case Counted Estimated
of all Test Testing
Test Case Size
Effort
Effort
Cases
Update
Parameters Historical
Data
Calibrate
Estimation Historical Data of this Project
Model
Test Cycle Size Actual Effort by
Effort Activity
…. …. …. ….
Test cycle i …. …. ….
…. …. …. ….
7

8. Count Size of Test Cycle
• Size of a test cycle is the total of TCPs of all test cases to be
executed in that test cycle
• Steps:
Count
Checkpoints
Adjust based on
Test Case Determine Set Up Unadjusted
Test Type TCPs
Complexity TCPs
(optional)
Determine Test
Data Complexity
8

9. Count Size of Test Cycle (cont’d)
• Checkpoints
– Checkpoint is the condition in which the tester verifies
whether the result produced by the target function
matches the expected criterion
– One test case consists of one or many checkpoints
One checkpoint is counted as one TCP
9

10. Count Size of Test Cycle (cont’d)
• Test Setup or Precondition
– Test setup specifies the condition to execute the test case
• Include setup steps to prepare environment for testing
• Mainly affect the cost to execute the test case
• May be related to data prepared for the test case
– Four levels of Test Setup complexity
• Each is assigned a number of TCPs
Number Complexity Description
of TCP(*) Level
0 None • The set up is not applicable or important to execute the test case
• Or, the set up is just reused from the previous test case to continue the current test case
1 Low • The condition for executing the test case is available with some simple modifications required
• Or, some simple set-up steps are needed
3 Medium • Some explicit preparation is needed to execute the test case
• Or, The condition for executing is available with some additional modifications required
• Or, some additional set-up steps are needed
5 High • Heavy hardware and/or software configurations are needed to execute the test case
(*) based on our survey of 18 senior QA engineers. You can adjust according to your project’s experience.
10

11. Count Size of Test Cycle (cont’d)
• Test Data
– Test Data is used to execute the test case
• It can be generated at the test case execution time, sourced from previous tests, or
generated by test scripts
• Test data is test case specific, or general to a group of test cases
– Four levels of Test Data complexity
• Each is assigned a number of TCPs
Number of TCP Complexity Description
(*) Level
0 None • No test data preparation is needed
1 Low • Simple test data is needed and can be created during the test case execution time
• Or, the test case uses a slightly modified version of existing test data and requires
little or no effort to modify the test data
3 Medium • Test data is deliberately prepared in advance with extra effort to ensure its
completeness, comprehensiveness, and consistency
6 High • Test data is prepared in advance with considerable effort to ensure its completeness,
comprehensiveness, and consistency
• This could include using support tools to generate data and a database to store and
manage test data
• Scripts may be required to generate test data
(*) based on our survey of 18 senior QA engineers. You can adjust according to your project’s experience.

12. Count Size of Test Cycle (cont’d)
• Adjust TCPs based on Type of Test
– This is an OPTIONAL step
– Adjustment is based on types of test cases
• Each type of test case is assigned a weight
• Adjusted TCP of the test case = Counted TCP x Weight(*)
(*) based on our survey of 18
senior QA engineers. You can
adjust according to your
project’s experience.
12

13. Estimate Effort of Test Cycle
• Overview
– Two estimation methods
• Based on Test Velocity
• Regression analysis of Size and Effort of completed test cycles
– Effort distributed by activity
• Test Planning
• Test Analysis and Design Each of these activities may
be performed multiple times
• Test Execution
• Test Tracking and Reporting
13

14. Estimate Effort of Test Cycle (cont’d)
• Estimate Effort based on Test Velocity
Effort(person-hour) = Size(TCP) / Test Velocity (TCP per person-hour)
– Test Velocity is measured as TCP/person-hour
• dependent on project
• calculated based on data from completed test cycles
of the same project
14

15. Estimate Effort of Test Cycle (cont’d)
• Estimate effort using Linear Regression Analysis
– Find out the equation of effort and size using similar completed test
cycles of a project
100
90
80 Equation of
70 Size and Effort
y = 0.0729x + 1.6408
Effort (PM)
60
50
40 The data analysis tool like
30 Excel can be used to find
20
out the equation
10
0
0 100 200 300 400 500 600 700 800 900 1000
Adjusted TCP
15

16. Calibrate the qEstimation Estimation Model
• Calibration: a process adjusting parameters for a
model using historical data or experiences
• With qEstimation, you can calibrate:
(1) TCP assigned to each complexity level of Test Setup
(2) TCP assigned to each complexity level of Test Data
(3) Test Velocity
(4) Effort distribution
(5) Weights of test case types
• Process can be done with the help of tools
Tool Demo
16

17. Conclusion
• qEstimation Analysis is an agile approach to estimating
size and effort of test cycle
– Estimate Size in TCP
– Estimate Effort using Test Velocity or Regression
– An Excel toolkit to simplify the approach
• Advantages and experiences learned
– Easy to implement
– Reflecting real complexity of test cases
– Independent with the level of details of test cases
– Found useful for estimating testing effort
• Limitations and future improvements
– A new approach
– Need more empirical validations
17

18. Thank You

19. References
• [1] Y. Yang, Q. Li, M. Li, Q. Wang, “An empirical analysis on distribution patterns of software
maintenance effort”, International Conference on Software Maintenance, 2008, pp. 456-
459
• [2] N. Patel, M. Govindrajan, S. Maharana, S. Ramdas, “Test Case Point Analysis”, Cognizant
Technology Solutions, White Paper, 2001
• [3] QASymphony: www.qasymphony.com
• [4] V. Lam, “Estimable”, Professional Tester,
http://www.professionaltester.com/magazine/backissue/PT015/ProfessionalTester-
July2012-Lam.pdf
• [5] QASymphony, “Test Case Point Analysis”, White Paper and Tool
http://www.qasymphony.com/white-papers.html