The Secret Ingredient of Test Management answers on such questions as "What is ROI?", "What are testing measures ?", "How to improve test process?". "What is Risk-based testing?", " How to manage the team?" etc.
2. ABOUT MYSELF
VLADIMIR ARUTIN
QA ENGINEER
at HYS ENTERPRISE
ISTQB CERTIFIED TEST MANAGER
ISTQB Training Instructor
Quality Assurance Training Instructor
Speaker
3.
4. TEST MANAGEMENT – planning, scheduling,
estimating, monitoring, reporting, control and
completion of test activities.
*Ref: ISO 29119
WHAT IS TEST MANAGEMENT?
5. WHY DO WE NEED TEST MANAGEMENT?
“Тестирование - это искусство
сравнивания невидимого с
противоречивым, чтобы
предотвратить немыслимое,
случившееся с неизвестным.”
8. TEST MANAGER’S RESPONSIBILITIES
• BULDING UP AND LEADING THE TESTING TEAM
• DEFINING THE SCOPE OF TESTING
• DEPLOYING AND MANAGING RECOURCES FOR TESTING
• APPLYING THE APPROPRIATE TEST MEASUREMENTS AND METRICS
• PLANNING, DEPLOYING AND MANGING
• ASSESSING THE PROGRESS AND EFFECTIVENESS OF THE TEST EFFORT
• ADVOCATING THE APPROPRIATE LEVEL OF TESTABILITY
• ADVOCATING THE APPROPRIATE LEVEL OF QUALITY
9. WHAT ARE THE CHALLENGES IN TEST MANAGEMENT?
NOT ENOUGH TIME TO TEST
NOT ENOUGH RECOURCES TO TEST
THE PROJECT BUDGET IS LOW
AND SCHEDULE IS TOO TIGHT
TESTING TEAMS ARE NOT ALWAYS IN ONE PLACE
THE REQUIREMENTS ARE TOO COMPLEX
TO CHECK AND VALIDATE
10. MAIN PROBLEM
HEY YOU!
TEST THIS APPLICATION
TO ENSURE HIGH
QUALITY!
OKAY, BUT I NEED
PEOPLE, BUDGET,
TIME… LET ME
CREATE A TEST PLAN
FOR HIGH QUALITY
TESTING.
BOSS
TM
11. LOOK! HERE IS MY TEST PLAN.
AT FIRST SIGHT IT COSTS
A LOT, BUT ALL THESE ACTIVITIES
ARE NECESSARY FOR GOOD
TESTING.
WHAT?!! COST IS TOO
HIGH! CUT YOUR
BUDGET IN 4 TIMES!!!
AND FINISH THE PROGECT
IN HALF TIME!!!
12. YOU HAVE 1 DAY TO PRESENT COMPLETE
TEST PLAN BASED ON THE APPROVED
BUDGET!
IF YOU CAN’T TO IT, WE’LL FIND ANOTHER
TEST MANGEMENT WHO CAN!
16. 1. Decide What to Automate
2. Prioritize & Divide Tasks
3. Prepare Test Cases & Scenario Beforehand
4. Create Tests that Don’t Affect UI Changes
5. Don’t Automate Every Test
6. Review Tests Periodically
7. Compare Your Costs And Benefits
CALCULATE YOUR COSTS
18. 7 Principles of TESTING
1. Exhaustive testing is not possible
2. Defect Clustering
3. Pesticide Paradox
4. Testing shows a presence of defects
5. Absence of Error - fallacy
6. Early Testing
7. Testing is context dependent
10 Principles for Agile
Testers
1. Provide Continuous Feedback
2. Deliver Value to the Customer
3. Enable Face-to-Face Communication
4. Have Courage
5. Keep It Simple
6. Practice Continuous Improvement
7. Respond to Change
8. Self-Organize
9. Focus on People
10. Enjoy
USE TEST MANAGEMENT PRINCIPLES
24. USE TEST METRICS. MEASURE AND RULE!
Test Effort
How long, how many,
and how much?
25. USE TEST METRICS. MEASURE AND RULE!
Test Effectiveness
Defect Detection Percentage =
(DDP)
Defects found by testing stage
Total defects found in all stages
27. Defect Density = No. of Defects identified / size
No. of Defects identified 30
No. of requirements 5
Others
Defect Density = (30 / 5) = 6
Defect Removal Efficiency (DRE)
No. of Defects found while Testing + No. of Defects found by User
DRE= х100%
120
No. of Defects found by User 42
No. of Defects found while Testing
No. of Defects found while Testing
DRE = [120 / (120 + 42)] * 100 = [120 /162] * 100 = 74.07%
28. Others
х100%
120
No. of Defects found by User 42
No. of Defects found while Testing
Example, QA team reported 100 defects out
of which 20 were invalid(not bugs, duplicates,
etc.) and only 65 were fixed
Defect gap % is: (65/100-20)X100= 81%
Defect Gap =
Total No. of Valid Defects reported)
Total No. of Defects Fixed
29. Others
Total RawTest Steps
TestCaseProductivity = Step(s)/hour
Efforts(hours)
Test Case Name Raw Steps
XYZ_1 30
XYZ_2 32
XYZ_3 40
XYZ_4 36
XYZ_5 45
Total Raw Steps 183
Efforts took for writing 183 steps
is 8 hours.
TCP=183/8=22.8
Test case productivity = 23 steps/hour
30. Test Automation Metrics
AP (%) =
Automation Progress
# of actual test cases automated
# of test cases automatable
Test Progress
TP (%) =
# of test cases (attempted or completed)
time (days/weeks/months, etc)
Percent of Automated Testing Test Coverage
PTC( (%) =
automation coverage )
total coverage
32. SCRIPTS FOR SUCCESS
BEGIN THE TESTING ACTIVITIES EARLY IN THE SDLC
REUSE TEST ASSETS AND IMPROVE TIME SPENT EVERY TIME
ENSURE PROPER COORDINATION BETWEEN CO-LOCATED TESTING
RECOURCES
DEFINE & ENFORCE FLEXIBLE TESTING PROCESSES
COORDINATE & INTERGRATE WITH DEV TEAMS –
USE DEVOPS SHIFT LEFT PRACTICE
COMMUNICATE THE CORRECT STATUS AT THE RIGHT TIME
TO SAVE TIME, AUTOMATE, AUTOMATE, AUTOMATE!
33. SCRIPTS FOR SUCCESS
PLAN, DO, CONTROL AND STANDARDIZE!
RECOGNIZE PROBLEMS.
IT’S COOL BECAUSE NOW YOU CAN FIX THEM.
DOCUMENT ALL SOLUTIONS AND RESULTS
EACH DAY –NEW SMALL IMPROVEMENTS
INVOLVE ALL STAFF WHILE MAKING DECISIONS
IF YOU DO SOMETHING BE SURE IT WILL STAY THAT WAY
YOU DID IT. SO DO IT PROPERLY AT ONCE.
ALL THE WORK ON THE PROJECT SHOULD BE CONSIDERED
THROUGH THE LENS OF COMPANY DISTANT TARGETS FOR
FUTURE GROWTH.
36. TEST PROCESS IMPROVEMENT (TPI) NEXT
TPI NEXT: A continuous business-driven framework for test process improvement
that describes the key elements of an effective and efficient test process.
16 KEY PROCESS AREAS:
1. Stakeholder commitment
2. Degree of involvement
3. Test Strategy
4. Test Organization
5. Communication
6. Reporting
7. Test Process Management
8. Estimating and Planning
9. Metrics
10. Defect Management
11. Testware Management
12. Methodology Practice
13. Tester Professionalism
14. Test Case Design
15. Test Tools
16. Test Environment
39. CRITICAL TESTING PROCESS (CTP)
CRITICAL TESTING PROCESSE (CTP): A content-based model for test process
improvement built around twelve critical processes.
12 CRITICAL PROCESSES:
1. TESTING
2. ESTABLISHING CONTEXT
3. QUALITY RISK ANALYSIS
4. TEST ESTIMATION
5. TEST PLANNING
6. TEST TEAM DEVELOPMENT
7. TEST SYSTEM DEVELOPMENT
8. TEST RELEASE MANAGEMENT
9. TEST EXECUTION
10. BUG REPORTING
11. RESULTS REPORTING
12. CHANGE MANAGEMENT
40. SYSTEMATIC TEST AND EVALUATION PROCESS
(STEP)
SYSTEM TEST AND EVALUATING PROCESS (STEP): A structured testing methodology,
also used as a content-based model for improving the testing process which doesn’t
require that improvements occur in a specific order.
Basic premises of the methodology include:
• A requirement-based strategy
• Testing starts at the beginning of the lifecycle
• Tests are used as requirements and usage models
• Testware design leads software design
• Defects are detected earlier or prevented
altogether
• Defects are systematically analyzed
• Testers and developers work together
41. TESTING MATURITY MODEL integration
(TMMi)
TESTING MATURITY MODEL intergation (TMMi) is composed of 5 maturity levels
and is intended to complement CMMI. Each of the maturity level contains defined
process areas that must be 85% complete by achieving specific and generic goals
before the organization can advance to the next level.
72. + –
” 0 Y^ y + ”
ПОКА ДЕФЕКТ НЕ ИСПРАВЛЕН, РИСК В ПРОДУКТЕ
ОСТАЕТСЯ
‘RISK ELIMINATION’: РИСК УСТРАНЯЕТСЯ В
РЕЗУЛЬТАТЕ СПРАВЛЕНИЯ И ВЕРИФИКАЦИИ
ДЕФЕКТА
‘RISK ACCEPTION’: РИСК ПРИНИМАЕТСЯ, КОГДА
ОБНАРУЖЕННЫЙ ДЕФЕКТ РЕШЕНО НЕ ИСПРАВЛЯТЬ
‘RISK SHARING’: КОГДА НЕ ИСПРАВЛЯЕТСЯ
СЕРЬЕЗНЫЙ ДЕФЕКТ, ЕГО ОБІЧНО ДЕЛЯТ:
- С МЕНЕДЖЕРОМ ПРОЕКТА (проектное совещание)
- С ЗАКАЗЧИКОМ (в ходе сдачи-приемки)
- С ПОЛЬЗОВАТЕЛЕМ (описание дефекта в
сопроводительной документации)
‘RISK AVOIDANCE’: ИНОГДА МЫ ПРОЬУЕМ ИЗБЕЖАТЬ
РИСКА
-переписать фрагмент кода с дефектами
- Урезать код (даже путем сокращения функционала)
103. You have completed the following skills assessment for your team, rating each
person from 1 (low) to 5 (high) for their competency in each area
You are in the middle of a very technical project with significant problems. Tester D
is quite junior and needs frequent guidance. You have to go on vacation for the
next two weeks. Who is the best person on your team to handle the test
management tasks while you are gone?
a. Tester A
b. Tester B
c. Tester C
d. Tester E
104. You have completed the following skills assessment for your team, rating each
person from 1 (low) to 5 (high) for their competency in each area
You are in the middle of a very technical project with significant problems. Tester D
is quite junior and needs frequent guidance. You have to go on vacation for the
next two weeks. Who is the best person on your team to handle the test
management tasks while you are gone?
a. Tester A
b. Tester B
c. Tester C
d. Tester E
105. You have completed the following skills assessment for your team, rating
each person from 1 (low) to 5 (high) for their competency in each area.
You have an opportunity to send one of your testers to speak at a conference on the
importance of technical testing skills in software testing. This person will also be expected to
attend other seminars and bring back and disseminate that information to the team. Who is
the best candidate for this opportunity?
a. Tester A
b. Tester C
c. Tester D
d. Tester E
106. You have completed the following skills assessment for your team, rating
each person from 1 (low) to 5 (high) for their competency in each area.
You have an opportunity to send one of your testers to speak at a conference on the
importance of technical testing skills in software testing. This person will also be expected to
attend other seminars and bring back and disseminate that information to the team. Who is
the best candidate for this opportunity?
a. Tester A
b. Tester C
c. Tester D
d. Tester E
107. You have completed the following skills assessment for your team, rating
each person from 1 (low) to 5 (high) for their competency in each area.
You have a limited training budget, but you can afford to send two people to a course on peer
coaching. In order to best benefit the department, who should you send to this training?
a. Testers C and D
b. Testers A and C
c. Testers B and E
d. Testers B and D
108. You have completed the following skills assessment for your team, rating
each person from 1 (low) to 5 (high) for their competency in each area.
You have a limited training budget, but you can afford to send two people to a course on peer
coaching. In order to best benefit the department, who should you send to this training?
a. Testers C and D
b. Testers A and C
c. Testers B and E
d. Testers B and D