As organizations mature in their testing processes, the perennial quest to achieve ultimate excellence has led many to establish a Testing Center of Excellence (TCoE). Many of these initiatives have been plagued with issues ranging from partial implementation to complete abandonment midway. Additionally, most TCoE initiatives meet heavy resistance and inertia from testing teams who perceive them as a threat to their independence and way of working. At the heart of these issues lie misalignment with business goals, faulty ROI analysis prior to investing, poor communication, and an incorrect organizational model. Drawing from their experience consulting with organizations on TCoE initiatives, Krishna Iyer and Mukesh Mulchandani share insights, experiences, and lessons learned from both their successes and failures. Learn how to go about creating your own TCoE while overcoming the common—and not so common—challenges you will face along the way. Draw on their experience to troubleshoot some of your unique problems.
2. Krishna Iyer
Zen Test Labs
With more than fifteen years of experience in QA, Krishna Iyer is the CEO of Zen Test
Labs, a niche software quality assurance firm focused on revolutionizing the way we
test software. A prolific speaker, Krishna has delivered multiple presentations,
keynotes, and tutorials at global conferences including STARWEST and STAREAST.
Krishna balances his time between driving financial growth for Zen Test Labs and
devoting himself to social causes.
3. Mukesh Mulchandani
Zen Test Labs
CTO of Zen Test Labs Mukesh Mulchandani is focused on dramatically reducing
redundant manual tasks in the software testing cycle. With fourteen years of
experience in developing solutions for complex testing challenges, Mukesh is the
principal architect behind all of Zen Test Labs’ testing solutions. A frequent presenter
at global conferences including the STAR conferences, he spends his time developing
cutting edge solutions for Zen Test Labs. Mukesh is passionate about developing the
skills of youth at large and, to this end, spends his spare time leading non-profit
training initiatives.
4. 2/4/2014
1
BUILDING A TEST CENTRE OF EXCELLENCE:
EXPERIENCES, INSIGHTS AND FAILURES
PRESENTED BY:
KRISHNA IYER AND MUKESH MULCHANDANI
ABOUT KRISHNA AND MUKESH
Krishna is the CEO of Zen Test Labs‐ a niche software quality assurance firm focussed on
revolutionizing the way we test software. Krishna comes with over 15 years of experience in QA and
is a prolific speaker He has delivered multiple track presentations keynotes and tutorials at global
KRISHNA IYER
is a prolific speaker. He has delivered multiple track presentations, keynotes and tutorials at global
conferences including STARWEST and STAREAST. Krishna balances his time between driving financial
growth for Zen Test Labs and devoting himself to social causes.
MUKESH MULCHANDANI
Mukesh is the CTO of Zen Test Labs‐ a niche software quality assurance firm focussed on dramatically
reducing redundant manual tasks in the software testing cycle. Mukesh brings with him 14 years of
i i d l i l ti f l t ti h ll d i th i i l hit t
2
experience in developing solutions for complex testing challenges and is the principal architect
behind all of Zen Test Labs’ testing solutions. Mukesh has also presented regularly at global
conferences including the STAR conferences. Mukesh spends his time developing cutting edge
solutions for Zen Test Labs and is passionate about developing the skills of the youth at large. To this
end, he spends his spare time leading non‐profit training initiatives.
5. 2/4/2014
2
AGENDA
TEST CENTRE OF EXCELLENCE (TCOE)
INSIGHTS ON SETTING UP A TCOE
BASELINE AND BENCHMARK
IMPLEMENTING REUSABILITY IN TESTING
SHAPING SKILLSETS, MINDSETS AND TOOLSETS
BUILDING QUALITY INTO SOFTWARE
ROI BASED AUTOMATION
INTEGRATING MULTIPLE TOOLS AND METHODOLOGIES UNDER A SINGLE FRAMEWORK
RECRUITING AND RETAINING GOOD TESTERS
CONCLUSION
3
TEST CENTRE OF EXCELLENCE
CENTRALIZED
KNOWLEDGE
MANAGEMENT
STANDARDIZED
TEST
PROCESSES
EXCELLENT
CENTRALIZED
RESOURCE
MANAGEMENT
MANAGEMENT
TOOLS AND
TECHNOLOGY
TEST CENTRE OF
EXCELLENCE
EXCELLENT
QUALITY
IMPROVED
TIME TO
MARKET
4
REUSABLE
ARTIFACTS
TECHNOLOGY
LOWER COSTS
INFRASTRUCTURE
6. 2/4/2014
3
INSIGHTS ON SETTING UP A TCOE
1. BASELINE
AND
BENCHMARK
2. REUSABILITY
7. RECRUITING
& R
IN TESTING
3. SKILLSET,
MINDSET AND
T
6.INTEGRATING
UNDER ONE
F
& RETAINING
TALENT
5
TOOLSET
4. BUILDING
QUALITY INTO
SOFTWARE
5. ROI BASED
AUTOMATION
FRAMEWORK
1. BASELINE AND BENCHMARK
SIZING TESTING USING TEST POINTS: A SINGLE SIZE METRIC FOR ESTIMATION
TEST COVERAGE METRIC = NUMBER OF
TEST CASES PER TEST POINT PER
MODULE
DEFECT DENSITY METRIC = NUMBER OF
DEFECTS PER TEST POINT (FOR EACH
MODULE )
TEST PRODUCTIVITY METRIC = NUMBER
OF TEST POINTS DELIVERED IN AN HOUR
(FOR EACH MODULE)
6
7. 2/4/2014
4
1. BASELINE AND BENCHMARK
SIMPLE AVERAGE COMPLEX
SIZE ESTIMATION MATRIX
SMALL 1 TP 2.5 TP 3 TP
MEDIUM 4 TP 8 TP 6 TP
7
LARGE 9 TP 10 TP 11 TP
2. IMPLEMENTING REUSABILITY IN TESTING
REUSABLE BUSINESS
FUNCTIONS
REUSABLE TEST
DATA
AUTOMATICALLY
GENERATED TEST CASES
•Create business
functions that can be
reused to build
manual test cases.
•Consider the login
module. Once a login
business function has
been created every
•Create reusable test
data for building
manual test cases.
•Significant time
savings as reusable
test data is stored
separately and called
using a test data id
•Automatically
generate descriptive
manual test cases (in
English) once
business function
and test data are
brought together in a
test flow.
8
been created, every
manual test case
requiring it for that
application can reuse
the function.
using a test data id
(primary key)
8. 2/4/2014
5
3. SHAPING SKILLSET, MINDSET AND TOOLSET
• EMPOWER TEAMS WITH
STANDARDIZED TRAINING
CREATIVE
THINKING
THINKING SKILLS
FOR TESTERS
• PROMOTE CONTINUOUS
LEARNING AND IMPROVEMENT
• MAINTAIN A TRAINED BENCH
TO ENABLE RAPID RAMP‐UP
FOR PROJECTS
9
CRITICAL
THINKING
COVERAGE
THINKING
4. BUILDING QUALITY INTO SOFTWARE
BUILDING QUALITY INTO
SOFTWARE INSTEAD OF TESTING
QUALITY INTO SOFTWARE
10
DEFECT PREVENTION THROUGH
DEFECT CAUSAL ANALYSIS
COLLABORATION WITH
DEVELOPMENT TEAM
9. 2/4/2014
6
5. ROI BASED AUTOMATION
INITIATE ALL AUTOMATION
REQUESTS THROUGH TCOE
TCOE TEAM RECEIVES DEMO OF
APPLICATION FUNCTIONALITY,
INTERVIEWS KEY STAKEHOLDERS
TCOE TEAM PERFORMS QUICK
SMOKE TEST TO CHECK APPLICATION
STABILITY
TCOE TEAM ARRIVES AT A DECISION
TO APPROVE OR REJECT
AUTOMATION REQUEST
11
5. ROI BASED AUTOMATION
SPECIALIZED METRICS TO KEEP TRACK OF AUTOMATION PROJECT
TYPE FORMULA DESCRIPTION
AUTOMATION BY RISK OR
PRIORITY
WEIGHTED TEST CASES AUTOMATED BASED ON
RISK
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
TOTAL WEIGHT OF MANUAL TEST CASES BASED ON
RISK
UNDER THIS METHOD ONE NEEDS TO COMPUTE THE
RISK/PRIORITY OF EACH MANUAL TEST CASE, THEN
COMPARE THE SUM OF RISK/PRIORITY OF ALL THE
TEST CASES AGAINST THE TOTAL RISK/PRIORITY OF
TEST CASES AUTOMATED.
AUTOMATION ROI BY TIME
TOTAL EXECUTION TIME FOR TEST CASES
AUTOMATED
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
TO COMPUTE USING THIS METHOD ONE NEEDS TO
CALCULATE TOTAL TIME OF RUNNING ALL THE
MANUAL TEST CASES IN AUTOMATED FASHION VS. THE
TOTAL MANUAL EXECUTION TIME FOR THESE TESTMANUAL EXECUTION TIME FOR THE TEST CASES
AUTOMATED
TOTAL MANUAL EXECUTION TIME FOR THESE TEST
CASES.
AUTOMATION ROI IN DOLLARS
AUTOMATION BENEFITS ACCRUED
‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐
TOTAL COST OF AUTOMATION
THIS IS STANDARD AUTOMATION CALCULATION IN
DOLLARS OR ANY OTHER CURRENCY.
12
10. 2/4/2014
7
6. INTEGRATING MULTIPLE TOOLS AND METHODOLOGIES UNDER A
SINGLE FRAMEWORK
TCOES CAN DEVELOP CUSTOM SOLUTIONS THAT INTEGRATE MULTIPLE TOOLS AND METHODOLOGIES
FRAMEWORKS CAN COMBINE THE
ADVANTAGES OF VARIOUS APPROACHES SUCH
AS FUNCTIONAL DECOMPOSITION, KEYWORD
DRIVEN AND DATA DRIVEN APPROACHES
SUCH FRAMEWORKS CAN BE HIGHLY
MAINTAINABLE AS UI, TEST SCRIPTS AND TEST
DATA ARE REUSED
CAN BE A GREAT VALUE ADD FOR NON‐
TECHNICAL AND BUSINESS USERS
REDUCED COST OF SCRIPT CREATION. ENABLE
LARGE SCALE AUTOMATION
13
7. RECRUITING AND RETAINING GOOD TESTERS
A TCOE CREATES
COMPELLING NEW
OPPORTUNITIES
FOR TESTING
PROFESSIONALS
A TCOE BRINGS
TOGETHER THE
BEST TALENT AND C
14
INCREASED JOB
SATISFACTION DUE
TO REMOVAL OF
REPETITIVE TASKS
BEST TALENT AND
CREATES TEST
SPECIALISTS
CHOOSE
THINKING SKILLS
OVER TECHNICAL
SKILLS,
EDUCATIONAL
BACKGROUND
AND TOOL SKILLS
WHILE HIRING
12. 2/4/2014
9
ABOUT ZEN TEST LABS
Domains
Banking & Financial Services
Banking & other ISV’s
Mobile and eLearning
BANKING
Technology, Tools and
Frameworks
Large scale Test Automation
IP based products‐
ZenFRAME, ZenWRITE, etc.
Testing tools Mercury‐ HP,
Test Complete, Selenium
(Illustrative only)
E ternall a dited ISO
Geographic Spread
USA operations based
out of Wilmington, DE
Customer base that spans
US, EU, APAC and India
Partner net ork that
S|One Corporation
BANKING SOFTWARE
Externally audited ISO
27001 processes
ROI driven engagement
models
Testing thought leadership‐
STAR conferences, etc.
Partner network that
spans US, EU, MEA and
APAC
17
OTHERS
