Snehasish Roy who is a Test Automation Manager & Enterprise Architect at Amdocs along with Rekha Shetty who is a Program Manager at Amdocs and Umashankar Reddy who is a Software Test Manager at Amdocs took a Session on "AIML Driven extensive reusable Automation Asset Management Process (AIM) for "QA/QE/Testing Org"" at Global Testing Retreat #ATAGTR2019 Please refer our following post for session details: https://atablogs.agiletestingalliance.org/2019/12/04/global-testing-retreat-atagtr2019-welcomes-snehasish-roy-as-our-esteemed-speaker/ https://atablogs.agiletestingalliance.org/2019/12/04/global-testing-retreat-atagtr2019-welcomes-rekha-shetty-as-our-esteemed-speaker/ https://atablogs.agiletestingalliance.org/2019/12/04/global-testing-retreat-atagtr2019-welcomes-umashankar-reddy-as-our-esteemed-speaker/

1. Main Author -> Snehasish Roy:- Automation Manager (Enterprise Architect)
Co-Author 1 -> Rekha Shetty:- Test Manager
Co-Author 2 -> Umasankar Reddy:- Test Manager
Amdocs
1
AIM ->
Back to Story Appendix I
AIML Driven extensive reusable Automation Asset Management Process (AIM) for “QA/QE/Testing Org”

2. Abstract
As days progress, number of software services used by human are increasing at very Rapid dynamic pace. Also today’s Digital World is surrounded with many
smart devices and complicated technologies.
To cope up with, over the period of time the Software Development Life Cycle has been changed from Waterfall to Agile and DevOps now, where testing is
always been the last quality gate in the cycle which has been drastically reduced with cost decline.
The above points making QA/QE/Testing very challenging and complex to handle without Organized Lean Automation Testing & Quality Design Process.
As per World Quality Report statistics -> Test Automation Level is below per low with several challenges been observed for both Automation and DevOps.
Automation Testing mostly in silos and at project level, not leveraged across Testing Pyramid for the same project and Cross Projects within Organization.
This Thought paper will cover the resolution of majority of the challenges faced today for achieving desired level of Automation and explain transformation need
for QA/QE/Testing Org to have of An Extensive Reusable Organized Automation Process at QA/QE/Testing org level.
QA/QE/Testing Org need:- In DevOps world Organized Intelligent Smart Automation with Increased coverage is needed for Continuous Integration and
Continuous Deployment with shrinking timelines to have Quality Product delivery.
These brings a driver for Quality Engineering leadership to transform existing “QA/QE/Testing Automation” process into “Automation Asset Management
Framework” to overcome associated challenges in order to sustain and stay ahead with the industry.
As an enabler for Quality Engineering leadership to break the silos, “AIM” is the QA/QE/Testing Organization level solution to leverage and utilize Testing
Automation Assets across.
“AIM” is the AIML Driven Centralized Governed Shared Cloud based Asset Management Framework at Organization Level for Business Units/Domains -> Verticals,
to revolutionize and standardize the “QA/QE/Testing” Automation Process to meet the Future purpose for QA/QE/Testing organization for years to come.
The Assets/Inventory can be shared within Inter Project Testing Pyramid and Cross Projects as well as per defined Compliant process.
All the implemented Assets are checked-in/out to Automation Inventory Management(“AIM”) with code branch & version control mechanism.
“AIM in a Nut-Shell” and Benefit Realization:-
 Using “AIM” there were 93+ assets were implemented across 29+ projects.
 Introduced new development ROI and KPI for code reuse:- ~50+ % of Development Time Saving.
 Automation Execution ROI and KPI introduction:- ~70-80% Execution Coverage increase.
 Benefit realization is in shorter time to market, quick and faster resolution, increased efficiency and reduced escaping production defects.
Also Benefit realization is observed in shorter time to market, quick and faster resolution, increased efficiency and reduced escaping production defects.
2 Back to Story Appendix I

3. 10 Million 1,000 Billion
1 Million 1 Billion
100,000 10 Million
10,000 100,000
1,000 1,000
100 100
1955 1975 1995 2005 2015 2025 2040
Interoperability-NumberofDataSources
ResponseTime-Seconds
1 1
Minutes
Hours
Seconds
Months
Weeks
Days
Software Industry Evaluation
1. Digitization Connects People, Devices and Businesses -> Exponential
6.08 6.85 7.58 8.24
8.85
0.36 1.96
5
10.9
17.15
0 5 30.73
125
200
0
500
0
20
2000 2010 2020 2030 2040
Year:- 2000 to 2040
Global Population Internet Connections Connected Devices
InBillions
Year:-2018
AIML
Large
Data
IOT
SOA
Cloud
3. Technology Complexity ->2. Response Time Interoperable data process -> Exponential, Real Time
Multiple OS
Platform &
Browsers
Digital
Fig 1 Fig 2
Fig 3 Fig 4
Year:-2018
75+ Billion Devices & Sensors by 2025
200 Billion by 2040
3 Back to Story Appendix I
Analytics

4. Software QA/QE/Testing Industry Today’s State -> Problems
Waterfall Agile DevOps
Scope
Complexity
Time to
Market
Cost
Quality Shift Left Right
SDLC Methodology compare due to Industry change QA/QE/Testing Automation Today -> In Silos and Scattered
Fig 5.1
Fig 6Market Size:- $12.85 Billion by 2025
DevOps is Future
CAGR 18.6%
Fig 5.2
4 Back to Story Appendix I

5. Software QA/QE/Testing Industry Today’s State -> Problems cntd
84%
16%
Testing Level
Manual Automation
84%
16%
All Testing Life Cycle
Activities
Manual Automation
Lies between 14% – 18 %
Software Testing
CAGR 13-15% by 2023
Size:- $35 Billion to $67 Billion
by 2023
Testing CAGR & Market SizeWorld Quality Report Stats
Automation Testing
CAGR 18-20% by 2023
Size:- $9 Billion to $22 Billion
By 2023
Automation Testing Market Size corresponding to Software
Testing Market size increase -> 26% to 32% by 2023
Fig 7.1 Fig 8.1Fig 7.2 Fig 8.2
5
Automation Testing Stats is Below par
Back to Story Appendix I
Executed using Automation Tools:-
• 16% overall
• 15% end-to-end business scenarios

6. Software QA/QE/Testing Industry Today’s State -> Problems cntd
Extensive Organized Automation Process is missing in QA/QE/Testing Org.
Automation reusability is not leveraged within project and across projects to handle complete coverage within Org.
Automation is done in Silos.1.
2.
3.
QA/QE/Testing Org Today’s state -> Consolidated Problem reasons
Duplication of Automation Inventory across organization:- In-efficient utilization of assets and existing Skilled resources.
4.
Fig 9
6 Back to Story Appendix II
Reference taken from:-
Main challenges in achieving desired level of test automation:-
• Statistics calculation done with interviewing CIOs/Executives and plotting %
• 3, 4 stats started in 2018. 5, 6, 7 continued as of previous year.
1. Lack of skilled
and experienced
test automation
resources
2. We don’t have
the right
Automation
Tools
3. We have too
many different
automation
tools
4. We started
too late with
testing and
Test
automation
5. We don’t
have the right
automation
testing
process/method
6. We find it difficult
in automating
because we use
multiple development
life cycle
7. Find it difficult to
Integrating test
automation into a
DevOps process

7. Software QA/QE/Testing Industry need and Solution
BU -> Business Unit
V -> Vertical
P -> Project
BU1-
V1-P1
Asset1
BU1-
V1-P2
Asset2
BU2-
V2-P2
Asset n
BU2-
V1-P1
Asset1
BU4-
V1-P1
Asset1
BU3-
V3-P3
Asset2
BU3-
V3-P1
Asset8
BUn-
Vn-Pn
Asset n
BU4-
V2-P1
Asset2
BU5-
V1-P4
Asset 4
BU8-
V3-P5
Asset 2
AIM Solution:- Brain Sketch
Legend:-
“AIM” is the Centralized Governed Shared Cloud based Automation Asset Management Framework at Organization Level for Business
Units/Domains.
Automation Assets leveraged across Project Testing Pyramid and Across Projects within or across Business unit and Verticals.
Due to Time Shrink and Cost decline Extensive Reusable Organized Automation Process is required at QA/QE/Testing org level.
QA/QE/Testing Organization transformation need to overcome Problems:-
QA/QE/Testing Organization transformation Solution:-
Fig 10
7 Back to Story Appendix II

8. AIM -> CONCEPTUAL POD BUSINESS ARCHITECTURE
Organization (All Accounts) - AIM
Asset List
Account
Asset List
Account Level Implementation
Account Level AIM
DashboardMeasurement
Intelligent
Search Engine
BU->V->P->
Assets category
BU –> V -> P ->
Hierarchy
Compliant
Filter
Profile
Security
Access
Management
Governance
Tool/Utility
Change
Tool/Utility
Add
Capability
Change
Capability
Add
Automation
Suite Change
Automation
Suite Add
Framework
Change
Framework
Add
• AIM:- Automation Inventory Management
• One Account can have multiple Projects
BU -> Business Unit, V -> Vertical, P -> Project
• D -> ROI* :- One time Development -> Return on Investment
• D -> KPI*:- Development -> Key performance Indicator
• E -> ROI:- Multiple Execution -> Return on Investment
• E -> KPI:- Multiple Execution -> Key performance Indicator
Cloud
Infrastructure
Reporting
DeleteChangeAdd
Asset
Evaluation
Legend:-
Fig 11
Development/
Maintenance
E -> KPIE -> ROID -> KPID -> ROIDashboardMeasurementReporting
D-> KPID-> ROIE-> KPIE-> ROIExecution
8 Back to Story Appendix II

9. Business Unit
Business Unit ->
Verticals
BU-> V -> P -> Asset Category
AIM -> 1. Governance Model -> Add/Change Asset
QA/QE/Testing Org Asset List
(AIM)
Add/Change
Request
Normal
Change
Risk
Assessment
Management
Approval
High
Risk?
CAB
Approval
Implement
Add/Change
Script
Add/Change
Standard
Add/Change
Access = Yes
NO
YES
1 2 3
4
5
6
7
8
8
8.1 8.2
8.3
9
9
Automation
Framework
Automation
Suite/Scripts
Automation
Capability
Automation
Tool/Utility
Compliant = Yes
Verticals ->
Projects
Add/Change
Fig 12
Compliant
Filter
9 Back to Story Appendix II

10. Reporting
Intelligent Search Engine
AIM -> 2. Governance Model -> Search Engine and Implementation
QA/QE/Testing
Org Asset List
(AIM -> Cloud)
Search Engine
AIML
Engine ->
Assets
Search
Result
1. NLP Prediction Model.
2. Pattern Search
3. Behavior Driven search
4. Update Engine for
successful & other results
Recursive ->
Update AIML Engine
Add/Change Asset
Implement
Existing
Assets from AIM
Follow 1. Governance Model -> Add/Change Asset
Access = Yes
Users
12
13
14
15
16
Search Asset
32
1
4
5
6
7
8
9
101112
13
14
15
16
Compliant
Filter
Business Unit
Business Unit ->
Verticals
Verticals ->
Projects
Fig 13
Dashboard &
Measurements
ROI & KPI
10 Back to Story Appendix III

11. QA/QE/Testing
Org Asset List (AIM)
AIM -> Search Engine -> AI & ML Engine -> Technical Architecture
13
11
ML
Engine
Data
Hub
AI
Engine
Standard
Change
Normal
Change
Service Invoker/Bus
1. ML_Get_Config_Template
1.1. ML_Supervised_Aut_ Framework
1.2. ML_Supervised_Aut_Script_Suite
1.3. ML_Supervised_Aut_Capability
1.4. ML_Supervised_Aut_Tools_Utilities
Data ingestion:-
1. ML_Unsupervised_Aut_ Framework
2. ML_Unsupervised_Aut_Script_Suite
3. ML_Unsupervised_Aut_Capability
4. ML_Unsupervised_Aut_Tools_Utilities
Search
1. AI_Refer_Previous_Search_Result
1.2. AI_Predict_Return_Update_Search
1.2.3. AI_Return_Search_Aut_Framework
1.2.4. AI_Return_Search_Aut_Script_Suite
1.2.4.1 AI_Return_Search_Function_Activity_Method
1.2.5. AI_Return_Search_Aut_Capability
1.2.6. AI_Return_Search_Aut_Tools_Utilities
1.2.6. AI_Update_Search_Results
Desktop &
Mobile App
Recursive
Recursive
• NLP techniques (Python/R/Matlab)
• Exact Match and AIM Match
• Probability & Prediction
• N-gram
• Bayes techniques
• Other AI techniques and rules
First time Supervised configuration Un-supervised learn
Assets
Add/Change/Delete
BU –> V -> P -> AIMs
AI Rules
Engine
Legend:- Major Services are mentionedFig 14
Back to Story Appendix III

12. QA/QE/Testing
Org Asset List (AIM)
AIM -> Reporting -> ML & Report Engine -> Technical Architecture
13
12
ML
Engine
Data
Hub
Report
Engine
Standard
Change
Service
Invoker
/Bus
Data ingestion:-
1. ML_Unsupervised_Aut_ Framework
2. ML_Unsupervised_Aut_Script_Suite
3. ML_Unsupervised_Aut_Capability
4. ML_Unsupervised_Aut_Tools_Utilities
Dashboard/
ROI/KPI
1.1 Report_Return_Development_ROI_Project
1.2 Report_Return_Development_KPI_Project
1.1.1 Report_Return_Development_ROI_BU_Account
1.2.1 Report_Return_Development_KPI_BU_Account
1.1.1.1 Report_Return_Development_ROI_Org
1.2.1.1 Report_Return_Development_KPI_Org
2.1 Report_Return_Execution_ROI_Project
2.2 Report_Return_Execution_KPI_Project
2.1.1 Report_Return_Execution_ROI_BU_Account
2.2.1 Report_Return_Execution_KPI_BU_Account
2.1.1.1 Report_Return_Execution_ROI_Org
2.2.1.1 Report_Return_Execution_KPI_Org
3. Report_Return_Dashboard_Usage
Desktop &
Mobile App
Recursive
Recursive • Development estimate calculation
• Execution utilization calculation
• Release wise Defects calculations
• ROI & KPI Calculations
Un-supervised learn
Assets
Implementation
BU –> V -> P -> AIMs
Report
Analytics
Engine
Legend:- Major Services are mentionedFig 15
Back to Story Appendix III

13. AIM Assets classification
QTP Versions
UFT Versions
Ginger Versions
Selenium + TestNG
Cucumber BDD
Robot
Adobe Reporting
BI
TOSCA
Hoverfly (Virtualization)
AppliTool
Appium
Katalon
Sikuli
Automation
Framework
Sanity:-
Customer relationship management X,X+I,..
Customer service management Y, Y+J,..
Regression:-
Customer relationship management X, X+I,..
Customer service management Y, Y+J,..
Order management system Z, Z+K,..
mS (micro Services)
Data creation:-
Order management system Z, Z+K,..
API
Progression:-
Customer relationship management X,X+I,..
Customer service management Y, Y+J,..
Order management system Z, Z+K,..
Check environment
DB validation
DB scan tool
Auto DB fetch
Auto file simulator
Single console execution
KIBANA -> Elastic search
Idle time
Usage tracking
Disk clean up
QC audit
Monitor defects
Toad network tool
Automation
Suite/Scripts
Automation
Capability
Automation Tool
and utilities
Suites:- BU = Telecom -> Vertical = Client A ->
Project1, Project 2,.., Project N.
Script, Script-less, Intelligent Smart
13 Back to Story Appendix III
Reuse within same Project Reuse with other Project
Reuse across Projects
Reuse across Projects
Legend :-Reuse post 1st time development ->
High (71-100%)
Medium (31-70%)
Low (1-30%)
Note-> Only some of the assets
Are mentioned as examples

14. QA/QE/Testing Organized Automation Process -> Value Propositions
Cost
Efficiency
Quality
Time to
Market
Flexibility
Scalability
Consolidation
Innovations
QA/QE/Testing
Challenges
AIM
Project 1
Automation
Project 2
Automation
Project 3
Automation
ROIs
KPIs
Continuous
Improvement
Automation
Maturity
Model
Innovation
Management
Reuse
Pool of automation
experts
Defined Governance
processes
Common Defined
Infrastructure:-
1.Frameworks,
2.Capabilities,
3.Tools & Utilities,
4.Automation Suites
Defined
Metrics
Established
Processes
Combined Solution
Accelerations
Fig 16
Back to Story Appendix III
Automation Infrastructure Management at QA/QE/Testing org
14

15. AIM -> Challenge-Solution-Value Model at a Glance
THE
SOLUTION
THE
CHALLENGE
97%
• QA/QE/Testing Org High
levels of reusability of
existing automation
assets across
• DevOps -> Development
& testing working
together for CICD with
Asset leverage
• QA/QE/Testing Org ->
Non-Organized
automation
• DevOps -> Shorter
Coverage at Project
level for short cycle
• Silo Automation
• Shorter delivery cycles
• Higher defect identification
in pre-production
• Reduction in
defect leakage to
Production
• Remove Asset duplicity ->
Waste removal
• Increased Automation
coverage at Project and Org
• Asset Duplication
• Effort Waste
• Organized extensive
automation at Project
level
93+ Assets implement in 29+ Projects:-
AIM:-
THE
BUSINESS VALUE
15 Back to Story Appendix III

16. Case Study -> Automation Framework, Capability, Tools -> ROI & KPI Measurement
16
Development (D) Business Values
Total Auto Frameworks:-
~16+
(Left shows example of 3
frameworks for 3 BUs)
26
Months
12.20
Months
With Reuse
effort
required
Development
effort required
With Reuse
~13.80 Man Months
Saved.
D -> ROI & Savings:-
D -> KPI - Reuse:-
~ 213% Coverage
D -> KPI – Automation coverage
Fig 17
Fig 19
Automation Framework, Capability, Tools D-> ROI* & KPI*:-
Fig 18
3 26
Total Auto Capabilities:-
~9+
(Left shows example of 5
capabilities for 3 BUs)
16.50
Months
10.65
Months
With Reuse
effort
required
Development
effort required
With Reuse
~5.85 Man Months
Saved.
D -> ROI & Savings:-
D -> KPI - Reuse:-
~ 155% Coverage
D -> KPI – Automation coverage
5 15
Total Auto Tools/Utilities:-
~26+
(Left shows example of
19 Tools/Utilities for 3 BUs)
64.15
Months
17.20
Months
With Reuse
effort
required
Development
effort required
With Reuse
~46.95 Man Months
Saved.
D -> ROI & Savings:-
D -> KPI - Reuse:-
~ 373% Coverage
D -> KPI – Automation coverage
19 90
Back to Story Appendix III

17. Case Study -> Automation Suite/Scripts -> ROI & KPI Measurement
17
Automation Suite/Script D-> ROI* & D-> KPI*:- Development(D) Business Values
54%
Overall Release over Release:-
Developed = 46%
Reused = 54%
Total Auto Development
of Flows/Scenarios/TCs:-
~47632 2415
Months
987
Months
With Reuse
effort
required
Development
effort required
With Reuse
~1428 Man Months
Saved.
D -> ROI & Savings:-
D -> KPI - Reuse:-
~ 244% Coverage
D -> KPI – Automation coverage
Development Reuse
47632
25266
22%
Overall Regression to Progression and vice
versa:-
Developed = 78%
Reused = 22%
Total Auto Execution of
Flows/Scenarios/TCs:-
~188490 4222
Months
Execution (E) Business Values
1681
Months
Automation
Execution
time
Manual
Execution
time
With Automation
~2541 Man Months
Saved.
E -> ROI & Savings:-
20
Days
10
Days
E -> KPI - Delivery Cycle:-
~ 251% Coverage
E -> KPI – Execution coverage
Automation Suite/Script E-> ROI & E-> KPI:-
Fig 20
Fig 21
Back to Story Appendix IV
BU2 start

18. Case Study-> Automation Suite/Scripts -> E->KPI -> PROD Defects Reduced drastically
18
Note:- One Release = 2 months combined data. R1 shows state before Inventory sharing. R5 & R6 shows post Inventory Sharing over period of
time with Increased Coverage. There are transformation projects as well in this Statistics. Also the Release size varies based on Scope.
Fig 22 Fig 23 Fig 24
Fig 25 Fig 26 Fig 27
Early Defect Identification rate increased significantly
Back to Story Appendix IV

19. WQR -> Main challenges to achieve desired test automation -> Solution Mitigation
19
We don’t have the right Automation Tools.
We started too late with testing and Test automation.
1
2
3
4
Lack of skilled and experienced test automation resources.
We have too many different automation tools.
We don’t have the right automation testing process/method.5
We find it difficult in automating because we use multiple development life cycle.6
Find it difficult to Integrating test automation into a DevOps process.7
Inventory reuse allows to free up existing skilled resources. Coverage increased.
Automation Infrastructure management with history allows to choose the right Tools.
For referring existing similar automation user is able to select existing utilized Tools.
Early Automation with defined Automation Strategy along with leveraging assets.
AIM is the required “QA/QE/Testing Organized Automation Process”.
AIM supports all SDLC Cycles along with multiple life cycles.
AIM allows to move Shift left and Right along with DevOps CICD pipeline.
Main challenges as WQR
Solution Mitigation
Back to Story Appendix V

20. Bibliography I
1. https://www.wikipedia.org/
2. https://www.sogeti.com/explore/reports/world-quality-report-201819/
3. https://en.wikipedia.org/wiki/Global_Internet_usage
4. https://royal.pingdom.com/2010/10/22/incredible-growth-of-the-internet-since-2000/
5. https://www.quantumrun.com/future-timeline/2020/future-timeline-subpost-technology
6. https://www.quantumrun.com/future-timeline/2030/future-timeline-subpost-technology
7. https://www.quantumrun.com/future-timeline/2040/future-timeline-subpost-technology
8. https://www.infoplease.com/world/population-statistics/total-population-world-decade-1950-2050
9. https://www.statista.com
10.https://www.statista.com/statistics/471264/iot-number-of-connected-devices-worldwide/
11.https://www.forbes.com/sites/louiscolumbus/2016/11/27/roundup-of-internet-of-things-forecasts-and-
market-estimates-2016/#6efe14d2292d
12.https://iot-analytics.com/state-of-the-iot-update-q1-q2-2018-number-of-iot-devices-now-7b/
13.https://futurism.com/by-2020-there-will-be-4-devices-for-every-human-on-earth/
14.http://singularity.com/charts/page67.html
15.http://singularity.com/charts/page70.html
16.https://www.grandviewresearch.com/press-release/global-development-to-operations-devops-market
17.https://www.marketscreener.com/CIGNITI-TECHNOLOGIES-LTD-18554003/news/Software-Testing-Market-
to-Grow-at-a-CAGR-of-13-by-2023-According-to-New-Research-For-Key-players-26167366/
20 Back to Story Appendix V

21. Bibliography II
21
18.https://www.businesswire.com/news/home/20180717005888/en/Global-Software-Testing-Services-
Market-2018-2022-Increasing
19.https://www.prnewswire.com/news-releases/global-automation-testing-market-2018-2023-expected-to-
grow-at-a-cagr-of-17-7-during-the-forecast-period-to-reach-19-27-billion-300652566.html
20.https://www.businesswire.com/news/home/20180610005033/en/Global-Automation-as-a-service-
Market-Post-20-CAGR-2018-2022
21.https://globenewswire.com/news-release/2018/05/11/1500942/0/en/Global-19-Billion-Automation-
Testing-Market-2018-2023-Increasing-Adoption-of-the-Devops-Methodology.html
Back to Story Appendix V

22. Author Biography
Snehasish Roy
Automation Manager
(Enterprise Architect)
1. 13+ years of Experience in Enterprise System Architecture, Design, Development, Testing &
Production Support.
2. Currently working in Amdocs. Carry prior experience from Syntel.
3. Worked in BNFS, Telecom, Insurance and other domains.
4. Automation Manager & Enterprise Architect.
5. Education:- BE in Electronics & Communication from Visvesvaraya Technological University.
Certifications:- TOGAF 9.1, PRINCE2.
6. Currently Responsible for NAM, ATT and Other regional account’s Automation.
1. 14+ experience in Testing of different domains.
2. Currently working as S/W Test Manager in Amdocs.
3. Education: B.E in Computer Science from Karnataka University
Certifications: CP-DOF.
4. Responsible for NAM region account E2E testing deliverables.
Rekha Shetty
S/W Test Manager
1. 14+ years of experience in Testing of different domains.
2. People management and End to End project governance of DevOps projects.
3. Led several testing programs with the teams spread across multiple global delivery
locations.
4. Supports both Transformation + BAU projects.
5. Education: B.Tech in Electronics and Communication from Jawaharlal Nehru Technological
University. Certification: DevOps certification.
6. Responsible for NAM and Canada regional accounts testing.
Umasankar Reddy
S/W Test Manager
22 Back to Story Appendix V

23. Question & Answers
23 Back to Story Appendix V

24. Thank You
24 Back to Story Appendix V

25. Appendix I - Paper Story Elaboration
Page 1:- Thought paper title. Author details.
Page 2:- Explains the Abstract of the paper. QA/QE/Testing Organization high level -> problem statements, solution and
benefits.
Page 3:- Software industry evaluation.
Fig1:- The Trend of Global population, Internet connections, Connected device plot.
Fig2:- How the World will look alike with Connected Devices and sensors with the predicted numbers in future.
Fig3:- The Trend of Exponential increase of Interoperability of the Data with Exponential decrease in processing
time due to continuous software industry change. (Moore’s law)
Fig4:- Due to the changes seen in Fig1, Fig2, Fig3 shows Software Technology evolution towards complexity.
Page 4:- Software QA/QE/Testing Industry Today’s Problems -> Automation
Fig5.1:- Shows due to the Software industry change, the SDLC Methodology change and comparison of SDLC
Models.
With DevOps Scope & Complexity increased while Time to market and Cost decreased however the Quality to
be sustained at High.
Fig5.2:- Market study shows that DevOps is the Future based on CAGR & market size and currently under
Adoption.
Fig6:- Shows QA/QE/Testing Automation Today’s state where Automation is performed in Silos and scattered
even within same project.
Page 5:- Software QA/QE/Testing Industry Today’s Problems -> Automation continued
Fig7.1:- World Quality Report (WQR) was created by interviewing 1700 executives from 32 countries and
benchmark for QA/QE/Testing industry. Based on WQR shows the Automation Level %.
Fig7.2:- Based on WQR shows All Testing Activities Automation %.
25 Back to Story Appendix V

26. Appendix II - Paper Story Elaboration
Page 5:- Software QA/QE/Testing Industry Today’s Problems -> Automation continued
Fig8.1:- Market study for Total Software Testing Market size and CAGR increase.
Fig8.2:- Market study for Software Automation Testing Market size and CAGR increase.
After comparing both markets, Automation Testing Market size increases due to bigger CARG rate shows that
future of QA/QE/Testing work without extensive Automation is not sustainable.
Page 6:- Software QA/QE/Testing Industry Today’s Problems -> Automation continued
Fig9:- As per WQR explains further challenges in achieving desired level of automation.
Finally QA/QE/Testing organization Consolidated problem reasons are thought, consolidated and articulated.
Page 7:- Software QA/QE/Testing Industry need and solution based on articulated problem reasoning.
Explanation of QA/QE/Testing Org Transformation is explained. Solution “AIM” is explained.
Fig10:- Solution Brain Sketch is designed along with Correct Organization Hierarchy required in QA/QE/Testing
Org for Assets/Inventory Reuse. QA/QE/Testing Org hierarchy -> Business Unit (BU) -> Vertical (V) -> Project (P).
Business Units will be BNFS, Telecom, Insurance, Healthcare, Retail etc on the same business domain.
Verticals will be defined by the same Line of business (LOBs) for same or different customers without business
conflict of interest according to Compliant regulations of Clients and Vendor providing QA/QE/Testing Service.
Projects will be under the same LOBs of the verticals.
Page 8:- AIM -> CONCEPTUAL POD ARCHITECTURE defining the transformed Reusable Organized Automation Process.
Fig11:- Detailed Component Level Architecture of AIM along with 4 types of assets/inventory classification.
Page 9:- AIM -> Further detailing the POD from page 8 to show -> 1. Governance Model -> Add/Change Asset.
Fig12:- How the Governance model works for Addition or changing Assets/Automation Inventory ?
26 Back to Story Appendix V

27. Appendix III - Paper Story Elaboration
Page 9:- Fig12:- Also a very critical point explained are the Compliant Filters to be Compliant as per Client Non-
Disclosures, Copy rights and Vendor Compliant process applicable for Products and Intellectual
Properties (IP).
Also explains the Management Approval, Risk Management and Change Advisory Board (CAB) Approval
process for AIM.
Page 10:- Continue detailing the POD from page 8 to show AIM -> 2. Governance Model -> Search Engine and
Implementation process capitalizing and reusing existing assets.
Fig13:- How the Governance model works for Asset/Inventory search and implementation model with
Compliant process in place ?
Explained the Search engine and Artificial Intelligence Machine Learning (AIML) engine interactions at high
level.
Reporting, Dashboard, Measurements, ROI & KPI model iterations are depicted.
Page 11:- Fig14:- AIM -> Search Engine -> AI & ML Engine -> Technical Architecture along with core services.
Page 12:- Fig15:- AIM -> Reporting -> ML & Report Engine -> Technical Architecture along with core services.
Page 13:- AIM Assets categorization and some examples provided under each category.
Page 14:- QA/QE/Testing Organized Automation Process -> Value Propositions.
Fig16:- QA/QE/Testing Organized Automation Process brings the value propositions and balance to overcome
Today’s QA/QE/Testing challenges.
Page 15:- AIM -> Challenge-Solution-Value Model at a Glance.
Page 16:- Case Study -> Automation Framework, Capability, Tools -> ROI & KPI Measurement.
This paper introduces new ROI & KPI parameters and calculation factors for Development. * refers new.
27 Back to Story Appendix V

28. Appendix IV - Paper Story Elaboration
Page 16:- Case Study -> Automation Framework, Capability, Tools -> ROI & KPI Measurement
All the ROI & KPI calculations are done after considering few BUs under Org, few projects under each BU and
few assets.
It is observed that actual QA/QE/Testing & Org level ROI & KPI factors increases even significantly as the
reusability increases much more while accounting more implementations.
D -> ROI* & D -> KPI* follows in correspondence to below for multiple projects under 3 BUs.
Fig17:- Same 3 Automation framework implementation -> Coverage & Savings.
Fig18:- Same 5 Automation capabilities implementation -> Coverage & Savings.
Fig19:- Same 19 Automation tools/utilities implementation -> Coverage & Savings.
Page 17:- Case Study -> Automation Suite/Scripts -> ROI & KPI Measurement. (Development & Execution)
D -> ROI* & D -> KPI* follows in correspondence to below for multiple projects under 2 Bus for past 1 year stats.
Fig20:- Automation Suite/Script Development ROI & KPI-> Reuse (%), Coverage (%) & Savings.
Fig21:- Automation Suite/Script Execution ROI & KPI-> Delivery Cycle shrink for DevOps, Coverage(%) & Savings.
Page 18:- Case Study -> Automation Suite/Scripts -> E-> KPI -> PROD Defects Reduced drastically
Pre Production release wise defects:-
Fig22:- Project Wise -> For 6 Projects under BU3 over 5 releases.
Fig23:- Vertical Wise -> For 2 Verticals/Accounts under BU3 over 5 releases.
Fig24:- Business unit and Organization wise -> For 2 BUs under Org and at Org over 6 releases.
Post Production release wise defects (Reduction observed for increased coverage):-
Fig25:- Project Wise -> For 6 Projects under BU3 over 5 releases.
Fig26:- Vertical Wise -> For 2 Verticals/Accounts under BU3 over 5 releases.
Fig27:- Business unit and Organization wise -> For 2 BUs under Org and at Org over 6 releases.
28 Back to Story Appendix V

29. Appendix V - Paper Story Elaboration
Page 19:- WQR -> Main challenges to achieve desired test automation -> Solution Mitigation explanation.
Page 20:- Research references in Bibliography I.
Page 21:- Research references continued in Bibliography II.
Page 22:- Author Biography details.
Page 23:- Questions and Answers ?
Page 24:- Thank You.
Page 25:- Appendix I - Paper Story Elaboration.
Page 26:- Appendix II - Paper Story Elaboration.
Page 27:- Appendix III - Paper Story Elaboration.
Page 28:- Appendix IV - Paper Story Elaboration.
Page 29:- Appendix V - Paper Story Elaboration.
Note:- All the details shared as per being compliant to Amdocs process. BU, Vertical, Automation
Framework, Capability, Tool & Utility details are mentioned generically as per protocols.
29 Back to Story Appendix V