1. Automate, Virtualize & Smart Test
Understanding the new realities of Testing
by
Manoj Narayanan
Director –Testing Services, Cognizant
@manoj7698
http:everydaytesting.blogspot.com
©2011, Cognizant

2. What does traditional testing offer?
Discrete, independent
Waterfall Methodology
Requirements Architecture Developm
Testing Release
Gathering & Design ent
Focused on defect identification
1

3. What does business want today?
24% 22% 17% 14% 13% 7%
Reducing risk Quickly Reducing costs Delivering Improving Delivering
5%
Compliance
by preventing responding to by improving applications on collaboration applications and
poor quality customer our time and on between before regulatory
from impacting requests and development budget development competition issues
satisfaction requirements and testing and QA
productivity
• 32% of organizations say they release software with too many defects
• 38% of organizations believe they lack an adequate quality assurance program
• 27% of organizations do not conduct any formal quality reviews
2
More than 60%
©2011, Cognizant
Source- QAI –Edista Testing

4. What does business want today?
Reducing Risk
Responding Fast
24% 22% 17% 14% 13% 7%
Reducing risk Quickly Reducing costs Delivering Improving Delivering
5%
Compliance
by preventing
poor quality
responding to
customer
by improving
our
applications on
time and on
Beat Competitors
collaboration
between
applications
before
and
regulatory
from impacting requests and development budget development competition issues
satisfaction requirements and testing and QA
productivity
• 32% of organizations say they release software with too many defects
• 38% of organizations believe they lack an adequate quality assurance program
• 27% of organizations do not conduct any formal quality reviews
3
More than 60%
©2011, Cognizant
Source- QAI –Edista Testing

5. Traditional testing approach
Discrete, independent
Waterfall Methodology
Requirements Architecture & Developmen
Testing Release
Gathering Design t
Focused on defect identification
4 ©2011, Cognizant

6. We need to move testing to the beginning of
SDLC – focus on prevention than identification
From Discrete and Independent Testing to…
Requirements Architecture
Development Testing Release
Gathering & Design
Waterfall Methodology
…Continuous and Integrated Testing
Scrum
5 ©2011, Cognizant

7. The “Shift-Left” Philosophy
Focus on Test Driven
Development or a Non-functional
variant testing
Performance & Security PoCs
Identify QA skill Early execution of QA, Automated end to
sets needed Non-GUI testing
and initial effort
end regression
estimate
6 ©2011, Cognizant testing

8. Shift-Left Philosophy is accompanied by the
need for a new testing approach
Unit
effort
Testing Unit Testing
spent
Percentage
Service Layer Testing Service Layer Testing
GUI
Testing
GUI Testing
Need to move away from traditional …to an approach focusing on increased
approach focusing on GUI based testing.. defect capture through Unit and Service
Layer Testing
7 Automation becomes critical
©2011, Cognizant

9. This results in the need for seeding Quality
Engineering into testing
• A QE approach focused on testing components
• A QA approach focused on regression and UAT
QE Approach QA Approach
8 ©2011, Cognizant

10. And “Smart Testing” dissolving boundaries of
traditional system & integration testing
QE Approach QA Approach
Regression Fast merging into a
Unit Testing Testing social media and
(automation) “crowd” aided
Community Testing
Service Layer
Beta Testing
Testing
Functional User
Testing Acceptance
(automation) Testing
9 ©2011, Cognizant

11. And “Smart Testing” dissolving boundaries of
traditional system & integration testing
QE Approach QA Approach
Regression
Unit Testing Testing
(automation)
Service Layer
Testing
Community
Functional Testing
Testing
(automation)
10 ©2011, Cognizant

12. Leverage optimal mix of automation across
the lifecycle
• Re-usable, automated test cases used by developers
Unit Testing
• Leveraging xUnit Frameworks
• Leverage re-usable frameworks like RIDE, Fitnesse,
Cucumber
Manual Testing Service Layer Testing
• Web Services testing
• Leverage Service Virutalization
• >90% automated regression test cases
Regression Testing • 100% automated smoke testing
• Continuous validation of test flow relevancy
11
©2011, Cognizant

13. Prioritizing Automation: ROI driven approach
Verify application paths that have high degree of frequency and other high volume
activities where software failures would occur
Validate applications with a high degree of risk which define or control the core of
a company’s business; Failure can cause extreme disruptions in critical operations
Any flow that needs to be run repeatedly in a build
Repetitive Testing
Automating applications that are or will be in production for a long time
Greater benefits can be reaped from Test Automation if the test execution
frequency is high
Test Automation will be an ideal testing solution if the testing has to be performed on
12 ©2011, Cognizant multiple environments

14. Prioritizing Automation: ROI driven approach
Verify application paths that have high degree of frequency and other high volume
activities where software failures would occur
Validate applications with a high degree of risk which define or control the core of
a company’s business; Failure can cause extreme disruptions in critical operations
Any flow that needs to be run repeatedly in a build
Repetitive Testing
Automating applications that are or will be in production for a long time
Greater benefits can be reaped from Test Automation if the test execution
frequency is high
Test Automation will be an ideal testing solution if the testing has to be performed on
13 ©2011, Cognizant multiple environments

15. The need for early testing and automation also
triggers environment related challenges
Environment availability challenges –
productivity losses, inability to duplicate
COM App Database
defects, performance testing
Mainframe
Ordering
Swing App
Business Process
App
Need for testing individual
App Server Additional Apps
components as part of “shift-left”
EJB, WS, Now & Future
ESB,
Processing Web
philosophy JMS,
MQ
App
Database
14 ©2011, Cognizant

16. Work Around: Virtualization & Service
Virtualization
UI Testing no longer the Environment availability challenges
dominant testing method mitigated – simulate performance
COM App Database
environment, component transactions
5
Mainframe
Ordering
Swing App
Re-usability increases, limited dependency on Business Process
App
development teams
Additional Apps
App Server Now & Future
EJB, WS, ESB,
JMS,
Processing Web MQ
App
Early testing is now feasible.
Defects identified at each Enables organization to adopt
component level “Shift Left” testing
15 ©2011, Cognizant Database LISA Virtualize
Picture: Courtesy ITKO

17. Explaining Service Virtualization
• Mainframe applications are consumed
through queues – in this case, MQ
• The MQ sends messages to the
application which gets transformed
through the adapter
“Listener”
©2011, Cognizant

18. This has to be complemented by a centralized
Test Environment Management approach
Cloud too is an option – even after the Amazon EC2 fiasco
17 ©2011, Cognizant

19. And an organizational transition plan based on
current maturity level
Stakeholder Discussions
Evaluate Industry best practices
Inputs Define
Current QA Inputs
Future State
Artifact Analysis
State Methodology Adopted
Focus on
• Process & Methodology Focus on
• Tools • Risk appetite
• Infrastructure • Cost constraints
• Operating Model Identify • Available time
• People: Skill set/Roles implementat • Priority areas
ion priorities • Current & Future
state gap
Stakeholder Validation Draw
Inputs Implementation
18
External dependencies Roadmap

20. But do not let the total cost creep up…
• The QA organization needs to keep a constant watch on the
Total Cost of Quality as it makes the transition
Cost of Production Cost of Compliance
Cost of Testing
Defects Audits
Total Cost of Quality
• During the transition phase, the Cost of Testing goes up due to
the increased focus in areas like skill ramp-up, automation and
process re-alignment
• The resultant increase in Total Cost of Quality needs to be
continuously adjusted to ensure that it falls within best practice
levels of 25–30% within 3 to 6 months of transition
19 ©2011, Cognizant

21. And most importantly, do not forget to
communicate
• Ensure Executive & Stakeholder buy-in on the
change
• Establish a communication channel for regular
updates and feedback
20

22. Summary
• Traditional testing approaches are being replaced by a “shift-
left” approach to accommodate current business realities
• Shift away from GUI based testing to component based testing
• Focus on both QE and QA based testing – prevention rather
than identification: Need for “Smart Testing”
• Automation becomes critical to maintain the ever reducing
testing window
• New facilitators have risen: Community Testing, Service
Virtualization, Test Environment Management
• However, it is very critical that the organization has a well
thought out transition plan to accommodate these changes
21
And, DO NOT forget to communicate!

23. @manoj7698
http:everydaytesting.blogspot.com
www.cognizant.com
©2011, Cognizant