This document discusses the importance of continuous testing in Agile and DevOps environments. It describes key elements of continuous testing including unit testing, static analysis, automation of functional and non-functional tests, service virtualization, continuous integration and delivery pipelines. The document also discusses how a collaborative DevTest platform can help by providing visibility into the testing process and enabling continuous feedback through automated reporting of test results and metrics.

1. Continuous Testing – A key
Ingredient for Success in Agile
& DevOps
By
ChakravarthyS Devarakonda
Asst. VicePresident

2. Page 2
Agenda
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
• AnalystSpeak
• DevOpsinanutshell
• DevOpsvsDevTestOps
• ContinuousTesting&DeliveryleveragingQE
• UnitTesting&SAST
• Automation,Performance&Security
• ServiceVirtualization
• ContinuousIntegration,Deployments&Delivery
• ContinuousAutomation&DeliveryPipeline
• CollaborativePlatform&Howdoesithelp?

3. ANALYST SPEAK
1
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies

4. Page 4
Continuous Testing is important in DevOps
Test is not a step. In a DevOps pipeline, it is a continuous discipline requiring
the need to visualize and assess the test from a plan, through production
and back.
Agile and DevOps practices are disrupting old ways of testing software and
applications to keep up with the increasing demand for quality at speed. Testing
comes earlier in the cycle (shifting left) and is becoming faster, better, and
smarter.
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies

5. Page 5
Continuous Testing is important in DevOps
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
DevOps toolchains are often built from
discrete and sometimes disconnected
tools, making it difficult to understand
where bottlenecks are in the
application delivery pipeline
DevOps initiatives continue to grow
in complexity, driving many IT
organizations to seek tools that
provide greater control and visibility
over the entire DevOps process
Toolchain orchestration promises to simplify the
DevOps toolchain; however, one tool is not able to
support the entire toolchain resulting in additional
integration, ownership and handoff challenges
when using multiple orchestration tools
Orchestrating the DevOps toolchain
can be compelling, but it can introduce
another set of challenges
01
02 03
04

6. Page 6
DevOps adoption Is no longer Just For unicorns
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
Source: Forrester’s Q1 2017 Global DevOps Benchmark Online Survey Source: Forrester Data Global Business Technographics Developer Survey, 2016 &
Forrester Data Global Business Technographics Infrastructure Survey, 2016
Culture and
people come First;
process follows
Release velocity is pivotal There’s no “easy” button

7. Page 7
DevOps Adoption is Growing
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies

8. Page 8
DevOps Tool Adoption is Growing as well
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies

9. Page 9
Adoption and Strategies vary acrossVertical Segments
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
$
Business Services And construction
Ponder The Leap Into Automation
Media, entertainment, &
Leisure embark On The
Journey To Differentiate
With DevOps
Public Sector And Healthcare cling To Manual Audit
And compliance Procedures
Retail And Wholesale DevOps Adoption Drive
Differentiated Interactions With clients
Compliance-wary companies need not fear automation
Utilities And Telecommunications
Lead The Pack To Drive Business
Transformation
Data must drive strategy, with services
that emphasize connectivityFinancial Services Accelerate To compete
With Fintechs
DevOps offers key benefits in fraud detection, time-to-market,
and scalability
Get ready for the next generation of digital stores
DevOps drives identi cation
to execution to improve
customer experience
Manufacturing Transitions
Beyond Automation To
Show DevOps Momentum
Manufacturers improve
efficiency and insight by
expanding the role of
technology
Technology is
becoming part of the
business

10. Page 10
DevTestOps & Continuous Testing is key for DevOps success
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
DevOps
End to End Automation
Overall Cost Reduction – Automation,
Reduce Cost of Quality
Fail – First, Fail - Fast, Fail- Often – Early
Defects Detection
Collaborative Communication utilizing
Dashboard
Continuous Testing
DevTestOps
Equal Importance for Dev, Operations
and Testing
All actions – Development --> Operations
Emphasis on Automation & Analytics
Continuous Integration + Continuous
Deployment (CI/CD)
Dev --> Ops Process Reduced from
Months to Days/Hours
Continuous Feedback & Improvement
Supported
Maturing Processes and Organizations
Focus

11. Page 11
Devops in a nutshell
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
Build Dev
Automated Unit
Testing & SAST
Build Packing &
Automated
Deployments
Continuous Testing
• Functional Automation
• Non Functional Automation
Production
Deployments
Post Production
Monitoring & Feedback
Amplification
Virtual Server(s)
Physical
server(s)
CI
Server
Hosted Tools
Infrastructure
Automation &
operations

12. Page 12
Continuous Testing & Delivery Leveraging QE elements
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
Client Layer Web Server App Server Component 1 Component 2 Component 3
Target System
Target System
Service
Virtualization
Data
Engineering Dashboard
Application Under Test
CI
Platform
UI
Services
App Component
Continuous Monitoring
Health Check
InfrastructureApplication
{Through scripts} {Through metrics configuration}
Functional Automation
Continuous Testing
Non - Functional Automation
Leveraging a collaborative
dashboard to monitor
application release
activities & health
Enabling continuous testing
by overcoming the
dependency factors involved
Centralised data management through
automated TDM tools & other mechanisms
for on demand testing
Ensuring faster feedback for
issues remediation through
continuous infrastructure &
application monitoring in
production & test environments
Adopting Quality Engineering
approach to provide end to end
automated functional and non
– functional test coverage for
each release
Ensuring quality during rapid and frequent build
deployments by leveraging robust CI tools
integrated with test automation frameworks
Third party system
Efficient Tools
Leverage

13. Page 13
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
DevOps Journey
DevOps Journey Automation Maturity
Partial Agile
No Automation
in place
Functional and Non
Functional
Automation
Adopted CI and
CD tools
Implemented Dual Shift
Approach
Build - Deployment –
Release Automation
Matured Agile
Organization
Transformed to TDD
/ ATDD / BDD
DevOps
Matured Collaborative
Agile ( Dev, QA and Ops)
Cost Elements ROI Factors
» Tools for Automation and
Non Functional Tests
» Process Standardization
» On Demand Test Envts.
» End-to-End Automation
and Test Coverage
» Tools and process for CI
and CD
» Infra Maintenance
» DevOps Process
» Org. Change Mgmt.
(People, Process and Systems)
» Cost of Maintenance
» Technology Adoption
» Unified Automation
Framework
» Best suited tools
» Tool optimization and
consolidation
» Quality Check gates
» Automation Stabilization
» Continuous Testing –
Continuous Feedback
Quality Engineering
approach for early testing
(Functional and Non Functional)
» Build Reliability
» Uninterrupted Service
Delivery

14. In Detail
KEY QUALITY
ENGINEERING ELEMENTS
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies

15. Page 15
Unit Testing & SAST
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
JUNIT/NUNIT
Static Code Analysis
Build Server2
2
CI Server
Unit Testing
1
Open Source/Commercial Tools
3
Results Analysis &
Vulnerable
identification
Manual Analysis
for false positive
removal
Final Summary
Code Commit
1
I D E
(Technology Agnostic)
Code Repository
» Unit testing is performed using respective open source tools that
integrated with the IDE used for code development
» CI Server picks the build from the code repository/version controlling
tool once the developer commits the build
» CI Server initiates the static code analysis by triggering a request to
the configured open source / commercial tools
» Input Validation
» Output Encoding
» Secure handling of credentials
» Session Management
» Cryptographic practices
» Error handling / Logging
» Data Protection
» Communication Security
» Memory management
Secure Coding Guidelines & Standards

16. Page 16
Automation,Performance & Security
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
CI PlatformBuild
Version
Controlling Tool
Test Automation Performance Testing
Security Testing (DAST)
Test Automation
Tools
Test Management
ToolsTest Automation
Framework
Performance Testing
Tools
Performance Testing
Scenarios
Monitoring
Tools
Trigger Monitoring High Level
Analysis Report
Trigger Automated
Test Execution
DAST Tools
Monitoring
Tools
Results Analysis &
Vulnerabilities
Identification
Manual Analysis
(False Positives)
Automated
Trigger
Final
Summary
Report

17. Page 17
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
TDD blended with DevOps
Function 1- Unit
Test case
Function 2- Unit
Test case
Function 3 - Unit
Test case
Feature broken
into functions
Add a Developer
test
Run Developer
tests at the unit level
Make code changesRun the
Developer testsPass,
Functionality completeDevelopment Ends
Fail
Fail
Pass
Pass,
Functionality
Incomplete
FinalizedFeature
Unit Testing, Integration
Tests and Performance
Tests in reference to
development under
DevOps practices.
DevOps Specific
input to feature
requirements
Feature Spec
file
Automated Configuration
Management frameworks
(Ansible/Chef/Puppet)
CI/CD framework
(Continuous Integration/
Continuous Delivery)
TDD
DevOps
Continuous
Feedback
DevOps Analytics Platform
Continuous
Deployment

18. Page 18
ServiceVirtualization
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
T
Legacy
ERP
Databases
Mainframe
» Automatically Capture realistic
behavior
» Build VS Image, collection
services requests and Service
responses
» Optimize models as desired
Create Virtual
Services
Playback against Virtual Service
Test
(Manual, Automated)
System Under
Test (SUT)
Virtual Service
SOAP Request
SOAP Response
JMS Publish
Exercise SUT
ServiceRequest
ServiceResponse
Service Response
Service Request
Service Response
Service Request
ServiceResponse
ServiceRequest
DevTest
Deploy SV tool
on cloud
Advantages
✓ Accessible from Multiple Geographies (Cloud & On-premise)
✓ Deploy & Bring Up/down to Optimize Resources & Cloud Spend
✓ Easily Customized as the UI or Business Needs Change
✓ Leverage Docker Containers, enabling Continuous Integration of
Code Changes and Parallel Test Instances
✓ Reduce CapEx (Capital Expenditure) by Hosting in AWS (Amazon
Web Services) Cloud
✓ Dynamically Support Infrastructure & Policy as Code, with Fugue…
✓ Configurations
✓ Configurations
✓ Configurations
✓ Configurations
Target Service
DevTest
Typical DevTest SV Usage Scenario

19. Page 19
Continuous- Integration vs.Delivery vs.Deployment
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
C o n t i n u o u s - I n t e g r a t i o n v s . D e l i v e r y v s . D e p l o y m e n t
Source Control
Commit Changes
Build
Run Build And Unit Tests
Staging
Deploy to test Environment Run Integration
Tests, Load Test, & Other Tests
Production
Deploy to Production Environment
V1.1
Automated Automated
Continuous
Deployment
Continuous
Delivery
Automatic Deploy
Approve Deploy
Continuous
Integration

20. Page 20
Continuous Automation & Delivery Pipeline
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
ORCESTRAED CONTINIOUS INTEGRATION PIPE LINE
Test
SCM Repo Adoption Build Package Deploy Host / Stage
Jenkins
Requirements Document & verify
Container as a SCM service using Rancher/Docker
Environment Environment
Resources Resources
Private Container Service
User Directory Container Registry
Ops ToolsDevOps
Pipeline
Enterprise Micro
service Stack
Reporting &
Monitoring
On demand test bed using
containers
Test Automation
Tools
Web & Mobile
browsers
Advance
Analytics
Dashboards
Tableau
DeliveryPipelineContinuous
Automation
Requirements
Automated
Testing
Test Automation
Scripts
CI Enabled
Execution
Results
Automated Performance
DAST
Regression & Risk Based
Continuous Development
Coding
Compile & Build
Code
Quality
SAST
Unit testing
Continuous Deployment
Deploy Apps
Release
Management
On Demand Infra
Collaborative
Dashboard
Continuous
Monitoring
Operations
Enablement

21. Page 21
Collaborative DevTest Platform & How does it help?
Unsolicited Distribution is Restricted. Copyright © 2017 - 18, Cigniti Technologies
Build OperateIntegrate Deploy
Application Health Release Readiness Quality Metrics Go – No Go Decision Enabler
Release Progression Build Status Test Status Predict Analysis
LiveDashboard
Test Automation
Tools
Test Management
Tools
Performance
Testing Tools
DAST Tools
Monitoring
Tools

22. Q & A
Thank You
Connect at:
chakri@cigniti.com
+1 469 955 0841