A Continuous Testing presentation, presented at DevOpsDays Cape Town 2017.

1. Continuous Testing –
The final frontier of DevOps
DevOpsDays Cape Town 2017
Jaco Greyling, CTO Enterprise DevOps
@jaco_greyling

2. Testing
Experiences
Ambiguous or no
requirements
Testing is the last line of
defense
Lack of test environments
creating delays
Test data? Pfft, I create my
own!
QA is not Agile, always one
Sprint behind
Manual, manual, manual
testing…
2
* TestMatick Software Testing Company

3. Copyright © 2017 CA. All rights reserved.
Eliminating the barriers
between ideas and outcomes?
How good are you at …
3

4. Copyright © 2017 CA. All rights reserved.
Your Software Factory must be an accelerator.
Is yours, or is it an inhibitor?
ProductConcept
EFFICIENT, HIGH-QUALITY,
PRODUCTIVE PIPELINE
MORE CUSTOMER
VALUE
BUSINESS
ALIGNMENT
4

5. Copyright © 2017 CA. All rights reserved.
CREATE AN AGILE BUSINESS
THE
MODERN
SOFTWARE
FACTORY
The capabilities you need,
from mobile to mainframe,
to deliver the experiences
customers want.
Agile DevOps Security
BUILD BETTER APPS, FASTER
MAKE SECURITY A COMPETITIVE ADVANTAGE
MAXIMIZE APPLICATION PERFORMANCEMAXIMIZE APPLICATION PERFORMANCE
5

6. Copyright © 2017 CA. All rights reserved.
Continuous Testing
How do you build better apps, faster?
6

7. Copyright © 2017 CA. All rights reserved.
What is your reality?
* Actual customer example if their testing strategy
Dev Tools
TOOLSET TEST
PYRAMID
UI Tools
Dev Tools
TOOLSET TEST
PYRAMID
API Tools
UI Tools
TESTING GOAL
API Tools
94%
3%
3%
5%
45%
50%
Non-functional: static, performance,
security, etc…
API
Unit
UI
API
Unit
UI
TESTING REALITY
7

8. Copyright © 2017 CA. All rights reserved.
32% 16% 22%
63%
21% 23% 30%
QA is STILL a Bottleneck… Even in DevOps
Shops
http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf
…of delays were occurring at
the Test/QA stage of the
cycle.
“Where are the main hold-ups in the software production process?”
63%A 2017 survey of self-
proclaimed DevOps
practitioners found that
…
8

9. Copyright © 2017 CA. All rights reserved.
Challenges to Achieving Continuous Delivery &
Testing
Ave. cost of downtime per minute
53% percent of visits are abandoned if
a mobile site takes more than three seconds
to load
Requirements
Development
QA / Testing
Operations
Release
70% of all
testing is still
manual4
63% of testers admit they
can’t test across all the different
devices and OS versions5
50% of time
spent looking for
test data6
79% of teams face
restrictions, time limits or
access fees on 3rd party services3
X X X
64% of total defect cost originate in the requirements analysis and design phase1
50% of developers time is spent
fixing and finding defects2
?
80% of teams experience delays in development and
QA due to unavailable dependencies3
?
57% are dissatisfied with the time it takes to deploy new features7
!
!
Greater than 50% production outages from
configuration management
!
= $4,700
1 Hyderabad Business School GITAM University Quality Flaws: Issues and Challenges in Software Development – 2012
2 Study conducted by the Judge Business School at Cambridge University – January 2013
3 Voke Market Snapshot Service Virtualization – January 2015
4 Bloor Report Automated test case generation - September 2014
5 Perfecto Mobile 2014 Benchmark Survey, Why Mobile Apps Fail - 2014
6 Forrester Global Modern Service Delivery Benchmark Online Survey - Q4 2014
7 BlazeMeter “Black Friday” study, 2016
9

10. Copyright © 2017 CA. All rights reserved.
What is Continuous Testing?
IT’S NOT… TEST AUTOMATION
IT’S NOT… CONTINUOUS INTEGRATION
IT’S NOT… UNIT TESTING
IT’S NOT… DONE ONLY IN THE TEST ENVIRONMENT
IT’S NOT… A QA-LED ACTIVITY
1
2
3
4
5
10
First we need to understand what it’s not!

11. The practice of testing across every activity in the SDLC
to uncover and fix unexpected behaviors as soon as they
are injected. 11
What is Continuous Testing?

12. Copyright © 2017 CA. All rights reserved.
Key Elements of Continuous Testing
Automatically generate test
automation scripts from
requirements
1
2 Simulate test
environment
3
Access test data on-
demand & ensure PII
compliance
4
Start multi-layer /
backend “requests &
response” testing
prior to UI
5 Democratize
performance testing
Integrate open source testing
tools
6
7 Ensure comprehensive,
cloud-based API testing
8 Built-in automated
application security testing
9
Promote artifacts when tests
pass by orchestrating &
automating the pipeline
10
Harness application insight
across the SDLC to improve
user experience
12

13. Most Customers Cannot Do
Continuous Testing Today,
But They Aspire To Get There….

14. Copyright © 2017 CA. All rights reserved.
If This is Your Current SDLC
You are waterfall
Dev Test
X WEEKS X WEEKS X WEEKS
Agile Team: Design / Development / Unit Testing Integration & Central QA Release & Operations

15. Copyright © 2017 CA. All rights reserved.
If This is Your Current SDLC
You are water-SCRUM-fall
Sprint 1 Sprint 2 Sprint 3
X WEEKS X WEEKS X WEEKS X WEEKS X WEEKS
Agile Team: Design / Development / Unit Testing Integration & Central QA Release & Operations

16. Copyright © 2017 CA. All rights reserved.
If This is Your Current SDLC
You are not “really” agile
Sprint 1 Sprint 2 Sprint 3 Sprint N
Sprint 2 Sprint N
Sprint 3 Sprint N
User
Stories
Dev and
Manual
Testing
Automated
Regression
Testing
X WEEKS X WEEKS X WEEKS X WEEKS
Sprint 1
Sprint 1 Sprint 2
X WEEKS X WEEKS
Sprint 3
Release

17. Copyright © 2017 CA. All rights reserved.
What Testing at the Speed of Agile Looks Like!
In-Sprint everything to achieve “Continuous Testing”
No Sprint-lag effect
Release in end of Sprint
In-Sprint Test Automation
Sprint 1 Sprint 2 Sprint 3 Sprint N
Sprint 2 Sprint N
Sprint 3 Sprint N
User
Stories
Dev and
Exploratory
Manual
Testing
Test Automated
Scripting &
Execution
X WEEKS X WEEKS X WEEKS X WEEKS
Sprint 1
Sprint 1 Sprint 2
Release
Sprint 3
ReleaseReleaseRelease
HIGHLIGHTS

18. Copyright © 2017 CA. All rights reserved.
Simple View of Continuous Testing
REQUIREMENTS
MODELING
TEST
EXECUTION
MONITORING
Continuous Integration Continuous Testing
Continuous
Delivery
Continuous
Deployment
WhatHow
Flowchart Diagrams
(Model-Driven)
DEVELOPMENT TEST
DESIGN

19. Copyright © 2017 CA. All rights reserved.
Simple View of Continuous Testing
REQUIREMENTS
MODELING
Continuous Integration Continuous Testing
Continuous
Delivery
Continuous
Deployment
WhatHow
Test-Driven Development
DEVELOPMENT
REQUIREMENTS
MODELING
TEST
EXECUTION
MONITORINGTEST
DESIGN

20. Copyright © 2017 CA. All rights reserved.
Simple View of Continuous Testing
REQUIREMENTS
MODELING
Continuous Integration Continuous Testing
Continuous
Delivery
Continuous
Deployment
WhatHow
Test Cases
Test Automation Scripts
Test Data
TEST
DESIGN
DEVELOPMENTREQUIREMENTS
MODELING
TEST
EXECUTION
MONITORING

21. Copyright © 2017 CA. All rights reserved.
Simple View of Continuous Testing
REQUIREMENTS
MODELING
Continuous Integration Continuous Testing
Continuous
Delivery
Continuous
Deployment
WhatHow
Test Environments
Test Automation Engines
Test Frameworks
DEVELOPMENTREQUIREMENTS
MODELING TEST
EXECUTION
MONITORINGTEST
DESIGN

22. Copyright © 2017 CA. All rights reserved.
Simple View of Continuous Testing
REQUIREMENTS
MODELING
Continuous Integration Continuous Testing
Continuous
Delivery
Continuous
Deployment
WhatHow
Analyze
Learn
Update Model
DEVELOPMENTREQUIREMENTS
MODELING
MONITORING
TEST
DESIGN
TEST
EXECUTION

23. Can You
Afford Not
To?
Any Focus
On Speed…
Without Quality
Will Result in
Their Company’s Future
DEPENDS on the QUALITY
of their software
* “Surviving Disruption, Leading Change: Winning in the Application Economy,” 2015
2/3
Of Business Leaders Say

24. Thank You.

25. CTO Enterprise DevOps
jaco.greyling@caafrica.co.za
Jaco Greyling
@jaco_greyling
slideshare.net/CAinc
linkedin.com/in/jaco-greyling/
ca.com/za