Led by Nicole Forsgren, Gene Kim, and Jez Humble, DORA (DevOps Research and Assessment) is recognized as the leading authority on the factors driving DevOps success. Their upcoming “2018 Accelerate State of DevOps” report is the result of five years of research, with over 30,000 data points from thousands of companies, to understand precisely what practices enable teams to deliver better software faster. Join Nicole Forsgren, PhD, CEO and Chief Scientist at DORA and Wolfgang Platz, Founder and Chief Strategy Officer at Tricentis, to explore what this year’s report reveals about the role of Continuous Testing in DevOps success. The focus will be on helping IT leaders understand: Why and how high-performing teams make Continuous Testing a linchpin of their DevOps process What specific strategies and practices enable high-performing teams to deliver faster and dedicate more resources to innovation Which quality-related technology and process investments have the greatest impact on DevOps success

1. © 2018 by
© 2018 by .
Wolfgang Platz, Founder & Chief Strategy Officer
The journey to Continuous Testing

2. © 2018 by
employees
600+
locations
Austria
Germany
Switzerland
United Kingdom
The Netherlands
Poland
USA
Australia
Singapore
India
founded
2008
customers Forbes Top 10 Global 2000 Companies
6
4
5
4
4
4
investment
computer services
major banks
Australian banks
life insurance
diversified insurance
support
awards
Leader
Gartner MQ
Leader
Forrester Wave
ASP Award 2016
& 2017

3. © 2018 by

4. © 2018 by
PLAN
Agile Dev Operationsmeets
Dev Ops
DevOps is all about Continuous Testing!
http://devops.com/2015/02/02/devops-continuous-testing/ 4

5. © 2018 by
Continuous testing is change
It’s way more than just UI automation

6. © 2018 by
Focus Area
Exploratory Testing
!
Explore
!
Risk Based Prioritization
Test Case Design
CT Integration (CI, CD)
UI Automation: Script-based
UI Automation: Model-based
API Testing
Active Test Data Management
Test Driven Service Virtualization
!
Integrate
!
Optimize
!
Automate
Test Management
Manual Testing
M1 M2 M3 M4 M5CTMM
Continuous
Testing
Maturity
Model

7. © 2018 by 7
DORA
CTMM
+
Provides comprehensive DevOps assessment based on a thorough
data analysis, best practices and benchmarks
Covers entire DevOps cycle
Is primarily a diagnosis instrument
Provides comprehensive Continuous Testing assessment based on a
best practices and benchmarks from Tricentis’ customer base
Specific focus on Continuous Testing
Is both a diagnosis and a therapy instrument

8. © 2018 by
Customer Example: Core Banking Solution
8

9. © 2018 by
> 80%
Manual Testing
< 20%
Automated Testing
UI
API
Today
The reliance on manual testing is the
#1 technical challenge in app development1.
1 … Source: World Quality Report 2015 - 2016
35%
IT Budget Spend on QA & Testing1
$35B
Annual Spend on Testing Services
Legacy platforms require weeks to execute testing.
9

10. © 2018 by
Focus Area
Exploratory Testing
!
Explore
!
Risk Based Prioritization
Test Case Design
CT Integration (CI, CD)
UI Automation: Script-based
UI Automation: Model-based
API Testing
Active Test Data Management
Test Driven Service Virtualization
!
Integrate
!
Optimize
!
Automate
Test Management
Manual Testing
M1 M2 M3 M4 M5CTMM
Continuous
Testing
Maturity
Model

11. © 2018 by
Focus Area
Exploratory Testing
!
Explore
!
Risk Based Prioritization
Test Case Design
CT Integration (CI, CD)
UI Automation: Script-based
UI Automation: Model-based
API Testing
Active Test Data Management
Test Driven Service Virtualization
!
Integrate
!
Optimize
!
Automate
Test Management
Manual Testing
M1 M2 M3 M4 M5
Risk weighted functional structure
Projected most relevant 600 test cases
onto it
UI test automation
most relevant 600 test cases

12. © 2018 by
Today’s reporting is based on numbers of test cases.
What does it tell us?
Version 7.5 42,278 10,086 910
Is 53,274 the right number of test cases?
Can I go productive?
12

13. © 2018 by 13
Front wingGear box
Tires
Telemetry Software
Steering Wheel
Cassis & MonocoqueEngineRear wing
BrakesExhaust

14. © 2018 by 14

15. © 2018 by
BusinessRiskcoverage[%]
Test Cases
(sorted by contribution)
t, $
0% 20% 40% 60% 80% 100%
20%
40%
60%
80%
100%

16. © 2018 by
We need to introduce a new currency in testing.
66% 9% 15% 10%Core Bank
Risk Coverage [%]
Risk Coverage
16

17. © 2018 by
Continuous Testing requires Automation
17
80%
Manual Testing
20%
Automated Testing
UI
API
Today Future
Exploratory Tests
Automated UI Tests
API Tests
Orchestrated Service Virtualization
+85%
- key enabler for high automation rates -

18. © 2018 by
BusinessRiskcoverage[%]
Test Cases
(sorted by contribution)
t, $
0% 20% 40% 60% 80% 100%
20%
40%
60%
80%
100%
Quickly obtain > 40% business risk coverage through automation
(aka Heartbeat 1) - Rapid Automation Package
M2
M2

19. © 2018 by
If you always do what you’ve always done, you
will always get what you’ve always got.
19
Albert Einstein, Relativity Theory
Henry Ford, Ford T Automobile
Mark Twain, Author
Anthony Robbins, Author & NLP Trainer

20. © 2018 by
Model-Based Browser
Assert.AreEqual(“331,00",
driver.FindElement(By.XPath("//table[@id='PriceList']/t
body/tr[10]/td[2]/b"))
.GetAttribute("value"));
Browser
Script-Based
Code
Synchronization Behavior
Code
Recovery Handling
Code
Data Driven Framework
Code
Keyword Driven Framework
1
2
3
4
20

21. © 2018 by
100 pax
3 weeks
ManualTesters
Test automation learning curves
21
100 pax
3 months
“We will train them 3 months on
Selenium based on our framework!”
Head of QA Services
“Go train them on Tricentis Tosca –
you got 3 weeks!”
15%
85%
With Tricentis Tosca, 85% of manual testers become productive
automation specialists in 3 weeks.
ManualTesters

22. © 2018 by
Focus Area
Exploratory Testing
!
Explore
!
Risk Based Prioritization
Test Case Design
CT Integration (CI, CD)
UI Automation: Script-based
UI Automation: Model-based
API Testing
Active Test Data Management
Test Driven Service Virtualization
!
Integrate
!
Optimize
!
Automate
Test Management
Manual Testing
M1 M3 M4 M5M2
Introduced test case design
… according to use-case relevance

23. © 2018 by
Test Cases
(sorted by contribution)
20% 40% 80% 100%60%
t, $
BusinessRiskcoverage[%]
0%
20%
40%
60%
80%
100%
M2
Risk coverage depends on application of test case design +
efficiency of test automation access (UI or API)
Push business risk coverage + maximize automation
Optimize TestingM3 M5…

24. © 2018 by
Focus Area
Exploratory Testing
!
Explore
!
Risk Based Prioritization
Test Case Design
CT Integration (CI, CD)
UI Automation: Script-based
UI Automation: Model-based
API Testing
Active Test Data Management
Test Driven Service Virtualization
!
Integrate
!
Optimize
!
Automate
Test Management
Manual Testing
M1 M3 M4 M5M2
Introduced test case design
… according to use-case relevance
Expand UI test automation (+ 500)
+ 900 test cases
Extend coverage through API testing

25. © 2018 by
Integration Tests
Unit Tests
UAT
System
Integration Tests
… Automated Test
The Agile test pyramid ←

26. © 2018 by
Set-up
Maintenance
Future Test Automation will focus on API testing
Task
Runtime
Timing
4x
6x
100+ x
Factor
API
testing
25%
16%
<1%
progressive
UI test
automation
100%
100%
100%
regressive

27. © 2018 by
Integration Tests
Unit Tests
UAT
System
Integration Tests
… API test automation
The Agile test pyramid ←
… UI Test automation

28. © 2018 by
Focus Area
Exploratory Testing
!
Explore
!
Risk Based Prioritization
Test Case Design
CT Integration (CI, CD)
UI Automation: Script-based
UI Automation: Model-based
API Testing
Active Test Data Management
Test Driven Service Virtualization
!
Integrate
!
Optimize
!
Automate
Test Management
Manual Testing
M1 M3 M4 M5M2
Introduced test case design
… according to use-case relevance
Expand UI test automation (+ 500)
+ 900 test cases
Extend coverage through API testing
Implement synthetic test data gen

29. © 2018 by
The challenge…
§ Complex (test) processes require test data in certain, well defined states.
§ Production data may not contain data for new functionality yet.
§ Production data may not be used due to privacy issues.
§ Test data may expire (e.g.: persons/contracts grow old).
§ Test data is altered through the test itself (consumption of test data).

30. © 2018 by
Requirements
Epics/
UserStories
Test Data
Design
Test Case
Data
Test Data
Basis
Manual Test Case
Automated Test Case
Test Data Management

31. © 2018 by
Focus Area
Exploratory Testing
!
Explore
!
Risk Based Prioritization
Test Case Design
CT Integration (CI, CD)
UI Automation: Script-based
UI Automation: Model-based
API Testing
Active Test Data Management
Test Driven Service Virtualization
!
Integrate
!
Optimize
!
Automate
Test Management
Manual Testing
M1 M3 M4 M5M2
Introduced test case design
… according to use-case relevance
Drive test execution through CI
Expand UI test automation (+ 500)
+ 900 test cases
Extend coverage through API testing
Implement synthetic test data gen

32. © 2018 by
DevOps Tool-Chain
Epics
Requirements
Code/ Config Versioning Analyze & Build
Test (Explore, Automate)
Optimize, Automate, (incl. Virtualize)
Staging
&
Deployment
Monitoring
XCode
Container
VM-config
PLAN CODE BUILD CONTINUOUS TESTING DEPLOY MONITOR

33. © 2018 by
Focus Area
Exploratory Testing
!
Explore
!
Risk Based Prioritization
Test Case Design
CT Integration (CI, CD)
UI Automation: Script-based
UI Automation: Model-based
API Testing
Active Test Data Management
Test Driven Service Virtualization
!
Integrate
!
Optimize
!
Automate
Test Management
Manual Testing
M1 M4 M5M3M2
Further expansion of test
automation (+ 6,000); mainly API
Implement Service Virtualization

34. © 2018 by
App to be
tested
Tester
performs
user-input
Tester verifies
results
Request
Response
System B System C
Request
Response
Step 1
Step 2
Step 3
Step 4
Step 5
Example: A large portion of the app’s business logic can only be tested with systems B, C available.
X
X
X

35. © 2018 by
… requires discipline
… requires Test Driven
Service Virtualization
Integration Tests
Unit Tests
UAT
System
Integration Tests
←The Agile test pyramid

36. © 2018 by
Focus Area
Exploratory Testing
!
Explore
!
Risk Based Prioritization
Test Case Design
CT Integration (CI, CD)
UI Automation: Script-based
UI Automation: Model-based
API Testing
Active Test Data Management
Test Driven Service Virtualization
!
Integrate
!
Optimize
!
Automate
Test Management
Manual Testing
M1 M5M3 M4M2
DevOps Contract:
Each use-case can be
administered via API
Introduced exploratory
testing

37. © 2018 by
1 2 3 4 5 … … … … … … …
Risks

38. © 2018 by
Risks
Exploratory Branching
New testing ideas continually occur
during exploratory testing.
New
Testing
Idea Exploratory Testing
Perfect counterpart to specification-based
automated & manual testing.
Test Cases vs. Scenarios
Pre-specified inputs vs. hypothetical situation

39. © 2018 by
Focus Area
Exploratory Testing
!
Explore
!
Risk Based Prioritization
Test Case Design
CT Integration (CI, CD)
UI Automation: Script-based
UI Automation: Model-based
API Testing
Active Test Data Management
Test Driven Service Virtualization
!
Integrate
!
Optimize
!
Automate
Test Management
Manual Testing
M1 M3 M4 M5M2

40. © 2018 by
Testing Lifecycle Continuous Integration Shift Left Testing Continuous Delivery
Continuous Testing
Maturity Model
M1 - M3 M3 - M4 M4 - M5
Roles & Responsibilities In-Team Collaboration Across-Team Collaboration Across-Stage Collaboration
2x 5x3x
Initiation Collaboration Optimization
Prevent testing to become the
bottleneck for Digital
Transformation
Focus on product, people and
process
Scaling can increase your
time-to-market up to 5x
The first phase is
achievable within 3 - 6 months
Scaling Continuous Testing
3 to 6 months up to 12 months up to 18 months

41. © 2018 by
Contact us
Wolfgang Platz
w.platz@tricentis.com

42. © 2018 by
Questions
Dr. Nicole Forsegren, Wolfgang Platz
?