The document discusses optimizing continuous testing in DevOps pipelines. It begins by defining continuous testing as executing automated tests as part of the software delivery pipeline to obtain feedback on business risks. It then covers challenges to continuous testing like lack of test stability. The key to success is having a unified solution that addresses the pillars of continuous testing: the test lab, execution, creation, and analysis. Personas like business testers, SDETs, and developers must be supported. Metrics should measure testing and pipeline activities. Codeless testing is introduced as complementing code-based testing by allowing automated tests to be created without writing code.

1. 1 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Agile vs. DevOps for Continuous Testing:
How to Optimize Your Pipeline

2. 2 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
• Chief Evangelist and author at Perfecto
• Blogger and speaker
• 19+ years in development & testing
• Author of
“The Digital Quality Handbook”
and“Continuous Testing for
DevOps Professionals”
• Twitter: @ek121268
• Blog: continuoustesting.blog
• Website: perfecto.io
Today’s Speaker:
Eran Kinsbruner

3. 3 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Today’s Agenda
1
2
3
What Is Continuous Testing and Why Is It Hard to Mature?
What Are the Key Elements for Continuous Testing in DevOps?
Trends and Personas Involved in CT and How to Balance R&R
4 Measuring CT and DevOps — Metrics
5 Codeless Overview

4. 4 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
The Agile Testing Manifesto
“Stemming from an Agile approach to software development,
DevOps expands on the cross-functional approach of building
and shipping applications in a faster and more iterative
manner. In adopting DevOps, one is making a decision to
improve the flow and value delivery of their application by
encouraging a more collaborative environment at all stages of
the development cycle.” (GitLab)

5. 5 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
What Is Continuous Testing?
Continuous testing is the process of executing automated
tests as a part of the software delivery pipeline in order to
obtain feedback on business risks associated with a
software release.

6. 6 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Top 5 Automation Challenges: Continuous Testing Blockers
Tighter release schedule
leaves less time to
automate and test
(designing for testability?).
(Lack of) test
automation stability
and reliability block
continuous testing.
Test execution
management isn’t
intelligent/optimized
enough to guide teams
through the DevOps
pipeline activities.
Evolving and maintaining
test sets and maximizing
productivity isn’t an
ongoing practice.
Overall test automation
tool stack isn’t in sync
with organizational
capabilities.
T E S T I N G I S K E Y T O R E L E A S E F A S T E R

7. 7 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
The Path to Continuous Testing
1 2 3 4 5
Stable automation Daily cycle Increase coverage Reach 95% Continuous testing
W H A T Y O U ’ L L G E T
W H A T Y O U ’ L L N E E D
• 99.9% availability lab
• Evidence collection
• Process integration
• CI
• Defect tracking
• Execution control
• Dashboarding
• Create scripts
• Maintain scripts
• Understand what’s
wrong
• Skillset matched
tool
• False negative
detection in reports
• Run scripts daily
• Get results in
minutes
• Valuable coverage
• Meaningful daily
feedback
• Advance validations
• Visual automation
• Basic orchestration
• Up-to-date lab
• Scaled lab
• Analysis grouping
• Automate all that is
possible
• Advanced
orchestration
• Elasticity
• Sharding
• Environment
control
• Scaled reporting
• Nightly 
Continuously
• Cloud execution
• Very high scale
D E V O P S - F R I E N D L Y Z O N E

8. 8 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
The Key Pillars of Continuous Testing in DevOps
ANALYSISCREATION EXECUTION LAB
Unified solution serves DevOps with greater effectiveness.
Lab for continuous testing
• Backbone for all testing
activities
• Hosting of devices,
management, & logistics
• Availability & scaling
• Supporting functions
Test execution within the
pipeline
• Scale
• Orchestration
• Frameworks
Test creation and
maintenance
• Authoring tool
• Validations
Test analysis
• Fast feedback
• Root cause analysis

9. 9 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Key Benefits of Mature DevOps
State of DevOps Report, 2018 — DORA

10. 10 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Key Personas in Continuous Testing

11. 11 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Continuous Testing: The Three Personas
B U S I N E S S T E S T E R
S D E T ( S O F T W A R E D E V E L O P E R I N T E S T I N G )
S O F T W A R E D E V E L O P E R
Performs exploratory and structured manual testing.
Creates functional and nonfunctional code-based test
automation scripts (via Selenium, Appium, etc.).
Creates unit and build acceptance tests.

12. 12 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Business Tester:
Manual? Exploratory?
BDD? Codeless?
Persona-Based Challenges in Continuous Testing
P L A N C O D E B U I L D T E S T D E P L O Y R U N
Test Platform
Test Framework
Trigger Feedback
Time
90% Emulator/Local Device
10% Cloud Device
Unit
API
Espresso / XCUITest
Exploratory
Pre-commit
Manually – On-demand
Few minutes
70% Emulator
30% Cloud Devices
API
Espresso / XCUITest
Commit / Few # day
Automatic
< 5-10 Min
100% Cloud Devices
Espresso/XCUITest
Appium
Nightly
Automatic
< 6 Hours
100% Real Devices (S)
Espresso/XCUITest
24/7
Automatic
< 5 Min
SDET
Functional/NF
Developer
Unit/BAT
?
D E V O P S C O N T I N U O U S F E E D B A C K U N O P T I M I Z E D M O D E LU N O P T I M I Z E D M O D E L

13. 13 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
I N T E R A C T I V E
T E S T S
UI/UX manual tests
Balancing Test Creation for the Three Different
Personas With the Right Tools
O P T I M I Z E D M O D E L
Developers & SDETs
(Code-Based)
Business Testers
Ownership (Codeless)
Business Testers
Ownership

14. 14 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Measuring Continuous Testing Success in
DevOps & Recommended Practices

15. 15 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Tighter Release Schedule Leaves Less Time to Automate and Test
• Consider ATDD/BDD/Codeless*
software development practices.
• Build a consistent and unified CI
process between Dev-QA-OPS and
eliminate “shadow CI.”
• Consider adjusting your test pyramid
and focus on the highly valuable and
fastest test suites.
• Leverage faster test frameworks (XCUITest,
Espresso, headless — Puppeteer, etc.).
*Codeless — Later in the session

16. 16 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
(Lack of) Test Automation Stability
and Reliability Block Continuous Testing
• Test authoring based on industry standards (POM,
automate what’s right).
• Continuously maintain your test as you
maintain code (SCM).
• Consider breaking your tests into sub test
suites based on context.
• Gain control over your test suites through measurements
and agreed upon KPIs.
• Stabilize your lab and testing environment.
• Make sure your target platforms are in “ready state” mode.

17. 17 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
• Start exploring innovative machine learning
tools that can optimize your test code.
• Refactor test code and include tagging,
analytics hooks, and other post-execution
debugging capabilities.
• Periodically test code throughout the pipeline.
Test Execution Management Isn’t
Intelligent and Optimized Enough

18. 18 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Test Authoring – Treat Test Code as Production Code
Test automation code is code — maintaining, versioning,
reviewing, and leveraging best practices are key for
stable automation, stable CI, and a stable pipeline.
• Pairing/coaching.
• Use the right object identification strategy.
• Use the right test framework.
• Measure test efficiency within the CI.
• Risk-based approach to test automation.
• Continuous test data analysis and improvement.
S O M E E X A M P L E S
B E S T P R A C T I C E

19. 19 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Some Recommended CT Measurements (KPIs)
• How fast are testing activities moving, and
what is slowing down these activities?
• Test flakiness.
• Test duration.
• Percentage of automated vs. manual tests.
• Application quality measurements
• Number of escaped defects and in which areas.
• MTTD — mean time to detection of defect.
• Build quality.
• Pipeline efficiency measurements
• Number of user stories implemented per iteration.
• Test automation as part of DoD across iterations.
• Broken builds with categories.
• CI length trending.
• Lab availability and utilization.
• Quality cost measurements
• Operational costs, lab availability issues.
• Cost of hardware/software.
• Costs of defects by severity and stage

20. 20 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
What Does Good Look Like?
State of DevOps Report, 2018 — DORA

21. 21 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Introduction to Codeless Testing

22. 22 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Introduction to Modern Codeless Testing
R E C O R D
& P L AY B A C K
S E L F H E A L I N G
( M L / A I )
S TA N D A R D
F U N C T I O N A L I T I E S
C O N N E C T T O
C I / C L O U D
C O M P L E M E N T S
C O D E - B A S E D T E S T I N G
Codeless testing –
the process of creating
automated tests
without writing a single
piece of code.

23. 23 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Code-Based AutomationCodeless Automation

24. 24 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
5 Recommended Practices for
Embedding Codeless Testing in CT
1
2
3
Identify manual and/or flaky test steps.
Mark these as “codeless” to disconnect them from
the code-based automation suite (avoid duplications).
Record the tests in a “SMART” manner:
• Flow-based and exploratory recording.
• Reusable components.
• Robust object locators.
• Tagging for reporting and tracking.
4 Connect codeless to CI and pipeline for continuous testing.
5 Identify maintenance and examination points for both suites.
B E S T P R A C T I C E

25. 25 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Introducing Perfecto Codeless

26. 26 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
A.I. Object Locator
Powerful Language Simplified & Smart Validations
Just Selenium
• No installations
• No modifications
• No plug-ins
• No extensions
• All platforms (Win, Mac, Linux)
• All browsers
• All versions
• Always updated, always ready
Multi-attribute
uniqueness matching
Over-time neural
network adaption
Framework &
app awarenessRun-time self-healing
P R O J E C T - L E V E L S H A R E D A D A P T I V E R E P O S I T O R Y
• Full browser control
• Object type awareness
• Data driven
• Functions
• Rich text validations
• Appearance & style validations
• Data validations
• Time validations
• Variables & data sets
• Branching
• JavaScript extensions
No installation,
no setup
Record on any
browser
Contextual point
& click recording
Object reusing
Interactive
debugger
T E S T S U I T E S
A S S E T S M A N A G E M E N T
P L AT F O R M S E T S
Integrated CI-connectors
S C H E D U L I N G
P R O J E C T M A N A G E M E N T
Perfecto Codeless
Web-Based Interactive Smart Recorder / Debugger Test Management

27. 27 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Perfecto Codeless Testing Benefits
Recorders
• Fully featured capabilities for additional
complex flows.
• Robust AI-driven bindings take care of
97.4% of app changes.
Scripting
• No coding background necessary
to create automation scripts.
• Reduce maintenance cost of
automation.
Setup
• No set up required.
• Easy to learn.
Process Fitness
• Efficiently reuse automation
scripts without the need to code.
• Accelerates the testing process
and provides developers and
QA fast feedback.
P E R F E C T O
C O D E L E S S
A U T O M AT I O N

28. 28 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
PERFECTO CODELESS SELENIUM DEMO

29. 29 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io

30. 30 | Agile vs. DevOps for Continuous Testing: How to Optimize Your Pipeline perfecto.io
Thank You