Full webinar recording -- In an era when testing is no longer limited to end-of-line waterfall-type process -- but is an integral and substantial part of the end-to-end development process -- it is ever more pressing to look under the hood of the tools, technologies, processes, and best practices utilized by top-notch testing teams that release world-class software. In this webinar, we took an in-depth look at the technical aspects required to successfully scale continuous testing -- and discussed the tools and technologies used by leading engineering teams, including their approach to managing functional, performance and visual testing at scale. We also took a hands-on approach: via a live demo, we showed how all of these come together to create a complete end-to-end quality pipeline that supports continuous testing at scale.

1. Building a Complete Pipeline
The Essential Components of
Continuous Testing

2. Today’s Speakers
2
Eran Barlev
Sr. Software Engineer
Guy Arieli
CTO

3. Agenda
● Continuous Testing Overview
● The Value of the Pipeline
● The Three Essential Components
○ Functional Testing
○ Performance Testing
○ Visual Testing
● Live Demo
● Recap and Q&A

4. Continuous Testing and the Value of CI/CD Pipeline
• Quality + Velocity = Continuous Testing
• Fast (~immediate) feedback to developers
• Solid pipeline
• Quality efforts aligned with business expectations
• Prioritized list of actionable tasks
• Outcome: Huge change in productivity
Quality is the biggest challenge in most technology organisations
Add ROI pillars

5. Development
Engineer
Fast Feedback with Continuous Testing
Commit step
definitions +
feature files
Trigger test
execution
Load app
to cloud
Tester + Test
Engineer
Pull request
to the main
branch
Performance
validation
Continuous Integration
reporttest
build Pull request
Approved?
no yes
Refuse pull request
Debug /
update test suite
1
Pull app
source2 Build
app3
Pull test
source4
Build test
project5
6 7 8
9
10
Git Hook
Merge to
master branch
Visual
validation

6. PERFORMANCE
UX changes against
a baseline
The Three Essential Components
FUNCTIONAL
Unit / API / UI
VISUAL
Visual changes against
a baseline

7. FUNCTIONAL TESTING

8. Functional Testing (Unit, API, UI)
●The most widespread type of functional
testing
○ Automated with Selenium, Appium, Cypress, …
●Tests the end-to-end functionality of a
system through its UI
●A test is a sequence of steps in which you
○ Simulate user inputs: navigation, clicks, swipes,
keystrokes, etc.
○ Validate the system output on the page using
assertions

9. PERFORMANCE TESTING

10. 10
Performance - User Perspective
● Duration: Long load time as 3 seconds can
drive away half your customers
● Resources hogging: An application that hogs
resources, CPU, battery, data traffic and
memory is 10x more likely to be removed by
customers
App Store: While functional testing can get you
to 3 stars, you need performance testing to
get to 5 stars
Performance is critical to customer user experience

11. Three Factors that Impact Performance
11
• Technology
• Mobility
• Coverage
• Congestion
Client
• Code
• Environment
- Server / device resources
- OS / Browser version
- Other applications
• Code
• Environment
- Architecture
- Resources
- Networking
Back-end Performance Testing
(Load, Stress, Spike, endurance..)
Jmeter, Loadrunner, Network Virtualization
(network types, mobility scenarios…)
Device Monitors
Back-end Network

12. Three Factors that Impact Performance
12
• Technology
• Mobility
• Coverage
• Congestion
Client
• Code
• Environment
- Server / device resources
- OS / Browser version
- Other applications
• Code
• Environment
- Architecture
- Resources
- Networking
Back-end Performance Testing
(Load, Stress, Spike, endurance..)
Jmeter, Loadrunner,
Back-end Network
User Experience Performance
Testing

13. Focusing on Transactions
Transaction
:An event in the UI that sends a request to the server, leading to a server
response and another event in the UI
A change of 2 seconds or 500 ms can be meaningful for a
transaction and for customer experience, but not in a regular
test duration that could take minutes
Transaction
Transaction only – no setup
A single step - action and outcome
only
Duration – seconds, ms
Enables granular measurement
5
UI Event Server

14. Gather Transaction Data
as Part of Functional Testing
Leverage existing efforts and resources of functional testing
for performance transaction testing
Transactions
Information
Database
Transactions
Analytics
Analysis and
Debug
Manual
Tests
Automated
Functional Tests
Transaction Info
(Start/Stop)
7
REST API

15. UX Performance Testing - Example of a Report
15

16. 1 s 2 s 3 s 4 s 5 s
10% 10% 10% 10% 100%
10% 50% 90% 97% 100%
What is Speed Index?

17. Network Traffic Analysis
Evaluate the amount of traffic used by the application and its speed
• HAR file and network transaction waterfall view

18. Transaction Performance - Analytics
18

20. VISUAL TESTING

21. Why are we
talking about
this?
Catching visual bugs
is not perceived as a
priority.
Nobody’s boss is
asking for that.

22. What is the #1
priority?
Everyone wants to
increase functional test
automation coverage
to safely speed up
releases

23. Types of Functional Testing

25. ●The most widespread type of functional testing
○ Automated with Selenium, Appium, Cypress, …
●Tests the end-to-end functionality of a system through
its UI
●A test is a sequence of steps in which you
○ Simulate user inputs: navigation, clicks, swipes, keystrokes, etc.
○ Validate the system output on the page using assertions
UI Testing

26. Test Required Login Details

27. A Cypress UI Test

28. What If the CSS Changes?

29. What If the CSS Changes?

30. …

31. How many
functional
assertions are
needed here?

32. How many
functional
assertions are
needed here?

33. When the UI Changes...

34. The Pain
● Most organizations struggle to succeed with UI testing
○ Every assertion increases the maintenance overhead - Is it really worth automating?
○ Coding skills required to create and maintain tests
○ Checks system functionality via the UI – Not the UI!
○ Can only catch expected bugs
○ Can’t assert on non-dom elements: image, canvas, css graphics, WebGL, etc.
○ Manual testing is still needed!

35. The Solution…
UI Testing with Visual
AI!

36. Adding a Visual Checkpoint

37. ●A single assertion for complete
functional coverage
●Does not break when the UI
changes
●No coding skills required to
maintain baselines
●Root cause analysis
●Bonus: validates the UI
Visual Assertions Advantages

38. Empirical evidence based on 278 completions (100 winners)
● Time to implement test suite: 6x faster (7.2x for winners)
● Total lines of code: 49% less code (72% for winners)
● Number of locators: 7.8x more stable (8.9x for winners)
● Bugs found: 100% vs 70%

39. UI vs. Visual Testing

40. DEMO

41. Thank You