How Artificial Intelligence (AI) is changing software quality
Hybrid test automation framework to test identified and unidentified UI properties
Demonstration of a use case with AI in UI test automation for any skill level
3. How is Artificial Intelligence (AI) transforming businesses?
How can AI impact Software Quality?
What areas is AI impacting software testing the most?
What is SmartBear doing with AI to help testers?
Key Questions to Consider
Proprietary & Confidential
3
4. How many people are using AI in
their day-to-day lives?
Proprietary & Confidential
4
5. AI-Driven technology is reshaping our day-to-day lives,
changing the way we see the world.
Proprietary & Confidential
5
10. Imagine using AI to explore 1000 possible application
designs from a list of requirements.
Proprietary & Confidential
10
Autodesk Dreamcatcher AI:
The Next Generation
of CAD
11. M87 – Black Hole
Proprietary & Confidential
11
Continuous High-resolution Image
Reconstruction using Patch priors – CHIRP
• Bayesian Algorithm
• Compensates for Low Signal/Noise Ratio in
Very long baseline interferometry
Katherine L. Bouman, Michael D. Johnson, Daniel Zoran, Vincent L. Fish, Sheperd S. Doeleman,
William T. Freeman; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
2016, pp. 913-922
12. What is AI?
Neural networks Machine learning Deep learning
or the networks of hardware and
software that approximate the
web of neurons in the human
brain
which is a technique using
algorithms to teach machines to
learn
which helps machines learn to
go deeper into data to
recognize patterns
Proprietary & Confidential
12
AI has come to represent the broad category of methodologies that teach a computer to perform tasks as
an “intelligent” person would
13. We Want Faster Release Cycles
Proprietary & Confidential
13
Automation Capability High
Agile
Testing Business Goals
Speed
Quality
Cost
Culture
Innovation
SDLCTimeTakenWeeks
Hours
Low
Test
Automation
Continuous
Testing
Autonomous
Testing
14. Current UI testing challenges
Proprietary & Confidential
14
• Unreliable Object Recognition
• Test Framework Design
• Inadequate Prioritization
• Test refactoring
• Test scalability
• Inadequate Documentation
TEST DESIGN
TEST MAINTENANCE AND
EXECUTION
15. How AI could address these challenges
Proprietary & Confidential
15
INTELLIGENT TEST DESIGN
INTELLIGENT TEST
MAINTENANCE AND EXECUTION
Advanced
Object
Recognition
Framework
Generation
Responsive
Web
Design
Risk
Profiling
Predictive
Self-Healing
Intelligent
Bug Hunting
Netflix
Scryer and
Simian Army
Process
Automation
16. What does Intelligent Test Design Look Like?
Proprietary & Confidential
16
Risk Profiling
Prioritize tests based on business risk by environment, configuration, and
different aspects of your application
Object Recognition
Automatically recognize new objects and updates to add them to the DOM
and structure without manual effort
Framework Generation
Automatically scan your application to recommend a test framework
17. Each application update and a new test case can act as
inputs to help maintain an application.
Proprietary & Confidential
17
Example Outputs:
• Recommendation
systems
• Reinforcement
learning
• Device selection
Example Inputs:
• Application controls
• Properties of
controls
• New test cases
• New test results
18. What does Intelligent Test Execution and Maintenance
Look Like?
Proprietary & Confidential
18
Predictive Self-healing
Dynamically update your test suite when
your application changes or evolves
Netflix Scryer and Simian Army
Achieving resilience by applying predictive
auto-scaling and continuous fitness
functions.
Intelligent Bug Hunting
Discover bugs in the application through
AI-powered exploratory testing
Process Automation
Automate business workflows for end-
to-end testing
19. Proprietary & Confidential
19
Without AI: a lot of time spent on…
Failures
Refactor Tests
Object Recognition
Defects Found
Repetition
Stability
Bottle Necks
Decision Time
Resource Allocation
20. Proprietary & Confidential
20
With AI: gives us more time to focus on…
Creativity
Scalability
Customer Satisfaction
Test Coverage
Business Impact
Results
21. Property-based and visual recognition together achieves
the maximum level of test coverage
Proprietary & Confidential
21
Visual Testing
• Dynamic properties
• Faster maintenance
• Scalability
Hybrid Recognition
• Max. test coverage
• Faster test design and
maintenance
Manual Testing
• Exploratory benefits
• High maintenance
• Low accuracy
• Low test coverage
Property-Based
• Accuracy / Stability
• Distinguish similar
objects
EaseofMaintenance
Level of Accuracy
High
Low
Low High
22. • Slack Chatbot
• Visual Objects Recognition by Shapes and Colors
• Intelligent Code Completion
• Intellectual Modern Documentation
• Controlling the WebUI by using gestures
Recent SmartBear Hackathon
Proprietary & Confidential
22
23. Design Commit Test Deploy Monitor
Manage
UI
API
Collaborate on user stories,
tests, and code
Commit and push new code and
tests to kick off CI
Create, manage, and execute
automated tests
Deploy to production once
tests and builds pass
Monitor the performance
of your APIs and web apps
Analyze and improve all of your testing
Reduce defects and improve trust with code review
SwaggerHub
Design, model, & share
API definitions
HipTest
Design features & tests
using BDD
Zephyr Native Jira & enterprise test management
Review
Collaborator Code, document, & artifact review
CrossBrowserTesting
Run Selenium & Appium tests in the cloud
LoadNinja
Automated UI performance testing
TestComplete
Automated UI functional testing
ServiceV Pro
Virtualize APIs, UIs, and web services
LoadUI Pro
Automated API performance testing
SoapUI Pro
Automated API functional & security testing
AlertSite
Global, synthetic API monitoring
Mention TestIO here as companies trying to address this complete
Applitools
Mabl
Lino to expand risk profiling – scientist and engineer being on the same project
Funny for input output joke?
Intelligent system is based on inputs and outputs
Link to expand on what other inputs and outputs are involved in an AI model
Each change to an application and a new test case can act as inputs to help update object definitions and descriptions.
And each can change quickly as software developers enhance the application, adding new objects with new functionality or updating object definitions and descriptions while they are fixing bugs.
Mention TestIO here as companies trying to address this complete
Applitools
Mabl
Lino to expand risk profiling – scientist and engineer being on the same project
Certainly, many companies have used AI to automate processes, but those that deploy it mainly to displace manual work will see only short-term productivity gains. In our research involving 1,500 companies, we found that firms achieve the most significant performance improvements when humans and machines work together. The same applies to today’s object recognition.
Through hybrid recognition, humans and AI actively enhance each other’s complementary strengths: stability, and accuracy, of the former, and the speed and scalability of the latter to perfect recognition reliability.
What comes simple to property-based recognition (identifying subtle differences between two images, for example) can be tricky for machines, and what’s straightforward for machines (analyzing gigabytes of data) remains virtually impossible for humans. Testing requires both kinds of capabilities.