In agile software development world, we are dealing with many test tasks such as user story testing, exploratory testing, check-list based testing, regression testing, performance testing, security testing in each sprint. Besides these testing activities, one of the test types which is considerably getting crucial is visual regression testing.
Visual regression testing focuses on to check visual contents and animations, page layout, and responsive design of a website/app. Because of the limits of human vision, human-based visual regression testing is generally error-prone and cumbersome. Hence, automation is inevitable. It enables us to run the tests much more precisely in a short time period. Also, it saves us a significant amount of time to deal with other manual test activities in each sprint.
In this talk, we will walk through well-known open-source and commercial solutions for visual test automation. We will learn which technologies they use, what type of visual tests they are suitable for, and their major differences between each other. Besides this overview, we will also make a real-life visual test automation demo by using Selenium, ImageMagick, and AShot.
3. Who am I?
▪ I am Onur Başkırt
▪ Head of Software Testing at
▪ Co-owner of
▪ Formerly: Teknosa, Ericsson, Huawei, STMicroelectronics, BAU & USCS
4. Online Employment and Recruitment Market Leader
in TURKEY since 1999
5.6 million
Visitors
180 million
Page views
25 million
Resumes
75.000+
Employers
6. Outline
• What is Visual Test Automation?
• Why Visual Testing?
• Why Automation?
• Challenges
• Visual Test Automation Tools
• Brief Overviews
• Detailed Reviews (Applitools Eyes, Percy.io, Galen Framework)
• Selenium & ImageMagick & AShot Example
• Q&A
7. What is Visual Test Automation?
• Verifies the GUI is displayed as expected to the end users.
• Focuses on visual contents:
• Colors
• Shapes
• Sizes
• Locations of elements
UI Test vs Visual Test
8. Why do We do Visual Testing?
WordPress
Template
CSS
animations
Widgets Sliders etc.
Mobile App
Generation
Engine
.ipa
.apk
GUI MUST APPEAR CORRECTLY
USER INTEARCTIONS
COMPANY BRAND
CRITICAL PRODUCTS and SERVICES
9. Why do We Automate Visual Testing?
Variety of Test Environments
• Browsers – Mobile Devices – Resolutions etc.
Time to Market Pressure
• Release cycles getting shorter.
Manual Visual Testing is Cumbersome
• Human eyes error-prone and getting blind.
In Mobile World, you will live with visual bugs longer
• Can not update mobile app daily.
19. What is Applitools Eyes?
• Automated Visual Testing Tool for Web sites and Mobile apps
• Finds Visual Differences with Several Match Levels
• Eyes SDK for Selenium & Appium
• Cloud-Based Clean and Informative Dashboard
20. How Does it Work?
Visually notifies the differences
At 2nd run and more it compares actual screenshots with the baselines.
At 1st run it saves all screenshots as baseline
Takes screenshots with Eyes SDK and sends to the Applitools cloud
21. Match Levels and Their Meanings?
• Pixel by Pixel ComparisonEXACT
• Mimics the human eyes.
• Ignores trivial mismatches (antialiasing, small pixel movements, etc.)
STRICT
• Similar with STRICT plus it ignores color changes.CONTENT
• Compares the layouts.
• Checks the elements locations.
LAYOUT
22. 4 Main Steps of Eyes SDK
Close the Test
eyes.close();
Add UI validation Check Points
eyes.checkWindow("Validation Window Name");
Start a Test
eyes.open(driver, APP_NAME, TEST_NAME);
Create and Configure an Eyes Object
Eyes eyes = new Eyes();
23. Mobile Visual Testing Example
Open BitbarSampleApp.apk
* Visually check that start screen as expected?
Click second option “Use Testdroid Cloud”
*Visually check that Second Option is clicked?
Write “SW Test Academy” to text bar
Click Answer button
Wait until the second page appear
*Visually check that the answer is correct?
Close the driver.
24. Mobile Visual Testing Example
Step-1: Install Eyes SDK with Maven Dependency
<dependency>
<groupId>com.applitools</groupId>
<artifactId>eyes-selenium-java</artifactId>
<version>RELEASE</version>
</dependency>
Step-2: Do Appium, Android SDK and GenyMotion Settings
26. Mobile Visual Testing Example
Step-4: Open Eyes Do Visual Check Close Eyes
Open Eyes!
Perform a Visual Check!
Close Eyes!
Abort if Eyes not closed!
34. Galen Reporting
• Error Reporting
Generates HTML reports where you can see
all your test objects
• Screenshots
Highlights the misaligned elements
• Image Comparison
Compares images and shows differences
35. Galen Example with JAVA & Webdriver & TestNG
Logo: 31 pixel
below from the
header
Header: 0px to
the right, left, top
Navigation Links:
Horizontally aligned.
Logo should displayed
correctly.
36. Galen Example with JAVA & Webdriver & TestNG
Step-1: Installation with Maven Dependency
<dependency>
<groupId>com.galenframework</groupId>
<artifactId>galen-java-support</artifactId>
<version>2.3.2</version>
</dependency>
37. Galen Example with JAVA & Webdriver & TestNG
Step-2: Write a Galen Specifications (.gspec)
web elementsfor desktop
Declaring locations
tagging
Image Comparison
loops
5
38. Galen Example with JAVA & Webdriver & TestNG
Step-3: Write your Test Code
Get Layout Report
Get Galen Test
Information from
Layout Report
Generate HTML Report based on
Galen Test Information
46. What is Percy?
Visual regression testing and review platform.
• COMPARES images, FINDS differences.
• We can REVIEW and APPROVE changes.
• Supports RESPONSIVE Design testing.
• Prevents FALSE-POSITIVES.
• PARALLEL TESTING support.
• Integrated with GitHub and many CI tools.
47. Which Technologies Does it Support?
CI Tool Integrations Languages/Clients
Travis CI Ember
CircleCI Ruby – Capybara
Semaphore Python – Selenium (BETA)
Codeship Static websites
Buildkite
Drone
Jenkins
48. How Does it Work?
When commit any code changes.
50. Percy Installation Steps (Percy Side)
• Go to Percy.io and click Sign In WITH GITHUB.
• Go to Percy Dashboard and Create an Organization
• Install GitHub Integration
• Create a project and get your CI integration variables:
PERCY_TOKEN & PERCY_PROJECT
51. Percy Installation Steps (Jenkins Side)
• Set your PERCY_TOKEN and PERCY_PROJECT environment variables
• Do GitHub Integration settings and Create a Jenkins Job.
• Do your Ember Project’s Repository settings.
• Add build and test Windows Batch commands:
• For BUILD:
• For TEST:
npm install && bower install
npm test
52. How to Write Test with Ember-Percy?
percySnapshot(name, options);
scope For element to snapshots. Default is full page.
percySnapshot('homepage', {scope: '#header'});
Takes ScreenShots
53. Sample Test Code Official TODO Example of Percy
Navigate the homepage
Take ScreenShot
Write sth to the bar
and
Press Enter Assertions
Take Final ScreenShot
59. What is ImageMagick?
• Open-source software suite that manipulates
images in a variety of formats (over 200) including
PNG, JPEG, GIF, etc.
• Edits, Converts, Resizes, Compares, Rotates,
Adjusts Colors etc.
• For JAVA it has Im4java which is a pure-java
interface.
60. What is AShot?
• Yandex's screenshot utility which takes, crops, prettifies and
compares screenshots.
• Entire Page
• Web Element
61. Example Test Scenario (Basic CSS Animation Check)
• Open www.kariyer.net
• Unhide text area of Uzman Photo
• Hover on the “UZMAN” weblement on main page.
• Wait 2 seconds to finish CSS animation
• Take screenshot of an element and save as Baseline
for the 1st Run.
• For 2nd and more runs COMPARE baseline image
with actual image.
• Put all differences in "Differences folder"
HOVER
BASELINE
62. Example Test Scenario (Basic CSS Animation Check)
• Step-1: Install ImageMagick with all options and add to system path.
63. Example Test Scenario (Basic CSS Animation Check)
• Step-2: Create a Selenium & TestNG project and add dependencies
64. Example Test Scenario (Basic CSS Animation Check)
Test Method’s Code
AShot
ImageMagick
JavascriptExecutor
- We have 5.6 million visitors and 180 million page views per month
- We hold more than 25 million resumes in our databases
- We have more than 75 (seventy five thousand) customers.
SW Test Academy is a technical software testing blog. We generally share technical articles on test automation, performance testing, security testing, DevOps practices and continuous integration. You can also join our newsletter and stay tuned for the latest articles.
What is Visual Test Automation?
Why do We Need to do Visual Testing?
Why do We Automate?
What are the Challenges?
Visual Test Automation Tools
Brief Overview of Visual Test Automation Tools
Detailed Reviews (Applitools Eyes, Percy.io, Galen Framework)
Selenium & ImageMagick & Ashot Example
Q&A
Visual test automation is a software test automation activity which verifies that the GUI is displayed as expected to the end users.
We check shapes, colors, locations of elements and verify that UI itself correctly displayed.
By using Selenium, we do UI test automation. We interrogate, manipulate web elements and add verifications to check system’s functionality.
There mustn't be any overlap between each element.
People may ridicule about your system.
If you are a WordPress Template company or if you provide a mobile app generation core which outputs ipa (IOS) and apk (Android) files, your product has to generate visually correct results. Beucase, customers rely on your product and it must be functionally and visually perfect.
- It is not feasible to cover all test matrix in a sprint. It takes too much time.
- In agile world, release cycles are getting shorter.
- If you do manual visual testing sprint by sprint, your eyes are getting blind and you can not detect the bugs.
Basically, antialiasing means making the edge of an image much smoother and nicer. This enhances image quality.
Because of graphics card’s rendering engine or antialiasing option settings, images rendered not the same in each machine. Thus, if we do pixel by pixel comparison, we will get mismathces and our test will fail.
The another big challenge is dynamic content in visual testing. You have to do something to eliminate this problem such as hiding those elements, stopping them or wait to finish animations etc.
Auto-kerning and dynamic letter movements sometimes causes mismatches thus visual testing tool has to be configured to handle this kind of movement problems.
For example, adding pixel paddings sometimes causes offset problems. Your tool has to be capable to handle this problem.
Also, there are many problems exists in visual testing. We have to handle them with strategies and our tool’s capabilities.
PhantomJS runs on Wekit and pure headless browser.
SlimerJS runs on Gecko (firefox) and not natively headless browser.
Resemble.js analyses and compares images with HTML5 canvas and JavaScript.
Define specific areas/Exclude specific areas/Sync Taken Images/Multiple ResolutionsWebdriverCSS has sync option to use your taken images any environment.
WebdriverCSS isn't yet compatible with WebdriverIO v3.0
Look-same is a node.js library for comparing images
It has strict mode like Applitools Eyes and you can modify tolerance, pixel ratio, antialiasing and text caret option.
Configurations done by .yaml files (snap_file, domains, directory, screen-widths etc.)
Add a Test Group
Add a Browser
Add Pages
Run Test
You can find the details of this example on swtestacademy.com. First, we open Bitbar apk….
Step2: Install Appium, Android SDK, and GenyMotion Emulator (because it is faster). Then do required settings.
Passing Test Scenario
Failing Test Scenario
Who knows GalenFramework?
LAYOUT: It tests location of objects relatively to each other on a page.
RESPONSIVE: It Opens Resizes Runs the Tests based on given Galen Specifications.
Grid: You can test responsive websites on different mobile devices and resolutions.Parallel: Tests runs faster and we save time.
First, we need to declare web elements
Then, we can put tags for our pages by using equality sign
Also, we can group elements by using @on desktop declaration
Under the desktop, we can write our elements location
We can do image comparison with given error ratio.
And we can use loops to handle list elements
First, we need to set the browser size with required dimensions.((JavascriptExecutor) driver).executeScript("returndocument.documentElement.clientWidth")((JavascriptExecutor) driver).executeScript("returndocument.documentElement.clientHeight")
http://isgarlo.tumblr.com/post/110815604942/webdriver-how-to-set-the-browser-viewport-size
Step1: We check Layout with Galen.checklayout() function and get layoutReport
Step2: We get Galen Test Information based on layoutReport
Step3: We build HTML report based on Galen Test Information
Step4: If there is a layout problem, test asserts a fail.
Does anyone know Percy? Or Did anybody use Percy before?
RESPONSIVE: We should simply provide a list of breakpoint widths and then Percy take care of the rest.
ANIMATIONS: It freezes different kinds of animations that can cause false-positive visual diffs.1) It freezes animated GIFs on the FIRST FRAME.
2) It freezes most CSS animation and transition styles
Static: A command-line client for static websites.
You can push your code changes or do pull request to the github repository then Jenkins triggered and run your tests, Percy takes DOM snapshots and uploads to Percy Server, and then it compares them with the Baseline images and you will see «4 visual diffs need review» and when you approved them you will see that "@OnurBaskirt approved 4 visual differences".
Add all installed tools to system path. Ember, Phantoms.js, Bower, npm, and node.
Save this variables to use Jenkins or another CI tool.
Start Jenkins with Admin User. It is so critical to prevents “is not recognized command” problems.
For taking elements snapshots, we can use scope option.
When we use header’s CSS path, it takes header screenshot
You can do responsive desing testing with Ember Percy very easily.
You need to declare your environments widths and give tags to them such as desktop, mobile, tablet.
Also you can define default Breakpoints.
When you run your tests, you can override default breakpoints at percySnapshot’s function.
When we run our test, at first run Percy does not do any comparison. It saves all images as baseline.
This is the Passing Test Scenario
I played with the test page and I added an extra line to the footer and Percy got the mismatch and highlight it.
doComparison method is in our BaseTest Class and it comprises of baselineScreenShotPath, actualScreenshotPath and differencesScreenShotPath variables.
-First we need to do ImageMagick configurations
Then, add expected, actual and difference image paths to the ImageOperation object
Finally we will use compare.run method to do comparison.
At first run, code creates Differences and ScreenShots folder and takes baseline screenshot and creates a folder by using test name and put baseline screenshot into this folder. It always takes actual screenshot and put it under test folder. It has not any side effect.
At seconde run, code checks that is there any baseline image exists? If yes, it compares baseline image with actual one and create a difference image.
In order to control fail case, I draw some random sketch on baseline image and then run the test again.
It gives an error and you can see differences both in test folder and differences folder.