Slides presented at Selenium Conference India 2016 Automated visual testing is a major emerging trend in the dev / test community. In this talk you will learn what visual testing is and why it should be automated. We will take a deep dive into some of the technological challenges involved with visual test automation and show how modern tools address them. We will review available Selenium-based open-source and commercial visual testing tools, demo cutting edge technologies that enable running cross browser and cross device visual tests at large scale, and show how visual test automation fits in the development / deployment lifecycle. If you don’t know what visual testing is, if you think that Sikuli is a visual test automation tool, if you are already automating your visual tests and want to learn more on what else is out there, if you are on your way to implement Continuous Deployment or just interested in seeing how cool image processing algorithms can be, this talk is for you!

1. Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
ADVANCED AUTOMATED
VISUAL TESTING WITH
SELENIUM

2. YOU CAN AND SHOULD
AUTOMATE YOUR
VISUAL TESTS!

4. AGENDA
 Why automated visual testing?
 Tools & Technology
 Where does it fit?
 Q & A

5. WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a Graphical
User Interface appears correctly to users

10. WHY AUTOMATE?
The test matrix is too big to cover manually
 Web browsers
 Devices
 Operating systems
 Screen resolutions
 Responsive design
 L10n
 3rd Party upgrades

11. WHY AUTOMATE?
Mobile apps quality is critical
 Harder to roll back changes
 Can’t push daily
 Updates take battery and data
 Higher quality bar

12. WHY AUTOMATE?
Release cycles keep getting shorter

13. AGENDA
 Why automated visual testing?
 Tools & Technology
 Where does it fit?
 Q & A

14. HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline

15. DEMOWebDriverCSS

16. HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline

24. AND MORE…
 1 pixel offsets in element positioning
 Dynamic content
 Moving elements
 Images of different size
 Performance

26. IMAGE COMPARISON
ENGINES

27. ImageMagick
 A powerful command line tool for image processing.
 Fuzzing is used to eliminate slight color differences
 An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246

28. THE JAVASCRIPT ENGINES
 Resemble.js
Pixel by pixel + error ratio + anti-aliasing
http://huddle.github.io/Resemble.js
 Blink-Diff
Pixel by pixel + error ratio + anti-aliasing + perceptual color distance
computation
https://github.com/yahoo/blink-diff
 Looks-Same
Pixel by pixel + perceptual color distance computation + ignore Caret
https://github.com/gemini-testing/looks-same

29. APPLITOOLS EYES
 Handles anti-aliasing, pixel offsets, color similarity, and image scaling
 Dynamic and moving content
 Compare images of different sizes
 No error ratio configuration required
 Validates full UI pages
 Fast!
 Supports layout / structure matching

30. DEMOVISUAL LAYOUT MATCHING

31. HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline

32. QUICK FEEDBACK TOOLS
MOTIVATION
 Get fast feedback on code changes
 Run tests locally in the background
SETUP
 Render screenshots with a headless
browser
 PhatomJS, SlimerJS
 Configuration file driven tests
CONS
 Partial coverage
 Chrome, IE?
 Old browser layout engine versions
 Limited navigation

33. VISUAL COVERAGE TOOLS
MOTIVATION
 Verify that the app / site looks right in
all real execution environments
 Test as many UI states as possible
 Execute many tests in parallel
SETUP
 Render screenshots using real browsers
on a variety of operating systems and
devices in parallel
 WebDriver, Grid
 Code driven tests
 WebDriver, DSLCONS
 Requires test infrastructure
 Implement and maintain test code

34. SELENIUM BASED TOOLS
CODE / SCRIPT
● Needle (Python WD)
● WebDriverCss (JS WebDriverIO)
● Gemini (JS DSL)
● Selenium Visual Diff (Java WD)
● VisualCeption (PHP CodeCeption)
● Pix-Diff (JS Protractor)
● Shoov (JS WebDriverIO)
● Vizregress (.NET WD)
● Rspec Page Regression (Capibara)
● Applitools Eyes (All WD + Appium)
● Fighting Layout Bugs (Java WebDriver)
● Huxley (Record Playback)
CONFIGURATION
● Viff
● CSS Visual Test
● GreenOnion
● Wraith-Selenium

35. HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline

36. REPORT DIFFERENCES
As files on the file system
WebDriverCSS

37. REPORT DIFFERENCES
As a gallery
Selenium Visual Diff

38. HOW DOES IT WORK?
Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline

39. UPDATE THE BASELINE
Gemini

41. DEMOAUTOMATED MAINTENANCE

42. AGENDA
 Why automated visual testing?
 Tools & Technology
 Where does it fit?
 Q & A

43. WHERE DOES IT FIT?
• Component
s
• Code
review
• Developers
• Designers
• QA
Visual testing of frontend components by
frontend developers

44. WHERE DOES IT FIT?
• Pages
• Page
sections
• Developers
• Designers
• QA
• Others
Full or partial validation of application screens
A collaboration tool

45. WHERE DOES IT FIT?
• Staging vs.
Production
• Ops
• QA
Validate your staging deployment using your
production deployment as a baseline

46. WHERE DOES IT FIT?
• Monitoring
• Ops
• QA
No missing resources in production
No corruption due to 3rd party data
No corruption due to browser / OS upgrades

47. Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
QUESTIONS?