Designing and implementing (or selecting) the right automation strategy, for functional testing, with visual testing, can help your project with greater test coverage while improving test scalability

1. Peter Kim
2017

2. INTRO – H20Dragon
Peter Kim
Email: h20dragon@outlook.com
https://www.linkedin.com/in/peterkim777

3. Agenda
ž Share how visual testing benefited our
functional test automation strategy
ž Share where visual testing successfully
provided coverage to other test types
ž Share how to get started with using
visual testing
ž Demos
ž Q&A

4. What is Visual Testing?

5. What is Visual Testing?
ž Verifies the UI/UX
— Test coverage of what was actually rendered
on the UI device
— Test coverage for responsive app. behaviors
Verifies as though you are using your eyes.

6. Automated Visual Testing

7. A need to change because ....
ž Catch defects earlier in the SDLC
ž Verify the UI/UX across target browsers/UI devices
ž Verify the UI/UX across different resolutions
ž Existing automation strategy didn’t include “visual” coverage.
ž Current process of asserting validations are too static
ž Brittle Selenium locators (e.g. XPATH)
ž Page Objects aren’t scaling quickly enough
ž Customers are reporting defects, in PROD, instead of us
ž Inability to scale functional (Regression) test coverage
ž Uncertainty of what type of assertions are needed
ž Lack domain knowledge in the app.
ž Test reports lack depth
ž Test reports need meaningful visuals
ž Customers feel that your UI/UX is as important as the functionality
ž QA team has limited resources
ž It’s not 2007 anymore, but your test automation design/strategy
thinks it is.
ž You want to repurpose your functional tests for other test types –
e.g. Localization / Accessibility

8. You need it because ..
Although, we were unable to create assertions to catch
potential defects, with visual testing, we now at least
have the ability to report and share with others what has
changed.
Here, you’ve empowered others, or possibly held
accountable, that they too were in a position to catch (or
see) potential defects or comment on changes (whether
intended or not per requirements/user story).

9. Automation with Selenium

10. Automation with Selenium
ž Challenges
— Takes time and resources to learn effective
Selenium/WebDriver
— Selenium is an automation library. You’re
responsible for creating and/or piecing together
the framework, designs, and strategies that will
leverage it.
— Poor test frameworks/design will mean brittle
tests and will most likely hinder your ability to
scale – you will be playing “catch-up” with the
current versions/releases of your app.

11. Automation with Selenium
ž Challenges
— Assertions are typically “static”, based on the
requirements/user stories/pre-canned
queries. E.g. Limited number of assertions
used to verify and validate your test case.
This means, if your application behaves
differently and none of your assertions cover
that change .. you are now exposed to risks
(defects, false confidence in your findings).

12. Are You Automating the
“Expected”?
“Obvious is the most dangerous word in
mathematics.”
- Eric Temple Bell (Mathematician)
1883-1960

13. Are You Automating the ‘Expected’?

14. Are You Automating the ‘Expected’?
ž Pesticide Paradox
Over the course of time, your automated
tests will diminish it’s ability to catch
defects.
Especially, the tests that seem to always
pass.

15. Improving automation
ž What was our criteria to advance
our existing automation strategy?

16. Automation Requirements
ž Need a technical solution that plays well with Selenium/WebDriver
since all our end-to-end tests are Selenium based.
ž Minimal impact, e.g. refactoring, of our existing automation test suites.
ž Avoid re-inventing designs based on old designs, thinking they’re new
designs.
ž Be cautious of “best practices” that have been around since 2007.
ž Short and simple learning curve, with powerful results.
ž Technology/tool must be well supported
ž Technology/tool must scale and be secure (e.g. cloud, multiple device
support)
ž Powerful and customizable test reports

17. Lessons Learned
ž .. Be cautious when investing in resources
with strategies which ultimately repurpose
the same old approach and processes.
ž Consider leveraging a new design/strategy
ž Consider leveraging proven technologies
that integrate with your existing testware.

18. DEMO – Functional Automated Test
Filters, for advanced search, returns correct set of Porsche 911s.

19. DEMO – Functional Automated Test
Filters, for advanced search, returns correct set of Porsche 911s.

20. DEMO I.
Create a baseline visual test
The visual baseline will be used for
validation of future visual test results.

21. DEMO I.
Create a baseline visual test
The visual baseline will be used for validation of future visual test results.

22. Demo I. Re-run the test.
Visually compare the new run against the baseline

23. Demo I. Recap
ž Created a visual baseline
ž Execute end-to-end with visual coverage
— Specified matchType (e.g. Strict, Content, Layout)
— Specified target page/regions
ž Automated coverage scales with more
assertions.
ž Easy to “see” visual results
ž Ability to manage/update assertion criteria

24. Demo II. Visual Defect

25. Demo II. Visual Defect Analysis

26. Demo II. View the results

27. What was actually
demonstrated?
ž Little resources are needed to add test coverage
(assertions) if you have a simple, yet powerful, ability
to leverage visual testing.
ž Coverage is dramatically improved
— Context (text)
— Layout
— Graphics
ž Less brittleness in your automated Selenium tests,
while decreasing coding efforts
ž Forgiveness of ones’ lack of insight of “what should
be covered?” or “what’s the scope?”
ž UI (Visual) assertions can facilitate functional testing
ž Minimize pesticide paradox
ž Fred Flintstone has an awesome car!

28. Leveraging VT
Localization Test Automation (Regression)

29. Leveraging VT
ž Localization Testing
Ensuring that all targeted and/or
served parties, regardless of
language, nationality, and culture are
equally and fairly respected and
presented with the same information
and content.

30. Localization Test Automation
ž Content
ž Images
ž Layout

31. Localization Test Automation
ž Content
— Headings
— Data formats
○ Date
○ Currency
— Translation Errors
ž Images
ž Layout

32. Localization Test Automation
ž Content
— Headings
— Data formats
○ Date
○ Currency
— Translation Errors
ž Images
ž Layout
Get more details on automating your Localization tests at https://applitools.com/concur/

33. Localization Test Auto

34. Tools –Applitools Eyes
ž Visual Test Automation
— Advanced Visual/Digital Analysis (AI)
○ Mitigates false positives (e.g. floating or shifting of elements)
○ Create/manage regions per inclusion/exclusion
— Applitools Account (It’s free)
https://applitools.com/sign-up/
— Applitools API Key
— Install the SDK (based on your programming language) for your
Selenium tests
— Understanding resolutions – e.g. “viewport”
Reference: https://applitools.com/videotutorials/

35. Summary
ž Functional testing can be improved by leveraging visual testing.
ž It’s simple to conduct Visual Test Automation
E.g. Applitools Eyes with Selenium/WebDriver based framework
ž Heavy reliance, using the pattern of asserting the “obvious”, being the “master of the obvious”, or
“happy path” is a great way to ship defects into PROD.
ž Run and validate your tests on the actual devices your end-users are using.
ž Run and validate your tests on the same resolutions (UI devices) based on the “rendered” pages (not
just the DOM).
ž Automate responsive apps visually.
ž Create a plan to deal with Pesticide Paradox
ž Advance your test automation framework with new designs and strategies (it’s not 2007) – also,
leverage new technologies that are proven to be reliable and scalable, while being simple yet
powerful.
ž For QA/QE teams – leverage “CYA”

36. Thank You
Peter Kim
Email: h20dragon@outlook.com
https://www.linkedin.com/in/peterkim777
https://dcast.io