1. Effective Test Automation
in an Agile Environment
Krishantha Samaraweera
Senior Technical Lead – Test Automation
WSO2

2. Agenda
● Automated testing
● Success stories
● Automation Concept – Test Pyramid
● WSO2 Test Automation Framework
● WSO2 Platform Automated Test Suite
● Lessons learned
● Automation Best Practices
● Continuous Integration and Delivery

3. What is Test Automation
● Automation testing – Testing which can be done programmatically
● Compare result – Actual and Expected
● Generate reports
● Goal is to reduce the number of test cases to run manually
● Not to eliminate Manual testing all together

4. Why Automated Testing
● To get early and instant feedback
● Safety net to save time and which is reinvented in to manual
testing.
● Doesn't require human intervention.
● Unattended (Overnight)
● Manual testing has high risk of mission out on something

5. Success Stories
● Google runs 100M+ automated test cases every day
● Google customer surveys team can deliver a change to
production 8 mins after code is committed.
● Ebay runs 35000 test cases in 5 hours
● At Facebook, each of 5,000 engineers commits to trunk HEAD
at least once a day and the code at trunk HEAD is pushed to
production once daily.
● Atlassian JIRA is protected by over 14,000 unit tests and
19,000 functional tests from regressions

6. Success Stories Cont'd..
● Etsy used to take 14 hours and an army of people to deploy
one release into production before CD kicked in. Now it takes
15 mins, 1 person. They did 729 deployments to production in
Nov 2010, only 6 deployment related incidents.
● At Amazon, new code is deployed to production at a
staggering rate of once every 11.6 seconds during a normal
business day.
● That’s 3,000 production deployments per day.

7. Test Automation Pyramid

8. Test Automation Pyramid Cont'd..

9. WSO2 Test Automation Framework
● Why our own framework
– Open source test scripts
– Stay lean
– Single framework for all products
– Facilitate backend testing + UI tests

10. Technology outlineTechnology outline

11. Test Automation Framework Architecture

12. Glimpse of Test Automation Framework
● Simple automation context API.
● Platform wide test execution support.
● Annotation based test execution management support.
● Test reports and coverage report generation.
● Test tooling integration (Jmeter).
● Selenium WebDriver integration and ability to run UI tests on
cross browser environments.
● Framework extensibility capability through pluggable modules.
● Easy retrieval of new configurations.

13. WSO2 Platform Automated Test Suite
● Package tests for other people to use
● Distribution contain all product integration tests and platform
tests jars.
● Test script management tool
● Ant based test executor
● Test execution on distributed product clustered or cloud
deployment.
● Cross platform test execution

14. Lesson Learned
● Once you abandon the maintenance of the automation, it is
likely to die. For a better chance of success, choose an
automation approach that will require the least maintenance.
● Automation requiring multiple cycles and project releases for it
to become fully effective and provide an acceptable ROI.
● Start by automating the most valuable tests.

15. Lesson Learned Cont'd
● Plan your test automation at the beginning of the project.
● Unit Testing FIRST – Fix your test Pyramid.
● Staff should know how to diagnose failures. Minimize calling
automators for every issue

16. Lesson Learned Cont'd..
● Automation development requires the same discipline as
software development.
● Organization wide commitment is required for successful test
automation methodology
● Everyone should know what is automated.

17. Test Automation Benefits – Efficiency ROI
Time savings of test execution with automation vs test execution without
automation
Product # of Tests Manual
time per
test case
Iterations Manual test
execution time
Automated
execution test
time
Saved time
ESB 922 8 min 5 384 hours 10 hours 77 man days
G-Reg 1557 8 min 5 1038 hours 15 hours 130 man days

18. Continuous Integration
● Continuous Integration is a software development practice
● Team integrate their work frequently
● Each commit is verified by automated build
● Rapid feedback
● Ships better code faster
● Fixing build breaks ASAP important.
● Should have a notification mechanism

19. Continuous Integration Cont'd..
Source : http://mikeciblogs.wordpress.com/2010/06/15/continuous-integration-for-agile-project-managers-part-3/

20. Continuous Delivery
How long would it take your organization to deploy a change [to
production] that involves just one single line of code? Do you do
this on a repeatable, reliable basis?”
- Mary and Tom Poppendieck,
Implementing Lean Software Development

21. Continuous Delivery
Build software in such a way that it ready to be released at any
time
– Continuous feedback
– Reduce risk of failures in production system
– Rapid, reliable and repeatable automated processes

22. Test Cases to be Automated
● HIGH RISK – Business critical test cases.
● Features tend to break frequently
● Repetitive tasks are good candidates for automation
● Test cases that are tedious or difficult to perform manually
● Test cases which are time consuming.
● Tests that use multiple data values for the same actions
● Smoke Tests

23. Test Cases not suitable for Automation
● Test cases that are newly designed – Not executed manually
at least once.
● Test cases for which the requirements are changing frequently.
● Usability testing – “How easy is the application to use?”
● One-time testing

24. Testing Philosophy
● You don't have to write 1,000 tests
● The benefit of the tests scales very nicely.
– 1,000 tests is good
– 100 tests is also good
– 10 tests is also good
● And even one test is better than no tests at all

25. Testing Philosophy
● So don't start out by saying "Oh, I have to write 1,000 tests for
this."
● Then you might not write any at all.
● Start by saying "I'll write one test".
● Then write it
● Then maybe write another one
● You will immediately see the benefit of having one test

26. Conclusion
● Fix your test pyramid right
● Test maintenance is the king
● Run you tests frequently
● Importance players of successful test automation are right
selection of tools, testing process and team.
● Manual testing is also important

27. References
[1]“GTAC 2013: Presentations - Google Test Automation Conference — Google Developers.” [Online]. Available:
https://developers.google.com/google-test-automation-conference/2013/presentations. [Accessed: 17-Mar-2014].
[2]“Continuous Integration for Agile Project Managers (Part 3) | Mike CI blogs.” [Online]. Available:
http://mikeciblogs.wordpress.com/2010/06/15/continuous-integration-for-agile-project-managers-part-3/. [Accessed:
17-Mar-2014].
[3]“All About Automated Testing.” [Online]. Available: http://www.guru99.com/automation-testing.html. [Accessed:
17-Mar-2014].
[4]“Writing Automated Tests.” [Online]. Available: http://perl.plover.com/yak/testing/samples/. [Accessed: 17-Mar-2014].
[5]“Continuous Integration.” [Online]. Available: http://martinfowler.com/articles/continuousIntegration.html. [Accessed:
17-Mar-2014].

28. Thank You