Working in the time box of a scrum sprint does present some challenges - especially if your team is building a product from scratch. Sprint after sprint you have to add new features and ensure that whatever you built previously continues to work. Having an automated testing framework, which takes care of both system and integration tests, adds a lot of firepower to such a team. It not only acts as a safety net against regressions caused by new development, but more importantly frees up a lot of precious developer and tester time - allowing them to focus on the things they do best. Hence a continuous test automation approach becomes inevitably necessary. Çevik metodolojilerle yazılım geliştirmenin kendine has bazı zorlukları bulunmaktadır. Her bir sprint ile eklenen özelliklerin, öncekileri bozmadığından emin olunması gerekmektedir. Bu durumda otomatize bir test altyapısına sahip olmak, ekibe oldukça büyük avantajlar sağlayacaktır. Bu avantaj regresyon test setinin kotarılması ile kalmayıp, yazılımcıların kendi işlerine odaklanmalarında da kendini göstermektedir. Böyle bir ortamda, sürekli test otomasyonu da kaçınılmaz bir şekilde gerekli hale gelmektedir. Please contact us for more information: www.keytorc.com

1. Software Test Automation in
Agile Projects

2. Agenda
» Test Automation Basics
» Continuous Integration/Delivery in Nutshell
» Stages of CD Pipeline
» Best Practices – Tools & Patterns
» Next Generation Approaches

3. Test Automation Basics

4. Test Automation
The major key to the success of software test automation:
To reduce manual testing activities and redundant test operations using a systematic
solution to achieve a better testing coverage.
Test automation is not always necessary,
appropriate, or cost effective. In cases
where we are making decisions
based upon an expected return on
investment, analysis can direct us to
where test automation can benefit us.
These returns are best computed by
comparing the costs and gains achieved
through test automation over
manual testing.

5. Automation Decision in Sequential Models
Frequent regression testing
Repeated test case execution is required
Faster feedback to the developers
Reduce the human effort
Test same application on multiple environments

6. Pyramid of Test Automation
Business Facing
Technology Facing

7. Continuous Integration/Delivery
İn Nutshell

8. Definition of Continuous Integration
Continuous Integration is a software development practice where members of a team
integrate their work frequently, usually each person integrates at least daily - leading to
multiple integrations per day. Each integration is verified by an automated build
(including test) to detect integration errors as quickly as possible.
(Martin Fowler – Thoughtworks)

9. Continuous Integration Practices
Maintain a Single Source Repository
Automate the Build
Make Your Build Self-Testing
Everyone Commits To the Mainline Every Day
Every Commit Should Build the Mainline on an Integration Machine
Fix Broken Builds Immediately
Keep the Build Fast
Test in a Clone of the Production Environment
Make it Easy for Anyone to Get the Latest Executable
Everyone can see what's happening
Automate Deployment

10. A Basic Pipeline

11. Continuous Delivery

12. What to Automate

13. Stages of
Continuos Delivery Pipeline

14. An Example Toolset

15. Testing Phase: Commit Stage
Commit
Stage
Automated
Acceptance
Stage
Non-
Functional
Test Stage
Manual
Tests
Sanity
Check
Rather be a scheduled
job

16. Commit Stage
• Run against every check-in
• If it fails, fix it immediately
• Creates a release candidate
Hint: If next stage(s) (Automated Acceptance Tests) take a significant amount of time
(e.g. More than 30 minutes), embed a small subset of them into commit stage. So,
feedback interval will be decreased to act fast on major incidents.

17. Testing Phase: Automated Acceptance Stage
Commit
Stage
Automated
Acceptance
Stage
Non-
Functional
Test Stage
Manual
Tests
Sanity
Check

18. Automated Acceptance Stage
Tips:
Focus on multi-threading for race conditions
Run tests in parallel (Use a supporting framework)
Design atomic scenarios
Manage data strictly
• End-to-end tests in production-like environment
• If it fails, fix it immediately triggered when upstream stage passes

19. Testing Phase: Non-Functional Test Stage
Commit
Stage
Automated
Acceptance
Stage
Non-
Functional
Test Stage
Manual
Tests
Sanity
Check

20. Non-Functional Test Stage (Performance)
• Select most suitable test cases (Either widely used or critical for a business)
• Test against a production replica environment, for example staging (As much
as possible)
• Focus on subject matter practices (Anything!)
• Use automate-acceptance tests with counters (As a first step maybe)

21. Testing Phase: Manual Tests
Commit
Stage
Automated
Acceptance
Stage
Non-
Functional
Test Stage
Manual
Tests
Sanity
Check

22. Manual Tests
• Focus on non-repetative, unautomated test cases, showcases
• Create room for creativity with exploratory testing
• Realize people dependent tests, like usability

23. Pipeline Integrated

24. Pipeline Integrated

25. Pipeline Patterns & AntiPatterns
Insufficient Parallelization
Ideal Time < 10-15 Mins
No more than 1 hour
Hint: Prefer a paralellizable unit testing framework. TestNG for Java and MbUnit for .Net
might be good choices.

26. Pipeline Patterns & AntiPatterns
Inflexible Workflow

27. References
http://www.romaniatesting.ro/to-automate-or-not-to-automate-testing
http://www.softwarequalitymethods.com/papers/star99%20model%20paper.pdf
http://www.continuousdelivery.com/2010/09/deployment-pipeline-anti-patterns/
http://www.testng.org/doc/documentation-main.html#parallel-tests
Referencing Lisa Crispin, Agile Testing Book 2009
Continuous Delivery Workshop Material created by Jez Humble, Martin Fowler, Tom Sulston, & Neal Ford

28. Teşekkürler
info@keytorc.com
Keytorc
Keytorc Software Testing Services