test automation strategy for Scrum Projects

1. INdT
TestLink
Quality
Bugtracker
Validation
VERIFICATION
Bug
SCRUM
Test Automation Strategy and Practices
for Scrum Projects
Nokia Institute of Technology
Test Automation
Eliane Collins
Paris, France - Sept 2013
Scrum Gathering 2013 - Paris
AGILITY

2. Company Confidential
Nokia Internal Use Only
2. Study Cases in Test
Automation
1. Introduction
Summary
3. Results and Lessons
Learned
4. Test Automation
Strategy
5. Execution
6. Conclusion

3. Company Confidential
Nokia Internal Use Only
3
Introduction

4. Company Confidential
Nokia Internal Use Only
INTRODUCTION
4
INTRODUCTION
• It is an independent and nonprofit
institute committed to conducting
research and development of
technological solutions through
application development, new
technologies and concepts.
Manaus
Recife
Brasília
São Paulo
Main Areas:
• Product Creation/Product Validation
• Apps and Services
• Hardware and Manufacturing
• Software User Interface
INdT – Nokia Institute of Technology
- Master degree in Software
Engineering.
- 8 years of experience in Software
Testing for web and mobile
platforms.
- Coordinator of Product Validation
team in INDT.
- Instructor and researcher of
projects in related area.
- International certifications CTAL-
TM, CSD, ITIL and COBIT.
Who I am?

5. Company Confidential
Nokia Internal Use Only
5
INTRODUCTION
Global Survey [LOGIGEAR, 2010], found that 40% of companies surveyed
believe that test automation is not easy, only 15% automate regression testing
and 56% fail to maintain test automation, even using the appropriate tools.
Technical survey by Kasurinen in 2010,
revealed only 26% of test cases in the
organizations automated.
Technical survey by DIAS-NETO in 2006,
indicated Brazilian companies from Manaus
do not execute formal tests and test
automation.

6. Company Confidential
Nokia Internal Use Only
6
INTRODUCTION
Difficulties:
• The Test Automation requires time, training and financial resources
to acquire tools .
• Constant maintenance of the test environment is needed.
• The development team do not
collaborate with test automation.

7. Company Confidential
Nokia Internal Use Only
7
Study Cases in Test Automation

8. Company Confidential
Nokia Internal Use Only
Study Case 1
8
 Strategy 1: Testers Automating Unit and System Tests :
 Team members: 4
 Testers decided to perform the unit testing tasks using
GoogleTest tool;
 Functional tests for the desktop installer and configuration
screen were automated using Sikuli tool ;

9. Company Confidential
Nokia Internal Use Only
 Strategy 2: Developers Automating Unit Tests and Testers
Automating System Tests
 Team members: 7
 Developers coding new features, unit testing using TDD
approach and Continuous Integration (Hudson).
 Testers automating UI Tests using Selenium.
Study Case 2
9

10. Company Confidential
Nokia Internal Use Only
 Strategy 3: Test team divided in Test team to analyze and test team
to automate tests.
 Team members: 8
 Developers just coding new features and unit tests.
 Test team to automate Integration tests, UI Tests using
Selenium, Load Tests and Security using JMeter.
 Test team to analyze and execute manual tests.
Study Case 3
10

11. Company Confidential
Nokia Internal Use Only
11
Results

12. Company Confidential
Nokia Internal Use Only
Study Case 1
12
 Positive and Negative points from Retrospective meetings:
Bugs found per story: 4

13. Company Confidential
Nokia Internal Use Only
 Positive and Negative points from Retrospective meetings:
Positive
Negative
Bugs found per story: 2,8
Positive Negative
 CI strategy improved testing
automation
 High coverage of GUI testing
automation
 Security and load tests
executed early
 Poor knowledge transferring
 Competition between testers and
developers
 High effort to automate tests with
record-and-play approach
 Test automation has been quit.
 Separation between testing and
development teams
Study Case 2
13

14. Company Confidential
Nokia Internal Use Only
 Positive and Negative points from Retrospective meetings:
Positive
Negative
Bugs found per story: 2.4
Positive Negative
 High coverage of GUI testing
automation
 Testers that code automated
scripts are closer to
development team
 Exploratory testing reveal
more important defects.
 Poor knowledge transferring
 High effort to automate tests with record-
and-play approach
 The developers give the responsibility for
quality to test team.
 Difficult synchronize the system changes
and the automated test suite.
 High effort to update test scripts.
 Test automation has been delayed
during some sprints.
 Separation between test analyst, testers
who automate scripts and development
teams
Study Case 3
14

15. Company Confidential
Nokia Internal Use Only
 Common problems
 Separated teams ;
 Separated work tools and environment infraestructure;
 No collaboration;
 Complex test tools and test environment;
 Centralized knowledge;
 The priority is time to delivery;
Results
15

16. Company Confidential
Nokia Internal Use Only
Results
All strategies have no co-operation between team project
16
Strategy 2
Strategy 3
Strategy 1
Lyndsay, 2006.
Source < http://www.mftrou.com/workplace-conflict-management-strategy.html>

17. Company Confidential
Nokia Internal Use Only
17
Test Automation Strategy
for Scrum Projects

18. Company Confidential
Nokia Internal Use Only
Test Automation Strategy for Scrum Projects
Iterative and Incremental:
• The test automation architecture should be developed and
updated in each iteration.
• The test scripts should be added to every story developed.
18

19. Company Confidential
Nokia Internal Use Only
Test Automation Strategy for Scrum Projects
Collaboration between team members:
• The test automation tasks are the responsibility of the whole
project team.
• Developers and testers should collaborate, interact and work
together in the activities of test automation.
• The test scripts should be added to every story developed.
19

20. Company Confidential
Nokia Internal Use Only
Test Automation Strategy for Scrum Projects
Encourage Collaboration:
• Co-location: scrum team must be work together in the same
place.
• Pair Programming: tester and programmers in pair to code
automated test scripts.
• Code and documents review.
• Get out of comfort zone
20

21. Company Confidential
Nokia Internal Use Only
Test Automation Strategy for Scrum Projects
Changing Control:
• The test tools must be under configuration management.
• The artifacts such as logs and access history and changes
should be available.
• Test automation under Continuous Integration.
21

22. Company Confidential
Nokia Internal Use Only
Test Automation Strategy for Scrum Projects
Process Improvement
• The evaluation of test automation related tools, tasks and
scripts must be made ​​at the meeting of the retrospective
• The lessons learned at this meeting are extracted and serve as
improvement for upcoming projects.
22

23. Company Confidential
Nokia Internal Use Only
Test Automation Strategy for Scrum Projects
The tasks can be shared.
23

24. Company Confidential
Nokia Internal Use Only
Test Automation Strategy for Scrum Projects
• Test tools under continuous integration.
• Test code under development tools and CI.
• Everyone in the project can generate builds and reports every
time.

25. Company Confidential
Nokia Internal Use Only
25
Execution and Results

26. Company Confidential
Nokia Internal Use Only
 Study Case 4: Developers and Testers Automating Unit Tests,
Integration Tests and System Tests
 Team members: 9.
 Developers and testers configured tools and programmed in
pair functional test scripts for unit tests, integration tests and
system tests;
 Tools: Junit and Fitnesse.
Execution and Results
26

27. Company Confidential
Nokia Internal Use Only
 Positive and Negative points from Retrospective meetings:
 Bugs per story: 1.42
Positive Negative
 CI facilitates testing automation.
 Security and load tests executed
early.
 Unit test review increased test
coverage.
 Intense team collaboration.
 Knowledge transferring regarding
the tools.
 Testing responsibilities distributed
to all team .
 Team motivation to learn new
solutions.
 High initial effort to set the
automation environment.
 Rework when customer requires
changes in the system.
Execution and Results
27

28. Company Confidential
Nokia Internal Use Only
28
Conclusion

29. Company Confidential
Nokia Internal Use Only
 Collaboration supports the Success of Agile Test Automation;
 Co-location facilitates collaboration;
 Automate Each Layer of Software When It Is Possible and Just for
Acceptance Tests in the Sprint;
 Test Automation in the Agile Project Should Be Simple, Reusable
and Maintainable for Project Team;
 Use Test Automation for Documentation and Information Feedback ;
Conclusions
29

30. Company Confidential
Nokia Internal Use Only
 It is feasible to adapt agile practices and values to test automation.
 The automation was a resource to document software, reduce cost,
and allocate tasks in smaller parts.
 The agile values allow the test automation maintenance.
 Next steps:
 Evaluate the test automation collaborative strategy as
framework;
 Evaluate the impact of team collaboration in distributed test
teams.
Conclusion
30

31. Company Confidential
Nokia Internal Use Only
Thank you! Merci! Gracias! Obrigada!
31
Contact: elianecollins@gmail.com
Blog: http://testautomationdiary.blogspot.com.br/