1. Load and Performance tests in Agile Scrum
framework
S Ravindra
S R V Subrahmaniam
Siemens Limited
An approach for complex product development context
Scrum Gathering India Regional 2013

2. Slide 2
Siemens
 Siemens a MNC with over 400000 people in more than 150 countries
 Siemens in India over 18000 people across locations
 Over 5000 people in software development in India
 Agile journey started in 2005
 Today more than 45% of the projects following agile scrum
 Basic Agile practices include: Time-boxed iterations, Continuous
integration, test early & Self-organized, cross-functional teams

3. Slide 3
Agenda
Context
Challenges
Discussion
Solution
Further actions

4. Slide 4
New Product development – project context for our presentation
Enterprise grade
Not a typical web
application
Niche product
(domain / target
users / use cases)
Critical Non-
Functional
requirements
Multiple deployment
environments
Centralized communication platform for use by emergency
intervention forces
Operators in the Command center will use it for
communication with public and on-field intervention forces
Safety critical application - used in Public safety and fire
emergency situations
Platform integrates Analog fixed, ISDN, Analog radio,
digital radio (Tetra)
Integration of heterogenous technologies

5. Slide 5
The Product : Communication platform
Touchscreen –
Thick client
application
Linux server Web server
IP PBX Digital & Analog Radio
Database
Gateways for different
types of incoming /
outgoing calls
Voice traffic carried over IP

6. Slide 6
Product Functionality
Client operator stations
Conference
Call forward
Make / Take calls
Send messages
Listen/talk in Radio
Hold/Unhold

7. Slide 7
Agenda
Context
Challenges
Discussion
Solution
Further actions

8. Slide 8
Uncertainty
over the
release dates
 Traditional waterfall
methodology
 Post code freeze, critical
Load and Performance test
defects are uncovered –
which sometime require fixes
with high impact
 The calendar duration for
Load and Perfomance testing
took 200% more time,
compared to initial estimate
 Technical challenges come to
the fore much later in the
project, when it is tested in a
close to real environment
Challenging environment
 Complex interplay of different
technologies including Java,
.NET, messaging platform over
VoIP /SIP.
 Dependencies with telecom
devices and equipments
 Mixing audio from various
streams and verifying for voice
quality
 Limited scope for test
automation (using conventional
tools) for this kind of a product
 Multiple deployment
environments
Technology Process
Uncertainty over Product quality. Management wanted to strengthen
the tests given the criticality of the application

9. Slide 9
Traditional delivery model
System test
 Design
 Coding
 Unit tests
 Functional tests
Requirements
Code freeze
Critical bugs identified
Related to:
Long run stability
Load and Performance
Environment
Heterogenous end-to-
end operations
Prolongs

10. Slide 10
Nature of Load and Performance defects pose a challenge
Defects with high turn-around time
Occurs during
long run
May require
architecture fixes
Bug verification –
again takes time
Fixes may have
functional impact
Hard to reproduce
Defect occurrance
does not follow a
pattern

11. Slide 11
Agenda
Context
Challenges
Discussion
Solution
Further actions

12. Slide 12
Agile Scrum is the right choice
Complex technology
New product & need for early
customer feedback
Positive Organizational mandate
Late testing needs to be avoided
Internal customer – less of contracting
hassles

13. Slide 13
Agile Scrum is fit for purpose
Niche
Generic
Simple /
Homegenous
Complex /
Heterogenous
Domain
Technical complexity

14. Slide 14
Key considerations in defining a methodology within Sprint framework
Thick client
application, more
susceptible to
stability issues
Used in emergency
call centers. Zero
tolerance for failure
Given nature of
application,
limited
automation
options
Device
integrations
are involved
Defect
turnaround time
high

15. Slide 15
Framing Agile testing for a Product in Safety critical environment
Sprint Test
Limited
feasibility
to do within
Sprint
Testing for
NFRs take
time
Accomodating the factors in the Agile test framework which is to be carried out
within a Sprint is the critical decision
NFRs need
to be
tested at
earliest

16. Slide 16
Sprint testing
IN Scope for Sprint Testing
 User story working at a functional
level
 User story working at an end-to-end
level
Horses for the courses
 Load and Performance tests
 Stability tests run over long duration in
multiple environments
 Complex concurrent workflows in a full
load scenario
 Exploratory tests
 Negative tests
Production-like environment
To be handled out of Sprint

17. Slide 17
Testing model practiced in the project
Sprint 1 Sprint 2
Functional tested package
Sprint 3
Dedicated
test phases
NFR test
The Sprint build is taken up for long running stability, load and performance tests,
multiple enviornment tests, exploratory tests and negative tests.
Defects
NFR test
.....
Four weeks
Product
Backlog

18. Slide 18
Testing methodology adopted
Sprint Testing
Enable test
Monitor resource utilization patterns; Focus on
tooling
Do tests where feasible – even if it is partial; Use
test stubs where needed
Plan the tests which need to be run; actively
collaborate with Product Owner
Bugs / tasks for forthcoming sprints
Early detection will enable certainty in product release schedules
Gather data
Perform test
Act on results

19. Slide 19
Test Organization
Within Scrum teams
- Testers part of the Scrum teams
- Operationally reporting to Scrum Master;
Officially reporting to Test Manager
- Apprisal, Competency development taken
care by Test Manager
- Testers need to be good in inter-personal skills
- Testers need to be aware of white box
techniques, interpretation of logs, configuring
and setting up parameters etc.,
System test team
- Testers do not belong to Scrum teams
- Testers report to Test Manager
- Work on testing the software at System level
and for non-functional requirements
- Diversity in terms of technical skills is key.
Each tester has to be a stand-alone expert in a
chosen technical area as the breadth of testing
would typically involve multiple areas

20. Slide 20
Other measures to foster Agile way of working
Management focus for moving towards Agile1
Visibility of the project status and key metrics for all stakeholders2
A possible dip in productivity initially and a rebound afterwards3
Facilitating the transition with suitable trainings and coaching4
Transition with new processes and tools, in tandem5
Co-location and office layout to reflect the new work culture6

21. Slide 21
Benefits accrued..1
Team
Product
Methodology
 Enhanced collaboration between the development and test teams
 Less stress compared to fixing critical issues, late in the project lifecycle
 Formulating Test organization for different test types
 Re-skilling and Competency development
 Riskier aspects of the Product tested upfront
 Design refactoring for performance and stability is a continuous process
 NFRs are given due focus - Enhanced visibility of the NFR related test cases
to development and product teams
 Product is as close to being production ready
 Clear separation between the different levels of testing
 NFR Tests planned and executed with due rigour without getting influenced
by Sprint dynamics
 Will support a move towards TDD
Compliance to schedule and lesser critical bugs after code freeze

22. Slide 22
Defect turnaround rate improved by 20%
50% defects found early – i.e. During the Sprint testing
NFR test cycles reduced by 25%
15% less of critical and major defects
Benefits accrued..2





23. Slide 23
Agenda
Context
Challenges
Discussion
Solution
Further actions

24. Slide 24
Next improvements – to our project
Continous review
and
improvement of
tools &
techniques
 Invest in developing an
automation framework at
end-to-end and
component Integration
levels
 Use of White-box tools and techniques for Load and
Performance tests
 Have specific user stories to improve performance
Enhance
Automation
Sprint development

25. Slide 25
Agenda
Context
Challenges
Discussion
Solution
Further actions

26. S. Ravindra (sm.ravindra@gmail.com)
S R V Subrahmaniam (srv.subbu@yahoo.in)
Thanks for your attention