2. Introduction
- Background : Claims & realities
- Quality Assurance in Agile
Motivation to explore quality assurance study in the Agile field
Research Gaps
Research Objective
Methodology
Case Study results
Future work & recommendations
Q&A
3. In the survey conducted by Ambler (2006), results have reported
66% increased in quality that has adopted one or more agile
development techniques.
Reduction in defect rates of 15% to 50%, using pair programming
(Cockburn et al, 2000)
Extreme Programming (XP) effect reported 50% increase in
productivity, a 65% improvement in pre-release quality and 35% in
post-release quality (Layman et al. 2004).
Pre-release defect density , measured as defects per thousand lines
of code, decreased between 40% and 90% relative to the projects
that did not use TDD. (Chris, 2009)
3
4. Recent survey conducted by versionone indicates that :
There continues to be an increasing % of respondents who do not know
the benefits of agile, or have not realized substantial benefits, especially
in the areas of managing distributed teams and cost reduction.
Agile enthusiasts claim significant increases in the quality of their
software while detractor cite instances where rapid development and
loose structure lead to decreases in quality. This happens because not
all ‘agile’ is created equally. Some agile practices are more likely, when
implemented correctly, to impact quality than others.
Organizations still struggling to balance between waterfall and agile
mindset to utilize the benefits claimed by agile.
4
5. Agile
Agile QA
adoption
Market Market
Qualifiers Winners
Agile Quality Service
Methods Cost Level
Lead Time
6. • Deliver the wrong functionality.
• Build to fragile design
• Hand offs inject defects
• Fixing defects is expensive
• Increased time to value
Traditional
• Greater ability to deliver required
Analysis Agile
functionality Design
• Active stakeholder participation Code
• Greater Discipline Testing &
• Shorter work cycle – More Value Verify
• Greater quality
7. Agile QA Plan driven /Waterfall QA
Who is responsible for All the development The QA team
software quality team members
When quality related All the time, quality is At the QA/testing stage (e.g.
topics are addressed one of the primary milestone declarations)
concerns of the
development process
Quality related Same as other Visibility is low, less frequent
activities status activities than other activities
Work style Collaboration with all Developers and QA people
parties might have conflicting
interests
8. Research Gaps
Studies done in past have revealed improvement in quality with agile
methods but not much dedicated study observed with regard to what
are the quality issues/challenges faced within Agile organizations and
how to overcome them.
Existing studies have revealed individual impact of one or other factor
on quality eg. TDD, pair programming, however what is the
relative/cumulative effect of these factors/practices on overall quality
needs to studied.
Lack of quality assurance framework in agile to help organizations for
implementing QA within agile to ensure sustainability.
Current studies are more based on practitioners report, however
research based empirical study needs still to be explored in this area.
8
9. Existing Challenges
Different levels of quality with different project agility levels and not
significant improvement visible.
Non functional requirements were ignored/not tested properly.
Pair Programming considered to decrease productivity.
Refactoring – How much is enough ? “Better don’t touch the code as it
might break existing functionality.”
Is it the quality of the code or the quality of the developers?
Mindset change between developers and testers impacting the delivery &
quality of software.
9
10. Problem question
How to ensure sustainability of quality improvement benefits
realized from agile practices ?
How particular context aspect (e.g. organizational culture,
organizational distribution, organizational maturity, or maturity of
the client’s organization) affect the potential of agile practices to
create value & quality software ?
What are product or process measures to evaluate Quality in agile
software mainly from customer point of view ?
How does different levels of implementing agile practice(s) or
agility impacts quality ?
10
11. To expand the existing study identifying the key enablers for sustaining
quality in Agile, using empirical analysis.
To identify critical links between conditions of success and reasons of
failure to achieve high quality Agile software.
Study how product quality impacts further cost reduction in terms of
reduction in Cost of poor Quality and improved business value in agile
environment
Propose Agile quality assurance framework that can guide organization
to establish effective quality assurance practices using Agile
development methodologies.
11
12. Research Methodology
Interviewed the stakeholders to identify how they perceive quality in agile
& probable factors influencing quality to base the further study.
Questionnaire based web survey designed.
Interpretive structural modeling (ISM) methodology to be applied to
develop contextual relationship among the control variables identified to
influence quality. Analytical Hierarchy Process (AHP) can then be used to
quantify relationships and weigh the significance of different factors
identified to impact quality.
Data will be put to statistical analysis using Structural equation modeling
(SEM) to derive the conclusions.
12
13. Most team members did not share the same understanding of
the concept of quality :
Few mentioned reduced
customer defects/defect
Improve in customer
density is a criterion for
satisfaction score
measuring or perceiving
quality
Delivering feature faster
as per the requirement
specification
13
14. But, If a team delivered defect free software , that is not used/valued
by the customer & delayed , they would not achieve the overall
iteration/release goals .
Quality would be perceived as low by customer !!
14
15. We identified what are the key factors that contribute to defining positive
perception of quality .
Intrinsic Quality Factors Extrinsic Quality Factors
Improved Code
Reduced Feature Cycle time
Quality
Reduced Technical
Better Customer Satisfaction Index
Debt
Green Build Reduced Cost of Poor Quality
Increased ability to meet with the
Reduced Defects
current customer requirements
100 % Requirement Increased flexibility to meet with
Coverage changing customer requirements
15
16.
17. Agile Core Characteristics Agile Influence on Quality
+ + Productivity/
- Customer Involvement - Defect Reduction
- Test Early and often - Early Defects Detection Cost
- Shorter Feedback - Cycle Time Improvement Reduction
- Prioritized Requirements - Code Quality
+
Control Variables Business Value
• Scrum Practices
• Continuous Integration
• Refactoring
• Experience of Team Members
• Geographical /Team distribution
• Done compliance
• Condition of Satisfaction
• Test Driven Development (TDD)
• Acceptance Test Driven Development (ATDD)
• Test coverage
• Sprint Commitment
17
18. Compliance requirement
Low risk Critical,
Audited
Geographical distribution Entrenched process,
people, and policy
Co-located Global
Minimal Significant
Agile
Development QA
Organization distribution
Application complexity (outsourcing, partnerships)
Simple, Complex,
single In-house Third party
multi-platform
platform
Team size Degree of Governance
Under 10 100’s of
Informal Formal
developers developers
19. 2. What do agile team members identify as the main
factors impacting on quality ?(+/-)
Development
Organization
Factors
Factors
Team H5(+) H1(+) Customer
Distribution/ Software Collaboration
Co-located Quality
H6(+/-
Assurance H2(+) Decision Time
Management )
In Agile
Culture H7(+/-
) H3(-)
Team Size Product
H8(+) Complexity
H4(+)
H9(+) H10(-)
Planning & Legacy Code
Control Base
Competence & Developer &
Learning Tester Mindset
19
22. Agile approaches Old ideas about
are changing the testing at the end
conversation about of the coding phase
software no longer
development applicable
Agile shifted our
Testers need to join
attention to small
in the conversations
teams incrementally
with developers and
delivering quality
users
software
23. Agile Quality Assurance Strategy
Goal/Objective: To analyze the Impact of Agile on Product Quality & identify the key Agile Quality Measures
Built-in Quality Visible Value Customer
Measures Creation Satisfaction
Stakeholder
Perspective Agile & Lean Principles
“What measurable impacts must we achieve to satisfy our stakeholders?”
Fault Reduction
Agile
Code Quality
Score Card
Metrics
Delivered Defect Density
Business Value
Green Build Success %
Cycle Time
Customer Loyalty Index
Outage Reduction
Fault Correction Response Time
Definition of Test Early & Daily Build & Fault Enhanced
Quality
Perspective
Done
Quality
+ Often
+ Integrating
often
+ detection &
fixing early
= Quality
(Hypothesis)
Improved Quality (Built-in)
“In order to achieve our desired quality, what Agile Practices we must do or enhance?”
Internal Process “What are the Quality concerns you have in Agile Mode of Operation? In order to satisfy our customers, what
Perspective Quality parameters /processes must we excel at and how?”
Study perception using Questionnaire
STRATEGIC Promote Define Recognize
THRUST Performance Excellence Performance Measures Performance Achieved
24. • Aims to provide software organizations deeper understanding about what
factors are important to sustain the quality benefits realized from in agile.
• Evaluating critical links between various success factors in
Agile development and their impact on achieving high quality
software
● Quality assurance framework in Agile that will help
organizations to drive towards achieving excellence with
improved software quality. .
What’s Next Data Collection is ongoing to validate the model
with large scale projects & study the contextual
factors influence on software quality
