Real case studies of QA management in big teams (60-100 people). How to setup robust QA processes and approaches in them. Main impediments and problems, how to solve them. SAFe.

1. QA Management in BIG Agile teams
Volodymyr Prymakov

2. Speaker info
Volodymyr Prymakov, Ukraine
Senior QA Manager at Ciklum, TCoE
Head of Performance QA Unit at TCoE
•14 years in QA
• 45 projects experience
• Certified: ISTQB Advanced Test Analyst &
Manager, ICAgile CP, SAFE
/in/vladimirprimakov/

3. Manual QA
Through manual
comparison of the
delivered product with
the requirements
Automation QA
Automation of
routine testing
activities to effectively
use time & resources
Performance QA
Verification of
stability and
performance of the
application
Security QA
Searching for
vulnerabilities in the
app to insure it’s safe
and secure for users
QA Audit and Consulting
Analysis of the QA Tools used, Processes and Approach to suggest and implement the best practices through
offering an efficient team composition and toolset to build up a comprehensive QA solution for use by the Client
Testing Center of Excellence:

4. Testing Center of Excellence:

5. Agenda
1. Big Team, What is it?
2. Process (Development and QA)
3. Infrastructure
4. Collaboration
5. Transparency
6. QA Team organization
7. Automation
8. Questions

6. What is a big team?

7. What is a big team?
Team

8. What is a big team?
Team Big team? - Expected!

9. What is a big team?
Team Big team? in Reality….

10. What is a big team?
Most of the problems in a big team
relates to
cross-team work and
common dependencies

11. Big agile team is a team of >50 people
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
1 PO
1 Tech Lead
5 Developers
2 QAs
PO Manager
Dev Manager
Architectors
QA Manager, Leads
SMSMSM
Program Manager
Release Manager
Dev. Teams
Leadership Team Release train

12. Products
1. Big (A lot of functionality or sub-products)
1. Complex (Architecture and infrastructure)
1. (Quite often) Monolete architecture or
interdependable components
1. A lot internal and external integrations.
2. A lot of functionality interdependencies.
High Regression risk!
3. Many end-users (sometimes, in many
countries)

13. PROCESS

14. Process - Overlapped Releases
Supported by the same teams
MASTER BRANCH
STABLE BRANCH

15. Process - Overlapped Releases
Supported by the same teams
STABLE BRANCH
MASTER BRANCH

16. Process - Overlapped Releases
Supported by the same teams
MASTER BRANCH
Bug HotfixesBug Hotfixes
STABLE BRANCH

17. Process - Overlapped Releases
Supported by the same teams
MASTER BRANCH
Bug HotfixesBug Hotfixes
STABLE BRANCHRegression ProblemsRegression Problems

18. Process - Overlapped Releases
!!!Delay because of merging,
regression and other problems
Supported by the same teams

19. Process - Overlapped Releases
!!!Delay because of merging,
regression and other problems
Supported by the same teams
Overlappingactivities

20. Process - Overlapped Releases
!!!More Delay
Supported by the same teams
Overlappingactivities
Overlappingactivities

21. Process - Overlapped Releases
!!!More Delay
SHIFTED RELEASES
Supported by the same teams
Overlappingactivities
Overlappingactivities

22. Process - Overlapped Releases
Supported by the same teams
PROBLEMS:
● No Buffer for human factor and
unpredictable problems

23. Process - Overlapped Releases
Supported by the same teams
PROBLEMS:
● No Buffer for human factor and
unpredictable problems
● Overlapping activities
● Big context switching

24. Process - Overlapped Releases
Supported by the same teams
PROBLEMS:
● No Buffer for human factor and
unpredictable problems
● Overlapping activities
● Big context switching
● Complicated branching
strategy and risky hot-fixing
process
● Not compatible with
Continuous Delivery

25. Alternative Process - Overlapped Releases
Regression and release support by Another team
PROBLEMS:
● Lack of new functionality
(implementation) knowledge
● Knowledge Transfer required
● Not enough expertise for bug
fixing
● Extra Collaboration needed
● Not Enough Capacity
● Motivation problem
System Dev & QA Team

26. STRAITFORWARD Release Pipeline
MASTER BRANCH
NOW

27. STRAITFORWARD Release Pipeline
MASTER BRANCH
FUTURE

28. STRAITFORWARD Release Pipeline
BENEFITS:
● SIMPLE and STRAIGHTFORWARD
● NO WASTE of TIME FOR
KNOWLEDGE TRANSFER
● NO CONTEXT SWITCHING
● RELEASES EVERY 2 WEEKS
● COMPATIBLE with CONTINUOUS
DELIVERY

29. Process Effect on Quality
NEW PROCESS
Bug Rate
Story
Points
Delivered
OLD PROCESS
STILL GOOD VELOCITY
30-40% LESS BUGS

30. STABILIZATION Period
● No Feature Merging in
master or release
branch
● Only Blocker and
Critical bugs fixing
● Regression testing
● Bug Retest.

31. Release management
FOR OTHER PRODUCTS
Monitor Release
Weight Impact
Prepare in advance
Test and Retest it!
Releases for Integrated Systems (internal and 3-rd parties)

32. Late Code Merging
Late code merging leads to bugs, no time
for testing and release delays!

33. Late Code Merging
To Avoid Merging Problems:
● Size stories optimally

34. Late Code Merging
To Avoid Merging Problems:
● Size stories optimally
● Merge as often as possible

35. Late Code Merging
To Avoid Merging Problems:
● Size stories optimally
● Merge as often as possible
● Early Deliver and Test (Automatically)

36. Late Code Merging
To Avoid Merging Problems:
● Size stories optimally
● Merge as often as possible
● Early Deliver and Test (Automatically)
● Deadline for merging + buffer for testing

37. Late Code Merging
To Avoid Merging Problems:
● Size stories optimally
● Merge as often as possible
● Early Deliver and Test (Automatically)
● Deadline for merging + buffer for testing
● Communicate, manage, and revert risky merges.

38. Regression testing approach (e.g.)
Country 2
Country 1
Country 3
Country 4
Markets

39. Regression testing approach (e.g.)
Fast
Feedback
about product
quality state.
Scope:
P1 Test cases
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1
Mostly p1, p2
bugs

40. Regression testing approach (e.g.)
Fast
Feedback
about product
quality state.
Scope:
P1 Test cases
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1 Regression Cycle 2
Cross-Platform
Compatibility
testing
Scope:
Exploratory
testing
Mostly p1, p2
bugs
Mostly p3 or lower
priority bugs

41. Regression testing approach (e.g.)
Fast
Feedback
about product
quality state.
Scope:
P1 Test cases
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1 Regression Cycle 2
Cross-Platform
Compatibility
testing
Scope:
Exploratory
testing
Regression Cycle 3
Validate
Quality after
Bug-fixes
verifications
Scope:
P1, P2 test
cases
Mostly p1, p2
bugs
Mostly p3 or lower
priority bugs
p1, p2, p3 or lower
priority bugs

42. Regression testing approach (e.g.)
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1 Regression Cycle 2
All other
platforms
Regression Cycle 3
Platform 2
(Chrome)
Platform 1 (IE)
Platform 1 (IE)
Platform 2
(Chrome)
Platform 1 (IE)
Platform 2
(Chrome)
Platform 2
(Chrome)
Platform 2 (IE)
Cross Platform Testing approach - Sprint N

43. Regression testing approach (e.g.)
Country 2
Country 1
Country 3
Country 4
Markets Regression Cycle 1 Regression Cycle 2
All other
platforms
Regression Cycle 3
Platform 1 (IE)
Platform 2
(Chrome)
Platform 2
(Chrome)
Platform 1 (IE)
Platform 2
(Chrome)
Platform 1 (IE)
Platform 1 (IE)
Platform 2
(Chrome)
Cross Platform Testing approach - Sprint N + 1

44. Regression testing approach (e.g.)
Type of Test Scope Outcome
Smoke End-to-end flow (most
popular booking)
Blocker bugs
Acceptance Alternative End-to-End
flows
P1 and P2 bugs
UAT User Acceptance Test
scenarios
P1, P2, P3
Some bugs may be missed
Regression P1 P1 test cases P1 and P2 bugs
Regression P2 P2 test cases P3 bugs
Exploratory Exploratory test cases P1, P2, P3
Testing Order!

45. System Intergration Testing
● Obligatory test integrations with
real life internal and 3d party
services
● Do confirmation testing for the
rest of functionality if needed
(Exploratory, Cross-platforms, or
more wider testing)

46. Cross team Bug Escalation
Priority
p3, p4, less
QA Coordinator /
Release Coordinator
Product
Backlog

47. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
QA Coordinator /
Release Coordinator
Product
Backlog
Assign to current FixVersionTESTER

48. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
QA Coordinator /
Release Coordinator
Product
Backlog
Assign to current FixVersionTESTER
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!Is this bug produced by the team?
Does the team has expertise to fast fix it?
Or does the team has capacity for bug fixing?

49. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER

50. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersion
Which team produced the bug?
Which team has better expertise to fix it?
Which team has more capacity for bug fixing?
TESTER

51. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER

52. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS
Which team has more capacity for bug
fixing?

53. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS

54. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS
POs
Which team has lower priority functionality
which can wait for the sake of bug fixing?

55. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS
POs

56. Cross team Bug Escalation
Priority
p3, p4, less
Priority
p1, p2
TEAM
TECH LEADS
QA Coordinator /
Release Coordinator
ASSIGN to A CERTAIN TEAM
Assign to Developer and Resolve Quickly!
Product
Backlog
Assign to current FixVersionTESTER
SCRUM MASTERS
POs
FACILITATE

57. Align QA approaches across teams

58. Align QA approaches across teams
• DoD for test cases
• DoD for Automation scripts
• Test suite Structure
• Test plan and run structure in test management
system
• Regression approach
• Bug Reporting
• Release readiness reporting
• and many others

59. Agreeing project
process/approach
Agree cross-team
approaches/processes on all levels

60. Agreeing project
process/approach
Agree cross-team
approaches/processes on all levels
Involve all concerned parties in the
discussions

61. Agreeing project
process/approach
Agree cross-team
approaches/processes on all levels
Involve all concerned parties in the
discussions
Align cross-team procedures

62. Agreeing project
process/approach
Agree cross-team
approaches/processes on all levels
Be tolerant and persistent
Involve all concerned parties in the
discussions
Align cross-team procedures

63. Agreeing project
process/approach
Agree cross-team
approaches/processes on all levels
Be tolerant and persistent
Document agreements
Involve all concerned parties in the
discussions
Align cross-team procedures

64. Infrastructure

65. Infrastructure problems
2h of infrastructure
downtime for the team in
size of 50 Devs and QAs
may cost 12.5 man days…
(several thousands USD)

66. Test Environments Requirements
SIMILAR to
PRODUCTION
AUTOMATED and
CONTROLLED
FAST and
POWERFUL
LESS
INTERDEPENDENT
ROBUST and
STABLE

67. Typical Infrastructure/environment
problems
Problem
Essential difference between environments (configuration,
integrations, data) leading to leaks of bugs to production
Uncontrolled changes and manual interventions
Configuration problems.
Slow and overloaded environment. Usage conflicts.
Integration issues (external and internal).
Services and DB compatibility.
Unreliable or absent test data.
Unpredictable downtimes. Connectivity and access issues.
Solution
Minimize the difference. At lease 1 test environment should
be 99% identical to production.
Automate Everything. Apply change control procedure. Limit
unauthorized access. Notify about changes in advance!
Increase Capacity. Build several test environments.
Don’t save on the environment. Use more powerful network.
Controlled 3d party releases. Component decoupling. Mocks
usage. Backward compatibility.
Mocking Data, Standard data sets usage, Agreeing test data
provision from 3-d parties, etc.
Plan B: Backups, alternative suppliers etc.

68. Environments (e.g.)
Testing Owners
Integrations
Hardware
capacity
DB

69. Devops should be part of teams
● ~1 Devop for 2 teams
● Open communication. Participation at
daily scrums and other sync-ups
● Proactive resolution of ongoing
infrastructure and environment
problems
● Devops backlog prioritization for sake
of teams’ needs

70. Cross team
Collaboration

71. Strategic Planning
Plans are hidden from teams

72. Strategic Planning

73. Strategic Planning
Strategic planning helps to build the right product
Product increment and other initiatives for a Quarter/Half-a-year

74. Sprint Planning (Devs)
Teams
Grooming

75. Sprint Planning (Devs)
Teams Cross-Team
Grooming Tech-leads
Preplanning

76. Sprint Planning (Devs)
Teams Cross-Team
Grooming
Common Risks
and
Dependencies
Identified
Tech-leads
Preplanning

77. Sprint Planning (Devs)
Teams Cross-Team Teams
Grooming Sprint
Planning
(adjusted)
Common Risks
and
Dependencies
Identified
Tech-leads
Preplanning

78. Sprint Planning (Devs)
Teams Cross-Team Teams
Grooming Sprint
Planning
(adjusted)
Common Risks
and
Dependencies
Identified
Sprint
Start
Tech-leads
Preplanning

79. Sprint Planning (QAs)
Cross-Team
QA Leads
Preplanning
Sprint Planning (QAs)

80. Sprint Planning (QAs)
Cross-Team
Additional sprint
testing scope
identified
QA Leads
Preplanning

81. Sprint Planning (QAs)
Cross-Team Teams
Sprint
Planning
(adjusted)
Additional sprint
testing scope
identified
QA Leads
Preplanning

82. Sprint Planning (QAs)
Cross-Team Teams
Sprint
Planning
(adjusted)
Additional sprint
testing scope
identified
Sprint
Start
QA Leads
Preplanning

83. Sprint Planning (QAs)
Cross-Team Teams
QA Leads
Preplanning
Sprint
Planning
(adjusted)
Additional sprint
testing scope
identified
Sprint
Start
Cross-Team
QA Sprint
Scope
Review
Next Slide =>

84. QA Sprint Scope Review & Planning
Participants
QA Manager,
QA Leads,
1 QA from
every team

85. QA Sprint Scope Review & Planning
Participants Goals
1. Review Sprint scope (Features)
2. Define Dependencies, risks, and impact
3. Adjust scope and type of testing
4. Define additional testing activities
5. Identify impact on automation and
performance scripts
QA Manager,
QA Leads,
1 QA from
every team

86. QA Sprint Scope Review & Planning
Participants Goals
1. Review Sprint scope (Features)
2. Define Dependencies, risks, and impact
3. Adjust scope and type of testing
4. Define additional testing activities
5. Identify impact on automation and
performance scripts
-------------------------------------------------
1. Discuss QA Automation coverage
increment and other achievements
2. QA automation scope adjusting
QA Manager,
QA Leads,
1 QA from
every team

87. QA Sprint Scope Review - Outcomes

88. Regular Team & Scrums
Teams Cross-Team
Scrum
Scrum of
Scrums
● Cross-team
Dependencies,
impediments, and help
needed
● Environmental Issues
● Overall Sprint Scope
covering (once)
● Are we ready for Code
Freeze?
● Discussing release
stoppers: Blocker and
critical bugs
Auditory:
Release Manager,
Scrum Masters,
Tech Leads,
QA Leads,
Devops Lead

89. Regular Cross Team Syncups
Dev Manager,
QA Manager,
Tech Leads,
DevOps Leads,
Release Coordinator
● Strategic Plans,
● Common
Approaches,
● Important Ongoing
Activities,
● Other Common
Questions and
Problems

90. Readiness Checkpoints
Ready for Regression?
Ready for Stage Testing?
Ready for UAT by
Business?
Ready For Production?
Regularly Sync-ups
on Product Readiness!
Release Manager,
Scum Masters,
PO Manager,
QA Leads/QA Coordinator,
(POs),
(Tech Leads)

91. Lessons Learned
Teams
Team
Retro
Sprint/Release
Finishes

92. Lessons Learned
Teams Cross-Team
Team
Retro
QAs
Retro
Sprint/Release
Finishes

93. Lessons Learned
Teams Cross-Team
Team
Retro
QAs
Retro
Release
Retro
Sprint/Release
Finishes
Cross-Team

94. Knowledge sharing
Spread Knowledge among the project!
!!!Record videos and/or document the stuff
● Business Domain
● Product Functionality
● Overall Architecture
● Component Technology
● 3-d party components
● Common approaches, etc.

95. Minimize frequency of sync-ups
Meet as often
as it benefits,
but do not
overdo

96. Minimize frequency of sync-ups
Meet as often
as it benefits,
but do not
overdo
!!!Avoid long
meetings at
essential
sprint/release
phases

97. Transparency

98. Scope understanding
Initiative/Epic level
Know and
align with
the
project
roadmap

99. Scope understanding
Sprint/Release level
Use Fix Version for
features and bugs
in advance to
track overall
release scope
(cross-team)

100. Realtime quality boards
Monitor Quality on daily bases or more often
1. Dynamic Jira Dashboards 2. Kanban Boards for Blocker and Critical bugs

101. Testing reporting
1. Report/Escalate on Blocker and
Critical bugs by a necessity.
2. Report regression testing results
on daily bases.
3. Involve all concerned parties in
the report.
4. Build quality
awareness/transparency on the
project!

102. Quality trends
Monitor and analyze Quality trends.
Apply Corrective actions if needed.

103. QA Team

104. TypicalQATeamStructure
QA Manager
QA Manual
Lead
Team 1
Senior/Middle
QA
Middle/Junior
QA
Team 2
Senior/Middle
QA
Middle/Junior
QA
Team 3
Senior/Middle
QA
Middle/Junior
QA
QA Manual
Lead
Team 4
Senior/Middle
QA
Middle
QA/Junior QA
Team 5
Senior/Middle
QA
Middle/Junior
QA
Team 6
Senior/Middle
QA
Middle/Junior
QA
QA Manual
Lead
Team 7
Senior/Middle
QA
Middle/Junior
QA
Team 8
Senior/Middle
QA
Middle/Junior
QA
Team 9
Senior/Middle
QA
Middle/Junior
QA
QA Automation
Lead
Core Automation
Team
Senior/Middle
Auto QA
Middle Auto
QA
Middle/Junior
Auto QA

105. QA Team Typical Roles
Defines and aligns QA Approaches on the project,
Strategic Planning and Reporting,
Monitor and adjust ongoing QA activities,
Resource planning, recruitment, competences, etc.
Escalation questions
QA
Manager
QA Approaches implementation,
Cross team ongoing QA work planning and coordination in a sprint (release),
Regular QA reporting,
People management in their teams (Mood, PDPs), etc.
QA Lead
Building and aligning QA automation approaches and expertize on the project,
Organizing the corresponding Knowledge Sharing,
Leading automation Framework development and support,
Automation team management.
QA
Automation
Lead

106. QA Component Lead role
1. Main knowledge holder / expert in the product area.
2. Responsible for the quality of the product area. Regularly
monitor its quality. Escalate problems if needed.
3. Define dependencies and quality risks in the area. Inform
others about them.
1. Defines manual and automation testing scope for the product
area.
2. Monitor and analyze automation testing results for the
product area. Execute automation tests if needed.
1. Plans and organizes regression and other kind of testing for
the product area.

107. QA Coordinator Role
1. Organize QA syncups and planning meetings.
2. Plans and organizes cross team testing activities
a. regression testing.
b. testing at production.
c. etc.
3. Monitor overall product quality state.
4. Escalate Blocker and Critical bugs if needed.
5. Prepare and send quality reports.
6. Play a representative role in release readiness
meetings.
Coordinate cross-team testing activities in a release.
Cycle the role
between QA leads.

108. Delegation on All levels
Other roles:
● QA Product Market Lead
● Platform statistics
management
● Devices Management
● A/B Testing management
● Payment Card and refund
Management
● Etc.

109. Automation

110. Why Automation?
Cross team
work
Regression Risk Automation
prevents the risk

111. Requirements to QA automation
All Levels Correct order Cross Platform

112. Requirements to QA automation
Sufficient
Coverage
Stable
and
Trustful
High Speed

113. Requirements to QA automation
Frequent
Executed on all
environments

114. Aligned with Manual QA
Manual QAAutomation QA

115. Aligned with Manual QA
Manual QAAutomation QA
Extend Coverage

116. Aligned with Manual QA
Integration with Test Management system

117. Non-Functional Automation

118. Parting Words

119. Questions