For organizations developing large-scale applications, transitioning to agile is challenging enough. If your organization has not yet adopted an automation culture, brace yourself for a big surprise because automation is essential to agile success. From the safety nets provided by automated unit and acceptance tests to the automation of build, build verification, and deployment processes, the iterative nature of agile demands a culture of automation across your engineering organization. Geoff Meyer shares lessons learned in adopting a test automation culture as the Dell Enterprise Systems Group simultaneously adopted Scrum and agile processes across its entire software product portfolio. Learn to address the practical challenges of establishing an automation culture at the outset by ensuring that your organizational makeover incorporates changes to your hiring, staffing, and training practices. Find out how you can apply automation beyond the Scrum team in areas including continuous integration, scale and stress testing, and performance testing.

1. W16
Agile Testing
5/1/2013 3:00:00 PM
Automation Culture: Essential to
Agile Success
Presented by:
Geoff Meyer
Dell, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Geoff Meyer
A test architect in the Dell Enterprise Solutions Group, Geoff Meyer has more than twenty-seven years of
experience as a software developer, manager, test architect, and business analyst. Geoff co-chairs the
Agile Steering committee within Dell Enterprise Solutions Group which guides the software development
practices of more than 600 development, test, and UX engineers across three Global Design Centers. He
is an active member of the Agile Austin community.

3. 4/16/2013
Automation Culture:
Essential to Agile Success
STAR East 2012
Geoff Meyer, geoffrey_meyer@dell.com
24 April 2013
Last updated: February 28, 2013
Session Objectives
•
•
Challenges automating in a HW-dependent
environment
•
NOT
Real-world examples − from a large organization
perspective
•
IS
Practical Solutions to What, When and What Not
to Automate
•
IS
Challenges to expect when adopting Test
automation during your transition to Agile
For Organizational Leaders driving Strategy
•
A Tools Discussion
•
Focused on how to develop Test Automation
1

4. 4/16/2013
Introductions
3
Geoff Meyer
• Dell Inc, 1998 – present
– Test Architect
› Agile Test & Automation Strategy
– Agile Steering Committee Co-Chair
– Global Projects
› 15 Scrum teams
› 9 scrum teams
› 7 Scrum teams
• NCR Corp. 1984 – 1998
– SW developer, Project Lead, SW Manager
• B.S. Computer Science, San Diego State University
• Masters Engineering Management - NTU
4
2

5. 4/16/2013
Agenda
• Why Automation is essential to Agile
• The Dell Landscape and Agile @ Dell
– Organizational Context Matters
• Challenges and Common Pitfalls
• The Automation Landscape
• Foundations of an Automation Culture
• Care and Feeding of the Automation Culture
5
The Need,
The Challenge,
The Pitfalls
6
3

6. 4/16/2013
Why is Automation So Important in Agile?
• Near-term
– Ensures that you don’t break what you just built
– Provides safety net for developers & rapid feedback to new changes
– Continuous Integration and use of Build Verification Test (BVT)
• Long-term
– Maximizes velocity of Scrum team
– Creates capacity for Exploratory and ad-hoc Testing
– Enables activities that can’t be done cost-effectively by humans
And if you don’t…
Projects rapidly build up an unmanageable regression suite
Differences with Automation in Waterfall?
In Waterfall…
• Automated tests are derived from the backlog of
completed features
– In Agile, Automation can be incorporated in the requirement
• Testing and automation is performed after Development
is complete
• Focus is on first-time discovery of defects and
optimizing your test coverage
vs. Agile… where automation provides immediate
feedback to defect introduction
8
4

7. 4/16/2013
The Challenge
Adopting Agile…
…In an organization that doesn’t have an
Automation Culture
“Apply Test Automation in the
context of your organizational
automation needs”
--- Bob Galen, iControl
The
Context @ Dell
http://amazngwallpapers.blogspot.com
10
5

8. 4/16/2013
Dell Enterprise Solutions Group
Global Design and Development using Agile Scrum
Silicon Valley
Design Center
Noida Design
Center
Austin
Design Center
Bangalore
Design Center
11
Dell Enterprise Solutions
Software Products
• Server Systems Management
• Converged Infrastructure Systems Management
• Private Cloud Server Management
• Console Plug-ins (i.e. for SCCM, vCenter…)
Common Product Characteristics:
•
Large hardware support test matrix
•
Software is installed the Data Center
•
Enterprise update cycles ~6 months
•
Products must function even as underlying FW, BIOS,
and Drivers are upgraded - SUSTAINING
12
6

9. 4/16/2013
Agile @ Dell Roles/Responsibilities
Product Management
Epic Prioritization & Business Value
Represents Customer - Profiles/Roles
Requirement Acceptance
Product
Owner
User Story Development
Customer rep. for Scrum team
Story Acceptance
Product Owner Proxy
1 per Scrum
Scrum Master
Facilitate, Remove Scrum impediments
1 per Scrum
Development
Architecture, Design, Development, Unit Test
4-5 per Scrum
Test
Test Design, Automation Development
2:1
InfoDev
User-facing text, error messages, online help
1 per 3 scrum team
UI
UI Design, Development, Unit Test
1 per Scrum
Scrum teams
13
Agile @ Dell with Adaptations
Feature
Complete
Release
Exit
Code Freeze
Release Plan
Sprints
1
1
2
2
1
2
3
3
…
3
N-1
N-1
N
N
N-1
…
N
Hardening
Stability
Extended Sprint Test
Software System Test
PRP
Define
Plan
Develop
Launch
14
7

10. 4/16/2013
Pitfalls encountered at
• Development didn’t historically automate unit tests
• Build teams were staffed with non-Build practitioners
• Minimal guidance beyond “Go forth and automate”
• Architecture(s) not optimized
for Automatability
• Automation was interpreted
by many as “Automate the UI”
• Insufficient SW engineering
background across Test
teams
15
Which Resulted In…
• Automation not keeping up within the sprint
• Feature Devotion
• Multiple automation tools & licenses
• Test scripts not designed for re-use
16
8

11. 4/16/2013
The Automation
Landscape
http://amazngwallpapers.blogspot.com
17
Opportunities for Automation
• It’s not just about Test cases
• Can provide efficiencies to:
–Test Preparation, Setup and Configuration
–Test Content Development and Execution
› Continuous Integration & Automated BVT
› Acceptance Testing
› Test Matrix coverage
–Non-Functional Testing
› Longevity, Scale and Performance Characterization
18
9

12. 4/16/2013
Test Preparation
• Setup & Configuration
– OS Provisioning and Configuration
• Deployment
• Test case staging
• Environment Cleanup/Baseline
• Virtualization-based test environments
19
Test Content Development
Whole team ownership of QA & Automation
• UI automation on Customer
Usage workflows
• Automated CI (UT, Build,
BVT)
• Web-services (or CLI) Test
Automation for Functional
Testing
• In-Sprint, Automated
Acceptance Tests
• Unit Test Automation
1
CAUTION: Application Architecture can be an enabler or inhibitor
1 - Agile Testing: A Practical Guide for Testers and Agile teams
20
10

13. 4/16/2013
Application Architecture Matters
• Does it facilitate testing at the API/Services level?
• Does the Business/Error Logic reside below the services
level?
• Does the UI architecture support test automation other
than record-playback?
For Large organizations:
– Standardize UI architectures across the product portfolio
› HTML5, Flex, Silverlight…
– Standardize Service architectures across the product portfolio
› SOAP, REST, CLI, API
21
Prioritize and Identify what NOT to Automate
Prioritize High ROI
1.
BVT Candidate Acceptance Test
–
Test of core functionality that executes in a short duration
2.
Core Functionality to be run in Nightly Regression
3.
Sustaining Test Candidate Test
–
Functional test that verifies the application can withstand subsystem
changes
What NOT to Automate
•
Everything
•
GUI – Except for High-value customer usage flows once
the UI is stable
•
Tests that would only find low severity bugs
22
11

14. 4/16/2013
Non-Functional Testing
• Performance Characterization
2
• Longevity
• Stress
• Scale
• Concurrency
Often times analysis of
these areas are simply
too cost-prohibitive to
be done manually
2 - http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/
23
Foundations of
an Automation
Culture
Images.yahoo.com
24
12

15. 4/16/2013
Keys To A Culture Transition
• Development vs. Test
• A different “School” of Test3
• Evolve from Functional Responsibility
…to
Whole Product Ownership
3 – Scott Barber “Approaches to Software Testing: An Introduction “
25
Establishing an Automation Culture
• Identify
• Near and long-term automation focus areas
• Inventory the culture and skillset of organization
• Establish
– Tooling and Infrastructure Standardization
– Develop your ‘community’
• Develop
– Workforce transition plan
• Organize
– Embedded vs. Specialized Automation team
– Re-align project staffing
• Operationalize
26
13

16. 4/16/2013
Identify
Focus Areas for Automation
4
4 - http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/
27
Establish
• Tooling and Infrastructure
–Tooling Standards and Frameworks
–Team to develop/manage home-grown tools
–Lead tool evaluations to meet new architectures &
technologies
Multiple automation Tools/FW’s may be
needed to automate at layers of testing:
•
•
•
Unit level
Middle tier
UI-driven
• Community: Internal and External
28
14

17. 4/16/2013
Develop & Organize
• Adjust Hiring Practices
• Re-align Project staffing
• Embedded Automation vs. Specialized
Automation team
• Training
–New-hire and ongoing
29
Confidential
Operationalize
• Whole team commitment in words and actions
• Include Test Automation in Acceptance Criteria
–Unit
–Acceptance
• Continuous Integration and BVT
• Establish Metrics and Governance
30
15

18. 4/16/2013
Care and
Feeding of the
Automation
Culture
31
Maintaining the Automation Culture
• Operationalize Automation
– Include Test Automation in the Acceptance Criteria
• Encourage Community Practice and Participation
– External - Industry User groups & Conferences
– Internal - Brown-bag sessions, Showcases, Mini-conference
• For Large organizations:
– Overcome Skillset deficiencies by adjusting staffing strategy
– Automation Leadership team, Automation Architect(s)
– Continually monitor the alignment of Arch, Dev and Test
– Develop reward systems
32
16

19. 4/16/2013
Recognizing a Successful Automation Culture
• Automation is a shared responsibility
• Automatability is a key architectural
consideration
• Continuous Integration & BVT
• Teams are staffed to include an
automation skillset
• Automation is operationalized
Resources
• Agile Manifesto
• Articles:
– http://support.smartbear.com/articles/testcomplete/automated-testing-agileenvironment/ - SmartBear
– http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ Lisa Crispin
– http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&
ObjectId=17793&tth=DYN&tt=siteemail&iDyn=2 – Rajini Padmanaban
• Books:
– Scaling Software Agility: Best Practices for Large Enterprises – Dean Leffingwell
– How Google Tests Software – James Whittaker, Jason Arbon, Jeff Carollo
– Agile Testing: A Practical Guide for Testers and Agile teams – Lisa Crispin, Janet
Gregory
• Presentations:
– Approaches to Software Testing: An Introduction – Scott Barber
– Agile Testing: Challenges Beyond the Easy Contexts – Bob Galen
34
17

20. 4/16/2013
Questions?
35
Backup
36
18