The document discusses how agile development and continuous deployment disrupt traditional functional testing processes. It describes how testing practices have evolved from waterfall development with long release cycles to frequent daily releases. This requires testing to be more automated, with practices like acceptance testing driven development (ATDD) where testing defines requirements and drives the engineering process. It presents a cloud-based platform for ATDD that supports collaboration and test management integration.

1. How Agile and Continuous Deployment
disrupt traditional functional testing process!
Learn about new testing techniques
Laurent PY
CEO, Smartesting
Laurent.py@smartesting.com
@py_laurent
Session 1279
© 2013 IBM Corporation

2. Agenda
 Introduction
 New testing challenges
 Language of the tester
 Demo
 Perspectives
 Q&A

3. Time for disruption ?

4. Testing: Time for disruption ?
Software testing vendors:
N°1: HP (QC, QTP, BPT)
N°2: IBM (RQM, RTW)
N°3: Micro Focus (SilkCentral,
SilkTest)
Where do we find the best practices?
Do you think they use Quality Center?

5. Smartesting engineering process (early years 2004-2006)
 Water fall process
–Very few tests made by developers (No TDD)
–1 release every 6 months
–1 month (x5 engineers) spent on acceptance testing before
rolling out
–Very poor QA impacting customers feedback

6. Introducing agile development (2006)
 Introduction of agile: Scrum, TDD (unit &
integration testing), pair programming…
–Continuous integration
–1 release every 3 months
–4 man/days spent on acceptance testing before
rolling out
–Very good Quality

7. Going to the cloud (2012)
 Introduction of continuous deployment and DevOps
–SEVERAL releases a DAY! Enable business agility
–Acceptance Testing Driven Development (ATDD)
–Engineering process driven by testing !
http://www.thucydides.info/blog/295-does-atdd-really-save-you-time
With ATDD and TDD projects delivered 31% faster with 4 times fewer defects

8. Agenda
 Introduction
 New testing challenges
 Language of the tester
 Demo
 Perspectives
 Q&A

9. 9
Req Management &
Definition
Test Planning Execution Defect management
 Agility leads to short iterations, less planning, and massive
automation (with rise of mobile, virtualization)
 Value chain is shrunk acceptance test is the requirement⇒
(elicitation)
 Testing drives the engineering process !!!
The testing value chain is being shaken

10. 10
The traditional approach
Requirement
Management
Tool
Test
Management
tool
Technical
Specification
Traceability
Traceability
Traceability
Ambiguous,
Incomplete…
Inadequate coverage,
Hard maintenance…
Lack of collaboration and
shared understanding
Disconnected with
Requirements…

11. 11
Define
ExecuteMeasure
Test automation
Test virtualization
Mobile testing
Scanner (capture & replay)
Analytics
Model Driven
Testing
Exploratory testing
ATDD
Accelerate the testing cycle
LEAN ?

12. 12
Acceptance Testing Driven Development (ATDD) in a
Nutshell
 Begin with the end in mind
 Test is THE definition of done
 Written prior to development
 Confirmed with stakeholders
 May or May not be automated
Support Critique
Code Level Unit Testing Inspections, code
quality analysis
Business Level Acceptance
testing
Exploratory
testing
Test in natural language
Test fixture
Code

13. 13
ATDD in a Nutshell
 Some Benefits
–Collaboration and Communication
–Team knows what successful implementation means
–Achieving better coverage of business expectations
–Shorten the feedback cycle (lean)
 And some Challenges:
–Change in process so must have management support
–Find the right Balance People/Process/Tool

14. 14
Define
ExecuteMeasure
LEAN!
Get feed-back faster
Stop planning it! Do it!Stop planning it! Do it!
Increase velocity with
More test automation
Requirements/Stories acceptance tests
ATDD: Accelerate the testing cycle

15. 15
Define
ExecuteMeasure
Test automation
Test virtualization
Mobile testing
Scanner (capture & replay)
Analytics
Model Driven
Testing
Exploratory testing
ATDD
Testing
language
Testing
language
Accelerate the testing cycle

16. Agenda
 Introduction
 New testing challenges
 Language of the tester
 Demo
 Perspectives
 Q&A

17. 17
Lessons learned
Concrete test Abstract test
Adoption
Productivity
Testers: find the right level of
abstraction for the best productivity

18. 18
Testing VS Development
Tests are a subset of the behavior of the application

19. 20
Lessons learned: The concepts
 Steps & scenarios are the key elements
Gherkin language
Scenario=list of free step

20. 21
Lessons learned: Abstraction
 Function/Keyword/Action Word…
 Data: physical VS logical
 Graphical flow (no semantic to strong semantic)
 Behavior: IF THEN ELSE, WHILE
 Rules
Easy
Easy
Medium
Hard
Hard

21. 22
Lessons learned: Action Word
 Keyword/Action Word are easy to understand
 Powerful abstraction for maintenance
 But, It can be hard to get the right factorization
 Not every step needs to be reusable
Definition of Smart: Right balance between two levels

22. 23
 Should be parameterized (use logical data & data tables)
 Can include several other steps and Action Words
 Enable different levels of abstraction (Workflow, feature)
Lessons learned: Action Word
AW1 AW2
AW3 AW4 AW5 AW5 AW6
Workflow
Feature

23. 24
Lessons learned: Scenario, Action Word, Step, data
With these simple concepts you can get:
Flexibility to describe business domain
Powerful refactoring capabilities

24. 25

25. 26
Platform in the cloud for ATDD
Use Case 1: You are agile and do continuous integration
ATDD Platform
Create acceptance tests
Tester
Validate
acceptance criteria
Business Analyst
use
acceptance criteria
Developer
Code
repository
Integrate with
• Smartesting DSL
• Smart refactoring capabilities
• Collaboration
• Smartesting DSL
• Smart refactoring capabilities
• Collaboration

26. 27
Platform in the cloud for ATDD
Use Case 2: You are in a more traditional environment and got a test
management tool
ATDD Platform
• Smartesting DSL
• Smart refactoring capabilities
• Collaboration
• Smartesting DSL
• Smart refactoring capabilities
• Collaboration
Test
Management
Tool QC/QTP
RQM/RTW
Requirement
Management
Tool
RRC

27. 28
Platform in the cloud for ATDD
Setup your project and be ready to start in
60 Seconds

28. Agenda
 Introduction
 New testing challenges
 Language of the tester
 Demo
 Perspectives
 Q&A

29. 30
ALM market dynamic
 New software delivery models
 Testing is the biggest ALM segment
 Distributed Testing segment represents more than $2b in
2013 10% (http://www.gartner.com/id=2098416)
 Increasingly focused on Web & mobile
 Security testing is the fastest growing segment

30. 31
More Cloud
 ALM paas: $315m in 2016 (http://www.gartner.com/id=2188816)
 Started by defect management, load tests, dev or test lab provisioning
 Drivers
–Agile development
–Speed of deployment and business agility
 Inhibitors
–Managing the transition
–Lack of integration (OSLC)

31. 32
Test earlier & more automation
 Definition level
–More ATDD
–Testing driving the engineering process
–More & smarter tools to support design and maintenance
 Test Execution
–Test lab, virtualization
–New generation of Automation Tools

32. 33
Laurent PY
CEO, Smartesting
Laurent.py@smartesting.com
@py_laurent

33. 34
Daily Apple TV giveaway
 Complete your session surveys online each day at a conference kiosk or on
your Innovate 2013 Portal!
 Each day that you complete all of that day’s session surveys, your name will
be entered to win the daily Apple TV!
 On Wednesday be sure to complete your full conference evaluation to receive
your free conference t-shirt!

34. 35
© Copyright IBM Corporation 2013. All rights reserved. The information
contained in these materials is provided for informational purposes only, and is
provided AS IS without warranty of any kind, express or implied. IBM shall not be
responsible for any damages arising out of the use of, or otherwise related to,
these materials. Nothing contained in these materials is intended to, nor shall
have the effect of, creating any warranties or representations from IBM or its
suppliers or licensors, or altering the terms and conditions of the applicable license
agreement governing the use of IBM software. References in these materials to
IBM products, programs, or services do not imply that they will be available in all
countries in which IBM operates. Product release dates and/or capabilities
referenced in these materials may change at any time at IBM’s sole discretion
based on market opportunities or other factors, and are not intended to be a
commitment to future product or feature availability in any way. IBM, the IBM logo,
Rational, the Rational logo, Telelogic, the Telelogic logo, and other IBM products
and services are trademarks of the International Business Machines Corporation,
in the United States, other countries or both. Other company, product, or service
names may be trademarks or service marks of others.