This document discusses testing software in high frequency delivery environments using continuous testing and DevOps practices. It outlines how continuous testing is not just about test automation, but also includes automated management of environments, application feedback through monitoring, and engaging in XP practices. DevOps helps by automating building, testing, and deployment to provide consistency and tools for teams. Characteristics of high frequency delivery environments include automating infrastructure, testing, and deployment to reduce errors and allow for smaller, more frequent deliveries. This allows for a single view of quality and faster restore times when issues arise.

1. Testing software in high
frequency delivery
environments.
Continuous
Testing &
DevOps

2. - Lead consultant & trainer at Lean
Software Services
- 16+ years working in software
development, based in London &
Madrid
Peter Marshall
MAIL
info@leansoftwareservices.com
TWITTER
@petemar5hall
LINKEDIN
https://www.linkedin.com/in/petedmarshall

3. Training
Coaching
Consulting
Madrid
Barcelona
London
Copenhagen
- Testing
- DevOps
- Continuous
Delivery
- Automation
- Virtualisation

4. Gain insight into how continuous testing, DevOps, and organisational
structure can alter the way we think about testing.
Understand how DevOps practices contribute to the test effort.
Objectives

5. Who is here today?

6. Creating
environments
deploying to
environments
testing
environments
looking at
requirements
pair
programming/
testing
creating
test data
writing
automated
checks
running
through
regression scripts
what else does a
tester do?
triage
mitigating
delays

7. Who is testing in an agile environment?
What about doing continuous delivery?
Who has a devops culture / team / role?

8. Sources: Applied Software Measurement: Global Analysis of
Productivity and Quality by Capers Jones, & IBM The Future of Testing
Most forms of testing find fewer than 30% of all bugs
almost 40% of testers have seen an increase in the frequency of release
Productivity and quality are directly coupled: projects with
high quality have high productivity and vice versa

9. How can devops
and continuous
testing help?

10. Agile gave us..

11. What does
this really
mean???

12. the test and QA
headache is
compounded
further by the need
to release faster
and more often
Amazon is on record as making changes
to production every 11.6 seconds on
average in May of 2011.
Facebook releases to production twice a
day.
Many Google services see releases
multiple times a week, and almost
everything in Google is developed on
mainline.
Etsy push unto 70 changes a day to its
monolithic software!

13. the presence of a
tester can sometimes
introduce more risk
than they mitigate..

14. Characteristics of
high frequency
delivery
environments

15. Key Concepts
…continuous
testing…
DevOps

16. Continuous Testing
…not just about test
automation! (ignore
wikipedia)
Actually includes, but not limited to:
Automated testing
Automated management of production and non
production environments
Automated application feedback through
monitoring
Automated evaluation of business objectives
Evaluating product concepts
Testing
engaging in XP practices
organisational change

17. The DevOps movement is built
around a group of people who
believe that the application of a
combination of appropriate
technology and attitude can
revolutionize the world of software
development and delivery.
Stephen Nelson-Smith - 2010
its about:
Automating building, testing, and
deployment
Bringing consistency to the way we
use and create technical assets
Providing teams with the tools
required to achieve technical
excellence
DevOps

18. vs.
more facts..
high performing organizations ship code 30 times faster (and
complete these deployments 8,000 times faster), have 50%
fewer failed deployments, and restore service 12 times faster
than their peers. The longer they had been employing these
practices, the better they performed.
source: ThoughtWorks, 2014

19. automate the
build
Reduce general coding errors, create a single source of
truth, ensure basic component integration

20. vs.script your configuration!
say NO to hand crafted configuration files!

21. automate the
infrastructure
Remove configuration issues by having a repeatable
mechanism for creating environments.

22. vs.automate the testing
continuous testing does away
with the concept of
regression testing.
..an effective software delivery process
requires no regression testing stages. It
becomes integral to everything we do..

23. automate deployment
say NO to sneaker net
say NO manually to copying files
say NO to manually pulling and pushing servers in and out of load balancers
deploying software is one of the
most mission critical aspects of
what we do, let a machine do it!

24. monitor
One of the greatest and
overlooked tools in the arsenal of
the tester is monitoring.
Realtime access to
application logs
Realtime access to
error logs
Realtime access to
infrastructure logs

25. “A good, easily accessible
event & error monitoring
system is essential for anyone
releasing software. Build an
early warning system into your
environment!”

26. “Understanding your systems
steady state allows you to react
quickly to any deviation”

27. Build Agent & Devel
The delivery pipeline
Pre commit tests
Code review
PO/UX review
Branch / Merge
Build
Unit tests
Contract tests
Component tests
Coverage
Static analysis
Component
performance
End to end
component tests
NFR tests
Exploratory Test
performance testing
Story sign off /
Demos
NFRs
Analytics
Monitoring
Production tests
canary or blue /
green release
CI Environments
Developer
Local
Pre-Production Production
Virtualised Full stack environments
Commit Auto-Deploy Promote Release
Build Pipeline:
There can be multiple pipelines, usually per system, should execute in minutes
Commit:
The commit can include changes to the infrastructure, database, or application code.
Outputs:
Software, infrastructure and configuration
Environment manifests
Release notes

28. Create a
single view
of quality

29. smaller,
more
frequent
deliveries

30. Demo

31. Thank
you!
TWITTER
@petemar5hall
Agile Consultant & Technical
Architect @planday
Peter Marshall
fiveMAIL
info@leansoftwareservices.com
LINKEDIN
https://www.linkedin.com/in/
petedmarshall