Using DevOps to Improve Software Quality in the Cloud

T21
Test Automation
5/8/2014 3:00:00 PM
Using DevOps to Improve
Software Quality in the Cloud
Presented by:
Glenn Buckholz
Coveros
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

GlennBuckholz
Coveros
Currently, Glenn Buckholz leads CI and deployment automations efforts at Coveros. Using his
fifteen years of industry experience, Glenn brings success to his customers. Beginning his
career as a consultant implementing automated test frameworks, he introduced the concept of
change management to many, many projects. Glenn then decided to become a part of honest
society and settled down at the Public Company Accounting Oversight Board as their full time
enterprise change manager. After several years, he ventured back into the real world at
Coveros, specializing in implementing agile practices, implementing CI, and engineering
configuration management instead of simply documenting it.

4/26/2014
1
© Copyright 2014 Coveros, Inc. All rights reserved.
Using DevOps to Improve Software
Quality in the Cloud
STAR EAST 2014 – 08 May 2014
Glenn Buckholz
glenn.buckholz@coveros.com
Agenda
What did I do?
What is it good for?
Demo
Architecture
Workflow
Orchestration
Cost vs Velocity
Data Analysis
What is Coveros doing with it for customers?

4/26/2014
2
What did I do?
Combined several technologies that Coveros is
familiar with plus Puppet to demonstrate a full
implementation of CI (Continuous Integration)
This could be titled “Everything I wanted to do on
Forge.mil but am not allowed.”
Specifically, I cobbled together:
Jenkins
Puppet
Puppet master
EC2
Linux
CURL
ANT
What is it good for?
Fail Faster
Used with permission from the people at Extra Credits.
http://www.youtube.com/extracredits

4/26/2014
3
Demo
Major Points
How is this DevOps?
Releases are engineered, the systems engineer
is simply pushing the button.
If the list of manual installation instructions is
more than one paragraph there is more
engineering work to do. Code is the
documentation.
Operations people now govern the access,
developers govern the how.
New testing concerns.
Deployment code.
OS templates.
Developers now have the job of convincing
operations to run their deployment code.

4/26/2014
4
Architecture
Workflow
Developer commits code.
Jenkins detects code change.
Proper RPMs are built and distributed to
repositories.
Jenkins contacts EC2 and spins up blank
instances.
Puppet is delivered to the blank instances.
Jenkins works with the Puppet Master for
orchestration.
Jenkins confirms that the application is
running.
Jenkins fires off automated tests.

4/26/2014
5
Code Commit – Different
Approaches
Something has to trigger the testing. A code
commit is typically a significant enough
change to require some measure of retesting.
Unit test are the minimum criteria to the
publishing a change.
Even if the code is bad the system must
still run.
By convention, code commits should be
complete and significant.
Each commit now carries a cost.
This should be enforced by training.
For larger projects with large volumes of
RPMs Packaging
To move quickly code must be delivered in a
consistent format.
Copying files is too unpredictable and error-
prone.
Permissions
Location
Clean up of old file
Dependency checking
Any of the following are good choices
RPM – Centos, RHEL
Emerge/Portage - Gentoo
Deb – Debian, Ubuntu

4/26/2014
6
Cloud Provider
The main issue is elasticity.
Number of servers will vary.
Tests will be running in parallel.
Need a way to catalog all the machines
Assign metadata to VMs
Ability to group VMs and test results
by deployment
Automatically mapping code commits
to cloud deployments is necessary for
proper bookkeeping.
OS templates
Each OS must originate with a standard template.
All change must be managed by Puppet or some
other deployment language
If the OS is not standard too much time will be spent
engineering around OS peculiarities
CM must be mastered
Every change to a standard OS must be recorded.
Every change must be programmed, no hand-
jamming
All deployments are now hands-off
You can almost get Continuous Delivery for free.
OS Customizations for only two reasons
Integrate into the Jenkins/Puppet
Integrate into the cloud provider from a standard

4/26/2014
7
OS Template Testing
Must Be Automated
RHEL, CENTOS, Windows change every
month, Manual checks must be limited to
keep up.
Security
STIG - Manual
OpenSCAP – STIG Automated
OWASP top 10 - Automated
Organizational Standards
Centralized Login
Standard directories
etc
Deployment Testing
This is a new area since deployments are now
code.
No traditional tools to test this type of
functionality
Currently methods are mostly manual.
Small automated tests must be constructed
to help with orchestration.
It either worked or it didn't
Manually sifting through logs to
troubleshoot.
Who does this?
Unfortunately, either a sysadmin or a

4/26/2014
8
Automated Testing
Automated testing must be an established
practice.
You cannot perform the volume of tests
required for CI efficiently without some sort
of automated check to separate the good
changes from the bad changes.
The cost and practice must be understood
within the development and testing
organization.
Testing resources must be able to handle max
capacity.
Tests must be able to run in parallel against
s
Role of Manual Testing
Manual Functional Testing DOES NOT GO AWAY!
Automated tests are not as effective as a person
because they are so specific.
Manual Testing is now a precious resource that
can only be applied in a limited fashion.
Time is what limits Manual testing
1-2 weeks of testing while code is written.
2-3 weeks of total testing effort.
Testing effort is divided.
Automation of new functionality testing.
Spot manual testing.
Manual free form regression.
Targeted regression based on automated results.

4/26/2014
9
Disposition of the Systems
Test reporting and the cloud provider must be
linked through a dashboard.
Manual testers must be able to verify failures.
Testers must have access to the broken
running system
Access will allow reproducibility and
verification
Developers must be able to identify broken
VMs
Access to the app to see the unexpected
behavior
Orchestration
This is the combination of architectural
components and services in a
meaningful fashion to produce a running
system.
For this example:
Database
App server
Database initialization
App server initialization
Coordinated starting of all the services

4/26/2014
10
Costs vs Velocity
Velocity comes with parallelism.
Tests running in parallel
Code being tested in a pipelined fashion.
Cost
Number of testing machines and infrastructure.
Number of parallel instances x machines per
instances
Storage – keeping failed instances around
Budget and Team size
Even with infinite money if you have a team of 5, the
400th VM is likely only giving very little marginal utility.
When money is the constraint, developer and tester
habits must best utilize the limited VMs wisely.
Data Analysis
In order to move quickly the right people must get
the right data.
Test Results
Quick access to the failed test
Quick access a screen shot of the failure
Ability to assign errors to the proper developers
Integrated bug tracking.
Server Analysis
Developers need to zero in on failed servers
quickly
Developers need to link failures to their code
quickly
Developers need to separate system/OS failures

4/26/2014
11
Dashboard (Data Analysis
Visualization)
Centralization of results.
Sonar Cube
Custom pages
Views for Management, Testers, and
Developers.
Traceability
Features all have GUIDs
GUIDs linked to code commits
Unit tests linked to GUIDs
Automated tests are linked to feature
GUIDs
VMs linked to list of new GUIDs
What is Coveros Doing with it for
Customers?
Training and Process improvement
Being the exemplar on a project to
move this practice to other projects.
Showing the change in the cost to test
a feature within an organization
Implementing pieces of the CI to show
marginal improvements.
Using the gains marginal gains to sell
the whole process.

4/26/2014
12
Level of Effort
The demonstration needs to be customized
for each environment.
Very few projects that I have dealt with are
“the same”. While this demo can be a guide,
I don't believe it can be generalized to a
solution, each environment must be
customized.
This implementation required research into
the APIs and inner workings of several
complicated software suites and services.
The engineering effort is commensurate with
the complexity of the architecture.
Why Link All These Technologies
Together?
The industry is moving this way in various forms.
AWS templates
Service Mesh
JuJu
Hadoop
Flexiant
Even though the products are varied, they all share
general themes.
Inventory
Relational Mapping
OS Templates
Virtual Network Capabilities
If you know one you can have a feel for them all.

4/26/2014
13
Thoughts? Questions?
Thank you for your time.

Using DevOps to Improve Software Quality in the Cloud

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Destacado

Destacado (16)

Similar a Using DevOps to Improve Software Quality in the Cloud

Similar a Using DevOps to Improve Software Quality in the Cloud (20)

Más de TechWell

Más de TechWell (20)

Último

Último (20)

Using DevOps to Improve Software Quality in the Cloud