1.
Testing 12-Factor Cloud Apps
Phillip Marlow
October 2022
Approved for Public Release; Distribution Unlimited. Case Number 22-3215
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

2.
Too Long; Didn’t Listen
 The flexibility and elasticity of cloud services allows better and more automated
testing – if applications are designed to take advantage of it
 Designing applications and services for the cloud provides increased testability
and security
 This makes applications more resilient against technical and environmental
failures as well as attacks
 It also improves the organization’s ability to deliver on their mission
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

3.
> iam list-user-tags
 Cloud Engineer:
Designed and built both AWS and Azure environments for
large teams
 Systems Engineer:
Focus on the overall system and process to deliver the system
 Developer:
10+ years
 DevOps Engineer:
Automating build, test, deployment, and monitoring
 Security Engineer:
GSE #263, SANS Master’s Degree
 Hacker:
Speaker at DEF CON Cloud Village
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

4.
Typical Application Promotion Process
Development.env Test.env Production.env
Application v1.0 Application v1.0
Application v1.0
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

5.
Application Development Process
Development Test Production
Application v1.0-
katherine
Application v1.0-jenny
Application v1.1
Application v1.0-
katherine
Application v1.0-jenny
Application v1.1 Application v1.1
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

6.
Mature Application Deployment Process
Development
Test
Production
Application v1.0-
katherine
Application v1.0-jenny
Application v1.1
Application v1.0-
katherine
Application v1.0-jenny
Application v1.1 –
instance 1
Application v1.1
Application v1.1 –
instance N
Test
App2 v2.1
App2 v2.1 App2 v2.1
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

7.
The Big Problem
 Can multiple versions of an application be hosted in each environment?
 This design creates choke points on work at each environment
 Especially problematic for the test environment which may be shared by many users
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

8.
Designing for the Cloud is Better
 The Twelve-Factor App, developed by Adam
Wiggins & Heroku
 https://12factor.net/
Apps that:
 Use declarative formats for setup automation,
to minimize time and cost for new developers
joining the project;
 Have a clean contract with the underlying
operating system, offering maximum
portability between execution environments;
 Are suitable for deployment on modern cloud
platforms, obviating the need for servers and
systems administration;
 Minimize divergence between development
and production, enabling continuous
deployment for maximum agility;
 And can scale without significant changes to
tooling, architecture, or development
practices.
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

9.
Twelve-Factor Alternatives
 Microservices Reference Architecture from NGINX
 https://www.nginx.com/blog/introducing-the-nginx-microservices-
reference-architecture/
 Beyond the Twelve-Factor App by Kevin Hoffman
 https://www.oreilly.com/library/view/beyond-the-twelve-
factor/9781492042631/
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

10.
I. Codebase
 Partially solves the big problem of multiple deploys in an environment
One codebase tracked in revision control, many deploys
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

11.
II. Dependencies
 No reliance on dependencies installed in the deployment environment
makes it possible to scale the number of deployments and environments
as needed
Explicitly declare and isolate dependencies
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

12.
III. Config
 Separating environment specific configuration allows consistent and
independent deployments
 It also ensures that no changes need to be made to the system between
environments, which could potentially compromise the integrity of
previously run tests
Store config in the environment
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

13.
IV. Backing Services
 By treating backing services, such as databases or APIs, as attached
resources, we ensure the application is loosely coupled to those resources
 This enforcement of loose coupling of components makes testing those
components easier
 While this may increase the number of integration tests, this approach
ensures we have a thorough understanding of those integration points
making developing integration tests easier
Treat backing services as attached resources
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

14.
V. Build, Release, Run
 Testing can be run more frequently when build is separated from run
 Ensures no code changes are possible at runtime, so earlier tests remain
valid in the production environment
Strictly separate build and run stages
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

15.
X. Dev/Prod Parity
 Independent tests results are applicable to the final deployment
Keep development, staging, and production as similar as possible
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

16.
Wins
 Tests can be run simultaneously AND independently
 It’s easy to add another instance of an app or a whole environment
 Applications are designed for easy integration with other tools, including test
orchestrators and cloud security platforms
 Common operational patterns can be used to make the application more resilient
against a variety of failures and attacks
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.

17.
Phillip Marlow
@wolramp
linkedin.com/in/phillipmarlow
Thank You!
pmarlow@mitre.org
© 2022 THE MITRE CORPORATION. ALL RIGHTS RESERVED.