The document discusses the process for deploying code changes to tier 1 services at Amazon. It involves code review, automated builds and testing, deployment through multiple environments including production regions, automated rollback capabilities, and auditing of changes. Amazon is able to deploy changes rapidly due to standardized tooling and processes, an internal culture of ownership, and a focus on continuously deploying changes and monitoring their impact.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Life of a Code Change to a Tier 1
Service
Andy Troutman, Senior Manager AWS Deployment Services

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why
do
you
care?

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
To PRODUCTION!

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
It is rather impressive…
50 MM Deployments in a year
10’s of thousands of builders.
That comes to more than a software
deployment every second

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
It’s more
sausage
making than
magic.

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Much of what enables
Amazon is in our tooling,
and we’re releasing most
of it to AWS customers.
You can build a
developer experience
SUPERIOR to Amazon.

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling Philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Let’s Meet Our Hero….
hello world.

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon’s Service Oriented Architecture
8 Engineers is still
the average sized
team at Amazon.

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DevOps and Ownership
1. A federation of startups
2. Everyone owns the product end to end

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
My one slide on the local development
process
Yeee Ha
• Agile… ish?
• “The Process” will ensure you
don’t cut corners.
• Iterate as quickly as you can.
• Your host is a service.

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling Philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Building
Blocks
Key Callouts
Fund
Winners
Marketplace

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Our Antagonist
It’s me, THE
PROCESS.

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A change is born.
• Replication
• Access Control
• Code Search on Commit
• AWS CodeCommit gives you a lot
of these same advantages (BTW)

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling Philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CodeReview and Pull Request
Does this
thing make me
look fat?
• The last step before “the
process” takes over.
• Teams can block commits to
mainline via a variety of rules.
• You find very little branching
strategy at Amazon.

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Build
Global Dependency Closure
Unit Testing
Static Analysis
Bad Software Blocked

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Begin with the Pre-Mortem

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling Philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Part 2 – The Journey
Test
Integration
Production
Ugh Oh.

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
16 Regions
54 Availability Zones
100 Cloudfront PoPs
Did I mention we also deploy for Amazon?

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Release Automation
This is going to be a long road trip.
It’s beautiful.

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Global Andon Cords
Manual Approval Time Windows Test Tasks
Release Primitives
Attribution and Roles
Neat.

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Changing Software
Auto-Rollback

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Changing Software
Incremental Release

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Changing Software
Health Tracking

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Changing Software
Change Providence

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Changing Software
Auto-Rollback
Incremental Release
Change Providence
Health Tracking
I don’t feel
trusted.
You’re not!

30. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling Philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

31. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Infrastructure Automation
• 1-Day region builds
• Infra and OS pipelined
like software
• Lock in best practices
with our own
templating engine

32. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
If All Goes Well…
I Made it!

33. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Auditing and Reducing Risk
My job is
never done.

34. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The rest of the talk:
1. Technical and cultural landscape at Amazon
2. Tooling Philosophy
3. Code review, build, and the pre-mortem
4. Pipelining and deployment
5. Managing infrastructure, and auditing
6. Final thoughts + Q&A

35. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key Enablers to Our Success
• Amazon standardizes a ton of the software
development process via common platforms.
• We encourage an open internal marketplace
for extending that platform, and fund winners.
• We ship so much because we continually
ship everything to production, not because
we go crazy fast.
• Amazon is doubling down on it’s AWS tools
to run it’s own business.