1.
Immutable Infrastructure
Rethinking Configuration in the Age of Easy Redeployment
RackN, Inc
May, 2019
Note: Graphics mainly from http://pexels.com
<<< Shift Left <<<
https://u.rackn.io/interop2019

2.
2
RackN, Inc. and Digital Rebar Provision
Your Presenter ….
Software for Dell “CloudEdge” Hyperscale Hardware effort
Learned many DevOps lessons in the field
then created first OpenStack installer: “Crowbar”
1) Operations is inconsistent, manual and heterogenous
2) Foundational automation makes many problems go away
3) “Apply, Rinse, Repeat” (aka Immutable Infrastructure)
Founding Member, Chief
Solutions Architect, and
Community Evangelist
Shane Gibson

3.
3
RackN, Inc. and Digital Rebar Provision
Your Presenter ….
Software for Dell “CloudEdge” Hyperscale Hardware effort
Learned many DevOps lessons in the field
then created first OpenStack installer: “Crowbar”
1) Operations is inconsistent, manual and heterogenous
2) Foundational automation makes many problems go away
3) “Apply, Rinse, Repeat” (aka Immutable Infrastructure)
Founding Member, Chief
Solutions Architect, and
Community Evangelist
Shane Gibson
But … really, who cares?
Let's get to the fun stuff …

4.
4
@rackngo #immutable
Storytime! “Self-Bootstrapping Kubernetes”
Kubecon in Nov 2017 we created this demo
Simple “immutable” Idea:
1) In Memory Boot Machines
2) Install Docker
3) Elect Leader
4) Run Kubeadm on Leader
5) Run Kubeadm on Remainder
But….it’s shockingly hard to maintain.
Dependencies breaks the installation
And they are constantly changing.

5.
5
@rackngo #immutable
Storytime! “Self-Bootstrapping Kubernetes”
So, while it’s pretty cool,
it’s not “real” immutability
Presentation & Demo
https://youtu.be/OowxF6GqK4I
sad!

6.
Why is configuration fragile?

7.
Why is configuration fragile?
mutation
V

8.
8
@rackngo #immutable
But… I 💜 Infrastructure as Code?!
Sorry. Mutability adds complexity
Traditional “build-in place” approaches
● Have hidden dependency graphs
● Create variation between environments
● Are harder to “lock down” due to config
AND OMG… updates and patches are even harder
● Idempotent operations are difficult
● Roll backward is next to impossible!
● Creating indeterminate state

9.
9
@rackngo #immutable
Traditional “build-in place” approaches
● Have hidden dependency graphs
● Create variation between environments
● Are harder to “lock down” due to config
AND OMG… updates and patches are even harder
● Idempotent operations are difficult
● Roll backward is impossible
● Creating indeterminate state
But… I 💜 Infrastructure as code?!
Sorry. Mutability adds complexity
Let’s lock it down!

10.
What is Immutable Infrastructure?

11.
What is Immutable Infrastructure?
Pre-deploy configured
V

12.
12
@rackngo #immutable
Traditional Deploy and Configure
System is configured in situ from
a least common denominator
baseline.
This can be “immutable-like”
under the right conditions.
We’ll come back to that...
Delivery Pipeline
Deployment
Code
Build
Integrate
Run
Configure

13.
13
@rackngo #immutable
Shifting Configuration BEFORE Deployment
In our ideal delivery pipeline,
configuration is before
deployment.
Running systems are delivered as
a complete runnable unit for
deployment.
Delivery Pipeline
Deployment
Code
Build
Integrate
Run
Configure

14.
14
@rackngo #immutable
Shifting Configuration BEFORE Deployment
In reality, it’s very hard to create a
distinct artifact for every running
instance; instead, we create
incremental versions.
So we do some initialization of the
reusable versioned instance.
Cloud init is the most commonly
known pattern for this.
Delivery Pipeline
Deployment
Code
Build
Integrate
Run
Configure
Initialize!
V

15.
Cloud Native
Infrastructure
CNIbook.info
Justin Garrison & Kris Nova
“Infrastructure as software”
“Intent-Based Platform”

16.
16
@rackngo #immutable
Which Enables… Delegating Operations
If you can make your artifacts
immutable then you can delegate
management of them to a
platform like Kubernetes.
Kubernetes does not configure
infrastructure. It maintains state
based on a manifest.
StateManager
(e.g.Kubernetes)
Code
Build
Integrate
Run
Configure
Delivery Pipeline

17.
17
@rackngo #immutable
Which Enables… Delegating Operations
If you can make your artifacts
immutable then you can delegate
management of them to a
platform like Kubernetes.
Kubernetes does not configure
infrastructure. It maintains state
based on a manifest.
StateManager
(e.g.Kubernetes)
Code
Build
Integrate
Run
Configure
Delivery Pipeline
Kubernetes? WTF!
Is Immutable > K8s?

18.
Immutable is a DevOps Pattern
<<< Shift Left & Create/Delete

19.
19
@rackngo #immutable
The Problem
Immutability <<< Shifting Left
package
server
image
provision
server
initial
config

20.
20
@rackngo #immutable
The Problem
Immutability <<< Shifting Left
patch 1
package
server
image
provision
server
initial
config

21.
21
@rackngo #immutable
The Problem
Immutability <<< Shifting Left
patch 1 patch 2
package
server
image
provision
server
initial
config

22.
22
@rackngo #immutable
The Problem
Immutability <<< Shifting Left
patch 1 patch 2
the madness doesn't
stop at patch 2!
package
server
image
provision
server
initial
config

23.
23
@rackngo #immutable
The Problem
Immutability <<< Shifting Left
patch 1 patch 2
the madness doesn't
stop at patch 2!
What Madness?
● We have to maintain root access
● Patches assume system state
● Patches create dependency graphs
● Coordination? Should we halt work?
● Drift is inevitable!
package
server
image
provision
server
initial
config

24.
24
@rackngo #immutable
The Problem
Immutability <<< Shifting Left
patch 1 patch 2
SAD !!package
server
image
provision
server
initial
config

25.
25
@rackngo #immutable
Apply cloud and container lessons to our Bare Metal …
Immutability <<< Shifting Left
package
server
image
provision
server
initial
config
destroy!!

26.
26
@rackngo #immutable
Apply cloud and container lessons to our Bare Metal …
Immutability <<< Shifting Left
destroy!!
destroy!!patch 1
package
server
image
provision
server
initial
config
package
server
image
provision
server
initial
config

27.
27
@rackngo #immutable
Apply cloud and container lessons to our Bare Metal …
Immutability <<< Shifting Left
destroy!!
destroy!!patch 1
depatch 2
package
server
image
provision
server
initial
config
package
server
image
provision
server
initial
config
package
server
image
provision
server
initial
config

28.
28
@rackngo #immutable
Apply cloud and container lessons to our Bare Metal …
Immutability <<< Shifting Left
destroy!!
destroy!!patch 1
patch N
depatch 2
package
server
image
provision
server
initial
config
package
server
image
provision
server
initial
config
package
server
image
provision
server
initial
config
package
server
image
provision
server
initial
config

29.
Cloud like behavior …

30.
30
Immutable Provisioning systems
treat infrastructure as a black box
Cloud-like Integration and Staged Workflow
Provisioning
System
Requested
State
Returned
State
REST
API
Event
Hook

31.
31
Cloud-like Integration and Staged Workflow
Immutable Provisioning systems
treat infrastructure as a black box
Provision requests are for a system state
with optional parameters.
The intermediate changes to achieve the
state are not exposed to the requester.
Provisioning
System
Reset Join
Install
Config
Test
Requested
State
Returned
State
REST
API
Event
Hook

32.
32
Cloud-like Integration and Staged Workflow
Immutable Provisioning systems
treat infrastructure as a black box
Provision requests are for a system state
with optional parameters.
The intermediate changes to achieve the
state are not exposed to the requester.
REMEMBER: Operators of the
provisioning system require high
transparency, stages and control.
Provisioning
System
Reset Join
Install
Config
Test
Requested
State
Returned
State
REST
API
Event
Hook
No hidden
operations!

33.
Immutable Patterns
1) Baseline + Configuration
2) Live Boot + Configuration
3) Image Deploy

34.
34
Provision
1: Baseline + Configuration
Benefit: Easiest to achieve with current tools, Safer than Patching
Challenge: Lots of Post-Configuration, Not Really “Immutable”, Slow
Instead of relying on patches, rely on starting from a pristine image
ResetBaseline Configure Run
Additional Reference https://thenewstack.io/immutable-hardware-ops-hygiene-security-efficiency/

35.
35
Provision
1: Baseline + Configuration
Benefit: Easiest to achieve with current tools, Safer than Patching
Challenge: Lots of Post-Configuration, Not Really “Immutable”, Slow
Instead of relying on patches, rely on starting from a pristine image
ResetBaseline Configure Run
Additional Reference https://thenewstack.io/immutable-hardware-ops-hygiene-security-efficiency/
You must control ALL
of your dependency
chain !!!

36.
36
Provision
1: Baseline + Configuration
Benefit: Easiest to achieve with current tools, Safer than Patching
Challenge: Lots of Post-Configuration, Not Really “Immutable”, Slow
Instead of relying on patches, rely on starting from a pristine image
ResetBaseline Configure Run
Additional Reference https://thenewstack.io/immutable-hardware-ops-hygiene-security-efficiency/
...every single (!!)
package from the
base OS on up ...

37.
37
Benefit: Fast reset times, forces good behavior
Challenge: Provisioning becomes critical path, still have dependency graph
Like #1 but clean-up is simply a reboot. Favors smaller footprint O/S.
2: Live Boot + Configuration
Provision RebootBaseline Configure Run

38.
38
How much of OS, Packages, and Application stack is in your Live Boot image
will determine how much work you have to do on dependency tracking and
version lock down.
2: Live Boot + Configuration
Provision RebootBaseline Configure Run

39.
39
3: Image Deploy
Benefit: Shorter time to ready, highly controlled (“shift left”), rollback
Challenge: Harder to create and deploy images
Image is deployed from source instead of Baseline + Configure
Provision
Deploy
Image
Run Provision
Deploy
Image
Run

40.
40
3: Image Deploy
Benefit: Shorter time to ready, highly controlled (“shift left”), rollback
Challenge: Harder to create and deploy images
Image is deployed from single artifact instead of Baseline + Configure
Provision
Deploy
Image
Run Provision
Deploy
Image
Run
Initialize!
V
Initialize!
V

41.
41
So… Let’s talk Image Creation
Ideally in an automation build process.
You DO THE CONFIGURATION on a
live system (so you still need
configuration tools) and then capture
the image into a portable format.
Tools like Hashicorp Packer, Image
Builder, WBIC or raw images are used
to create source files (e.g. Root FS
Tarball, Disk Image, AMI, OVS).

42.
42
So… Let’s talk Image Creation
Ideally in an automation build process.
You DO THE CONFIGURATION on a
live system (so you still need
configuration tools) and then capture
the image into a portable format.
Tools like Hashicorp Packer, Image
Builder, WBIC or raw images are used
to create source files (e.g. Root FS
Tarball, Disk Image, AMI, OVS).
That sounds like a lot
of work & really slow!

43.
Yes, But…
It’s faster, safer & more scalable.

44.
44
Build Pipeline
Immutable Demo
Prep: Image is pre-created from reference system.
Reference
System

45.
45
Build Pipeline
Immutable Demo
Prep: Image is pre-created from reference system.
Reference
System
customize

46.
46
Build Pipeline
Immutable Demo
Prep: Image is pre-created from reference system.
Reference
System
Image
Read
customize

47.
47
Deploy Pipeline
Build Pipeline
Immutable Demo
Prep: Image is pre-created from reference system.
Stage: Boot RAM image and write image to disk(s)
Reference
System
Image
Target System
RAM BOOT
Write
Read
customize

48.
48
Deploy Pipeline
Build Pipeline
Immutable Demo
Prep: Image is pre-created from reference system.
Stage: Boot RAM image and write image to disk(s)
Deploy: Reboot and run
Reference
System
Image
Target System
RAM BOOT
Target System
RUNNING
Write
Run
Read V
Reboot
Initialize!customize

49.
49
Build Pipeline
Deploy Pipeline
Immutable Demo
Hopefully (!!): You're pushing your images through a
rigorous CI/CD pipeline before you Deploy !!
Reference
System
Image
Read
customize

50.
50
Build Pipeline
Deploy Pipeline
Immutable Demo
Hopefully (!!): You're pushing your images through a
rigorous CI/CD pipeline before you Deploy !!
Reference
System
Image
Read
CI/CD Pipeline
TEST !!

51.
51
Build Pipeline
Deploy Pipeline
Immutable Demo
Hopefully (!!): You're pushing your images through a
rigorous CI/CD pipeline before you Deploy !!
Reference
System
Image
Read
CI/CD Pipeline
TEST !!
FAIL
???

52.
52
Build Pipeline
Deploy Pipeline
Immutable Demo
Hopefully (!!): You're pushing your images through a
rigorous CI/CD pipeline before you Deploy !!
Reference
System
Image
Target System
RAM BOOT
Write
Read
CI/CD Pipeline
TEST !!
Pass!

53.
53
Build Pipeline
Deploy Pipeline
Immutable Demo
Hopefully (!!): You're pushing your images through a
rigorous CI/CD pipeline before you Deploy !!
Reference
System
Image
Target System
RAM BOOT
Target System
RUNNING
Write
Run
Read V
Reboot
Initialize!
CI/CD Pipeline
TEST !!
Pass!

54.
Thank you!
Questions?
Interested in IMMUTABLE METAL?
It’s complicated, but we can get you there.
Start at http://portal.rackn.io
• Quickstart takes about 30 minutes
• Use your own hardware, VirtualBox or Packet.net account
– use “RACKN100” on Packet.net for credit