1.
Using Digital Rebar Provision for Immutable Infrastructure
RackN, Inc
January, 2018
<<< Shift Left <<<

2.
2
Sr. Architect and Community Evangelist
shane@rackn.com
Sr. Architect and
Community Evangelist
Shane Gibson
Shane Gibson is an Architect, Engineer, and Community Evangelist working at
RackN; leaders in physical and hybrid DevOps software. His experience spans
over two decades and includes Mainframe Hardware and Operations, Unix
Systems, Network Engineering, Computer Security, Virtualization Technologies,
Cloud Platforms, and Production Operations. He is passionate about automation,
full stack infrastructure architecture, and modern computing practices as applied to
Production Operations environments.

3.
3
An eight year history building physical automation
Learned many DevOps lessons in the field:
“Crowbar” … "Open Crowbar" …
"Digital Rebar ver 2" … "Digital Rebar Provision ver3"
1)
2)

4.
5
Create / Destroy Patterns.
● Mimics the Cloud/Container pattern of "Create"
and "Destroy"

5.
6
Create / Destroy Patterns.
● Mimics the Cloud/Container pattern of "Create"
and "Destroy"
● Favors minimal one-time configuration of runtime
state and secrets

6.
7
Create / Destroy Patterns.
● Mimics the Cloud/Container pattern of "Create"
and "Destroy"
● Favors minimal one-time configuration of runtime
state and secrets
● "Shift Left" in our Dev → CI/CD → Pre-Prod →
Prod pipeline

7.
8
Create / Destroy Patterns.
● Mimics the Cloud/Container pattern of "Create"
and "Destroy"
● Favors minimal one-time configuration of runtime
state and secrets
● "Shift Left" in our Dev → CI/CD → Pre-Prod →
Prod pipeline
● Both Image based and Package based patterns
supported

8.
9
As our deployments get
more complex and
interconnected, the
immutable pattern helps
limit drift between our
intended and actual
production environment.
The benefits of an immutable
infrastructure include more
consistency and reliability in
your infrastructure and a
simpler, more predictable
deployment process.
Immutable deployment is not
zero-configuration: the process
relies on minimal run-once
post-boot initialization.

9.
11
● Single Golang binary **
● Lightweight
● API first
● Fast
● Modern
● PXE/DHCP provisioning
● Composable Workflow (stages)
● Webevents
** requires: 7zip, bsdtar, unzip
(dependencies to be removed soon)

10.
12
HTTPS
DHCP
HTTP
Provisioning connects multiple
services and protocols
PXE is a very limited bootstrap
environment and restricts how the
system can work.
These protocols are deeply
embedded into infrastructure and
impossible to change.
Digital Rebar Provision Service
Core Services:
*DHCP or Provision services are optional
depending on configuration requirements.
API / UI
Service API (/api/v3)
UI Redirect (/ui)
8092/tcp
DHCP*:
Address Management
Next Boot Services
67/udp
TFTP PROVISION*:
PXE Boot Images
Dynamic Templates
Static Files
8091/tcp
Data
File
System
69/udp
No External
Database
Required

11.
13
Digital Rebar Scaffold
Distributed On-Premises
O
C
P
Stages
API
Plug-In
API
Template
API
Discover Customer
Burn-In
Install
Register
Inventory
Log
Notify
Verify
Alert
Ops Ticket
Reset
Chain Next
Install
Network
Grant SSH
Notify
RAID &
BIOS
Inventory
BIOS
RAID
IPMI
Chain Next
Post-
Provision
Grant SSH
Secure
CMDB
Notify
Hand OffChain Next Chain Next
P = provision C = control O = orchestrate

12.
14
● Always API first …
● CLI generated from API
● UX consumes API
● webhooks
● integrations inbound
...and outbound

13.
15

14.
16
Provisioning
System
Requested
State
Returned
State

15.
17
Provision requests are for a system state
with optional parameters.
The intermediate changes to achieve the
state are not exposed to the requester.
Operators of the provisioning system
require high transparency, staged
operation and control.
Provisioning
System
Reset Join
Install
Config
Test
Requested
State
Returned
State

16.
19
provisioned systems
Lightweight. Atomic. Coordination. Not Orchestration.
performed during provisioning activities
"secret"
master
DRP endpoint:
coordinates secret
management - locks minions
until master is ready
grantor:
provides "secret"
minions

17.
20
provisioned systems
Lightweight. Atomic. Coordination. Not Orchestration.
performed during provisioning activities
"secret"
master
DRP endpoint:
coordinates secret
management - locks minions
until master is ready
grantor:
provides "secret"
minions
followers:
use "secret" for cluster
action - typically "join"
join

18.
22
A Bootstrapping Illustration
1
node01 node02 node03 node04

19.
23
A Bootstrapping Illustration
Install
O/S
Install
Docker
Install
O/S
Install
Docker
1
2
Install
O/S
Install
Docker
Install
O/S
Install
Docker
node01 node02 node03 node04
Install Stage

20.
24
A Bootstrapping Illustration
Install
O/S
Install
Docker
Kubeadm
-- Init
Cluster
Token
Install
O/S
Install
Docker
1
2
3
4
Install
O/S
Install
Docker
Install
O/S
Install
Docker
node01 node02 node03 node04
selected master

21.
25
A Bootstrapping Illustration
Install
O/S
Install
Docker
Kubeadm
-- Init
Cluster
Token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
1
2
3
4 5
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
node01 node02 node03 node04

22.
26
Later...
A Bootstrapping Illustration
Install
O/S
Install
Docker
Kubeadm
-- Init
Cluster
Token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
1
2
3
4 5
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
node01 node02 node03 node04 nodeN

23.
27
A Bootstrapping Illustration
Install
O/S
Install
Docker
Kubeadm
-- Init
Cluster
Token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
1
2
3
4 5
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
Install
O/S
Install
Docker
Kubeadm
-- Join
wait for
token
node01 node02 node03 node04 nodeN

24.
<<< Shift Left

25.
29
The Problem
package
server
image
provision
server
initial
config

26.
30
The Problem
package
server
image
provision
server
initial
config
patch 1 patch 2

27.
31
The Problem
package
server
image
provision
server
initial
config
patch 1 patch 2
(the madness doesn't
stop at "patch 2" !!)

28.
32
The Problem
package
server
image
provision
server
initial
config
patch 1 patch 2
bad !!

29.
33
Apply cloud and container lessons to our Bare Metal …
package
server
image
provision
server
initial
config

30.
34
Apply cloud and container lessons to our Bare Metal …
package
server
image
provision
server
initial
config
package
server
image
patch 1
patch 2

31.
35
Apply cloud and container lessons to our Bare Metal …
package
server
image
provision
server
initial
config
package
server
image
provision
server
initial
config
destroy!!
patch 1
patch 2

32.
36
Apply cloud and container lessons to our Bare Metal …
package
server
image
provision
server
initial
config
package
server
image
provision
server
initial
config
destroy!!
destroy!!
patch 1
patch 2
package
provision

33.
37
Refining Our Immutable Pattern
● live boot - in ram/memory
○ reboot to apply updates
○ fast
○ consume additional memory
○ makes Provisioner more "critical path"
● Installed - to local disk
○ slower to install
○ frees memory resources
○ provisioner is less critical to Update operations

34.
38
Refining Our Immutable Pattern
● Packages (repos, kickstart, preseed)
○ VERY hard to control dependencies (pkg/lib versions)
○ easy pattern to implement
○ can define the configuration of system immutably
○ MUST own your own repos with controlled updates to pkgs
● Image based (raw, tarball, wim)
○ super FAST to install
○ requires a image build solution (eg "Packer")
○ enables very "far left shift"
○ stronger guarantees of Production deployment matching CI/CD, Dev/Test, etc.

35.
For self-trial, start at http://rebar.digital
• Quickstart takes about 30 minutes
• Use your own hardware, VirtualBox or Packet.net
account
– use “RACKN100” on Packet.net for credit
Digital Rebar UX is the RackN.com Portal
- https://rackn.github.io/provision-ux/
• Portal use is free
• Registration unlocks additional packages

36.
40
Give this thing a kick …
● QuickStart
○ http://provision.readthedocs.io/en/latest/doc/quickstart.html
● KRIB - Kubernetes Rebar Immutable Bootstrap
○ http://provision.readthedocs.io/en/latest/doc/integrations/krib.html

37.
41
Introductory:
Feature Landscape: https://goo.gl/GYtwNS
Installation: https://goo.gl/BoQG8J
Configuration: https://goo.gl/BzJzTP
Content Introduction: https://goo.gl/LChN6r
Understanding Stages: https://goo.gl/iUjNNJ

38.
42
Digital Rebar: http://rebar.digital/
RackN: https://www.rackn.com/
Doc for Digital Rebar: http://provision.readthedocs.io/en/latest/#
Digital Rebar Meetup: https://www.meetup.com/digitalrebar/
Slack Sign Up: https://www.rackn.com/support/slack
IRC Freenode: https://webchat.freenode.net/?channels=%23digitalrebar
Gitter: https://gitter.im/digitalrebar/core
Twitter: https://twitter.com/rackngo/
LinkedIN: https://www.linkedin.com/company/4809859/
YouTube Channel: https://www.youtube.com/channel/UCr3bBtP-pMsDQ5c0IDjt_LQ/
YouTube Playlist: https://www.youtube.com/playlist?list=PLXPBeIrpXjfilUi7Qj1Sl0UhjxNRSC7nx
RackN Blog: https://www.rackn.com/blog/