2. About
Kenzan
Core offerings
Application Development, Platform as a service,
cloud virtualization, platform engineering,
consulting services and business transformation.
Primary Clients
Multi billion dollar companies and media/content
providers such as Thompson Reuters, Charter &
Cablevision
Locations
Providence (RI), New York (NY), Denver (CO), Los
Angeles (CA), and a London presence
Founded in 2004.
We are a software engineering and digital
consulting firm that has been helping
clients Make Next Possible for over a
decade:
Full Service Consulting Firm
Architecture, front and back end development,
business analysis and DevTest.
Cloud Virtualization Experts And Enablers
AWS, Netflix stack, enterprise architecture and
beyond.
DevOps Leadership
Platform builds, continuous delivery and scalable
resourcing.
Veterans of the Media Industry
Migrations, enterprise wide solutions, digital experts
and thought leaders.
Employee focused
Collaboration, communication and culture are key.
3. Agenda
● What is CI/CD
● Deployment types
● On prem physical servers
● Calculating health against SLO’s and SLA’s
● Canary Deployments
● Common causes of outages
7. Deployment Pipelines
Complex
Unit Tests Integration Tests
End to End Tests
Stress Tests
Test AWS account
Git Push
Stable AWS
account
Manual Judgment Production
8. Cattle Not Pets
Pets:
Servers or server pairs that are treated as indispensable or unique systems that
can never be down. Typically they are manually built, managed, and “hand fed”.
Cattle:
Arrays of more than two servers, that are built using automated tools, and are
designed for failure, where no one, two, or even three servers are irreplaceable.
Typically, during failure events no human intervention is required as the array
exhibits attributes of “routing around failures” by restarting failed servers or
replicating data through strategies like triple replication or erasure coding.
15. Cloud Bursting
Cloud bursting is an application deployment model in which
an application runs in a private cloud or data center and
bursts into a public cloud when the demand for computing
capacity spikes.
17. Pilot Light
Design:
● Images (AMI’s, containers, ect) are copied to the cloud
● Auto Scale Groups update to use latest image
● Cluster sizes set to 0
Benefits:
● Low overhead costs
● Can be activated fairly quickly.
18. Warm Standby
Design:
● Images (AMI’s, containers, ect) are copied to the cloud
● Auto Scale Groups update to use latest image
● Cluster sizes set to a reasonable number, no less than 2.
Benefits:
● Can be activated instantly
19. Multi-Site
Design:
● Images (AMI’s, containers, ect) are copied to the cloud
● Auto Scale Groups update to use latest image
● Cluster sizes set to a reasonable number, no less than
2.
● Some traffic is always directed to the cloud servers.
Benefits:
● Always active
● Far away regions can use the cloud for reduced latency.
21. Service Level Objectives
&
Service Level Agreements
SLO:
SLOs are specific measurable characteristics of the SLA
such as availability, throughput, frequency, response time, or
quality.
SLA:
The SLA is the entire agreement that specifies what service
is to be provided, how it is supported, times, locations, costs,
performance, and responsibilities of the parties involved.
22. Uptime and Automation
99.9% 8 hrs 45 mins
99.99% 52 mins
99.999% 5 mins
Uptime Percentage Acceptable yearly outages
26. Slow is better
We want to make sure our software works in the real world.
But...
Users are both predictable and unpredictable
Different regions and devices may behave differently
Some issues (memory leaks) only appear overtime.
27. Canary Watcher
Simple script run every 10 mins and monitors health / logs.
Keeps track of the deployment state (10%, 50%, ect)
Automatically remove from LB if an SLO is missed.
Can be run ad hoc