Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Wayin devops-2013
1. DevOps at Wayin
Orchestration with AWS Cloud Formation
David M. Johnson
Software Developer - Wayin, Inc.
Friday, June 21, 13
2. Agenda
•What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13
3. “Orchestration describes the
automated arrangement,
coordination, and management
of complex computer systems,
middleware, and services.
It is often discussed as having
an inherent intelligence or even
implicitly autonomic control, but
those are largely aspirations or
analogies rather than technical
descriptions.”
http://en.wikipedia.org/wiki/Orchestration_(computing)
Friday, June 21, 13
4. “cloud orchestration =
automation
+ integration
+ best practices”
https://www.ibm.com/developerworks/mydeveloperworks/blogs/9e696bfa-94af-4f5a-ab50-
c955cca76fd0/entry/orchestrating_the_cloud_to_simplify_and_accelerate_service_delivery1
Friday, June 21, 13
5. Cloud Orchestration is...
Automating the infrastructure
needed to provide a computing
service so that resources are
automatically allocated and de-
allocated to meet changing demand
Friday, June 21, 13
8. Agenda
• What is Cloud Orchestration
•Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13
16. Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
•Wayin Hub deployment & app architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13
17. Glossary
• EC2 - server virtualization
• EBS - mountable block storage
• ELB - load balancer
• R53 - domain name services (DNS)
• S3 - file / bucket storage service
• CloudFront - content distribution network (CDN)
• Autoscale - automatically stop & start servers
• CloudWatch / Cloud Alarms - monitoring & triggering
• Security Group - restrict port access on group of instances
• SQS - Simple Queuing Service
Friday, June 21, 13
20. Cloud Formation
“An easy way to create and manage a collection of related
AWS resources, provisioning and updating them in an orderly
and predictable fashion.
You can use AWS CloudFormation’s sample templates or create
your own. Once deployed, you can modify and update the AWS
resources in a controlled and predictable way allowing you to
version control your AWS infrastructure in the same way as
you version control your software.
You can deploy and update a template and its associated collection
of resources (called a stack) via the AWS Management Console,
CloudFormation command line tools or APIs. CloudFormation is
available at no additional charge, and you pay only for the
AWS resources needed to run your applications.”
Friday, June 21, 13
21. Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
•How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13
22. Separate auto-scale groups
Ubuntu
API server
WAR
Jetty
Ubuntu
API server
WAR
Jetty
Ubuntu
API server
WAR
Jetty
Min: 3
Max: 9
Scale up (quickly):
CPU > X% forY minutes
Scale down (slowly):
CPU < A% for B minutes
Ubuntu
Cassandra
Redis
Memcache
DB serverUbuntu
Cassandra
Redis
Memcache
DB server
Min: 2
Max: 2
We decide when to scale
up and scale down our
DB server instances
Ubuntu
Tracker
WAR
Jetty
Ubuntu
Tracker
WAR
Jetty
Ubuntu
Tracker
WAR
Jetty
Min: 2
Max: 20
Scale up (quickly):
Queue has > X
messages forY minutes
Scale down (slowly):
Queue has < X message
forY minutes
Friday, June 21, 13
23. CFN Launches
new API server
instances
1
Self-configuring API servers
DB Security Group
S3 ELB
CloudInit script
downloads the
deployment bundle
2
CloudInit scripts
find DB servers by
Security Group,
starts Jetty
3
CloudInit
scripts put
server into
ELB rotation
4
API Security Group
CFN
Cloud
Watch
Friday, June 21, 13
24. Self-configuring DB servers
DB Security Group
S3
CloudInit scripts finds other
DB servers by Server Group
3
CloudInit scripts
download latest
deployment bundle
2Human runs the
command to increase
DB servers by one
1
Ubuntu
New DB server
New Cassandra node joins
cluster when ready
7
CFN
CloudInit scripts write
Memcache config, start it
4
MemcacheCloudInit scripts write Redis config,
assume first server is master, start it
5
Redis
CloudInit scripts write
Cassandra config, start it
6
Cassandra
Friday, June 21, 13
25. Automated deployment
API Security Group
GitDeveloper merges code
into deployment branch
1
Developer kicks off
Jenkins build
2
S3
Jenkins copies
bundle to S3 bucket
3
Developer kicks off
deployment build
4
Jenkins runs
deploy command
via SSH
5
On each server the script
runs locally, downloads
and installs the bundle
6
Ubuntu
Jenkins
Tracker Security Group
Friday, June 21, 13
26. Incremental backups
copied to S3 bucket
5
Automated Backups
S3
Jenkins runs
daily backup job
1
Backup script
creates EBS snapshot 3
DB Security Group
Cronjob kicks off
incremental backups
4
Cassandra snapshot
backups happen
simultaneously
2
Ubuntu
Jenkins
EBS
Friday, June 21, 13
27. Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
•About Cloud Formation Templates
• About vendor lock-in
• Designing for orchestration
Friday, June 21, 13
35. Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
•About vendor lock-in
• Designing for orchestration
Friday, June 21, 13
36. Lock-in in automation
• Cloud Formation itself is lock-in
• Plenty of AWS dependencies in our:
• Instance init scripts
• Deployment scripts
• Backup scripts
Friday, June 21, 13
37. No lock-in in app code
• Bulk of our code is in our application
• AWS dependencies are pluggable
Friday, June 21, 13
38. Agenda
• What is Cloud Orchestration
• Quick intro to Wayin Hub
• Wayin Hub deployment & application architecture
• How we scale, deploy and do backups
• About Cloud Formation Templates
• About vendor lock-in
•Designing for orchestration
Friday, June 21, 13
39. Choose right components
• Good
• Peer-to-peer
• Linear scalability
• Bad
• Master-slave
• “Special” servers
• Single point-of-failure
Friday, June 21, 13