San Francisco AdvancedAWS Meetup, 2016-03-17

1. (and AWS Lambda)

3. & aoepeople!

4. E-Commerce:
Magento
CMS:
TYPO3
Portals:
ZF, Symfony,…
Mobile Searchperience:
ElasticSearch
250+ people
world-wide
(in 8 locations)
Global
Enterprise
Projects
Infrastructure:
AWS

5. CloudFormation Lambda “Immutable”

6. /i(m)ˈmyo͞ odəb(ə)l/
adjective
unchanging over time or unable to be changed.
“disposable”
“ephemeral”

7. Pet
Cattle

13. not disposable disposable

14. disposable
not disposable
disposable

15. Static Resources BuildVPC Build

17. Build Bucket
IAM Setup
VPC
Static Resources
Build
Manual Setup
separate
CloudFormation
Stacks
use resources
from underlying
stacks as input
parameters
Lambda functions,
Monitoring,…

18. Build Bucket
IAM Setup
Environment A (e.g. prod) Environment B (e.g. stage)
VPC VPC
Static Resources Static Resources
Build BuildBuild Build
Build X Build X+1 Build X Build X+1
Manual Setup
different scopes
(build, environment,
account, global…)

19. Build Bucket
IAM Setup
Environment A (e.g. prod) Environment B (e.g. stage) Environment C (e.g. qa) Environment D (e.g. dev)
IAM Setup
VPC VPC VPC VPC
Static Resources Static Resources Static Resources Static Resources
Build BuildBuild Build Build Build Build Build
Build X Build X+1 Build X Build X+1 Build X Build X+1 Build X Build X+1
Manual Setup Manual Setup
Build Bucket Access

20. Private subnets
Public subnets
ElastiCache (Redis)
with replication groups
for cache and sessions
RDS (multi-az) with
DB subnet group
Bastion
server
s3: media
storage*
Route 53: DNS
configuration
CloudFront
distribution
SSL
Certificates
Security group for Varnish servers
Security group for Magento servers
Security group for Load Balancer
Static Resources

21. Build
Auto Scaling group
Auto Scaling group
Elastic Load
Balancer
Auto-
Scaling
Group
Launch
Configurati
on
Scaling
Policy

22. Build
Auto Scaling group
Auto Scaling group
Elastic Load
Balancer
Auto-
Scaling
Group
Launch
Configurati
on
Scaling
Policy
Auto Scaling group Auto Scaling group

23. +1
Baking AMIs

24. “Chef vs. Puppet?”
http://fbrnc.net/blog/2015/11/
how-to-provision-an-ec2-instance
“…Ansible!”
“BASH!”

25. Keep it simple…!

26. ✔ ✔ (✔)

27. the “last mile”
most
underestimated
CFN feature!

29. var r = require('cfn-response');
exports.handler = function (event, context) {
[…]
var res = {};
if (event.RequestType == 'Create') {
res.Password = randomPassword(20);
}
r.send(event, context, r.SUCCESS, res);
};

30. "IndexerDb": {
"Type": "AWS::RDS::DBInstance",
"Properties": {
[...]
"MasterUserPassword": {"Fn::GetAtt": ["GenerateDbPassword", "Password"]},
[...]
}
},
"GenerateDbPassword": {
"Type": "Custom::PasswordGenerator",
"Properties": {
"ServiceToken": {"Ref": "PasswordGeneratorArn"}
}
},

32. • launch some ASGs (set DesiredCapacity)
• create database passwords
• tag all resources (incl. ElastiCache!)
• restore database and media files
• (one-time) install scripts (db migrations,…)
• detect Varnish backends
• wait for healthy backends in the ELBs
• run infrastructure tests
• cache warming
• update Route53 records sets
• delete old stacks
• …

34. DesiredCapacity =
Number of Instances in current ASG x 1.2
better safe
than sorry…

35. DesiredCapacity =
Number of Instances in current ASG x 1.2

36. DesiredCapacity =
Number of Instances in current ASG x 1.2
ScalingPolicy
takes care…

37. "CountInstances": {
"Type": "Custom::InstanceCounter",
"Properties": {
"ServiceToken": {"Ref": "InstanceCounter"},
"AutoScalingGroupTags": [
{"Key": "Environment", "Value": "prod"},
{"Key": "Type", "Value": "Frontend"}
],
"Min": 1, "Max": 10, "Factor": "1.5"
}
},
"FrontendAsg": {
"Type": "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
[...]
"DesiredCapacity": {"Fn::GetAtt": ["CountInstances", "Count"]},
"Tags": [
{"Key": "Environment", "Value": "prod", "PropagateAtLaunch": true},
{"Key": "Type", "Value": “Frontend", "PropagateAtLaunch": true}
]
}
},

38. Update Route53 record set

39. Update Route53 record set
Delete old stacks

40. "UpdateR53": {
"DependsOn": [ "MagentoWaitCondition", "MagentoWorkerWaitCondition", "Elb"],
"Type": "Custom::Route53Update",
"Properties": {
"ServiceToken": {"Ref": "R53Updater"},
"Name": {"Fn::Join": ["", [ "www-", {"Ref": "EnvironmentName"} ]]},
"HostedZoneId": {"Ref": "HostedZoneId"},
"AliasTargetDNSName": {"Fn::GetAtt": ["Elb", "DNSName"]},
"AliasTargetHostedZoneId": {"Fn::GetAtt": ["Elb", "CanonicalHostedZoneNameID"]},
"Comment": "Updated via CloudFormation/Lambda"
}
},
"DeleteStacks": {
"Condition": "DeleteOldStacks",
"DependsOn": ["UpdateR53"],
"Type": "Custom::StackDeleter",
"Properties": {
"ServiceToken": {"Ref": "StackDeleter"},
"TagFilter": { "Environment": {"Ref": "EnvironmentName"}, "Type": "Deployment"},
"ExceptStackName": {"Ref": "AWS::StackName"}
}
}

42. +1
Baking AMIs
Tools+CI

43. Have fun
filling out
47 form
fields! :)

44. aoepeople/stackformation
command-line tool
(Symfony console, uses
AWS SDK for PHP)
integrates nicely into
your CI (Jenknis,…)

45. pre-process
Userdata
Lambda JS
“Template”
superset of CFN
template (JSON)
Parameterslookup from other
stacks, env vars,…
Template
Parameter
Values
create/
update
Stack policies, Tags,…

46. Stack
magento-stage-build-5
Stack
magento-stage-build-6
Stack
magento-prod-build-6
Stack
magento-prod-build-5
CloudFormation
Template
merge &
pre-process
“CloudFormation+X”
Template(s)
+
Dynamic Parameters
+
Stack Policies
+
Behavior
+
Tags
…
Blueprint
magento-{env:ENVIRONMENT}-build-{env:BUILD}
Blueprint
magento-{env:ENVIRONMENT}-setup
Stack
magento-stage-setup
Stack
magento-prod-setup

47. blueprints:
- stackname: 'magento-{env:BUILD}'
template: 'magento.template'
stackPolicy: 'policy.json'
OnFailure: 'DO_NOTHING'
parameters:
Build: '{env:BUILD}'
KeyPair: '{var:KeyPair}'
VPC: '{resource:setupstack:VPC}'
Subnet: '{resource:setupstack:Subnet}'
InstanceSg: '{resource:setupstack:InstanceSg}'
InstanceProfile: '{output:setupstack:InstanceProfile}'
BootAmi: 'ami-06116566'
tags:
Environment: 'prod'
Build: '{env:BUILD}'
enforce
“immutability” by
denying updates!

48. aoepeople/cfn-vpc
aoepeople/cfn-lambdahelper
aoepeople/cfn-amibaker
via composer

49. so we can
integrate this into
our CI pipeline…

50. •

52. aoepeople/awsinspector
command-line tool
(Symfony console, uses
AWS SDK for PHP)
Domain models for PHP

53. $repository = new AwsInspectorModelElbRepository();
$dns = $repository->findElbsByTags([
'Environment' => 'deploy',
'Build' => 554,
'Type' => 'Frontend’
])->getFirst()->getDNSName();

54. > bin/awsinspector.php ec2:ssh -t Environment:prod –c Type –c Build
filter by tag
Please select an instance
[0] i-1033ed9b (Type: Frontend; Environment: prod; Build: 477)
[1] i-4ff36ec8 (Type: Backend; Environment: prod ; Build: 477)
[2] i-5ab4322b (Type: Worker; Environment: prod; Build: 477)
[3] i-705ad42f (Type: Worker; Environment: prod; Build: 476)
>
• will take jump hosts into
account (ProxyCommand)
• auto-detects your local
(encrypted) private keys
• multiplexed ssh connections
• run commands directly

56. logins,
orders,…
deployments,
scaling activity,…
JMeter
response time,
error rate,…
CPU, load,
network I/O,…

57. correlate
metrics from
various sources

58. GrafanaElasticSearch
(Service)
time-series
database

59. leftovers from
“disposed” resources

60. …
“Ready for testing”
(WaitCondition)
setup
terminate
instances
and wait
• Create Security
allowing access
to all IPs and
attach to UAT’s
ELB
environment
(incl. restoring db+media
snapshot from prod)
environment*
Delete self*
*via Lambda custom resource
Pushing samples in near real-time
via custom JMeter backend
listener
“Testing Completed”
(WaitCondition)
JMeter test
CloudFormation
Stack
GrafanaElasticSearch
(Service)
CloudWatch
Lambda
find relevant resources for the
current deployment (via tags),
collect metrics,
and push them to ElasticSearch

61. CloudFormation Stack
• Install JMeter
• Download
testcase from S3
• Signal “Ready”
and wait
• Create SG
allowing access
to all IPs and
attach to UAT’s
ELB
Auto-Scaling Group of Load Generator Instances
Run Stress Test2.
Time
timeout 3600 jmeter –n –t testcase.jmx
Delete test
environment*
Delete self*
3. 4.
*via Lambda custom resource
…
Spin up test
environment
(incl. restoring db+media
snapshot from prod)
1.
CloudFormation
Stack
setup
terminate
instances
JMeter test
Pushing samples in near real-time
via custom JMeter backend listener
“Ready for testing”
(WaitCondition)
“Testing Completed”
(WaitCondition)

65. https://blogs.aws.amazon.com/application-
management/post/Tx38Z5CAM5WWRXW/Faster-Auto-Scaling-in-AWS-
CloudFormation-Stacks-with-Lambda-backed-Custom-Resou

66. Provisioning
“Ready for baking”
(WaitCondition)
EC2
AMI Baker
(Lambda) Node.js SDK
AWS CLI
“Baking Completed”
(WaitCondition)
shutdown
(-> terminate)
ec2.createImage
aws ec2 wait image-available
AMI Baker
(Lambda) Node.js SDK
ec2.deleteImage
Custom
CloudFormation
Resource
Time

68. •

69. Follow me on twitter!
My blog