Owain will talk about the journey JustGiving.com have gone through to get to Continuous delivery on their Windows environment. He will talk about what they did, how they did it and lessons learned along the way
Video: https://www.youtube.com/watch?v=MVXaR6oEK60
Join DevOps Exchange London here: http://www.meetup.com/DevOps-Exchange-London
Follow DOXLON on twitter http://www.twitter.com/doxlon
Owain Perry (Just Giving) - Continuous Delivery of Windows Micro-Services in the Cloud
1. Continuous Delivery of Windows
microservices in the cloud
@owainperry
http://www.slideshare.net/perryofpeek/continuous-delivery-of-windows-micro-services-in-the-cloud
2. What?
● The journey of the last 18 months,
Justgiving has taken to ship more in less
time
● How we have implemented supporting
infrastructure around micro services
● Where we went wrong
3. ● Raise Money for Charities
● Turnover ~£400 Million a year
● Process UK gift aid
● Text Giving
● Crowd funding
● ~1000 nodes
7. JustGiving micro service?
● It's a small service!
– Usually a component makes up part of a
business domain
– Has one purpose
– Is simple to reason about
– Reality - various levels of granularity
8. Where we went next
● New stuff:
– built using micro services
– Hosted in the cloud
– New pipelines for deployment
● Monolith stuff:
– Chop out choice areas, bite size chunks at a time
– Build a simpler pipeline
– Simpler branching strategy
9. Infrastructure principles
● “Very” Available
● Repeatable through automation
● Development define and build their
infrastructure
● Keep environments isolated
● Be pragmatic….
10. Train track automation
● Development - self service infrastructure
– Fast to spin up infrastructure
– Less Ops bottleneck
● Reduce the need for AWS keys
● Consistency benefits:
– Service architecture
– Names and name conventions
– Tags
– Security groups
– Scripting
11. Types of service
● 1. Windows service
Either:
– Self hosted nancyfx web server
● Application
● Health endpoint
– Message based service
● Health endpoint
● 2. IIS hosted application
– Health endpoint
12. Core tools
● AWS
● Artifactory – package repository
● Chef – open source v11
● GO CD
● Team City
● Chocolatey
13. Auto scale groupAuto scale group
Micro service hosting
architecture
● Varnish is the load balancer
Elastic Load BalancerElastic Load Balancer
Varnish
Node 1 Node 2 Node 1Node 2
14. CD of micro services
Each micro service has its own
● Git repository
● GO Pipeline
● Team City build
● Chef cookbook
● Tests
● AWS Infrastructure
● Operations run book
● Health check endpoint - /status/health
● Blue / Green Deployment
15. Steps for a new micro
service
● Think of a name
● Create a git repository
● Open up micro service template in visual studio
– Rename a few things
● Teamcity build (copy n paste)
● Go pipeline (copy n paste)
● Define Infrastructure via json
18. AWS infrastructure
● Treat AWS as another resource
● Heavy Weight Resource Providers for AWS
● Light weight resource Providers to abstract the HWRP
● json data_bags based infrastructure
● Infrastructure builder node executes chef client for
infrastructure
● “Most” of our infrastructure is built this way, not just
business application services
21. Micro service cookbook
● Chef is used to configure the node
● Chef is used to deploy the application
● “deployment” - is just version constraint in a
chef environment
22. Deployment
● Set the version constraint for the inactive
environment
● Chef search all nodes
● For each node
{ remote in an kick off chef client scheduled task}
● Switch active environment
● Poll for nodes to be on correct version.
● Post deployment tests
24. Bootup
● Chef is prebaked into AMI (faster)
● Instance userdata script contains the bootup
script (inc runlist)
● Auto scale groups used to instantiate new
instances
25. Automated change
● Change log automated into Jira for releases
– Staging and production
● Change log automated for cookbook
releases
● Platform to include more over time.
28. What we didn't get right...
– Zombies….
– Chocolatey (exe / bat)
– AWS reboot
– Cost control
– Abstraction of infrastructure json
– Chef search to configure load balancer
– Not automating our base AMI's from the outset
29. Future
● Rolling instance recycling – daily / weekly?
● Bake images in production
● Ohai for sensu subscriptions
● Service discovery rather than chef search
● Application secret security
● Code security scanning in the pipelines
● Infrastructure penetration testing
● Infrastructure security checks
● Packer to build base ami's
30. So….
– Micro services – has helped productivity
– AWS automation in hands of developers
– Simple json defined infrastructure
– Chef is core at the centre
– We have made mistakes
– But it's been a worthwhile journey so far.
33. Chef resources
● Elastic Load Balancer
● Auto scale groups
● Security groups
● IAM roles
● Launch configurations
● DNS
● Subnets
● Ec2 instances
● Route tables
34. HWRP example
aws_subnet 'subtest2' do
accesskey accesskey
secretkey secretkey
region “eu-west-1”
vpc “DEV-VPC”
azone 'eu-west-1a'
cidr '10.20.30.0/24'
routetable 'RTB-STG-PRIVATE'
tags [ { "a" => "1"},{"c" => "2" }]
action :create
end
36. Chocolatey
● Rebuild all packages for internal hosting.
– Don't want a dependency on 3rd party download
– Control versions of software in use
● Common pattern for git and builds
– Take ~10 mins to repackge
● ( This is a bit of a chore )