The virtues of continuous delivery are widely understood and accepted by organisations which value fast feedback cycles, reduced risk through incremental delivery of smaller changes and the ability to respond quickly to external factors. Furthermore if microservices are part of your architecture, then the ability to rapidly deploy multiple components of a system become increasingly important. The foundations of scripting, automation and more recently containers made *nix-based systems the first target for automated deployments and subsequently continuous delivery. With the advent of some new tooling and a bit of courage these principles can now be applied to more heterogeneous environments including those from Redmond. Using their backgrounds in automating large-scale ruby and java-based deployments, Warner and Matt embarked on a journey with SEEK to increase their agility by enabling continuous delivery – typically multiple times per day. This is their story.

1. Rise of the Machines
Continuous Delivery at SEEK
CO PRESENTED BY:

2. CD @ SEEK
“To the cloud”

3. Scorched Earth Deployments
● Create a pristine environment for every release.
● No need to migrate state of environment.
○ ∴ Configuration Management tools not
required.
● State should not be changed at runtime.
○ ∴ Immutable Server
● Blue/Green deployment strategy:
○ Monitor health during deployment.
○ Auto-rollback if new environment is unhealthy.

4. Automate all the things!
● Delivery team is responsible for the automation of:
○ Server installation
○ Infrastructure creation
○ Deployment
● Automation is impossible without team managed
infrastructure.

5. Infrastructure as Code
● SEEK use AWS CloudFormation to automate their infrastructure.
● Operations are responsible for the network-level infrastructure,
including:
○ VPC
○ Routing
○ Users and Roles
○ Shared services
● Delivery team is responsible for infrastructure within the VPC, such
as:
○ Elastic Load Balancer
○ Security Group
○ Auto Scaling Group
○ Launch Configuration
○ Instance Profile

6. Server automation
● Provisioning Linux:
○ SSH ⇒ Easy!
● Provisioning Windows:
○ WinRM ⇒ Hard!
○ AWS makes bootstrapping easier:
■ EC2 User Data
■ cfn-init
○ Packer Windows Plugins can be used to create Windows Amazon
Machine Images
● Combination of Packer and CloudFormation can be used regardless
of target platform

7. Packaging
● Applications should be self-installable.
● Application installation should be separate from server automation.
● Installation should be independent of Continuous Deployment tool.
● Native package manager is ideal solution:
○ Linux: apt-get or yum
○ Windows:
■ One-Get (available in next version of Windows)
■ Chocolatey (available now!)
● Package manager is platform independent pattern
○ <package-manager> install <package-name>
● Windows Coding Horror:
○ Interacting with Windows resources is not trivial, custom
PowerShell modules or PowerShell DSC Resources make this
easier.

8. Pre-Baked Images
● Install a package on a base image.
○ But… don’t include configuration.
● Then, create an image (AMI).
● AMI is our environment independent deployable
artifact.
● Future is containerised...
○ Docker is the next evolution of our approach.

9. Late-binding of context
● Keep your artifact environment agnostic:
○ Frameworks want you to apply configuration at build or
package-time.
○ Deploy tools want to apply configuration at install-time.
○ When deploying to the cloud, you should apply configuration at
launch-time.
● When AMI is launched it doesn’t know where it is:
○ Need to inject environment-specific context at launch-time.
○ How do you do this?

10. Environment Variables
A platform independent way of specifying environment-specific config that’s
been around since 1979
● Use environment variables for:
○ DB connection parameters.
○ Location of service endpoints.
● Don’t use them for:
○ Altering runtime behaviour.
● Overriding .NET configuration with environment variables:
○ .NET 4.5.x: write your own Configuration Provider.
○ .NET 4.6: works out of the box.
○ Azure + .NET 4.5.x: works out of the box.
● Service discovery is on the horizon:
○ SEEK is investigating.
○ Externalised config is first step.

11. Confidence to Release
● Put enough steps and checkpoints in pipeline to get enough
confidence to release.
● Too many checks will result in a slow pipeline and kill the benefits of
Continuous Delivery.
● Take things away until you cry.
○ Then... compensate some other way
○ For example, consumer-driven contract testing makes sure we
honour API contracts without the need to integrate.

12. Ship It!
● Distinction between Deployment and release:
○ Deploy: create Blue environment and deploy application.
○ Release: cut-over from Green to Blue environment.
● Smoke test in Blue environment before release using synthetic
transaction.
● Same smoke test can be used to monitor health after release.
● Choose to enable manual release if change set is risky and additional
manual testing is required.

13. Towards Polyglot Architecture
● Continuous Delivery + Microservices gives teams the ability to choose
the right stack for the job.
● For SEEK, CD is opening up the possibility migrating legacy apps from
Windows to Linux.
○ Who knows, in the future, running Windows containers will be a
viable alternative.
● SEEKs approach to Continuous Delivery provides better logical
separation automation steps.
● Time to first commit still a problem and this will only get worse with
the proliferation of different stacks and tools.

14. Machine Factories

15. Manual Development Environments
■ Are a bad experience for your developers.
■ Are time consuming and therefore costly.
■ Make it difficult for devs to move between projects.
■ Are rarely accurately reconstructed.
■ Encourages manual configuration rather than the more
reliable and repeatable practice of Infrastructure-as-code.
Can lead to environments that have all of the parts, but not quite
in the right order.

17. “It works on my box”
■ Sign of environmental divergence.
■ The Butterfly effect: Minor differences in environment
configuration can produce obscure bugs in Prod.
■ This issue is compounded when deployments carried out by
another person (e.g. Ops).

19. Dev vs Production Parity
■ Environment Parity = Confidence.
■ Overcompensate lack of parity with more, possibly manual tests.
■ Environment Parity…
○ enables continuous delivery .
○ requires diligence and automation for accurate replication.
○ encourages infrastructure-as-code.

20. Wouldn’t it be great if we...
■ Could use declarative formats for setup automation, to
minimize time and cost for new developers joining the
project.
■ Had a clean contract with the underlying operating system,
offering maximum portability between execution
environments.

21. Wouldn’t it be great if we...
■ Built apps suitable for deployment on modern cloud
platforms, obviating the need for servers and systems
admin.
■ Could minimize divergence between development and
production, enabling continuous deployment.
■ Could scale up without significant changes to tooling,
architecture, or development practices.

22. The “12 Factor App”
■ These 5 principles are what loosely describe the 12 factor
App, a guideline for modern SaaS products.
■ 12 Factor Apps, Factor #10: Dev/prod parity
■ “Keep development, staging, and production as similar as
possible”.
■ Vagrant can help us solve the environment parity issue.

23. Vagrant, Docker Compose &
Friends
Onward to immutable development environments

24. Scorched Earth Dev Environments
■ Dev automation typically come in 2 flavours:
○ Use of CM Tools like Puppet, Chef (Boxen, Babushka)
○ Virtual Machine Runners (Vagrant, Docker Compose)
■ CM Tools fraught with challenges:
○ Not isolated
○ Hard to automate across OS/Platforms
○ Doesn’t force developers to automate

25. What is Vagrant?
“A tool to help create and configure lightweight, reproducible, and
portable development environments.”
■ A replacement for manually configured boxes.
○ Enforces infrastructure-as-code.
■ A machine runner (typically a VM or a Container)
■ Separates authoring environment from development runtime
■ Isolated & repeatable

26. What is Vagrant?
The following is a logical view of how Vagrant interacts with your
authoring environment:
■ Vagrant has spun up Virtualbox VM with the name ‘Web’,
and is running it on a Mac OSX desktop
■ Code is synchronised from /c/proj/x -> /vagrant of
the running environment

27. What is Vagrant?
■ The Web application spins up a server using the code in
/vagrant, which binds to a private network on the VM on
port 80
■ Port 8080 is exposed to the outside ‘world’ (in this case, a
Mac OSX desktop) which forwards requests to port 80 in the
running VM
■ The Developer is free to author in whichever IDE of her
choosing

29. The TAO of Vagrant
■ vagrant up to get a fully working dev runtime.
■ Development occurs in preferred IDE: Visual Studio,
Sublime, ViM, with your terminal, your shortcuts - etc.
■ vagrant destroy && vagrant up will destroy the
machine and create a new, clean, isolated environment.
■ Workflow repeatable and transferrable to other projects.

30. Developing on Ecosystems
From monoliths to ecosystems

31. From Monolith to Microservices
■ Applications are usually more interesting than a single UI,
they are usually connected to things.
○ They form part of a functioning ecosystem.
■ Vagrant good for Monoliths and ecosystems.
■ Explosion of microservices highlights this need.

33. From Monolith to Microservices
Commentary on Architecture:
■ Multiple components, in disparate technologies
■ Distributed system
■ Layered architecture
■ = complexity
Increased complexity increases the number of things that can
go wrong. We can combat this with parity and local simulation.

34. From Monolith to Microservices
■ Vagrant can replicate ecosystems locally, while still having
the benefits of isolation, repeatability, immutability etc.
■ Running ecosystems locally gives us flexibility.
■ Vagrant can simulate or test a number of scenarios, for
example to:
○ Model distributed systems behaviour
○ Perform disaster testing; outages, network partitions ...
○ Replace components independently
○ Test OS/DB upgrades
○ Run entire production replica ecosystems, locally

35. Local Ecosystem Setup
The following is a Logical View of this fictional ecosystem with
Web and API applications exposed to the ‘World’ and a database
hidden ‘behind the firewall’:

37. Local Ecosystem Setup
■ Note that we are running multiple virtualisation
technologies here as if they were all running locally
■ Vagrant will spin up 3 local VMs:
○ Web and API applications exposed to the ‘World’ and a
database box hidden ‘behind the firewall’:
■ The 4 VMs are able to communicate through a private
network on their native ports (80, 80 and 27017
respectively).

38. Native to Docker and is fast, but some caveats:
■ Unless on linux, networking is not automatically exposed to
the host machine.
■ You need to ensure that your apps are compiled/built within
the Docker containers - otherwise you’ll still need to
bootstrap your dev box anyway!
■ Hybrid VM/Docker setups not supported.
Docker Compose

39. Machine Image Pipelines with
Packer
Where environments are born

40. Cattle in Production, Pets elsewhere
■ Dev == Prod - but what about our build/other servers?
■ We don't want cattle in production and pets elsewhere.
■ Quality starts before coding begins
○ Must think about environments we develop, build and
deploy to.
■ How do we ensure other environments are configured
identically?
-> Immutable servers everywhere!

41. The Machine Factory
■ How do we go about creating and managing all of these
machine images?
■ At SEEK, we needed a way to generate all of these machine
images in a sensible, standardised and organised way.
■ Enter: The Machine Factory:
○ “A process for generating machine images for use in
different contexts such as development, test, build, prod
and so on.”
■ So what does this look like?

42. [Insert diagram here]

43. Machine Factory
■ Assembly-line style manufacturing:
○ Input images (AMI, ISO, Docker Image etc.)
○ Apply Build Steps with shared infrastructure code, using
CM tools, shell scripts etc.
○ Output Images (AMI, ISO, Docker etc.)
■ Images put in stasis for their destined environment/context
where they will be reanimated.
■ CM tools as a code smell:
○ Your application might be too big. For this step,
lightweight options usually suffice.

44. Tuning the assembling line: Pipelines
■ Booting machines and provisioning from scratch is slow.
■ 2 main strategies: Intermediate images & enrichment.
■ Create intermediate images to shortcut future builds:
○ Install common infra such as log forwarders, statsd, AV,
patches/updates etc.
○ Engineering wide (Platform), team wide (Base) or specific
Application images.

45. Tuning the assembling line: Enrichment
■ Prod should have only the runtime dependencies
○ Better Security.
○ Better Performance.
■ Dev will have the most. This can be expressed as:
Prod ⊊ Build Server ⊊ Dev
■ Enrich machines from left to right in your pipeline, feeding
previous image into next step.

46. Machine Factory @ SEEK
Walking the walk

47. Applying CD to Talent Search
■ Talentsearch previously part of .NET monolith.
■ Talent Search is a web-based tool that Recruiters use to
connect with candidates, and was previously part of the
monolith Warner discussed earlier.

49. Carving out the relevant bits
■ Isolate scope, cut out vertical slices from the monolith.
■ Enables us to Focus our automation efforts.
■ Broadly this is achieved using 2 approaches:
○ Mocking out downstream APIs for local dev.
○ Consumer Driven Contracts, using Pact.
■ These give us isolation and speed.

50. Automating Windows
■ Windows tooling support in past year is much improved.
■ AWS & Vagrant all have first class support.
■ Seek have contributed a number of plugins for Vagrant to
make Windows development easier,
■ ...a number of useful chocolatey packages,
■ ...and a large suite of DSC resources to assist configuring
Windows machines.
■ And then there is Packer...

51. Automating Windows: Packer
“A tool for creating identical machine images for multiple
platforms from a single source configuration”
■ Packer is the core of our Machine Factory.
○ It will boot a machine from an image such as an ISO
○ Configure it with scripts or CM tools.
○ Export it into multiple formats - such as a Vagrant box,
Docker image or an AMI.
■ Uses a simple JSON DSL to easily comprehend and visualise
the steps applied in the manufacturing process.

52. Automating Windows: Packer
■ Packer uses SSH, which is very fragile on Windows.
■ SEEK have helped create a port of Packer to communicate
natively to Windows over WinRM.
■ The result is Packer Community, a set of plugins for Packer
that make Windows machine automation a breeze.
■ Using Packer we have automated:
○ our local dev boxes in Vagrant,
○ our build/test/prod servers in AWS,
○ ...across Linux, Windows Server and Mac OSX.

53. #winning
■ Dev env < 10 mins for .NET API + UI
■ Dev -> Build -> Prod from Factory
○ Vagrant: Vbox, Parallels, Mac +
Windows
○ AWS
■ Commit to Release in 7 mins.
■ Proven ‘vertical slices’ strategy.

54. Bonus: Challenges &
Lessons on Windows
Tips on effective use of Vagrant, Packer and DSC.

55. Vagrant
■ Other authoring environment dependencies = anti-pattern.
○ Installing VM provider (e.g. Virtualbox) + Vagrant should
be the goal
○ Consider scripting the install (DSC, BoxStarter,
Powershell scripts etc.) if you can’t avoid.
■ Shared folders and IIS Applications don’t play nicely
○ Use rsync synced folder type instead
■ When provisioners get too big/slow, move into a Packer
script - vagrant up should be fast and reliable

56. Vagrant
■ Always set $VAGRANT_HOME on multi-partitioned machines
due to disk space restrictions (n+1 space requirements)
■ Distributing Images
○ Use Versions + Atlas, if possible
○ S3 also good option
■ GUI Testing (e.g. Protractor tests)
○ Consider having a GUI fallback (at the cost of slightly
bigger images)
○ X-Forwarding on *nix hosts not ideal

57. Vagrant
■ Cross-OS concerns do occasionally come into play:
○ NPM for example has issues writing to a linux
networked file system where the underlying host is
Windows.
■ Domain access can be problematic and increases
complexity - avoid if possible.
○ Use Vagrant Windows Domain plugin if you can’t.
■ Use multiple NICs with private_network if application
requires multiple SSL certificates.

58. Vagrant
Box cleanup:
■ Clean up after yourself and remove any
binaries/installers/cruft not required for day-to-day use
■ Defrag and compress virtual disk images if possible
■ Matt Wrock has a number of good tips on how to cut it right
down.

59. Vagrant
Build .NET apps in Vagrant without Visual Studio:
■ Install MS Build Tools and VS Agents
■ Package reference assemblies from VS installation into
guest machine.
Install the remote debugger and port forward on 4018 to debug
from Visual Studio

60. DSC
■ Expect pain/effort, it’s in its infancy and is immature.
■ Error messages can be very unhelpful/confusing, and you
should be prepared to get comfortable with Windows
debugging in the event viewer and the CLI.
■ IMHO, DSC is still a better choice than the *nix open source
alternatives such as Chef/Ansible (although support is
improving).

61. DSC
■ Use the Vagrant and the DSC Plugin to iterate on a box;
Vagrant allows you to test your DSC configurations in a
clean environment and test any potential issues with
conditions/ordering, without polluting your host
environment
■ Prefer the feature rich SEEK-Jobs/DSC community resources
over the Powershell team (experimental) xModules
■ Avoid the Script resource - It is rarely a good idea to use it.
Write your own custom resource instead.

62. Packer
■ Don't try to do it all in one go. Build complexity
■ When building a new image, create scripts, run them
manually and then automate once you’re comfortable they’
re working as expected.
■ Iteration times can be slow, particularly when booting and
installing an OS
○ Create intermediate boxes to build off to avoid that
overhead.
■ Using Vagrant to test those scripts is often a good starting
point.

63. Packer
■ Sysprep Windows machines so licenses don’t expire.
■ If you’re creating a Windows image, Use Packer Community
(only if you can’t use Linux! :)).
■ Don’t forget to disable Windows Updates (at least for dev
machines).
■ Windows boxes often need restarting, use the restart-
windows Packer Community provisioner to do so safely.

64. Packer
■ Build Packer images in pipelines - feed output of Packer
runs into the start of another Packer run.
■ For example, use the Virtualbox ISO builder to boot and
install a Windows machine with updates applied.
■ That box can now be fed into the Virtualbox OVF builder
which runs the application specific provisioners (typically
this is the more error prone step).

65. Thank you
@SEEK_Geek
CO PRESENTED BY:
@terrencebenade
@dius_au @warnergodfrey
@matthewfellows

66. Resources
■ Vagrant (http://vagrantup.com)
■ The TAO of Vagrant (http://mitchellh.com/the-tao-of-vagrant)
■ Vagrant DSC Plugin (https://github.com/mefellows/vagrant-dsc/)
■ Vagrant Windows Domain Plugin (https://github.com/SEEK-
Jobs/vagrant-windows-domain)
■ DSC (http://blogs.technet.
com/b/privatecloud/archive/2013/08/30/introducing-powershell-
desired-state-configuration-dsc.aspx)
■ Read the free DSC eBook (https://onedrive.live.com/?
cid=7f868aa697b937fe&id=7F868AA697B937FE!107) for a small but
comprehensive guide to DSC

67. Resources
■ SEEK-Jobs/DSC resource (https://github.com/SEEK-Jobs/DSC/)
■ Setting up a server for use with DSC (see packer templates https:
//github.com/mefellows/packer-community-templates)
■ Packer (http://packer.io)
■ Packer Windows Plugins (aka Packer for Windows https://github.
com/packer-community/packer-windows-plugins)
■ Sample Windows Packer Templates (https://github.
com/mefellows/packer-community-templates)
■ Legacy Windows Packer Templates and provisioning scripts (Uses
OpenSSH, however there is plenty of inspiration to be found -
https://github.com/joefitzgerald/packer-windows)