5. Actual Deployment plan
Check if access to the deployment packages is available.
• R:Callcredit Development DMLLLBuildsRelease- v6.1.56.14856.zip
Extract
the relevant (Web or App) files zip within the release to c:program filescallcreditLL
on each Web and App servers. Ensure the correct rollback version location is known.
Deployment Steps
Instruction Done (1,2&3) Done (4,5&6)
Run script to take the first 3 pairs of Web and App servers out of the pool
“C:scriptsscripts” on WIN**WB01 server….(run Phase 1 script).
ENSURE YOU ARE NOT REMOVING ALL SERVERS FROM THE POOL
On the app server:
• Open a command prompt in the "Release" folder, e.g. "C:Program FilescallcreditLL v6.1.56.14856Release"
• Enter the command:
• Teardown_APP
On the app server:
• Open a command prompt in the "Release" folder, e.g. "C:Program FilescallcreditLL v6.1.56.14856Release"
• Enter the command:
• nant -D:"config=siteconfigid" -D:"encrypt.key=yourencryptionkey“
ptionkey = see app support passwordsafe)
(where siteconfigid:
• Live = “LV”
• Site 2 QA = “qa”
• Site 2 CT = “ct”
On the web server, open INETMGR and delete the following:
From IIS ? Callcredit Web Site ? services
• Admin
• Call
• DL1
• DL2
• LL
From IIS ? Callcredit Web Site
• Call
• DL
• LL
On the web server:
• Open a command prompt in the "Release" folder, e.g. "C:Program FilescallcreditLL
v6.1.56.14856Release"
• Enter the command:
• Teardown_WEB
On the web server:
• Open a command prompt in the "Release" folder, e.g. "C:Program FilesLL v6.1.56.14856Release"
• Enter the command:
• nant -D:"config=siteconfigid"-D:"encrypt.key=yourencryptionkey"
(where yourencryptionkey= see app support passwordsafe)
(where siteconfigid:
• Live = “LV”
• Site 2 QA = “qa”
• Site 2 CT = “ct”
On each of the App servers:
Under component servicescomputersCOM+ApplicationsCAST Job controllerRolesPermittedUsers
Add the user: ServerName_LL_CS_WebSvc
e.g. OLSWINxxAP0xLL_CS_WebSvc
the user should already be present under computer Managementusers
On each of the App servers:
Open DCOMCNFG -> servicescomputersCOM+ApplicationsCAST Job controller
Right click on the component and:
• Shut down
• Disable
• Enable
DBA should run following script on ML_Lookup_Vws5 databse:
UPDATE vw_IDType
SET [Desc] = 'Passport'
WHERE [ID] = 5
Rerun the pairing scripts to update COM+ configs (if COM+ has been updated)
ENSURE YOU ARE RUNNING THE CORRECT “PHASE” SCRIPT TO MAINTAIN SEPERATION BETWEEN UPGRADED AND NOT UPGRADED
VERSIONS OF THE RELEASE!!
QA can then verify the web and app tier install
Add the 3 pairs back into the pool and remove next 3 pairs (run phase 2 script).
ENSURE YOU ARE RUNNING THE CORRECT “PHASE” SCRIPT
Repeat above steps for each pair
Once paired testing is complete, add all servers back into pool (run Phase 3 script)
ENSURE YOU ARE RUNNING THE CORRECT “PHASE” SCRIPT
QA will perform a final BIGIP test.
14. Death to Manual Deployments!
Numbers from : OctopusDeployment AutomationSurvey 2013, State of SoftwareDelivery: Trends in the market now and in the future (2014), PuppetLabs State of
DevOps Report 2014
15. Death to Manual Deployments!
Numbers from : OctopusDeployment AutomationSurvey 2013, State of SoftwareDelivery: Trends in the market now and in the future (2014), PuppetLabs State of
DevOps Report 2014
16. Death to Manual Deployments!
Numbers from : OctopusDeployment AutomationSurvey 2013, State of SoftwareDelivery: Trends in the market now and in the future (2014), PuppetLabs State of
DevOps Report 2014
17. Deployment automation - Picking a tool
https://xebialabs.com/periodic-table-of-devops-tools/
19. Continuous Delivery
What :
We make sure our software is always potentially shippable throughout its
entire lifecycle and that any build could potentially be released to users at the
touch of a button using a fully automated process.
Why :
We can put the release schedule in the hands of the business rather than IS. Be
this with a regular rhythm or on demand.
34. Next steps along the road - Culture
- Complete the move from Project Teams to Product Teams
- Really treating Pre-Production and Production the same
- We need to breed more champions across the company – Development, Security,
Operations, Business
35. Next steps along the road - Monitoring
- MUCH more monitoring – proactive rather than reactive and visible to everyone
- Log aggregation – choose a tool and allow access to Dev and Ops
- Get development teams delivering monitoring as a standard part of any system
36. Next steps along the road - Environments
- IaaS – internally and externally
- MUCH more monitoring – proactive rather than reactive and visible to everyone
- The Cloud – use the best technology for the required solution
37. Next steps along the road – Delivery
- Blue/Green deployments with zero downtime
- Continuous performance testing
Introduce me!
Introduce Callcredit
…this is the story of the last 18 months at Callcredit as we’ve worked to move into a DevOps way of working
First a bit of background for those of you that don’t know us. CC has only existed as a company for about 15 years, when the Skipton Building society decided it would be a cunning plan to set up a 3rd Credit Reference Agency to challenge the big 2What followed was a decade and a half of rapid growth – and it’s that velocity that had created us some….challenges.
So…
Traditionally with Agile development we have a beautiful process of iterative development – with actual working functionality at the end of each sprint… GREAT!and then you hit this…..
....in fact from the Ops perspective this is what they see…
By way of example –
Who fancies following this manually…
on 12 servers….
at 4am…
and expect not to make a single mistake or miss a single line???
So we’d kind of ended up with the classic Dev vs Ops Silos. With Developers constantly ‘sabotaging’ the live service and Ops “not letting us deploy changes”
Server builds took in the region of a day to just build a server - let alone provisioning storage and configuring firewalls
Bottleneck on DBA and Application Support teams when they're also trying to keep live services running
Can't get people from Ops to contribute to projects (especially since Live is a dark hidden mystery to the dev teams)
Can't get time from Ops to do deployments
Painful deployments
So once you did manage to secure someone from Ops to do a deployment for you , what was presented to them (often for the first time) was an, often long, list of manual instruction
So, with new owners and a stated aim to double the revenue of the company in the next 5 years we clearly needed to stop sprinting to stay put and start getting to a place where we’re actually moving somewhere
At the same time the then IT Director moved 'across the floor' into delivery Director role.
It had already been flagged up by the Delivery teams struggled to get IT resource - mostly just deployment monkeys - so some members of IT Ops moved with him.
This was the moment of opportunity and DevOps was flagged as a thing we should look at…. So we did….
We did A LOT of reading and talking to people
Trying to understand what we were really trying to do. Fortunately there are Millions of websites, blogs and books …
(As an aside – If you’re looking into this sort of transformation read the Phoenix Project, and get everyone else to read it too. From the CEO downwards ideally)
It’s a lot of different things –
Most obviously it’s a simple combination of Development and Operations and really bringing those two disciplines together is what we’re talking about.
That’s why the biggest word up there is ‘culture’… which is also, incidentally, the biggest challenge
Some things you’ll also sometimes hear about DevOps
….fortunately we didn’t know enough about DevOps at this point to be aware of these!
And in reality that’s‘ completely…. Wrong. Anyone can work towards a DevOps culture and gain the associated benefits
Looked at the potential topologies - we know what's ideal, but that's too big a leap for us. So we aimed for (Venn diagram)
Created 2 new teams *alarm bells* for anyone who's read into DevOps before, but don't panic - remember the initial topology we're aiming for.
Type A – where we startedType 1 – where we’d like to be
Type Y – a good start
System Build
Embedded in Development teams
Bringing the Ops perspective into the Development teams
‘non-functional’ requirements
understanding of live environments
Championing other DevOps principles – automation, collaboration
Allow the team to be autonomous in terms of getting code from idea all the way into production
Platform Build
Focus on providing services to help rapidly build the infrastructure, OS and middleware platforms which underpin these
Operating in a “Platform-as-a-Service”
Aiming at Infrastructure as code
Also provide guidance and advice on infrastructure design for new systems and solutions
Once we had a decent idea what it was we were talking about we looked at where we were currently and then at (a potentially unattainable) AWESOME Nirvana state
Broken down into 4 areas Culture, Monitoring, Environments and Continuous Delivery….
Broadcast, broadcast, broadcast is SO important I can’t over-emphasize
Speaking of sharing information – another thing that worked well for us was a small weekly email sent to all the IT and delivery teams. To feed them information whether they like it or not. Regardless it was just another way of getting DevOps to start seeping into their everyday working life
During of research we’d come across some interesting surveys – and they pretty much showed us that we weren’t alone in our challenges around software delivery
During of research we’d come across some interesting surveys – and they pretty much showed us that we weren’t alone in our challenges around software delivery
During of research we’d come across some interesting surveys – and they pretty much showed us that we weren’t alone in our challenges around software delivery
One of the first things System Build did was to look at a deployment automation tool – and be in no doubt there are a LOT of tools out there
It’s IMPORTANT TO PICK THE ONE THAT’S RIGHT FOR YOU RIGHT NOW
We specifically looked for
ease of Integration
Ease of use
Visualisation
Traceability
Costs/Support
Focus on automated pipelines (using Octopus Deploy)
small deployments delivered often all the way to live
Product rather than project
work with change management to define a new Standard Release process – in fact we ended up with an actual DevOps champion in change management
– side point ITIL and DevOps absolutely DO play nicely together
In short we focussed on….
In many ways this is the reason we’re doing all of this – and is also one of the easier sells to the business side of the world
What’s the thing that makes databases trickier than web or app tiers? It’s that pesky data….
When we looked we found that, actually, once your dB is established most of the changes aren't destructive to data.
In which case we can mostly treat database deployment like we do web or app…. As long as we ALWAYS keep thinking about the effect changes have on data.
For our SQL Server deployments there only really seemed to be 2 tools – Microsoft’s SSDT and Redgate’s SQL deploy. Both work just fine
DATABASES CAN BE BOTH VERSION CONTROLLED AND DEPLOYED IN A PACKAGED FASHION
Would anyone like to guess when we put in the deployment automation tool?
Tools used
So, we had our deployment automation tool of choice for software, at the same time we looked at the same question for infrastructure
There are a lot of “DevOps tools“ out there, but the realisation that platform came to was that there was a long way we could go with the ones we already had and just weren’t leveraging
Powershell – Because if you’re automating in Windows, why wouldn’t you, fast, powerful scripting language
SCCM – was used to deliver our base image, rather than templates
Why: Because we could port it between hypervisors, physical servers and cloud providers easily, means we are adaptable, yet could maintain an identical base image
System Center Orchestrator – Combining Powershell, SCCM and C#. Community of orchestrator guru’s delivering integration packs
SCOM – immediately able to give devs a view of live server performance leading to areas quickly identified for code improvements
Leankit – to manage our demand, and capture metrics
Who meet regularly to discuss
Finally the infrastructure wasn’t always the bottleneck to delivery!
Statistics showed us to be 17x faster at deploying servers than the old manual method, this fact was advertised to senior IT Leadership, the result: an ovation!!!
There’s still some teams and individuals unwilling to change how they work
We’re still periodically asking Ops to stop ‘fixing’ live manually
And there’s still developers who don’t think it’s their job to worry about if their code works in Production environments so long as it works on their computer
Infrastructure as code almost instantly showed up every little mismatch in our environments, forcing us to fix and improve along the way
Automating our complex legacy network architecture was going to be no small task
In short we’re learning a whole lot about our own environments that were previously hidden
Another challenge was - especially with the infrastructure - we were now starting to move at a pace that the service teams were struggling to support
Such as…
Being able to support an increasing number of appearing and disappearing VMs
and
Purchase and provision of storage
were starting to be the new bottlenecks – so clearly we’ll now need to focus on helping them mitigate these challenges
While we have made huge leaps, there are still challenges around working together.
We got some great grass roots support – as one engineer noted, if we do this right then being On-call is FREE MONEY!
We’re now working with Ops leadership to also drive this from the top down as well – especially around aligning objectives around business requirements rather than clashing things like uptime vs change
We’re in a highly regulated industry
Working with large amounts of very sensitive data
And some very complex legacy systems and infrastructure
Every company’s DevOps strategy will be unique, and every company’s plan for overcoming challenges will be just as unique. Whatever the circumstances, communication will be crucial to any adoption, as it ensures that everyone understands why such a change is necessary…
So, we’ll keep pushing towards our ever-changing definition of “Awesome”
I assume eventually it’ll look like this…