18. Our DevOps and Continuous Delivery evolution
Start Mobile
app
development
Start full
automation
of IT
processes
Continuous
Delivery
Discussions
who runs
production
Today
2009 2010 2011 2012 2013 2014 2015
18
New
management
and the will to
change
First scrum
team and
start agile
development
Full
reorganization
towards 180
DevOps teams
First
initiatives
to create a
private
cloud
34. Team vs Enterprise
34
Best solution, cool factor, little rules,
Lot of learning, build it yourself, high
maintenance, little collaboration
Standardization, collaboration, re-use
36. Team vs Enterprise
36
Best solution, cool factor, little rules,
Lot of learning, build it yourself, high
maintenance, little collaboration
Standardization, collaboration, re-use
High license costs, maintenance, version
management, knowledge share, life cycle
management issues
Low license costs, best practices, less
maintenance, knowledge by community
38. Team vs Enterprise
38
Best solution, cool factor, little rules,
Lot of learning, build it yourself, high
maintenance, little collaboration
Standardization, collaboration, re-use
High license costs, maintenance, version
management, knowledge share, life cycle
management issues
Low license costs, best practices, less
maintenance, knowledge by community
Work around rules and regulations Make rules and regulations part of the pipeline
39. So HOW do we do this for Continuous Delivery @ING?
39
Ideas
Build
Product
Measure
Learn
Data
Feature Request list from
IT user communities
Standard Continuous
Delivery pipeline for ING
CD expert team
collaborating with IT
communities
Metrics from all parts of
the CD Pipeline
Collect all data through
an Eventbus and store it
in a database
Analyze all data and
present it to IT engineers
and IT Managers
43. Principles:
• All hardware, middleware and application components are constantly monitored
• Automated alerts are generated whenever a deviation occurs
System health. Availability of the pipeline
43
48. Started as a service and now strategy driven
Take small steps to achieve your goals Learn and adapt
It is also a cultural change. It takes time
Get rid of silos
Recap
48
This talk will take you on a journey of the changing role for IT within a financial enterprise organization.
How it changed from service driven IT towards strategy driven IT in the last decade
At work I’ve had the great opportunity to start up the DevOps transition at ING by organizing several DevOps days within ING, ‘performing’ on several IT conferences, but also by breaking down several heavily shaped processes within ING and the last years specifically being part of the ING Continuous Delivery Team ……….And therefore to be part of the journey towards DevOps and Continuous Delivery
In this talk I will cover the following topics;
Some history of IT at ING. Where do we come from and what was the role of IT?
How did our organization evolve towards a state of strategy driven IT?
How do our customers benefit of all this?
And finally … where are we now concerning DevOps & Continuous Delivery and what are we going to do in near future?
I am going to tell you where we come from as IT within ING. To do so I created some context ……If we look at the first online initiatives to a broader public ……. (next slides)
Like the newspaper….. One was used to the paper version and an online news paper was not convenient. You had to start your PC, browse to the newspaper….maybe waiting to download the articles etc..
But.. It was there and you had an alternative for your paper version
Or to fill out your tax forms. Though is became a lot easier to do your input via a floppy or send it online to the tax company. One was used to fill out forms and had little trust in security of a computer program or the internet.
But there was a possibility to do it via IT focused channel
Buying tickets. Get out of bed early and go to your local post office. At least you could see the queue and the first one there was the first one with a ticket.
But their was also the online opportunity
And what about banking? Fill out your transfer form, put them in a envelope and mail it or bring it to your bank. Easy does it.
But there were also possibilities to do it at home without any paper work.
Back in these days… IT was a nice to have.
It brought us an additional channel for doing our day to day business.
Did we really care about any service loss or downtime due to maintenance windows?
It was not as hell breaks loose……. as the old non-IT channels where still primairly and the IT focussed channels where nice to haves.
Which also meant our organization didn’t have to be very responsive for market changes as market changes where mainly physical changes with low usage of IT and thus not super fast
As seen in the examples the usage of channels to reach people was changing. IT got involved.
From fysical channels, like a shop or bank office, towards digital channels, like a webshop or internet banking.
So let’s move forward on the Banking example.
It all started with the bank office. The office was your only point of contact with a bank. The single channel.
IT started to change this. It was possible to offer more channels for the customers, like
The good old bank office
A call desk or voice repsonse system
Internet banking or home banking
That also means that customers are going to rely on the offered IT services. We got more and more used to it.
But as said. IT was mainly service driven and our organization was not very responsive. What does that mean? ….. Next slide….
If you look at this model, you see a triangle which you would want to have in balance in order to deliver proper IT services.
You need competent people to the work.
You need all kind of processes to do your work in a controlled manner. Like quality – security – maintenance – change processes
And you need proper technology to do your work. State of the art hardware, software, development tools etc.
The focus of our service driven IT was mainly on people and processes. We tried to outsource most of the IT services……… Service Level Agreements, contracts and maintaining these did the job.
For example; ITIL & PRINCE 2 processes where leading and not used as a way to do your work
……. Circle can float in ………
So you can see the focus did not include technology. Here we have a sub optimized or out of balance triangle
So this was a brief introduction and some context for the topics to come.
There was a need for change.
The distance between IT and business had been increased and they were working more against each other than collaborating with each other
The focus of the organization was on the professionalization of IT and Operations and less on business or customer needs
As mentioned earlier we tried to outsource a significant part of our activities. That was about 9 years ago
This caused the focus was mainly on organizing external activities, especially process control, and less on the execution of typical IT development and operations.
As you probably know ING is a merge of Postbank and ING in 2009. During this merge of PostBank and ING there were less investments in new systems. First migrate, than upgrade to make the merge possible.
So we were very busy as IT doing our own stuff and meanwhile our business wanted to do change.
Off course this created a gap
The GAP between IT and business could be best compared with the grand canyon, neither had an idea why & what the other was talking about.. Resulting in less collaboration to get great results for our customer..
Besides our outsource activities, merge activities and maintaining all systems, we had a legacy and bureaucratic organization.
Changes easily took us more than a year to implement..
We were used to do large and long projects like outsource activities.
We were always trying to eat the whole elephant at once………….
And guess what… In the end we were always over time, over budget and overscoped
and the GAPs between business? ……… IT kept growing as nobody was happy with the results. it never ended as planned!
This was not the only GAP in our organization… When we take a look into our process minded IT organization we see the second GAP in two departments having different focus areas…..
Changing the bank and running the bank.
Most of the communication between these two departments were captured in processes and procedures.
For running the bank we had all the ITIL processes in place and very strict in following these processes, focus points where the stability of the system,
within changing the bank we had our Prince 2 processes in place and were also very strict in following them, performance on this side was measured by release dates
Resulting in the famous wall of confusion
Changing the bank had their deadlines and wanted to deliver as fast as possible.
Running the bank did not have much control of what was getting over that wall and tried to regulate that as much as possible to mitigate the risks of downtime.
Because in the end, if there is a fire, it was not the DEV problem. It was the OPS problem. Worked fine in DEV
So we had our challenges and needed to deal with these in order to stay in business as a bank.
How to change? What to do?
Mobile Banking at ING was introduced
A new way for our customers to do their banking and a great opportunity for ING to keep up with their competitors.
Our competitors, like ABNAMRO, already had a mature Mobile Banking app and they were way ahead of us.
As we were used to work on large and long projects we thought to create the best Mobile app with all feautures the customers want. As you can guess …… a project like that probably should have last for one year or longer. Even worse.. We really tried it that way. AND FAILED
So we changed the way of working!
We started to do the development ourselves. We created a minimal viable product where you could check your balance and do a money transfer. Only from your current account. The next feature in the Mobile app was not a feature as wished by business or IT, but as wished by the customer. In this case they wanted to have their savings account visible instead their address book.
The first Agile development had been started.
(And just to promote . We catched up with our competitors and even created a better rated app in just 2 years time)
The start of developing in small steps was a huge change in the way of working!
Following the success of the aglile development of our Mobile app a big reorganization had been announced.
What was this reorganization all about?
We created DevOps teams where teams became responsible for their application from the “gradle to the grave”. Or as a very wise man ever told me “womb to tomb”
The way of working in these teams was Continuous Delivery
As shown in this picture you see how we transformed from waterfall to Continuous Delivery.
The first step was working with Scrum. But after the build we still had a long test cycle as well as a long deployment process. Because these activities were executed manually and by other teams
The second step is Continuous Delivery. Where Analyse, Design, Build, Test and Deploy were done in the same team and in one sprint in a (as far a possible) automated way.
******
Big advantage … you can change scope every sprint
Here you see our DevOps and Contiunuous Delivery evolution.
***Summarize*** l
Key is a management team that is willing to change
Moving from service driven towards strategy driven.
An example of strategy driven is the development of the Mobile app. We are able to respond on customers needs in a short time because of the short feedbackloop.
What does this mean for the People-Processes-Technology triangle?
Technology is getting into the triangle, but we are not there yet. Still a lot of challenges are ahead to have everything in balance.
People needed to shift from process minded to engineering minded. All technology needs to be build up in order to facilitate people and processes.
Well so far a nice story, but what’s in it for me as a customer?
Let me first take you into our mission as ING which is focussed on Omnichannel…
which is specifically what the journey I’ve been talking so far has brought the customer as well..
due to this journey we as IT will be able to provide our customer a omnichannel experience
What is Omnichannel?
In this picture you see how the usage of channels evolves in time
Years ago most companies just had one channel : the fysical store of bank office
The development of technology brought us more channels which could be used next to each other: MULITCHANNEL. In our case the ATM’s, voice banking, internet banking
There channels were indepentend and there was hardly any cooperation between these channels
The next step is CROSSCHANNEL. More collaboration and aligment between the channels. This is the phase ING is in at the moment
In OMNICHANNEL we get rid of all borders between the channels. Customer data is stored centrally and anywhere realtime available.
There are still different channels, but the customer experience is the same everywhere. Same look and feel, same offers, same functionality etc. The customer does not experience differences between the channels and can easily switch from one to another.
The customer is central and the company organizes around the customer
Omnichannel is nice, but why should we do it?
Three reasons
First of all,Technical innovators have a huge impact on customer behaviour. This changes the customer behaviour and also customers of banks. Who will expect another way of service.
The second reason is the great opportunities it offers to create a better relationship with our customers
And the third reason. It makes our organisation more efficient and therefore we can reduce costst
One can not just say: “Let’s do Omnichannel”
It has a huge impact on both the organization and IT.
Because we did invest in taking small steps, burning down silo’s and shorten the feedbackloop we are able to make the change to Omnichannel.
What these changes are, I will show you these in the next slides.
Teams who are experts and responsible for the application DevOps
I will eloborate this in later
Create an organization that is able to change Create an Agile organization/enterprise.
I will eloborate this later
Deliver fast and high quality Continuous Delivery. I will eloborate this later
Well almost. But I couldn’t find that meme
Since we started the change from service driven IT towards strategy driven IT we are able to contribute the Omnichannel journey from an IT perspective. We made really big improvements towards the agile way of working and Continuous Delivery
“Start driving your strategy from IT”
I have been telling about ING moving towards a strategy driven IT and the development of an Omnichannel Experience.
So where are we now considering DevOps, Organization and Continuous Delivery?
We created teams with business and IT . Let’s call it BusinessDevOps.
There are multiple teams/squads in a tribe. A tribe is a number of squads an has its own purpose. For example: Omnichannel Experience. This tribe does do all the work needed for that purpose, omnichannel
There is no hierarchy in the squads. It is all driven by the purpose of the tribe
Leg dit model maar gewoon helemaal uit
So we just talked about the to organize. Now we will dive in to the way to automate at scale
So we need to learn and adapt to stay ahead of our competitors. Misschien wat meer uitleg over de lean circle?
We want to share three experiences of scaling Continuous Delivery by learning and adapting;
Automation: Team level vs enterprise level?
Tools: Inventing the wheel vs standardization?
Security
As mentioned earlier. We had a reorganisation where we stated “Continuous Delivery the way we work and DevOps the way we organize”.
This brought huge enthusiasm within the teams and their first reaction was to automate all the things. They were eager to start and use all kind of automation tools in order to create a continuous delivery pipeline and learning to automate.
This also resulted in doing “cool” things instead of adding direct value to the products. Most of the tools were new to the teams and they really need to learn the tools and after that putting effort in the tools to use them.
Besides many Continuous Delivery pipelines, this also resulted in many tools. There was no standardization in way of working as well as tooling. Furthermore, there was no collaboration between teams to share their experiences. The wheel was invented over and over again.
Standardize to the max!
Many license costs, not all on the same version, no good lcm etc…
What about security?
Lastig om anders te werken als je niet toegang tot alles hebt (wat je nodig hebt). Blijkt allemaal niet zo makkelijk te realiseren door geschiedenis etc…
Begint met Agile, dan werk je om security heen. Onderstaande afdelingen/gevestigde instituten ga je omheen werken
Security
Compliance
Governance
Op enterprise niveau kan dat niet en ga je zorgen dat die afdelingen ook Agile worden en je gaat ze intergreren in de CD pipeline.
Op onze manier van werken worden het meer geautomatiseerde controls
e.g. Fortify (code security check), integrated backlog management system
Normally a security framework with strict rules where you have to comply to, but we now have a system where actually mitigate risks for the bank by learning from experience.
Risk department is also co-repsonsible for security measurements.
Everyone did explore CD.
We had a best pratice at ING, which we made a standard. We are now improving this standard by (the circle)
How do we do this with our current standard pipeline?
En dan het plaatje toelichten
Continuous Delivery as a Service…… what does that mean?
Well .. There is some history in this..
This CDaaS pipeline has also been created step by step.
Our Channels department were very advanced in their continuous delivery. Not only in automating IT processen, but also measuring the whole pipeline.
Extra layer which tells us about behaviour of the pipeline for teams and releases
Our Continuous Delivery Pipeline collects metrics from all parts of the process.
Collect data
Teams
Learn and adapt
Releases
Is it safe to bring it to PRD
All in balance (for) now.
We really started to embed technology. Like engineer desktops, tools, hardware etc…
Processes are helping us to do our work
People started to move away of the process mindset
DRAFT To be pimped
Be like a flock of birds. Make sure you can adapt to all circumstances