5. The police protects the
democracy, maintains the law
and is the authority on the
streets. Around 65.000 people
work at the Dutch police, of
which over 1500 IT
professionals.
Dutch National Police
6. CLOUD
PLATFORM
ANALYSE
PATRONEN
BIG DATA
SECURITY5 DevOps teams are building
high tech big data web
applications in a private cloud
environment. These
applications support police
related themes.
Product line
Cloud | Big Data | Internet
8. • 5 teams, separate backlogs
• Overall planning at start of sprint
• Minimal planning ritual
• Usability tests as part of sprint
• (Almost) no meetings
• Phabricator as tool of choice
Methodology
9. • Continuous Delivery & DevOps
• Short feedback loops
• Embrace change
• Minimal dependencies outside team
• Invest in people, not in products
• Open, transparent, verifiable
Culture
Source: http://kids.nationalgeographic.com/explore/countries/netherlands/#netherlands-tulip-fields.jpg
10. • How to handle support, monitoring and
(pro-active) maintenance during sprint?
Make this a role that cycles through the
team.
“Operator of the day” and “Developer of
the day”
Culture
Source: http://kids.nationalgeographic.com/explore/countries/netherlands/#netherlands-tulip-fields.jpg
12. • End-to-end security and encryption
• Version control for everything
• Horizontally scalable, no single points of failure
• No runtime dependencies on external systems
• Standardised naming
• Right tool for the job (polyglot)
Architecture
24. Source: https://www.google.com/about/datacenters
• How to manage 100’s of physical servers,
1000’s of virtual servers and 1000’s of
desktops?
Infrastructure as code.
Terraform for cloud, Puppet for servers,
Ansible for desktops
Platform
26. • Angular 5.x, TypeScript, RxJS
• Bootstrap, responsive design
• Graceful degradation when backend fails
• All console logs/errors go to centralised
logging system
Frontend
27. • How to decouple building a feature from
release and deployment?
Use feature toggles.
Frontend
28. • How to prevent creating a monolithic
frontend?
Apply microservice principles to the
frontend too.
Create re-usable frontend components as
standalone component libraries.
Frontend
30. • Small in size, single responsibility
• Runs in its own process
• Independently develop, deploy, upgrade, scale
• Has its own data store
• Distributed by default
• Potentially heterogeneous/polyglot
• Light-weight communication
Anatomy of a microservice
31. • Did we build microservices right from the start?
Nope.
• Why did we move to a microservices architecture?
• Scalability: both for performance/load and
development teams
• Modularity: independent development and
deployment of each part of the system
• The ‘cool factor’ helped bit ;-)
The path to microservices
32. • How did we move to microservices?
• Split the existing system in modules (bounded
contexts) - defined by business functionality
• For each module, create a microservice
• For every microservice, check if it now only
serves 1 part of the domain
• If so: cool. If not: repeat - split it up more.
The path to microservices
33. • Spring Boot, Java 8, Maven
• Stateless
• 1 service in 1 jar on 1 JVM on 1 host (and 1 git project)
• Minimal amount of shared code:
• Security
• Logging and metrics
• Past: high available via load balancers
• Present: service discovery
Backend
34. Service discovery is an advanced pattern.
Do you need it?
Probably not.
Do we need it?
We think we do.
Service discovery
35. Service discovery
Why do we use service discovery?
• HA & load balancing without single points of failure
• Direct secure end-to-end client-service comm.
• Kerberos: picky on DNS verification
• Dynamic horizontal scaling
• Resilience: automated response to failure
• Location transparency
• Zero downtime deployments
38. • Feature branch based development
• Master branch must always be releasable
• Test environment on OpenStack runs the
master branch
• Feature branches only live locally and on
the CI server
Development
39. • How to locally run a system that consists
of ~50 services?
Run only the component(s) you work on
locally.
For other components, local env connects
to test env on OpenStack
Development
40. • Unit tests
• Mutation tests
• Service/integration tests: Spring boot
integration, embedded in-memory data
stores, REST assured
• End-to-end test: Protractor
• Load tests: Gatling
Testing
41. • How to test feature branches in a
microservices environment?
Spin up branch version of component in a
container on the CI environment.
For dependencies, connect to test env.
Avoid changing multiple components at once.
Testing
44. • How to manage >100 builds with a single
team?
Create modular, reusable build
definitions.
(See https://virtualjug.com/pipeline-as-code-building-
continuous-delivery-pipelines-with-jenkins-2/)
Build tools
45. • Every push to master is a release
• Config embedded in executable jar
• Deployments
• Rundeck and Puppet (old)
• Nomad (new)
Deployments
46. • How to know when, what and which version(s)
to deploy?
• Minimise administration and think time.
• Test environment: deploy component on commit
• Everything from test -> acc during sprint
• Everything from acc -> prod after sprint
• Single component test -> acc -> prod when needed
Deployments
47. • Logging and dashboards via Graylog
• Metrics:
• Spring Boot actuator
• Distributed tracing with Zipkin
• Grafana for graphs
• Monitoring via Sensu and Flapjack
Running in production
49. Challenges
01
Share as little as possible; prefer
duplication over coupling.
Sharing code between services
04Authentication and authorisation
happen at every request. Find the
balance between performance and
security.
Running stateless has a cost
When moving fast,
don’t forget to finish up before
starting something new.
Switching focus has a cost
06
Throwing something away and
starting over can work out better
than refactoring.
Don’t be afraid to rebuild03
Microservices are not just for the
backend. Modularity is just as
important on the frontend.
Monolithic frontend
02
Minimalize dependencies on
other teams, or it will slow you
down.
Cross functional team
composition is vital
05
and lessons learned
@bjschrijver
50. Looking ahead
Upgrades and fixes without users even
noticing.
0-downtime deployments
Our plans for the (near) future.
@bjschrijver
Cross-functional teams with vertical
(full stack) responsibilities.
Product teams
Split the frontend in products and re-
usable components.
Modular frontend
There is no silver bullet here, but useful
tools and practices do exist.
Automated security testing
Get the teams the information they
need, but only when they need it.
Better dashboards and alerting