Evolution of infrastructure as code, a framework that can drastically improve deployment speed and development efficiency. Youtube version: https://www.youtube.com/watch?v=z2kHFpCPum8

1. 2021 | intellias.com
is the best modern practice for CD with Kubernetes
GitOps approach
by Volodymyr Shynkar
Senior Lead DevOps Engineer

2. #!/BIN/ABOUT_ME
Volodymyr Shynkar
Senior Lead DevOps Engineer at Intellias
6+ years of commercial DevOps experience. Overall 8+ years of Engineering
Successfully migrated, rolled out, consulted over 15 projects in the healthcare,
gambling, automotive, e-commerce industries
Member of the Center of Excellence
Lead of the CCI team in the automotive segment
Certified SAFe Agile Software Engineer
Addicted to IoT and Smart Home
Cyclist, promoter of a healthy lifestyle

3. Contents
01
Introducing to GitOps
02
The Principles of GitOps
03
Secure your pipeline
04
CD - Why? How?

4. 01
Introducing to GitOps

5. GitOps is
An evolution of infrastructure as code, a framework
that can drastically improve deployment speed and
development efficiency.

6. GitOps is…
v An operations model for building Cloud Native apps

7. GitOps is…
v An operations model for building Cloud Native apps
v Based on years of operations experience

8. GitOps is…
v An operations model for building Cloud Native apps
v Based on years of operations experience
v A set of principles

9. GitOps is…
v An operations model for building Cloud Native apps
v Based on years of operations experience
v A set of principles
v A way to speed up your team

10. GitOps is…
v An operations model for building Cloud Native apps
v Based on years of operations experience
v A set of principles
v A way to speed up your team
v GitOps applies to everything
v Config is a code
v Code must be version controlled
v Config must be version controlled too!

11. 02
The Principles of GitOps

12. The Principles of GitOps
The entire system is described
declaratively
The canonical desired system
state is versioned in git
Approved changes can be
automatically applied to the
system
Software agents
ensure correctness and alert
on divergence

13. Benefits from GitOps
Productivity Enhanced Experience Stability
• Allows for simplified continuous
delivery, which lowers the
“Mean Time to Deployment”
• Increases overall development
output 2-3 times.
• Pushes code and not containers.
• Manages Kubernetes resources
without having to know the internal
of Kubernetes.
• Automatically gains a convenient
audit log of all cluster changes
outside of Kubernetes.
• Always kept in sync

14. Benefits from GitOps
Reliability Consistency Security / Audit
• In case of a problem, rollback is as easy as
revert commit in Git. Single source of
truth simplifies the process and reduces
delays
• One entry point for changes. Everything
is driven through commits in a Git
repository. Everything is described at one
place (Single pane of glass). PR - Review –
Merge – Apply
• Leveraging Git build-in security and
signature. Allows for easy tracking of
changes and their approvals

15. Three pillars of GitOps
Pipelines Observability
• Continuous Integration
• Continuous Deployment
• Release Automation
Git as a source of truth
for desired system state
• Monitoring
• Logging
• Tracing & Visualization
Holistic view of the real
system state “right now”
• Everything via Git (updates, policy,
security)
• Orchestration
• Diff and Sync
Automation is Convergence
Control

16. 03
Secure your pipeline

17. Move the burden of security
Move from access to cluster to access to repository!
…So how to secure your repository?

18. It seems to be a fashion these days to share
your deepest secrets with everyone.
v Third party companies who want an admin
access to your cloud env or prod cluster
v Third party (opensource) repositories/providers
v Outdated software or plugins used on dev
workstation or on CI server.
v Shared space/server with other teams
Sharing is not caring

19. What is solution for this?
v Move humans as far away from that final
production deployment as possible
v Statically analyse everything
v Run through multiple layers of checks
v Have individuals sign off the change, but not
modify the change that goes through
v All above are fundamentally based on using git
as the basis for everything
Root Causes of Unplanned Outages
Source: https://www.ecmweb.com/power-quality-reliability/article/20900947/data-center-outage-costs-continue-to-rise

20. The OWASP project lists ten principles that should
be applied when designing secure applications:
v Minimize attack surface area
v Establish secure defaults
v Principle of Least privilege
v Principle of Defence in depth
v Fail securely
v Don’t trust services
v Separation of duties
v Avoid security by obscurity
v Keep security simple
v Fix security issues correctly
Security By Design
Source: https://wiki.owasp.org/index.php/Security_by_Design_Principles#Minimize_attack_surface_area

21. Threats:
v Git users can impersonate each other
• Mitigation: Enforce strong identity in VCS with GPG
Signed Commits
v Malicious user rewrites history
• Mitigation: Prevent force pushes to Master/Main branch
• Mitigation: Backup Git repositories
v Malicious user removes security features
• Mitigation: Configure Git provider with IaC
• Mitigation: Verify commits to Master/Main
Continuous Deployment and Continuous Security
Source: https://www.ecmweb.com/power-quality-reliability/article/20900947/data-center-outage-costs-continue-to-rise

22. v Everyone on the team can easily pull the deploy
secrets
v To pull the secrets out you just need to push changes
to a branch that print out the environment variables
during a job run, or POST them to an endpoint
v In the above scenario, compromising a single
engineer is enough to gain access to production
through the CI.
Free-for-all secrets

23. What is solution for this?
v Paper/USB/CDR two fireproof safes?
v Vault (or actually Consul)?
Where do I version control my secrets?

24. What is solution for this?
v Sealed Secrets (a Kubernetes controller and a
tool for one-way encrypted Secrets):
https://github.com/bitnami-labs/sealed-secrets
v Git Crypt - transparent file encryption in git:
https://www.agwa.name/projects/git-crypt/
Where do I version control my secrets?

25. Sealed Secrets as templates for secrets

26. Sealed Secrets as templates for secrets
The controller will unseal that into something like:

27. 04
CD - Why? How?

28. Two ways to GitOps
• There are two ways to implement the deployment
strategy for GitOps:
Push-based and Pull-based.

29. The push-based CD
The push-based strategy is much closer to the classic deployment pipelines and
also easier to implement.
Main components of such infrastructure are:
v repository with configuration
v automation tool
v target environment

30. Questions to ask include:
• Does your CI environment have direct access to the
container image repository?
• Does your CI environment have direct access to the
production cluster?

31. The push-based CD
The push-based strategy is much closer to the classic deployment pipelines and
also easier to implement.
Main components of such infrastructure are:
v repository with configuration
v automation tool
v target environment

32. The pull-based CD
The pull-based GitOps strategy is mostly used for Kubernetes environments
because this approach requires unusual tooling
Main components of such infrastructure are:
v repository with configuration
v automation tool
v target environment with an agent inside

33. Multicluster architecture
The App of Apps pattern:
• The Application is the Kubernetes resource
representing a deployed application in an
environment.
Two key pieces of information define it:
• source reference to the desired state in Git
• destination reference to the target cluster and namespace
• Create an app that creates other apps — which, in its
turn, can create different apps.
• This allows you to declaratively manage a group of
applications that can be deployed and configured in
concert.
Source: https://argoproj.github.io/argo-cd/operator-manual/architecture/

34. ArgoCD CRD
You must add the finalizer if you want to perform a cascade resource deletion
A minimal Application spec is as follows:

35. ArgoCD CRD
The AppProject is the Kubernetes resource object of a logical grouping of applications.
clusterResourceWhitelist – Specify which cluster-scoped resources can be created
namespaceResourceBlacklist – Specify which namespaced-scoped resources can be created
namespaceResourceWhitelist - Specify which namespaced-scoped resources cannot be created

36. Demo time

37. Questions?

38. 2021 | intellias.com
Thank you!
by Volodymyr Shynkar
Senior Lead DevOps Engineer
https://www.linkedin.com/in/vshynkar/
https://github.com/sqerison