JFrog's Kubernetes Journey: Lessons Learned Taking Apps to K8s

Copyright @ 2019 JFrog - All rights reserved
Kubernetes is hard!
Lessons learned taking our apps
to Kubernetes
Eldad Assis | DevOps Architect | JFrog

Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW!
Me
Email: eldada@jfrog.com
Twitter: @eldadak
LinkedIn: Eldad Assis
What I do
- I (try to) solve problems @ JFrog
- DevOps Architect
- Bringing Dev and Ops closer for over 15 years
- Doing CI/CD since the turn of the century

● Docker
Containers
● Kubernetes
An open-source system for automating deployment, scaling, and
management of containerized applications.
No deep diving. No Demo… sorry
Come see me after the talk for more!
For this talk, I assume you know a bit
Recommended knowledge

1. I need a running environment. Now!
a. Developers, QA, Support, Product Managers, Solution… anyone!
2. Per branch CI/CD
a. Full CI to Integrate my branch with other products development
branches
3. Better utilize our resources for dev and production
4. Support a new distribution for JFrog products
Why? (The problems we wanted to solve)
The journey to Kubernetes begins...

Why? (Taking containers to production)
Docker in development vs production
Docker container Docker container
Networking
Security
Monitoring
Logging
Auto scaling
Auto healing
Development
Production

● Start small
● Get the application ready before jumping into Kubernetes
● Security - control what’s running in your cluster
● Limits
● Health probes
● Visibility - usable and accessible monitoring and logging systems
● Spread the word and work with the community!
The End
What you should take from here

Kubernetes - the myth
ZZZZ….

Kubernetes - the reality
Hmmm….

1. See a tweet on a new technology! (WOW - THIS IS COOL!)
2. See video showing how easy it is (LOOKS EASY!)
3. Try running it yourself (HMM… )
4. Fail miserably (WOW - THIS IS HARD!)
5. Feel stupid
6. Try again (GO TO 3)
How I started
A recipe for a successful new technology adoption

● Start with a small application example (nginx hello world)
● Use existing demos
○ Understand it. What each line in the yaml actually means
● Set a minimal goal for getting your app to run in Kubernetes
● Get comfortable before you move on
○ Start with managed K8s for easy setup (AKS, EKS, GKE)
Where should I start?
Start small!

A lot has to be done on your application before you
can comfortably run it in Kubernetes
Here are some key points to consider when planning
your move to Kubernetes
Start with the application!
Is your application ready to run in Kubernetes?

● Logging
○ STDOUT/STDERR
○ Handling multiple log files
● Data Persistency
○ What kind of data needs persistency (if at all)?
● Proper handling of termination signals to init a proper shutdown
○ Controlled shutdown of the application
○ Easier recovery (after a controlled shutdown)
● Survive a restart
○ Managing leftovers from previous run
Is your app ready for Kubernetes?
It’s not just pushing it to Docker...

● Improve durability and availability
● Support running multiple instances of your application with load balancing between them
○ Scaling up and down will be easy (horizontal scaling)
● Rolling upgrades for zero downtime
○ Backward compatibility
● Zero service unavailability due to pod rescheduling
○ Cluster scaling (down)
○ Pod evicted or crashed
○ Unplanned outage of a node
High availability as the new default!

I. Codebase - One codebase tracked in revision control, many deploys
II. Dependencies - Explicitly declare and isolate dependencies
III. Config - Store config in the environment
IV. Backing services - Treat backing services as attached resources
V. Build, release, run - Strictly separate build and run stages
VI. Processes - Execute the app as one or more stateless processes
The Twelve-Factor App (https://12factor.net/)
VII. Port binding - Export services via port binding
VIII. Concurrency - Scale out via the process model
IX. Disposability - Maximize robustness with fast startup and graceful shutdown
X. Dev/prod parity - Keep development, staging, and production as similar as possible
XI. Logs - Treat logs as event streams
XII. Admin processes - Run admin/management tasks as one-off processes

● Your application is rarely the only thing in its pod
○ OS packages, OSS libs, 3rd party processes
● Security vulnerabilities
● Do you use public images from unknown sources?
● FOSS licenses compliance
Security
What’s running in your cluster?

Once your application is ready, let’s talk about to how to properly
manage your applications run-time and configuration in Kubernetes
And now - Kubernetes
Properly running in Kubernetes

● Pod limits. Always!
● The app inside might need its own limits. For example:
○ Java apps
■ Limit the java process with `-Xms=1g -Xmx=2g`
■ Pod memory limit should be higher than Xmx
○ RabbitMQ
■ [rabbitmq.conf]
total_memory_available_override_value = 1GB
○ MongoDB
■ --wiredTigerCacheSizeGB=1
Know your limits
There might be more than you know….
…
resources:
requests:
memory: "1Gi"
cpu: "100m"
limits:
memory: "2Gi"
cpu: "250m"
...

● The total sum of limits on a node can be way over 100%
● This allows for resource sharing between pods
● Pods will crash! Nodes will crash!
● See out of resources handling by Kubernetes
● Be prepared
○ Think about the requested and limits values
○ Spread your application across nodes using multiple
replicas (HA)
○ Use pods priority and preemption to take control
Know your limits - don’t overload
There might be more than you know….
…
resources:
requests:
memory: "64Mi"
cpu: "20m"
limits:
memory: "2Gi"
cpu: "4"
...

● Your application must have a reliable metric for health
● readinessProbe
○ Is the app ready to start serving
● livenessProbe
○ Is the app good to continue serving
● Types
○ Exec - return 0 on success
○ Http - return < 400 on success
○ Tcp - succeed to open a socket on a given port
● For complex checks, write a script and use the exec probe
Know your app’s health
Application’s readiness and health
…
readinessProbe:
httpGet:
path: /api/system/health
port: 8080
…
livenessProbe:
exec:
command:
- mongo
- --eval
- "db.adminCommand('ping')"
…
livenessProbe:
tcpSocket:
port: 5672
…

● Init Containers - run before main container
○ Prepare storage
○ Setup configuration
● Sidecar containers - run alongside main container
○ Maintenance
○ Log collection
○ Monitoring
○ Proxy
Multiple containers in a Pod
When a single container is not enough
Application pod example.
Multiple artifactory logs forwarded by a
Fluentbit container to a log aggregator.
Pod
Application
container
Fluentbit
container
Logs
Log
collector

● Your application is made up of multiple components
○ Each represented as a yaml file or snippet
● Versioning of an application that’s made up of several yaml files is challenging
● Having all configuration in the yaml files is great, but
○ How can I use the same yaml for different environments?
■ Local (minikube)
■ Dev cluster
■ Production
● Rolling back to earlier versions of the application
Managing an application’s lifecycle
The static nature of the yaml descriptors is challenging

● https://helm.sh/
● Helm is the package manager for Kubernetes. Like ‘yum’ for
CentOS/RedHat
● Describes the whole application in a single package - helm
chart (template yamls)
● Default configuration values (values.yaml)
● Single version for every chart (Chart.yaml)
Here comes Helm!
Dynamic control over your application’s deployment

● Use same chart for dev, staging and production!
○ Each environment’s config managed in its own copy of values.yaml
that is version controlled (values-stg.yaml, values-prod.yaml)
○ The default values.yaml should be for dev or local, so a developer can
use it locally without hassle
○ Everything is configurable!
● External charts as requirements (dependencies)
○ 3rd parties like databases
Helm - YES!
Use same chart in all environments

Helm - YES! YES!
Useful helm commands for understanding what the helm is going on...
# Lint your chart for errors and recommendations
$ helm lint <chart path>
# Download a chart for local viewing
$ helm fetch <chart>
# Get a release (application already deployed with helm) actual configuration
$ helm get <release>
# Get the status of all the resources included in a release
$ helm status <release>
# Get the actual resolved configuration without deploying anything
$ helm template <chart>
$ helm install --debug --dry-run <chart>
# Test your release (need to write test pods in your chart)
$ helm test <release>

More Helm charts
Find existing charts
Official Helm charts hub
https://hub.helm.sh/

● No more “ssh to the server and get me the logs”!
● Developers should not need kubectl access to debug their
applications
● Provide your Dev and Ops easy and reliable data
○ Monitoring
○ Logging
● Managed solutions provide OOB tooling
Visibility in Kubernetes?
We are not in Kansas anymore...

Dashboard for nodes
and pods
* Prometheus
* Grafana
Monitoring

Logs from all pods
EFK stack
* Fluentd
* ElasticSearch
* Kibana
Logging

Use with your CI
Plug your CI/CD to a Kubernetes cluster for easy environment deployment

● CI/CD for our products using Kubernetes
○ Using internal dev versions with our official Helm charts
○ On demand, fully isolated environment per branch per developer
○ 100’s of custom testing environments a week made up of ~50+ services
● Some of our cloud based applications already running on K8s
○ Internal and external
● Official JFrog Helm charts for all our products
JFrog and Kubernetes
What are we doing with Kubernetes?

● You are not the first one to stumble on that particular problem
● It’s as if you have more developers in your team
● Fast turnaround
● Examples
○ RabbitMQ HA
○ MongoDB (Bitnami)
The community
Use already tested and managed helm charts

● Start small
● Get the application ready before jumping into Kubernetes
● Security - control what’s running in your cluster
● Limits
● Health probes
● Visibility - usable and accessible monitoring and logging systems
● Spread the word and work with the community!
The (real) End
What you should take from here

Be Good!
Questions?

JFrog's Kubernetes Journey: Lessons Learned Taking Apps to K8s

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a JFrog's Kubernetes Journey: Lessons Learned Taking Apps to K8s

Similar a JFrog's Kubernetes Journey: Lessons Learned Taking Apps to K8s (20)

Más de AgileSparks

Más de AgileSparks (20)

Último

Último (16)

JFrog's Kubernetes Journey: Lessons Learned Taking Apps to K8s