Honestbee has been running Kubernetes in production since mid 2016. We have heavily invested in Helm and have real experience managing our Kubernetes service deployments as code with Helm. We would like to share how Helm is used internally at Honestbee. In the presentation we will tackle the following examples and use cases: * Continuous Delivery using Helm * Secret management with VaultController * Building and hosting our own chart repositories (and the iterations we did) * Plugins to help manage values * Helm Chart best practices

1. How Honestbee Does CI/CD On Kubernetes
Honestbee
@vincentdesmet

2. $ whoami
DevOps at Honestbee
tech.honestbee.com
@vincentdesmet / github.com/so0k
Singapore-Kubernetes-User-Group
Cloud-Native-Singapore

3. What is Honestbee?

4. Overview
- Honestbee technology choices
- Containers
- Kubernetes
- Helm
- Vault

5. What are Containers?
● Packages up software binaries & dependencies
● Immutable & testable
● Isolate software from each other
● Portable across environments
a self-contained process
ref: kubernetes-comic

6. Why Containers?
image: ruby:2.1
services:
- postgres
stages:
- Build
- Test
- Staging
- Production
...
source: GitLab CI
Lightweight
Reproducible builds

7. Versioning
$ make get-tags-prod
getting apse1a tag:
master-ebf3205b
getting apse1b tag:
master-ebf3205b
$ make get-tags-staging
getting apse1a tag:
staging-98a2aa4f
getting apse1b tag:
staging-98a2aa4f
latest is not a version

9. More than just packing and
Isolation
- Scheduling: Where should the containers run?
- Resource Optimisation: How much resources does each container
really need?
- Monitoring: What’s happening with the containers?
- Lifecycle and health: Keep containers running despite failures
- Auth{n,z}: Control who can do what with the containers?
- Scaling: Handle higher load by adding more instances
- Discovery: How can I connect to the containers?
- …
Source

10. ref: kubernetes-comic

11. origin story
Star Trek?

12. 1
2
3
4
5
6
7

14. Version 1: sed
kube-install: update-manifests
kubectl create -f manifests/${SHORT_NAME}-deployment.tmp.yaml
kube-update: update-manifests
kubectl patch deployment ${SHORT_NAME} -p
'{"spec":{"template":{"spec":{"containers":[{"name":"'"${SHORT_NAME}"'","image":"'"$
{IMAGE}"'"}]}}}}'
update-manifests:
@sed 's#(image:) .*#1 $(IMAGE)#' manifests/${SHORT_NAME}-deployment.yaml
> manifests/${SHORT_NAME}-deployment.tmp.yaml

15. Version 2: jinja

16. $ kubectl get deploy -l release=backend
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE
backend-admin 16 16 16 16
backend-core 32 32 32 32
backend-core-bee 10 10 10 10
backend-karafka 1 1 1 1
backend-scheduler 1 1 1 1
backend-worker-all 8 8 8 8
backend-worker-critical 2 2 2 2
backend-worker-food-import 1 1 1 1
backend-worker-high 2 2 2 2
backend-worker-low 12 12 12 12
$ kubectl get hpa -l release=backend
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS
backend-admin Deployment/backend-admin 9% / 70% 16 24 16
backend-core Deployment/backend-core 28% / 70% 32 48 32
backend-core-bee Deployment/backend-core-bee 20% / 80% 10 16 10
$ kubectl get scheduledscaler -l release=backend
NAME AGE
backend-admin 9d
backend-core 9d
backend-core-bee 9d

17. Different requirements

18. Kubernetes Deployment
challenges
- Each application has multiple components
- Every component has its own k8s resources
How to:
- Deploy, Manage, Edit and Update multiple k8s configurations
- Deploy multiple k8s configurations as a single application
- Parameterise and support multiple environments
- Manage application releases: rollout, rollback, history, ...
- ...

19. apiVersion: apps/v1beta1
kind: Deployment
metadata:
labels:
app: backend
component: service-core
name: backend-core
spec:
replicas: 32
...
template:
...
spec:
containers:
- env:
- name: NEW_RELIC_APP
value: backend-core-apse1a;backend-core
...
image: myregistry/backend:master-ebf3205b
volumeMounts:
- mountPath: /app/config/application.yml
name: backend-config
subPath: application.yml
volumes:
- name: backend-config
secret:
items:
- key: config
apiVersion: apps/v1beta1
kind: Deployment
metadata:
labels:
app: backend
component: scheduler
name: backend-scheduler
spec:
replicas: 1
...
template:
...
spec:
containers:
- command:
- bin/rake
- resque:scheduler
env:
- name: NEW_RELIC_APP
...
image: myregistry/backend:master-ebf3205b
volumeMounts:
- mountPath: /app/config/application.yml
name: backend-config
subPath: application.yml
volumes:
- name: backend-config
secret:
items:
- key: config

20. Helm
Package Manager for Kubernetes
- Aims to provide Apt/Yum/Homebrew User Experience
- Ensure collaboration
- Shareable Packages
Deployment Manager
- Repeatable deployments
- Manage multiple configurations
- Update, Rollback and test application deployments

21. Chart, Repositories, Releases
- Chart: “Package”, “Bundle”
- Repository: Package Repository
- Release: Installed Chart (same chart can be installed multiple times)

22. Helm Charts
Render(Templates + Values) = Release
Templates:
- The Go Template language: {{ .foo | quote }}
- Variables, simple control structures (looping, conditionals, ... )
- 50+ functions from Go/Sprig template libraries ...

23. {{- range $api_name, $api := .Values.api.types }}
{{- if gt $api.replicas 0.0 -}}
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: {{ template "fullname" $ }}-{{ $api_name }}
labels: {{ include "labels.standard" $ | indent 4 }}
component: service-{{ $api_name }}
spec:
replicas: {{ $api.replicas }}
...
---
apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
name: {{ template "fullname" $ }}-{{ $api_name }}
labels: {{ include "labels.standard" $ | indent 4 }}
component: {{ $api_name }}-service
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: {{ template "fullname" $ }}-{{ $api_name }}
minReplicas: {{ $api.autoscaling.min }}
maxReplicas: {{ $api.autoscaling.max }}
targetCPUUtilizationPercentage: {{ $api.autoscaling.cpuTarget }}
---
{{- with $api.autoscaling.steps }}
apiVersion: "scaling.k8s.restdev.com/v1alpha1"
kind: ScheduledScaler
...
steps:
{{ toYaml . | indent 2 }}
{{- end }}
---
{{ end -}}
api:
types:
admin:
autoscaling:
min: 4
max: 12
cpuTarget: 70
steps:
- runat: '0 30 11 * * *'
mode: range
minReplicas: 8 # 11:30am: pre-lunch schedule
maxReplicas: 12
...
resources:
limits:
cpu: 2
memory: 6Gi
requests:
cpu: 2
memory: 6Gi
config:
admin-panel-enabled: true
url: foo.honestbee.com
core-bee:
autoscaling:
min: 2
max: 16
cpuTarget: 80
steps:
- runat: '0 00 08 * * *'
mode: range
minReplicas: 4 # 08:00am: day schedule
maxReplicas: 16

24. Easy to manage deployments
$ helm install backend --values production.yaml
$ helm history backend
REVISION UPDATED STATUS CHART DESCRIPTION
110 Tue Apr 24 17:53 SUPERSEDED backend-2.0.7 Upgrade complete
111 Wed Apr 25 10:53 SUPERSEDED backend-2.0.7 Upgrade complete
112 Wed Apr 25 14:58 SUPERSEDED backend-2.0.7 Upgrade complete
113 Wed Apr 25 15:02 SUPERSEDED backend-2.0.7 Upgrade complete
...
$ helm rollback backend 112

25. Helm Values
Define your application configuration per environment!
- Should be source controlled
- roboll/helmfile
- skuid/helm-value-store
- Should not contain secrets
SECRETS

26. Public Charts hub.kubeapps.com

27. Private Chart
Repository
Honestbee: Version 1
- Self-host:
- Using s3 Bucket + VPC endpoint
- CI/CD:
- Package
- Generate Index
- Push to s3
articles/devops/2017-07/drone-helm-
repository
resource "aws_s3_bucket" "b" {
bucket = "${var.bucket_name_prefix}.${var.domain_name}"
policy = "${data.aws_iam_policy_document.s3-read.json}"
website {
index_document = "index.html"
}
cors_rule {
allowed_headers = ["*"]
allowed_methods = ["PUT", "POST"]
allowed_origins = [
"https://${var.bucket_name_prefix}.${var.domain_name}",
]
}
}
data "aws_iam_policy_document" "s3-read" {
statement {
sid = "Access-from-specific-VPC-only"
effect = "Allow"
actions = [ "s3:GetObject" ]
resources = [
"arn:aws:s3:::${var.bucket_name_prefix}.${var.domain_name}/*",
]
condition {
test = "StringEquals"
variable = "aws:sourceVpce"
values = [ "${values(data.terraform_remote_state.kops-
state.vpc_endpoint)}",
]
}
}
}

28. Integration with CI/CD

29. Private Chart Repository
Honestbee Version 2:
- Use ChartMuseum
- CI/CD:
- Package and push
honestbee/drone-chartmuseum
ChartMuseum is an open-source Helm Chart Repository written in Go (Golang), with support for cloud
storage backends, including Google Cloud Storage and Amazon S3.
Works as a valid Helm Chart Repository, and also provides an API for uploading new chart packages to
storage etc.

30. Secrets?
Vault + VaultController
- Hashicorp Vault
- Supports k8s auth
- roboll/kube-vault-controller
- Add k8s custom resource: secretClaim
kind: SecretClaim
apiVersion: vaultproject.io/v1
metadata:
name: {{ template "fullname" . }}
labels:
{{- include "labels.standard" . | indent 4 }}
spec:
type: Opaque
path: "secret/{{ .Values.env }}/{{ .Release.Name }}"
renew: 3900

32. Join us https://careers.honestbee.com/departments/engineering/