In a productive cloud native development workflow, individual teams can build and ship software independently from each other. But with a rapidly evolving cloud-native landscape, creating an effective developer workflow using a platform based on something like Kubernetes can be challenging. You are all creating software to support the delivery of value to your customers and to the business, and therefore, the developer experience from idea generation to running (and observing) in production must be fast, reliable, and provide good feedback. During this talk, Daniel will share with you several lessons learned from real world consulting experience working with teams deploying to Kubernetes. Key takeaways include: Why an efficient development workflow is so important; a series of questions to ask in order to understand if you should attempt to build a PaaS on top of Kubernetes (everyone needs a platform, but how much should be built versus integrated versus bought?); a brief overview of developer experience tooling for Kubernetes, and how this domain could evolve in the future; the role of Kubernetes, Envoy, Prometheus, and other popular cloud-native tools in your workflow; and key considerations in implementing a cloud-native workflow.

1. Creating an Effective Developer
Experience for Cloud-Native Apps
Daniel Bryant
@danielbryantuk | @datawireio

2. “Developer Experience”

3. @danielbryantuk
Independent Technical Consultant, Product Architect at Datawire
Previously: Academic, software developer (from startups to gov),
architect, consultant, CTO, trainer, conference tourist…
Leading change through technology and teams

4. DevEx 101

5. Developer Experience (DevEx) is about...
“...reducing engineering friction between creating a hypothesis, to
delivering an observable experiment (or business value) in production”
- Adrian Trenaman (SVP Engineering, HBC)
https://www.infoq.com/news/2017/07/remove-friction-dev-ex

6. DevEx isn’t new, but it is important
● Lead time
● Deployment frequency
● Mean time to restore (MTTR)
● Change fail percentage
● Rapid provisioning
● Basic monitoring
● Rapid app deployment
https://martinfowler.com/bliki/MicroservicePrerequisites.html

7. DevEx: Three Components

8. DevEx,
Workflow,
Platforms

9. The Ideal Workflow

10. “Progressive Delivery”
https://redmonk.com/jgovernor/2018/08/06/towards-progressive-delivery/ https://launchdarkly.com/blog/progressive-delivery-a-history-condensed/

11. https://speakerdeck.com/stilkov/microservices-patterns-and-antipatterns-1

12. Decentralised Biz/Product Teams;
Centralised Specialists and Platform
Team Z:
Prototyping
Team A: Mission-
Critical
Team T:
Production Phase

13. https://twitter.com/kelseyhightower/status/851935087532945409

14. https://www.infoq.com/news/2017/06/paved-paas-netflix
https://www.infoq.com/news/2018/07/shopify-kubernetes-paas

15. Should I Build a
PaaS on k8s?

16. Fundamental questions
Do you understand your domain?
Is your problem domain complex?
Do you have product/market fit?
Question
Is your solution event-driven (and simple)?
Should you be adding value elsewhere?

17. SOLID K8s: Open for Extension...
● Kubernetes becoming de facto CoaaS (the new cloud broker?)
○ Lots of hosted options
● Know the extension points
○ Custom Resources & Controllers
○ Operators
○ CloudBuddies
● Extension enables custom workflow
○ “Kubernetes Custom Resource, Controller and Operator Development Tools”

18. https://github.com/weaveworks/flux

19. How quick do you need feedback?
Question
https://mitchdenny.com/the-inner-loop/
https://bit.ly/2RXfokz

20. Automate Inner Dev Loop
https://blog.hasura.io/draft-vs-gitkube-vs-helm-vs-
ksonnet-vs-metaparticle-vs-skaffold-f5aa9561f948
https://codeengineered.com/blog/2018/kubernet
es-helm-related-tools/

21. ● Draft
○ Automates “inner loop” build-push-deploy
○ Utilises Helm
● Gitkube
○ Automates build-push-deploy
○ Provides heroku / CF like experience
● Skaffold
○ Automates build-push-deploy
○ Watches source code
○ Provides “dev” and “run” (CD) modes
● Tilt
○ Automates “inner loop” build-test-deploy
● Garden
○ Automates local build-push-test-deploy
Automate Inner Dev Loop
● Helm (*)
○ Package manager for k8s
○ Deploy and manage (ready-made) charts
● Ksonnet
○ Define k8s manifests in jsonnet
○ Create composable config/services
● Telepresence (*)
○ Enables local-to-prod development
(*) CNCF projects

22. Develop and test services locally, or
within the cluster (or both)?
● Working locally has many advantages
○ Reduce ops cost of multi-cluster
● However, some systems are simply too
large to run locally (for integration tests)
● Local/remote container dev tools like
Telepresence and Squash allow hybrid
Question

23. “Bring the cloud to you” “Put you in the cloud”

24. How do want to verify your system?
● Pre-prod testing in complex systems
● Canary testing is very powerful
https://medium.com/@copyconstruct/testing-microservices-
the-sane-way-9bb31d158c16
Question

25. Envoy for Managing L7 Traffic
● “Service-mesh all the things”?
● Allows fine-grained release
● Many control planes (for Envoy)
○ Ambassador
○ Gloo
○ Istio
○ Consul Connect
https://www.infoq.com/articles/ambassador-api-gateway-kubernetes

26. Canary UX

27. https://github.com/weaveworks/flagger

28. Canary gotchas (and mitigations)
● Observability is a prerequisite
○ Service Level Indicators (SLIs)
○ Service Level Objectives (SLOs)
○ Key Performance Indicators (KPIs)
● Needs high volume of diverse
(representative) traffic
● Take care with side effects
● Focus on “golden signals”
○ Latency, traffic, errors, saturation
○ Okay to initially “eyeball” data
○ Create actionable alerts
● Load test (with flag header)
● Run synthetic transactions
● Service virtualisation (Hoverfly)

29. Observability UX

30. Do you want to implement “guard rails”
for your development teams?
● Larger teams often want to provide
comprehensive guard rails
● Startups and SMEs may instead value
team independence
● Hybrid? Offer platform, but allow service
teams freedom and responsibility
https://blog.openshift.com/multiple-deployment-methods-openshift/
Question

31. Build vs buy
https://www.infoq.com/news/2019/03/airbnb-kubernetes-workflow
https://docs.openshift.com/container-platform/3.9/dev_guide/openshift_pipeline.html

32. Security guard rails

33. Getting Started:
Where to Focus

34. Decentralised Biz/Product Teams;
Centralised Specialists and Platform
Team Z:
Prototyping
Team A: Mission-
Critical
Team T:
Production Phase

35. Some thoughts on where to focus...
Prototype Production Mission Critical
Dev and test Local / hybrid Hybrid / local / staged Local / (hybrid) staged
Release Canary
(synthetic shadow)
Canary / pre-prod test Pre-prod test / Canary
Guide rails “YOLO” Limited Strong
Where to focus? Inner development
loop & CI/CD
Observability and
scaffolding (codifying
best practices)
Observability, debugging
and “recreatability”
(environment & data)

36. Conclusion

37. In Summary
The developer experience is primarily about minimising the friction between having
an idea, to dev/test, to release, to delivering observable business value
How you construct your ‘platform’ impacts the developer experience greatly
You must intentionally curate the experience of: local development, continuous
delivery, release control, observability, debuggability, and more...

38. Thanks for Listening!
Questions, comments, thoughts…
db@datawire.io
@danielbryantuk
More info: getambassador.io | telepresence.io | Canary Releasing

39. Bonus

40. Do you have a strong opinion on code
repository structure?
● Monorepo:
○ Coordination of integration and testing
across services is generally easier,
○ Service dependency management easier
● Multi-repo:
○ Clearer ownership
○ Can promote loose coupling
○ Refactoring and code-level standardization
can be challenging
http://blog.shippable.com/our-journey-to-microservices-and-a-mono-repository

41. https://blog.gardeviance.org/2015/03/on-pioneers-settlers-town-planners-and.html

42. Workflow
Tooling and
Techniques

43. https://www.youtube.com/watch?v=MGeUUmdtdKA

44. Skaffold

45. Telepresence

46. Pattern: Self-Service Init and Deploy

47. Pattern: CI/CD Enforces Policy
● Make is easy to do the right thing
○ Self-service pipeline creations
○ Bake-in hooks/slots for platform functionality
○ Sensible policy defaults
● Testing of NFRs is vital
○ Security
○ Performance
○ Quality
https://www.youtube.com/watch?v=hJkhPP2OLA8

48. Pattern: Observability > Testing
● Essential part of the platform and developer
workflow/experience
○ Monitoring, logging and tracing
○ Bake-in hooks to scaffolding
● Global && service-to-service dashboards
● “Observability and Avoiding Alert Overload
from Microservices at the Financial Times”

49. UX, it matters...