Everything as Code principles and processes we used at Hepsiburada from Workstation Provisioning to Infrastructure. Learnings from our production experience Agenda: - Infrastructure As Code - Workstation Provisioning - Pipelines as Code - Observability As Code - ADRs - Test Driven Everything

1. EVERYTHING AS CODE

2. TECHNOLOGY
About Us
SERHAN GÜNEY
● Full-Stack Developer with 8 years of experience
● Currently working in Hepsiburada Marketplace
team as Senior Full-Stack Developer
https://www.linkedin.com/in/serhanguney/
serhangny@gmail.com
https://gitlab.com/serhan
ERDEM ERBAŞ
● Full-Stack developer with 2 years of experience
● Currently working in Hepsiburada Marketplace
team as Full-Stack Developer
https://www.linkedin.com/in/erdemerbas/
erdemerbas12699@gmail.com

3. TECHNOLOGY
Disclaimer:
In this presentation, We are going to try
to give you an overall picture.
E

4. TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code
❖ ADRs
❖ Test Driven Everything
EE

5. TECHNOLOGY
Agenda
❖ Introduction <= You are here now
➢ Why Everything As Code
➢ Automation
➢ Gitops
➢ Code Review
➢ Testing
➢ Immutability
➢ Disposability
❖ Workstation Provisioning
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code
❖ ADRs
❖ Test Driven Everything
E

6. TECHNOLOGY
The idea behind the Everything as Code
concept is that infrastructure codified in a
declarative specification and treated like
application code such that they follow the same
software development lifecycle practices.
E

7. TECHNOLOGY Why Everything As Code
❖ Automation
❖ GitOps
❖ Code Review
❖ Testing
❖ Immutability
❖ Disposability
E

8. TECHNOLOGY Automation
❖ Higher production rates
❖ Increased productivity
❖ More efficient use of materials
❖ Better product quality
E

9. TECHNOLOGY GitOps
❖ Git as the source of truth
❖ Versioned CI/CD on top of declarative infrastructure
❖ Immutable deployment
❖ Stable and reproducible rollbacks
E

10. TECHNOLOGY Code Review
❖ Coding standards compliance
❖ Team cohesion
❖ Teaching and sharing knowledge
❖ Higher code quality
E

11. TECHNOLOGY Testing
❖ Testing verifies that the system meets the different requirements including,
functional, performance, reliability, security, usability and so on.
❖ Testing validates that the system being developed is what the user needs
E

12. TECHNOLOGY Immutability
❖ Build components to an exact set of specifications
❖ If a change to a specification is required, then a whole new set is
provisioned based on the updated requirements, and the previous is taken
out of service
❖ It is always possible to go back to the desired state
E

13. TECHNOLOGY Disposability
❖ Don’t Fix it, Throw it Away!
❖ It can be compared and contrasted with an application's disposability, one
of the characteristics of the 12 factor app
E

14. TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning <= You are here now
➢ What is workstation provisioning
➢ Onboarding
➢ Windows
➢ Linux
➢ Benefits
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code
❖ ADRs
❖ Test Driven Everything
S

15. TECHNOLOGY S

16. TECHNOLOGY Onboarding
❖ Preconfigured workstation
❖ Instead of following a guideline just press start
❖ Configuration drift
❖ Time effective
❖ Before/After
S

17. TECHNOLOGY
Workstation Provisioning
❖Windows
➢ Puppet
➢ Choco
➢ IDE
➢ Tools
E

18. TECHNOLOGY
Puppet
➢ Puppet is a desired state based configuration management automation
tool.
➢ Includes its own declarative language to describe system configuration
➢ Puppet supports all powershell commands(Powershell DSC)
E

19. TECHNOLOGY
Windows Provisioning
❖ Package
➢ Use chocolatey for default
package provider
➢ Taking advantage of powershell
desired state configuration
➢ Use specific versions(to avoid
breaking changes)
E

20. TECHNOLOGY
Windows Provisioning
❖ Configuration
➢ Configure installed
packages
➢ Every developer has
the same config
➢ Provides
immutability
E

21. TECHNOLOGY
Workstation Provisioning
❖Linux
➢ Image Bakery
➢ Vagrant
➢ Multiple Boxes
➢ Image Bakery vs CM
S

22. TECHNOLOGY
Image Bakery
❖ Developer Box
➢ Specialized ubuntu images are created using
Packer
➢ Minimized image
■ https://github.com/chef/bento
S

23. TECHNOLOGY
Vagrant
❖Vagrant orchestrates developer box provisioning
❖Virtualbox is the preferred provider
❖Ansible is the preferred provisioner
S

24. TECHNOLOGY
Ansible
❖Like Puppet, Ansible is another CM tool
❖Agent-less architecture
➢ everything is done by using SSH
S

25. TECHNOLOGY
ShowCase
❖ Image Bakery
❖ CM
➢ Vagrant
➢ Ansible
S

26. TECHNOLOGY
Multiple Boxes
❖ Differences of boxes are as it is in the image
➢ Seperate box for each business vertical
❖ In Common they all have
➢ required docker containers to ease local testing and debugging
➢ Git config to specify user
➢ ssh config to ease connection to build,qa and prod envs
E

27. TECHNOLOGY
Image Bakery vs CM
❖ Immutables like ansible should be in image bakery.
❖ Mutables like docker version should be configured by CM
E

28. TECHNOLOGY
Benefits
❖ Onboarding
❖ Dev/Prod parity
➢ Keep development, staging, and production as
similar as possible
❖ Testable infrastructure
E

29. TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning
❖ Infrastructure As Code <= You are here now
➢ Build Infra
➢ Environment Infra
❖ Pipelines As Code
❖ Observability As Code
❖ ADRs
❖ TDE
E

30. TECHNOLOGY
S

31. TECHNOLOGY
Infrastructure as Code
❖ Both On Premise
❖ Everything is a Docker Container
❖ Nginx as reverse-proxy
❖ No app-specific published ports
➢ Internal DNS
➢ Taking advantage of docker swarm mode SDN
E

32. TECHNOLOGY E

33. TECHNOLOGY
Build - Environment Infra
❖ Separation of Concerns
❖ Build generates artifacts - Environment runs artifacts
❖ Reduces complexity
❖ Different networks and docker swarms
➢ Build swarm,
➢ Environment swarms(eg. qa, prod)
S

34. TECHNOLOGY
Agenda
❖ Workstation Provisioning
❖ Infrastructure As Code
➢ Build Infra <= You are here now
■ Bootstrap
■ Gitlab
■ GoCD
■ Docker Registry (Harbour)
■ Nexus
➢ Environment Infra
❖ Pipelines As Code
❖ Observability As Code
❖ ADRs
❖ TDE
S

35. TECHNOLOGY
Build Infra
S

36. TECHNOLOGY
Build Infra
❖ Where everything gets built
❖ Artifacts and Images are pushed to appropriate registries
➢ eg. docker registry, nexus
❖ Automated provisioning of build infra via bootstrap.sh
➢ cloud or on premise
❖ Uses Docker swarm mode for orchestration
S

37. TECHNOLOGY
Bootstrap
❖ Bootstrapping is only necessary when the build infrastructure is
provisioned for the first time
❖ Bootstrapping will install barebone build infra services
➢ Gitlab(with groups and project)
➢ GoCD
➢ Docker Registry , etc
❖ Just invoke the `bootstrap.sh` and chill
❖ Can be disposible `dispose.sh` and cry
S

38. TECHNOLOGY
Bootstrap
❖ bootstrap.sh creates base-images required during Bootstrap
➢ eg. alpine, curl, golang
❖ Solves dependencies with health checks
❖ Applicable even to devbox
➢ for testing purposes
❖ Two different configurations
➢ One for barebone bootstrap image
➢ Specialized one for the environment
S

39. TECHNOLOGY
Gitlab
❖ Our preferred VCS
❖ Holds all our code and configuration
❖ Initialization steps are done by bootstrapper
❖ Configuration
E

40. TECHNOLOGY
Gitlab
❖ Showcase
➢ Gitlab omnibus conf
➢ user authorization
➢ slack integration
E

41. TECHNOLOGY
GoCD
❖ Our preferred CI/CD tool
❖ Pull based
❖ Agent based
➢ Capability based task scheduling
➢ Only docker capability(Only one agent to rule them all)
❖ Pipelines As Code
S

42. TECHNOLOGY
GoCD
❖ Base Build Infra pipelines are created by bootstrapper
❖ Updates itself via docker volume
❖ Slack Integration
❖ Pipeline as Code
❖ User Authentication
S

43. TECHNOLOGY
Docker Registry
❖ Holds all Docker images
❖ Initially used private docker registry
❖ Recently Harbor is used as container registry
➢ Registry authorization
E

44. TECHNOLOGY
Nexus
❖ Stores component type artifacts
❖ Keeping it internal helps you overcome external network issues
❖ Vagrant Box
➢ Raw repository
➢ Index.json (hyperlink)
❖ Apt package plug-in
E

45. TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning
❖ Infrastructure As Code
➢ Build Infra
➢ Environment Infra <= You are here now
■ Nginx
■ Consul
■ ElasticSearch
■ RabbitMQ
❖ Pipelines As Code
❖ Observability As Code
❖ ADRs
❖ TDE
E

46. TECHNOLOGY
Environment Infra
❖ Where Build Infra artifacts get to run
❖ Artifacts and Images are pulled from appropriate registries
➢ eg. docker registry, nexus
❖ Build once, run anywhere
➢ Configures image artifacts wrt environments
❖ Uses swarm as container scheduler
➢ 260 containers in total
➢ Will be replaced by k8s w/ istio
❖ Deployments are managed by constraints
E

47. TECHNOLOGY
Nginx
❖ High-performance HTTP server and reverse proxy
❖ Deploys on all nodes
❖ Creates all external overlay network definitions on deployment
❖ Configured via nginx.conf
➢ regexp to match wildcards
S

48. TECHNOLOGY
❖ SHOWCASE
➢ nginx.conf
➢ wildcards
➢ docker file
S

49. TECHNOLOGY
Consul
❖ Tool for service discovery and configuration
❖ Clustered
❖ Used for dynamic service configuration data and feature toggles
❖ Service discovery as RabbitMQ cluster peer discovery mechanisms
➢ RabbitMQ Cluster with HAProxy and Consul @Hepsiburada Tech
Blog
S

50. TECHNOLOGY
ElasticSearch
❖ Open-source search engine built on top of Lucene, a full-text search-
engine library.
❖ You know, for Search…
E

51. TECHNOLOGY
RabbitMQ
❖ The most widely deployed open source message broker.
❖ Uses consul based rabbitmq clustering
❖ Migrating RabbitMQ in a high traffic setup @Hepsiburada Tech Blog
❖ Manages plugins via enabled plugin configuration
❖ Can be configured via env variables or rabbitmq.conf
E

52. TECHNOLOGY
Environment Infra Also Includes
❖ Redis HA
❖ MongoDB
❖ Varnish
❖ ELK
❖ Prometheus
❖ Grafana
❖ Alert Manager
❖ Exporters
❖ Jaeger
Observability as Code
E

53. TECHNOLOGY
Agenda
❖ Introduction
❖ Workstation Provisioning
❖ Infrastructure as Code
❖ Pipelines as Code <= You are here now
➢ Definition of PaC
➢ Template Oriented Pipelines
➢ Examples
➢ Configuration
❖ Observability As Code
❖ ADRs
❖ TDE
S

54. TECHNOLOGY
Pipelines As Code
❖ Pipeline config is stored in VCS either with your application code or in
a separate repository
❖ Pipeline config repos are referenced at GoCD
❖ Application repos may contain .gocd.yml file
S

55. TECHNOLOGY
Template-based Pipelines
❖ Templates are stored at GoCD
❖ Docker eases templating
➢ same templates for all EaC,
➢ e.g. Golang API, React App, .Net Core API, ...
❖ Enables DSL-like pipeline definitions
➢ build-and-push,
➢ retag-and-push,
➢ auto-deploy,
➢ manual-deploy,
➢ deploy-prod, …
S

56. TECHNOLOGY
Template-based Pipelines
S

57. TECHNOLOGY
Showcase
❖ .gocd.yml
E

58. TECHNOLOGY
Agenda
❖ Workstation Provisioning
❖ ADRs
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code <= You are here now
➢ ELK
➢ Grafana
➢ Prometheus
➢ Tracing
❖ ADRs
❖ TDE
S

59. TECHNOLOGY
Observability As Code
❖ Where Build and Environment Infra are observed
❖ Resides with Environment Infra Code
❖ Three pillars of observability
➢ Logging
➢ Metrics
➢ Tracing
❖ Alerts
S

60. TECHNOLOGY
Logging
❖ ELK
❖ Logs shipped to ELK
➢ App logging
➢ Infra logging
❖ Watcher
❖ GELF for dockerized apps
❖ FileBeats for legacy apps
❖ Logstash pipeline
➢ grok
➢ gelf
S

61. TECHNOLOGY
ELK
❖ ElasticSearch
➢ Curator
➢ Watcher
➢ Slack Integration
❖ Logstash
❖ Kibana
E

62. TECHNOLOGY
Watcher
❖ Cluster Health alerts
❖ Application alerts
E

63. TECHNOLOGY
ELK
❖ ShowCase
➢ Watcher alerts
➢ Healthchecks
➢ Curator
E

64. TECHNOLOGY
Metrics
❖ Prometheus
❖ Exporters
❖ Alerts
❖ Monitors
S

65. TECHNOLOGY
Metric Monitoring Stack
S

66. TECHNOLOGY
Prometheus
❖ Holds all infra and app metrics
❖ Uses metrics exposed via exporters
❖ Constraints and docker volume utilized to persist data
❖ Generic and Environment based rules
❖ Slack Integration
➢ Alert Manager
S

67. TECHNOLOGY
Prometheus
❖ ShowCase
➢ Scrape conf
➢ Prometheus Rules
➢ Alert Manager
E

68. TECHNOLOGY
Exporters
❖ Export metrics in prometheus style
❖ Exporters used:
➢ Node Exporter
➢ cAdvisor
➢ Rabbitmq Exporter
➢ Consul Exporter
➢ Varnish Exporter
➢ Redis Exporter
E

69. TECHNOLOGY
Grafana
❖ Visualize prometheus data
❖ Provisioning
➢ Datasource
➢ Dashboard
➢ Env based provisioning
➢ V4 vs V5
E

70. TECHNOLOGY
Grafana
E

71. TECHNOLOGY
Grafana
E

72. TECHNOLOGY
Tracing
❖ Jaeger
❖ Open tracing
E

73. TECHNOLOGY
Agenda
❖ Workstation Provisioning
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code
❖ ADRs <= You are here now
❖ TDE
E

74. TECHNOLOGY
ADRs
❖ An architectural decision (AD) is a software design choice that addresses
a significant requirement.
❖ MarkDown syntax is used for documents.
❖ ADR cli utilities
E

75. TECHNOLOGY
ADRs
E

76. TECHNOLOGY
Agenda
❖ Workstation Provisioning
❖ Infrastructure As Code
❖ Pipelines as Code
❖ Observability As Code
❖ ADRs
❖ TDE <= You are here now
S

77. TECHNOLOGY
Test Driven Everything
❖ Since Everything is code
➢ Everything should be tested
S

78. TECHNOLOGY
Test Driven Everything
❖ Workstation Provisioning
➢ Windows
■ ServerSpec
➢ Linux
■ Goss
❖ IaC
➢ Varnish
❖ Test Suite
➢ Ginkgo/Gomega
➢ Mountebank
■ Gobank - https://github.com/durmaze/gobank
S

79. TECHNOLOGY
Test Driven Everything
❖ ShowCase
➢ goss
➢ serverspec
S

80. TECHNOLOGY
Future Roadmap
❖ Machine Provisioning
➢ Packer based images used in build/env infra
➢ SysAdmins manually provisions using this image
➢ Will be replaced by Terraform
❖ K8s w/ Istio migration
➢ Will be done after migrating the remaining legacy apps
❖ Secret Management

81. 81