So you're done developing your web service, but what will happen when you deploy it to production? In other words: how ready is your service to receive live traffic? In this talk you'll learn what it takes to transition your web service from "done" to "ready" and make sure that you can release to production seamlessly and with confidence.

1. Readying your Go webservice
Ralph Ligtenberg
September 12, 2019
https://www.pexels.com/photo/web-close-up-photography-2632059/

2. Ralph Ligtenberg
Tech Lead @ Travix
C#, Go, Google Cloud Platform
twitter.com/prutswonder
medium.com/@prutswonder
linkedin.com/in/ralphligtenberg

3. Done
❏ “Works on my machine”
❏ Merged to master
❏ Unit & Integration tests
❏ Works as expected
Ready
❏ “Works anywhere”
❏ Deployed & tested
❏ Acceptance & load tests
❏ Performs as expected
https://www.pexels.com/photo/photo-of-green-data-matrix-1089438/

4. ● CI/CD: build & deployment automation
● Observability: logging, monitoring & tracing
● Resilience: Circuit breaker & panic recovery
● Lifecycle management: warm-up & shutdown
● Security: authentication & authorization
From “Done” to “Ready”
https://www.pexels.com/photo/abstract-art-circle-clockwork-414579/

5. CI/CD: Kubernetes & Estafette
https://www.pexels.com/photo/red-and-gray-industrial-machinery-2569842/

7. Estafette build stage
# https://estafette.io/usage/manifest/#build-stages
stages:
test-lint-build:
image: golang:1.13.0
env:
CGO_ENABLED: 0
GOOS: linux
GO111MODULE: "on"
GOGC: off
commands:
- go get -u golang.org/x/lint/golint
- CGO_ENABLED=1 go test -race -cover ./...
- golint -set_exit_status ./...
- go build -a -installsuffix cgo -o ./publish/${ESTAFETTE_LABEL_APP} .

8. Estafette docker image creation
# https://estafette.io/usage/extensions/estafette-extensions/#extensions-docker
bake:
image: extensions/docker:stable
action: build
path: ./publish
push:
image: extensions/docker:stable
action: push

9. Estafette deployment
# https://estafette.io/usage/manifest/#releases
releases:
development:
stages:
deploy:
image: extensions/gke:stable
container:
port: 8080
env:
CORS_ORIGINS: "*"
cpu:
request: 10m
limit: 100m
memory:
request: 25Mi
limit: 100Mi
# Continued on the right -->
liveness:
path: /live
readiness:
path: /ready
sidecar:
healthcheckpath: /live
slack-notify:
image: extensions/slack-build-status:stable
workspace: travix
channels:
- '#{ESTAFETTE_LABEL_TEAM}-builds'
when:
status == 'failed'

10. Logging: ELK Stack & zerolog
https://www.pexels.com/photo/bark-brown-cut-daylight-296333/

11. Logging
import (
"os"
"github.com/rs/zerolog"
"github.com/rs/zerolog/log"
)
// Logger is used to write log messages.
type Logger struct {
level zerolog.Level
logger zerolog.Logger
}
// NewLogger sets and returns a new global-level logger.
func NewLogger(appGroup, appName, logLevel string) Logger {
zLogLevel := ToZeroLogLevel(logLevel)
logger := log.Output(v3FormatWriter{Out: os.Stderr}).With().
Str("appgroup", appGroup).Str("appname", appName).
Logger().Level(zLogLevel)
log.Logger = logger
return Logger{level: zLogLevel, logger: logger}
}

12. Monitoring: Prometheus & Grafana
https://www.pexels.com/photo/black-and-white-business-chart-computer-241544/

13. Custom metrics
import p8s "github.com/prometheus/client_golang/prometheus"
var (
labelNames = []string{"affiliate", "status"}
responseCounter = p8s.NewCounterVec(p8s.CounterOpts{Name: "app_response_count"}, labelNames)
)
func init() {
p8s.MustRegister(responseCounter)
}
func countFailedResponse(affiliate string, reason string) {
responseCounter.WithLabelValues(affiliate, reason).Inc()
}
func countSuccessResponse(affiliate string) {
responseCounter.WithLabelValues(affiliate, "success").Inc()
}

15. Tracing: Jaeger
https://www.pexels.com/photo/white-airplane-728824/

16. Tracing middleware
// WithTracing creates a span for each request
func WithTracing(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// retrieve span context from upstream caller if available
tracingCtx, _ := opentracing.GlobalTracer().Extract(opentracing.HTTPHeaders,
opentracing.HTTPHeadersCarrier(r.Header))
span := opentracing.StartSpan(fmt.Sprintf("%v %v", r.Method, r.URL.Path),
ext.RPCServerOption(tracingCtx))
defer span.Finish()
ext.HTTPMethod.Set(span, r.Method)
ext.HTTPUrl.Set(span, r.URL.String())
tw := tracedResponseWriter{ResponseWriter: w}
rc := r.WithContext(opentracing.ContextWithSpan(r.Context(), span))
next.ServeHTTP(&tw, rc)
ext.HTTPStatusCode.Set(span, uint16(tw.Status()))
})
}

17. Tracing: Jaeger

18. Circuit breakers
https://www.pexels.com/photo/2-man-on-construction-site-during-daytime-159306/

19. Circuit breaker
// Try wraps a function with circuit breaking
func (b *Breaker) Try(cmdKey string, tryFunc func() (
interface{}, error)) (interface{}, error) {
type result struct {
response interface{}
err error
}
startTime := time.Now()
cmd := b.getOrAddCommand(cmdKey)
if !cmd.allowRequest() {
cmd.registerShortCircuited()
return nil, ErrRequestShortCircuited
}
// Continued on the right -->
ch := make(chan result, 1)
go func(ch chan result) {
defer close(ch)
response, err := tryFunc()
ch <- result{response: response, err: err}
}(ch)
timer := time.NewTimer(cmd.timeout)
defer timer.Stop()
select {
case res := <-ch:
if res.err != nil {
cmd.registerFailure(time.Since(startTime))
return nil, res.err
}
cmd.registerSuccess(time.Since(startTime))
return res.response, nil
case <-timer.C:
cmd.registerTimeout(time.Since(startTime))
return nil, &ErrRequestTimeout{commandKey: cmdKey,
timeout: cmd.timeout}
}
}

20. Panic recovery middleware
// WithPanicRecoveryFunc recovers any panics that occur in the next http handler.
func WithPanicRecoveryFunc(logger logging.Logger, next http.HandlerFunc) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
defer func() {
if rec := recover(); rec != nil {
logger.Error("PanicRecovered").
Bytes("stacktrace", debug.Stack()).
LogErr(fmt.Errorf("%s", rec))
w.WriteHeader(http.StatusInternalServerError)
}
}()
next(w, r)
}
}

21. Application start-up
https://www.pexels.com/photo/close-up-engine-landscape-locomotive-533608/

22. Server & start-up
type shutdownFunc func() error
type server struct {
ctx context.Context
version versionInfo
shutdown shutdownFunc
}
func (s *server) Serve() {
// (...)
mux := http.NewServeMux()
svr := http.Server{
ReadTimeout: time.Second * 3,
ReadHeaderTimeout: time.Second * 2,
WriteTimeout: time.Second * 10,
Addr: ":8080",
Handler: mux,
}
// Continued on the right -->
mux.HandleFunc("/ready", s.ready)
mux.HandleFunc("/live", s.live)
// Put your own endpoints here
err = s.setControllers(mux)
if err != nil {
log.Error("ServerControllerError").LogErr(err)
return
}
_ = svr.ListenAndServe() // Blocks until server stops
}
func (s *server) live(w http.ResponseWriter, r *http.Request) {
_, _ = io.WriteString(w, "Alive!")
}
func (s *server) ready(w http.ResponseWriter, r *http.Request) {
//TODO: Determine readiness
_, _ = io.WriteString(w, "Ready!")
}

23. Graceful shutdown
https://www.pexels.com/photo/bulb-close-up-conceptual-current-207420/

24. Server with graceful shutdown
func (s *server) Serve() {
// (...)
s.shutdown = func() error {
cancelCtx, cancel := context.WithTimeout(s.ctx,
time.Second * 10)
defer cancel()
err := svr.Shutdown(cancelCtx)
return err
}
signalCh := make(chan os.Signal, 1)
go s.listenToShutdownSignals(signalCh)
_ = svr.ListenAndServe() // Blocks until server stops
}
func (s *server) listenToShutdownSignals(
signalCh chan os.Signal) {
signal.Notify(signalCh,
syscall.SIGINT, // kill -SIGINT XXXX or Ctrl+c
syscall.SIGTERM, // kill -SIGTERM XXXX
syscall.SIGQUIT) // kill -SIGQUIT XXXX
for {
sig := <-signalCh
switch sig {
case syscall.SIGINT:
fallthrough
case syscall.SIGTERM:
fallthrough
case syscall.SIGQUIT:
s.shutdown()
return
}
}
}

25. Security
https://www.pexels.com/photo/door-handle-key-keyhole-279810/

26. Estafette & cloud credentials
releases:
development:
stages:
deploy:
image: extensions/gke:stable
visibility: public-whitelist
container:
# (...)
sidecar:
healthcheckpath: /live
useGoogleCloudCredentials: true

27. Estafette secrets
production:
# https://estafette.io/usage/manifest/#release-actions
actions:
- name: deploy-canary
- name: deploy-stable
- name: rollback-canary
stages:
deploy:
image: extensions/gke:stable
visibility: public
container:
port: 8080
env:
# AES-256-encrypted value
AUTH_BASIC_TOKEN: estafette.secret(6J1RtwwpvMIhMtbE.QuRJhnTgO8FcHP-cEtJy72pTzaTw2L5-1zeT0A==)

28. Authorization middleware
import "net/http"
// WithAuthBasic performs a basic authorization check
func WithAuthBasic(token string, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
switch r.Header.Get("Authorization") {
case "Basic " + token:
next.ServeHTTP(w, r)
case "":
w.Header().Add("WWW-Authenticate", "Basic")
w.WriteHeader(http.StatusUnauthorized)
default:
w.WriteHeader(http.StatusForbidden)
}
})
}

29. CORS middleware
import "github.com/rs/cors"
// WithCORS wraps the handler with CORS options.
func WithCORS(handlerFunc http.HandlerFunc) http.Handler {
corsOptions := cors.Options{
AllowedOrigins: []string{os.Getenv("CORS_ORIGINS")},
AllowedMethods: []string{"HEAD", "OPTIONS", "GET", "PUT", "POST"},
AllowedHeaders: []string{"Origin", "Accept", "Content-Type", "X-Requested-With", "X-CSRF-Token"},
ExposedHeaders: []string{
"Access-Control-Allow-Headers",
"Access-Control-Allow-Methods",
"Access-Control-Max-Age",
},
MaxAge: 0,
}
return cors.New(corsOptions).Handler(handlerFunc)
}

30. ● Automate builds & deployments for your webservice
● Observe your webservice
● Make your webservice resilient
● Warm up your webservice
● Shut down your webservice gracefully
● Secure your webservice
Getting “Ready”
https://www.pexels.com/photo/abstract-art-circle-clockwork-414579/

31. Estafette: https://estafette.io/
Zerolog: https://github.com/rs/zerolog
Prometheus: https://github.com/prometheus/client_golang
Grafana: https://grafana.com/
Jaeger: https://www.jaegertracing.io/
CORS: https://github.com/rs/cors
Circuit breaker: https://martinfowler.com/bliki/CircuitBreaker.html
References
https://www.pexels.com/photo/abstract-art-circle-clockwork-414579/

32. Thank you!
Ralph Ligtenberg - @prutswonder
stock images downloaded from Pexels.com
https://www.pexels.com/photo/silhouette-of-airplanes-47044/