This document provides recipes for containerizing and deploying common C2 frameworks like Cobalt Strike and Merlin on Kubernetes. It discusses using Docker to containerize the C2 payloads, creating Kubernetes deployments, services, and ingress resources to expose the containers. It also covers using Nginx Ingress and Istio service mesh with Envoy sidecars to provide redirection and traffic manipulation capabilities for C2 infrastructure on Kubernetes.

1. Recipes for C2 Operations on
Kubernetes

2. Who we are
◎Larry Suto - Larry is an independent
security consultant based out of Oakland,
CA. He spends a lot of time researching
using cloud infrastructure for all types of
security testing. He spends some time on
Windows security as well.
Twitter: @larrysuto
2

3. Who we are
◎Jeff Holden – Works at the largest college
system in the united states. CISO by day,
hacker by night. Currently living in an RV
with 3 dogs, 1 wife, and 1 kid.
◎@jeffh

4. Goals
◎Working in AWS, GCE, Azure
◎Automated
○ Set domains, access keys, etc
◎Portable
◎Scriptable

5. What is this Docker thing?

6. Docker

7. Why Kubernetes?
◎Orchestration
◎Infrastructure as code
◎Portable
◎Speed
◎Efficiency

8. Kubernetes
Open source container-orchestration system for automating application
deployment, scaling, and management.

9. Creating a Kubernetes Deployment
◎Configmap/Secret
◎Deployment yaml
◎Service yaml
◎Helm chart (optional)

10. What is this Kops thing?
• Takes care of the infrastructure grunt work
• Creates Network
• Creates Base VMs
• Creates load balancer
• Creates Access list
• Supports
• AWS
• GCE

11. Kops
◎kops create cluster --zones us-west-2a,us-
west-2b --topology private
◎--networking calico --master-size t3.micro -
-master-count 3 --node-size
◎ t3.large --name <kube cluster name>
◎--state=s3://<Your S3 Bucket Name> --yes

12. External DNS
◎Support all major cloud services
◎ Pod adds and updates DNS entries to
cloud providers DNS
○ Need to use cloud provider specific tool to create
hosted zone
○ Can automate domain registration
12

13. SSL Cert Manager
13

14. SSL Cert Manager
◎https://docs.cert-manager.io/en/latest/
◎Automates LetsEncrypt Certificate
Management
◎TLS Passthrough
14

15. Background
◎Placing Apache mod_rewrite in front of C2 infrastructure has been common
place for many years now
◎Many advances have been made in redirection technology recently driven
by the move to container-based cloud systems such as Kubernetes
◎Adoption of Docker by the red team is common but we have seen little in
the way of container management systems and service mesh technology
which provide sophisticated mechanisms for ingress and egress
manipulation
◎Service mesh technology holds a lot of promise for sophisticated c2
redirection and traffic manipulation and can leverage multiple clouds with
many simultaneous ingress points
◎With Kubernetes its quite straight forward to introduce new redirection
point dynamically

16. Recipes for Containerizing C2
◎Candidates
○ Cobalt Strike
○ Merlin (http/2)
○ Many others (Empire, Faction, silver, Covenant,..)

17. Dockerizing Cobalt Strike
docker build --build-arg cskey="$(cat license)" -t cobalt/ub18:1.0

18. FROM ubuntu:18.04 as base
MAINTAINER kubered
LABEL version=”1.0"
LABEL description="CobaltStrike."
ARG cskey
ENV cs_key ${cskey}
SHELL ["/bin/bash", "-c"]
RUN apt-get update &&
apt-get install -y wget curl net-tools sudo software-properties-common apt-utils --no-install-recommends &&
apt install -y openjdk-11-jdk &&
update-java-alternatives -s java-1.11.0-openjdk-amd64 &&
rm -rf /var/lib/apt/lists/*
RUN var=$(curl 'https://www.cobaltstrike.com/download' -XPOST -H 'Referer: https://www.cobaltstrike.com/download' -H 'Content-Type:
application/x-www-form-urlencoded' -H 'Origin: https://www.cobaltstrike.com' -H 'Host: www.cobaltstrike.com' -H 'Accept:
text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8' -H 'Connection: keep-alive' -H 'Accept-Language: en-us' -H 'User-Agent:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_1) AppleWebKit/604.3.5 (KHTML, like Gecko) Version/11.0.1 Safari/604.3.5' --data
"dlkey=$cs_key" | sed -n 's/.*href="([^"]*).*/1/p' | grep /downloads/ | cut -d '.' -f 1) &&
cd /opt &&
wget https://www.cobaltstrike.com$var.tgz &&
tar xvf cobaltstrike-trial.tgz &&
cd cobaltstrike &&
echo $cs_key > ~/.cobaltstrike.license &&
./update
RUN apt-get -y clean &&
apt-get -y autoremove
COPY cobalt-kube.profile /opt/cobaltstrike/profiles/
# set entry point
WORKDIR "/opt/cobaltstrike"
ENTRYPOINT ["./teamserver"]

19. Dockerizing Merlin
FROM golang:stretch
MAINTAINER @audibleblink
RUN apt-get update && apt-get install -y git make
RUN go get github.com/Ne0nd0g/merlin/...
WORKDIR $GOPATH/src/github.com/Ne0nd0g/merlin
VOLUME ["data/temp"]
EXPOSE 443
CMD ["go", "run", "cmd/merlinserver/main.go", "-i", "0.0.0.0"]

20. Configmap and Secret
kubectl create –f cobalt-config.yaml
---
apiVersion: v1
kind: Secret
metadata:
name: cobalt-password
data:
password: cbs76654A
Configmaps allow you modify application behavior without respinning an
image
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Release.Name }}-{{ .Values.malConfig }}
data:
randomized.profile: |-
{{ .Files.Get "randomized.profile" | indent 4 }}

21. Deployment yaml
(Helm template)
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ .Release.Name }}-deployment
namespace: {{ .Values.default }}
spec:
selector:
matchLabels:
app: {{ .Release.Name }}
replicas: 1
template:
metadata:
labels:
app: {{ .Release.Name }}
spec:
securityContext:
runAsUser: 0
imagePullSecrets:
- name: {{ .Values.pullSecret }}
containers:
- name: {{ .Release.Name }}
image: {{ .Values.image }}:{{ .Values.tag }}
imagePullPolicy: IfNotPresent
volumeMounts:
- name: {{ .Release.Name }}-{{ .Values.malConfig }}
mountPath: {{ .Values.malc2path }}
subPath: {{ .Values.subpath }}

22. Deployment yaml cont
ports:
- containerPort: 80
- containerPort: 50050
- containerPort 443
env:
- name: COBALT_PASSWD
value: {{ .Values.password | quote }}
- name: MY_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: MAL_C2_PATH
value: {{ .Values.malc2path | quote }}
args: ["$(MY_POD_IP)", "$(COBALT_PASSWD)",
$(MAL_C2_PATH)]
volumes:
- name: {{ .Release.Name }}-{{ .Values.malConfig }}
configMap:
name: {{ .Release.Name }}-{{ .Values.malConfig }}

23. Service Definition (teamserver mgmt internal)
apiVersion: v1
kind: Service
metadata:
name: cobalt-console
annotations:
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
spec:
ports:
- port: 50050
protocol: TCP
targetPort: 50050
selector:
app: teamserver1-deployment
type: LoadBalancer
status:
loadBalancer:
ingress:
- hostname: a54cae28bd42b11e7b2c7020a3f15370-27798109.us-west-2.elb.amazonaws.com

24. Service Definition (Cobalt Strike Listener)
apiVersion: v1
kind: Service
metadata:
name: cobalt-listener
labels:
app: team-server
spec:
ports:
- port: 80
protocol: TCP
selector:
app: team-server

25. Mounting SSL Certificates to Containers pt 1
apiVersion: v1
data:
merlin.key:
LS0tLS1CRUdJTiBQUklWQVRFIEtFWS0tLS0tCk1JSUV2QUlCQURBTkJ
na3Foa2lHOXcwQkFRRUZBQVNDQktZd2dnU2lBZ0VBQW9JQkFRQ1lrL
2hMaEMzalh2Y3kKUHY1VDdNcU1OMWR5STlQNVM5MlpUUllNT1VZb2
JiUXREeE1KbWxMd3g4c0owQURlWjVzTWRSQkYwWjJzNVBrMApHL3V
2d2c2c2JpSTFCaXVqaVBzdnRwWVpIaC9nZVdJUG5zS....
merlin.crt:
S0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURHRENDQWdBQ
0NRRHJDajdxWHFhR1VqQU5CZ2txaGtpRzl3MEJBUXN….
kind: Secret
metadata:
name: merlin-ssl
type: Opaque
$ cat ssl.secret

26. Mounting SSL Certificates to Containers pt 2
spec:
containers:
- image: merlin
name: merlin
volumeMounts:
- mountPath: "/opt/merlin/data/x509"
name: merlin-ssl
readOnly: true
ports:
- containerPort: 443
volumes:
- name: merlin-ssl
secret:
secretName: merlin-ssl

27. Deploying nginx-ingress
◎https://github.com/kubernetes/ingress-
nginx/blob/master/docs/deploy/index.md#
aws
◎Layer 4: use TCP as the listener protocol for
ports 80 and 443.

28. ConfigMaps in nginx-controller
◎The ConfigMap API resource stores configuration data
as key-value pairs. The data provides the
configurations for system components for the nginx-
controller.
◎https://kubernetes.github.io/ingress-nginx/user-
guide/nginx-configuration/configmap/
◎use-http2
○ Enables or disables HTTP/2 support in secure connections.

29. nginx-ingress service definition
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: ingress-teamserver
annotations:
# use the shared ingress-nginx
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: ts.example.com
http:
paths:
- path: /
backend:
serviceName: cobalt-listener
servicePort: 80

30. Redirection Recipes
◎nginx-ingress
○ Can be deployed with YAML or a Helm Chart
◎Service Mesh (Istio)

31. Nginx Ingress as a Redirector
Implants
AWS L4
Loadbalancer
type: loadbalancer
Nginx Ingress Controller
C2 Service Payload Service
Ingress Resource

32. Nginx-Ingress Annotations
◎https://kubernetes.github.io/ingress-nginx/user-
guide/nginx-configuration/annotations/
◎You can add these Kubernetes annotations to specific
Ingress objects to customize their behavior.
◎There are annotations to perform rewriting:
○ nginx.ingress.kubernetes.io/rewrite-target
◎ Using the nginx.ingress.kubernetes.io/configuration-
snippet you can add to the controller nginx directives that
would go to a location block on a normal nginx.

33. Convert mod_rewrite to nginx-ingress
◎https://winginx.com/en/htaccess
RewriteCond %{HTTP_USER_AGENT} "wget|curl|HTTrack|crawl|google|bot|b-o-t|spider|baidu" [NC,OR]
RewriteCond %{HTTP_USER_AGENT} =""
RewriteRule ^.*$ http://COMPANYDOMAIN.com/404.html/? [L,R=302]
location / {
if ($http_user_agent ~* ""wget|curl|HTTrack|crawl|google|bot|b-o-t|spider|baidu""){
rewrite ^(.*)$ http://www.google.com/404.html/? redirect;
}
}
Location block can be dropped into an Ingress definition using the configuration
snippet we just described
nginx rewrite
mod_rewrite

34. Create External Back End
kind: Service
apiVersion: v1
metadata:
name: google
namespace: default
spec:
type: ExternalName
externalName: www.google.com

35. Final Ingress Definition
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: cobalt-ingress
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/upstream-vhost: "www.google.com"
nginx.ingress.kubernetes.io/configuration-snippet: |
if ($http_user_agent ~* ""wget|curl|HTTrack|crawl|google|bot|b-o-
t|spider|baidu""){
rewrite ^(.*)$ http://www.google.com/404.html/? redirect;
}

36. Final Ingress Definition (Cont)
spec:
rules:
- host: ts.example.com
http:
paths:
- path: /
backend:
serviceName: cobalt-listener
servicePort: 80
- host: www.google.com
http:
paths:
- path: /
backend:
serviceName: google
servicePort: 80

37. Nginx Ingress Rewrite Rules
◎With server-snippet annotations
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/server-snippet: |
set $agentflag 0;
if ($http_user_agent ~* "(Mobile)" ){
set $agentflag 1;
}
if ( $agentflag = 1 ) {
return 301 https://m.example.com;
}

38. Nginx-ingress SSL passthrough
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: merlin-ingress
namespace: default
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
tls:
- hosts:
- c2.example.com
rules:
- host: c2.example.com
http:
paths:
- path: /
backend:
serviceName: merlin1
servicePort: 443

39. Validate the Addition of the Annotations
kubectl exec nginx-ingress-controller-873061567-4n3k2 –n
kube-system cat /etc/nginx/nginx.conf

40. Istio Service Mesh
Mixer Istio-AuthPilot
Envoy Envoy
SvcA SvcB
Envoy
Istio-Ingress
(Gateway)
Service A (C2) Service B
http, https,
http/2 http, https,
http/2
http, https,
http/2

41. Envoy Proxy
◎Envoy supports an HTTP level filter stack within the connection
manager.
◎Filters can be written that operate on HTTP level messages
without knowledge of the underlying physical protocol
(HTTP/1.1, HTTP/2, etc.) or multiplexing capabilities.
◎Three Types:
○ Decoder: Decoder filters are invoked when the connection manager is decoding
parts of the request stream (headers, body, and trailers).
○ Encoder: Encoder filters are invoked when the connection manager is about to
encode parts of the response stream (headers, body, and trailers).
○ Decoder/Encoder:
◎Istio can manage Envoy configuration
◎Envoy supports custom LUA scripts for HTTP filtering

42. Installing Istio
◎kubectl create namespace istio-system
◎Install CRD with kubectl apply
◎helm install install/kubernetes/helm/istio --name istio --namespace istio-system
--values install/kubernetes/helm/istio/values-istio-demo.yaml
◎Inject the Istio/Envoy sidecar manually or using injection webhook
○ istioctl kube-inject -f samples/sleep/sleep.yaml | kubectl apply -f -
○ kubectl label namespace default istio-injection=enabled –overwrite
◉ Injection occurs when the pod restarts

43. Istio Ingress
◎kubectl get svc istio-ingressgateway -n istio-system
◎Handles http or tcp ingress
◎Overcomes weaknesses of Kubernetes Ingress
○ Kubernetes Ingress mostly focused on http/https
◎Generally auto-provisions a load balancer
43

44. Multiple Ingress
C2 Services
Kubernetes
cluster
istioingressB
nginxingressB
nginxingressA
istioingressA
IstioegressA
C2 Services

45. Modifying the Ingress Gateway
45
istio-ingressgateway:
enabled: true
…
gateway.
image: node-agent-k8s
…
labels:
app: istio-ingressgateway
istio: ingressgateway
…
ports:
- port: 15020
targetPort: 15020
name: status-port
- port: 80
targetPort: 80
name: http2
nodePort: 31380
- port: 443
name: https
nodePort: 31390
- port: 50050
name: tcp
nodePort: 31400
This file can be modified to add multiple ingress points:
install/kubernetes/helm/istio/charts/gateways/values.yaml

46. Deploy your own Ingress Gateway
46
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: istio-ingressgateway
namespace: istio-system
spec:
replicas: 1
template:
metadata:
labels:
app: istio-ingressgateway
istio: ingressgateway
visibility: internal # put a custom label here.
annotations:
sidecar.istio.io/inject: "false"
scheduler.alpha.kubernetes.io/critical-pod: ""
spec:
...

47. Exposing Services via Istio-ingeress gateway
◎To expose a service using ingressgateway you have to create at
least 2 objects
○ Gateway
○ Virtual Service

48. Istio Gateway
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: teamserver-gateway
spec:
selector:
istio: ingressgateway # use istio default ingress gateway
servers:
- port:
number: 80
name: http-system
protocol: HTTP
hosts:
- "*"
- port:
number: 50050 # exposes teamserver admin port
name: tcp-admin
protocol: TCP
hosts:
- "*"
- port:
number: 443
name: https
protocol: HTTPS
tls:
mode: PASSTHROUGH
hosts:
- "*"
Gateway describes a load
balancer operating at the edge
of the mesh receiving incoming
or outgoing HTTP/TCP
connections.

49. Virtual Services
◎A Virtual Service defines a set of traffic routing rules to
apply when a host is addressed. A routing rule defines
matching criteria for traffic of a specific protocol.
◎If the traffic is matched, then it is sent to a named
destination service (or subset/version of it) defined in the
registry.
◎Supports many protocols : http, https, http/2

50. Virtual Service Example
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: teamserver1-basic
spec:
hosts:
- kube.cccco.net
gateways:
- teamserver-gateway
tcp:
- match:
- port: 80
route:
- destination:
host: teamserver1-service
port:
number: 80

51. Virtual Service
-Multiple Conditions
51
gateways:
- teamserver-gateway
http:
- match:
- headers:
user-agent:
regex: "Trident/7.0;srv:11.0"
uri:
prefix: "/zC"
route:
- destination:
host: teamserver1-service
- route:
- destination:
host: www.google.com
tls:
- match:
- port: 443
sni_hosts:
- kube.cccco.net
route:
- destination:
host: teamserver1-service
port:
number: 443
-route …..www.google.com

52. StringMatch for HTTP Headers
Field Type Description
exact string (oneof ) exact string match
prefix string (oneof ) prefix-based match
regex string (oneof ) ECMAscript style regex-based match
Case sensitive

53. Istio ingressgateway Tips
◎A selector is used by istio to select the ingressgateway.
This is important when there are multiple
ingressgateways
◎istio uses port naming in some routing logic
◎Port names are of the form protocol-suffix with grpc,
http, http2, https, mongo, redis, tcp, tls or udp as the
protocol.

54. Routing to External Destinations
◎Envoy passthrough to external services
○ On by default
○ global.outboundTrafficPolicy.mode option set to ALLOW_ANY
○ kubectl get configmap istio -n istio-system -o yaml | grep -o "mode:
ALLOW_ANY"

55. Notes on Ingress Capabilities
◎A mesh can have any number of gateways, and multiple different implementations of the
gateway can coexist
◎The Kubernetes Ingress API cannot express the routing needs of Istio. Kubernetes Ingress
looks for a common intersection between different HTTP proxies. It only supports the most
basic of HTTP routing.
◎Kubernetes Ingress itself does not support the TCP protocol. Kubernetes Ingress cannot be
set up to configure an NGINX Ingress Controller for TCP load balancing – requires a special
configmap
◎Istio Gateway has overcome the above shortcomings of Ingress by separating the L4-L6
configuration from the L7 configuration
◎Gateway is only used to configure L4-L6 functions
○ exposed ports, TLS configuration
◎A VirtualService is bound a Gateway in order to control inbound TCP and HTTP traffic

56. Drone
◎Drone is a modern CI/CD platform built with
containers as first class citizens. Pipelines are
configured using a yaml file that you check-in to your
git repository.
◎Can be deployed as a Docker container to a VM or a
Kubernetes cluster
◎Integrates with git repository and can deploy pipelines
to a Kubernetes cluster

57. Using Drone
57
pipeline:
build:
image: gradle:4.5.1-jdk9
commands: gradle --help
docker:
image: plugins/docker
secrets: [ docker_username, docker_password ]
repo: komljen/test-image
tags: ${DRONE_COMMIT_SHA}
kubectl:
image: komljen/drone-kubectl-helm
secrets: [ kubernetes_server, kubernetes_cert,
kubernetes_token ]
kubectl: "get pods"
helm:
image: komljen/drone-kubectl-helm
secrets: [ kubernetes_server, kubernetes_cert,
kubernetes_token ]
helm: list
notify:
image: plugins/slack
channel: drone-notification
secrets: [ slack_webhook ]

58. Monitoring C2 with Prometheus
◎Open source metrics based monitoring system
◎Can be used to instrument application
◎Applications without instrumentation support can be monitored
using exporters and other legacy methods
○ extract whitebox monitoring data from application logs for collection
in a timeseries database
○ Google mtail
○ Grok Exporter
○ JMX (Cobalt Strike teamserver is a Java application so theoretically it
could be supported)
◎C2 authors and developers can instrument code to provide fine
grained monitoring support

59. Google mtail
◎Can be used to create timeseries data from standard logs
○ https://github.com/google/mtail/blob/master/docs/Programming-
Guide.md
◎Prometheus exporter is available
◎Cobalt Strike teamserver logs are in the Cobalt Strike working
directory:
○ Ie /opt/cobaltstrike/logs
○ Weblogs and beacon session logs are available
◎Use mtail to create counters by matching events in log with
regular expressions

60. Grok Exporter
◎https://github.com/fstab/grok_exporter
◎Can be used to convert arbitrary logs into
Prometheus compatible time series data
◎An example would be the Cobalt Strike
beacon log checkin entry
○ This interval can be collected as a metric to
monitor beacon health
◎Grok Exporter uses same language as

61. Prometheus Visuals (with Sysdig)

62. References and Thanks
◎https://bluescreenofjeff.com/
◎https://www.nginx.com/blog/creating-
nginx-rewrite-rules/
◎https://istio.io/docs/concepts/traffic-
management/
62

63. Questions?
◎https://github.com/cloudc2/kubered
63

64. Envoy LUA Filters
◎Inspection of headers, body, and trailers while streaming in either the
request flow, response flow, or both.
◎Modification of headers and trailers.
◎Blocking and buffering the full request/response body for inspection.
◎Performing an outbound async HTTP call to an upstream host. Such a
call can be performed while buffering body data so that when the call
completes upstream headers can be modified.
◎Performing a direct response and skipping further filter iteration. For
example, a script could make an upstream HTTP call for authentication,
and then directly respond with a 403 response code.
64

65. ISTIO Envoy and LUA Filters
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: csc2-lua
spec:
workloadLabels:
app: csc2
filters:
- listenerMatch:
portNumber: 8080
listenerType: SIDECAR_INBOUND # will match with the
inbound listener for csc2:8080
listenerProtocol: HTTP
filterName: csc2.lua
filterType: HTTP
filterConfig:
inlineCode: |
... lua code ...

66. Example LUA Filter
66
-- Called on the request path.
function envoy_on_request(request_handle)
-- Wait for the entire request body and add a request header with the body size.
request_handle:headers():add("request_body_size", request_handle:body():length())
end
-- Called on the response path.
function envoy_on_response(response_handle)
-- Wait for the entire response body and a response header with the the body size.
response_handle:headers():add("response_body_size",
response_handle:body():length())
-- Remove a response header named 'foo'
response_handle:headers():remove("foo")
end

67. Egress Gateway apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: istio-egressgateway
spec:
selector:
istio: egressgateway
servers:
- port:
number: 443
name: tls
protocol: TLS
hosts:
- www.google.com
tls:
mode: PASSTHROUGH
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: egressgateway-for-google
spec:
host: istio-egressgateway.istio-system.svc.cluster.local
subsets:
- name: google

68. Service Entry
apiVersion:
networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
name: google
spec:
hosts:
- www.google.com
ports:
- number: 443
name: tls
protocol: TLS
resolution: DNS