CNCF의 오픈소스 기반 애플리케이션 Observability 프레임워크인 OpenTelemetry를 기반으로 AWS에서 텔레메트리 데이터를 수집, 추적 및 시각화하는 방법을 살펴봅니다. 또한 이를 통해 서비스의 문제점을 발견하고 그 원인을 분석하는 방법을 이야기합니다.

1. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
S E O U L | M A Y 4 , 2 0 2 3

2. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS OpenTelemetry
Observability
Senior Solutions Architect
AWS

3. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
OpenTelemetry
AWS OpenTelemetry
ECS, EKS OpenTelemetry
Observability
Agenda

4. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
OpenTelemetry

5. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Observability?
Observability
,
Telemetry
Metrics, Logs, Traces

6. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Telemetry
Metric library
Tracing Library
Logging Library
Application
logic
Metrics Agent
Tracing Agent
Logs Agent
app logs
system logs
file logs
app metrics
host metrics
Metrics Backend
Tracing Backend
Logs Backend
application
Host/Pod/Container
?

7. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
OpenTelemetry (OTel)
• - x
• /
• SDK, API (instrument)
•
•
• CNCF

8. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
OpenTelemetry
Tracing Metrics Logs Baggage
OpenTelemetry Protocol (OTLP)
API
- stable,
- feature-freeze
SDK: stable
Protocol: stable
API: stable
SDK: mixed
Protocol: stable
API: draft
SDK: draft
Protocal: stable
Span
API
- stable,
- feature-freeze
SDK: stable
Protocol: N/A
https://opentelemetry.io/docs/reference/specification/status/ 2023 5 2

9. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
OpenTelemetry
Tracing Metrics Logs Baggage
OpenTelemetry Protocol (OTLP)
API
- stable,
- feature-freeze
SDK: stable
Protocol: stable
API: stable
SDK: mixed
Protocol: stable
API: draft
SDK: draft
Protocal: stable
Span
API
- stable,
- feature-freeze
SDK: stable
Protocol: N/A
https://opentelemetry.io/docs/reference/specification/status/ 2023 5 2

10. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
OpenTelemetry
application
logic
Metrics
Tracing
Logs
app logs
system logs
app metrics
host metrics
Metrics
Tracing
Logs
application
Host/Pod/Container
Auto/Manual
Instrumentation
Exporter
traces
OpenTelemetry Collector Backend
Processor
correlate
correlate

11. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Distro for OpenTelemetry (ADOT)
CNCF Project
API, Library, Agent, Collector
AWS
Amazon EC2, Amazon ECS, Amazon EKS
AWS Lambda
AWS Support
Amazon CloudWatch, AWS X-Ray, Amazon OpenSearch
Service, Prometheus, AWS
Exporter

12. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
OpenTelemetry Collector
OTel
instrumentation
Backend 1
Backend 2
Backend N
…
Receiver Exporter
Processor
OTel collector
Extension
Receiver
:
prometheusreceiver
Processor
:
filterprocessor
Exporter
:
awsxrayexporter
Extension
Collector
:
sigv4authextension

13. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agent
S A M P L E D O C K E R F I L E
ADD https://github.com/aws-observability/
aws-otel-java-instrumentation/releases/download/
version/aws-opentelemetry-agent.jar /app/aws-opentelemetry-agent.jar
ENV JAVA_TOOL_OPTIONS="-javaagent:/app/aws-opentelemetry-agent.jar"
ENV OTEL_RESOURCE_ATTRIBUTES="service.name=MyApp"
ENV OTEL_IMR_EXPORT_INTERVAL="10000”
ENV OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4317"

14. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Collector
- https://github.com/aws-observability/aws-otel-collector.git
$ docker run --rm -p 4317:4317 -p 55680:55680 -p 8889:8888
-e AWS_REGION=us-west-2
-e AWS_PROFILE=default
-v ~/.aws:/root/.aws
-v "${PWD}/examples/docker/config-test.yaml":/otel-local-config.yaml
--name awscollector public.ecr.aws/aws-observability/aws-otel-collector:latest
--config otel-local-config.yaml;

15. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Collector
- https://github.com/aws-observability/aws-otel-collector.git
$ docker run --rm -p 4317:4317 -p 55680:55680 -p 8889:8888
-e AWS_REGION=us-west-2
-e AWS_PROFILE=default
-v ~/.aws:/root/.aws
-v "${PWD}/examples/docker/config-test.yaml":/otel-local-config.yaml
--name awscollector public.ecr.aws/aws-observability/aws-otel-collector:latest
--config otel-local-config.yaml;

16. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Collector
- https://github.com/aws-observability/aws-otel-collector.git
$ docker run --rm -p 4317:4317 -p 55680:55680 -p 8889:8888
-e AWS_REGION=us-west-2
-e AWS_PROFILE=default
-v ~/.aws:/root/.aws
-v "${PWD}/examples/docker/config-test.yaml":/otel-local-config.yaml
--name awscollector public.ecr.aws/aws-observability/aws-otel-collector:latest
--config otel-local-config.yaml;

17. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ADOT Collector
RECEIVERS PROCESSORS EXPORTERS EXTENSIONS
prometheusreceiver attributesprocessor awsxrayexporter healthcheckextension
otlpreceiver resourceprocessor awsemfexporter pprofextension
awsecscontainermetricsreceiver batchprocessor prometheusremotewriteexporter zpagesextension
awsxrayreceiver memorylimiterprocessor loggingexporter ecsobserver
statsdreceiver probabilisticsamplerprocessor otlpexporter awsproxy
zipkinreceiver metricstransformprocessor fileexporter ballastextention
jaegerreceiver spanprocessor otlphttpexporter sigv4authextension
awscontainerinsightreceiver filterprocessor prometheusexporter
resourcedetectionprocessor datadogexporter
metricsgenerationprocessor dynatraceexporter
cumulativetodeltaprocessor sapmexporter
deltatorateprocessor signalfxexporter
logzioexporter
Amazon Managed Service
for Prometheus

18. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ADOT Collector
RECEIVERS PROCESSORS EXPORTERS EXTENSIONS
prometheusreceiver attributesprocessor awsxrayexporter healthcheckextension
otlpreceiver resourceprocessor awsemfexporter pprofextension
awsecscontainermetricsreceiver batchprocessor prometheusremotewriteexporter zpagesextension
awsxrayreceiver memorylimiterprocessor loggingexporter ecsobserver
statsdreceiver probabilisticsamplerprocessor otlpexporter awsproxy
zipkinreceiver metricstransformprocessor fileexporter ballastextention
jaegerreceiver spanprocessor otlphttpexporter sigv4authextension
awscontainerinsightreceiver filterprocessor prometheusexporter
resourcedetectionprocessor datadogexporter
metricsgenerationprocessor dynatraceexporter
cumulativetodeltaprocessor sapmexporter
deltatorateprocessor signalfxexporter
logzioexporter
AWS X-Ray

19. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon ECS

20. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon ECS ADOT
TaskDefinition
ECS Cluster
Service
Service
TaskDefinition
trace, metric

21. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sidecar
AWS Cloud
Amazon Virtual Private Cloud (Amazon VPC)
Amazon ECS
ADOT
container
Application
container
Amazon ECS task
/metrics
VPC endpoint
Amazon Managed
Service
for Prometheus
/remote_write
AWS X-Ray
VPC endpoint
/PutTraceSegments

22. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Collector
https://github.com/aws-observability/aws-otel-collector/tree/main/config/ecs

23. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
ADOT
AWS Cloud
Amazon Virtual Private Cloud (Amazon VPC)
Amazon ECS
Amazon ECS task
ADOT
container
VPC endpoint
Amazon Managed
Service
for Prometheus
/remote_write
Application
container
Amazon ECS task
Application
container
Amazon ECS task
Application
container
Amazon ECS task
/metrics /metrics /metrics

24. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EKS

25. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EKS ADOT
Collector - Deployment
Collector - StatefulSet
Collector 0 Collector 1 Collector 2
Kubernetes node
Application pod
Application
container
Container
Sidecar
Pod A
Pod B
Collector
DaemonSet
Kubernetes node
Scrape
trace/metrics
Amazon Managed Service
for Prometheus
CloudWatch
X-Ray
Scrape targets
Persistent volume
Collector
metrics data
traces data
Amazon EKS
• Deployment
• DaemonSet
• StatefulSet
• Sidecar

26. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon EKS add-on for ADOT
3. kubectl apply
EKS Cluster
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
...
mode: deployment
...
config: |
receivers:
processors:
exporters:
...
OpenTelemetryCollector
Custom Resource
2. watch
ADOT Add-on
5.
1. EKS ADOT
Add-on
26
Amazon Managed Service
for Prometheus
AWS X-Ray
Kubernetes
API Server
ADOT Collector
Deployment
Service
Service
Service
6. Telemetry
3.
$ aws eks create-addon
--addon-name adot
--addon-version v0.51.0-eksbuild.1
--cluster-name $CLUSTER_NAME

27. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sidecar
AWS Cloud
Amazon VPC
Amazon EKS
VPC endpoint
Application
pod
Application
container
ADOT
Collector
pod
ADOT
container
Amazon Managed Service
for Prometheus
(AWS Account X
Workspace1)
/metrics
/remote_write
VPC endpoint
/remote_write
Amazon Managed Service
for Prometheus
(AWS Account Y
Workspace2)

28. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Collector

29. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
OpenTelemetry
Observability

30. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
OpenTelemetry
awsxrayexporter
awsemfexporter
prometheusremotew
riteexporter
ADOT Collector
ECS
EKS
Lambda
EC2
X-Ray
CloudWatch
Prometheus
otlpreceiver
awsecscontainermet
ricsreceiver
… exporter
prometheusreceiver

31. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tracing
Application Receiver Processor Exporter
ADOT Collector
OTEL SDK
Traces in OTLP format
OTLP Trace receiver
X-Ray exporter
X-Ray

32. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.

33. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
OpenTelemetry Observability
OpenTelemetry
•
•
•
•
Amazon
CloudWatch logs
AWS X-Ray traces
Amazon
CloudWatch metrics
AWS observability
tools

34. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS : http://aws.amazon.com/otel
: https://aws-otel.github.io
: https://github.com/orgs/aws-observability
: https://github.com/orgs/aws-observability/projects/4
, : https://github.com/aws-observability/aws-otel-community/issues
Gitter: https://gitter.im/aws-otel-distro/community
: https://observability.workshop.aws

35. © 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.
감사합니다
© 2023, Amazon Web Services, Inc. or its affiliates. All rights reserved.