Containers and other forms of dynamic infrastructure can prove challenging to monitor. How do you define “normal” when your infrastructure is intentionally in motion and changing every minute, or when there are no hosts to monitor at all? Join us as we share proven strategies for monitoring your containerized infrastructure on AWS, Amazon ECS, and AWS Fargate. This session is brought to you by AWS Partner, Datadog.

1. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Charly Fontaine - Software Engineer | Datadog
- Monitoring in Motion -
Best Practices for Monitoring Amazon ECS
Containers Launched with AWS Fargate
1

2. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Today’s story
2

3. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
[charly@datadoghq.com]
Name: Charly Fontaine
Role: Software Engineer
Interests:
* Containerized Infrastructures
* Monitoring all the things
* Motorcycles & bouldering

4. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
• SaaS based infrastructure and app monitoring
• Open Source Agent
• Logs
• Application Performance Monitoring
• Time series data (metrics and events)
• Processing trillions of data points per day
• Intelligent Alerting
Datadog Overview
4

5. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
What is Fargate
• Easy way to deploy your containers on AWS.
• Use containers as a fundamental compute primitive.
• No need to manage the underlying instances.
5

6. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Why use Fargate
6
• Speed up deployments.
• Focus on building apps not maintaining infrastructure.
• Optimisation of costs.

7. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
How Fargate works
• Two launch types: Amazon EC2 or Fargate.
• Task based approach.
• Amazon ECS agent starts and manages tasks.
7

8. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Monitoring
8

9. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
What is monitoring
9

10. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Why monitoring
10

11. 11

12. 12

13. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Why monitoring ?
Collecting data is cheap;
not having it when you
need it can be expensive
13

14. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Why monitoring ?
14

15. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. 15

16. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
How Monitoring ?
$ docker run -d --name dd-agent
-v /var/run/docker.sock:/var/run/docker.sock:ro
-v /proc/:/host/proc/:ro
-v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
-e DD_API_KEY=<secret>
datadog/agent:latest
16

17. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
How monitoring ?
$ cat /sys/fs/cgroup/cpuacct/docker/$CONTAINER_ID/cpuacct.stat
> user 2451 # time spent running processes since boot
> system 966 # time spent executing system calls since boot
$ cat /sys/fs/cgroup/cpu/docker/$CONTAINER_ID/cpu.stat
> nr_periods 565 # Number of enforcement intervals that have
elapsed
> nr_throttled 559 # Number of times the group has been throttled
> throttled_time 12119585961 # Total time that members of the
group were throttled (12.12 seconds)
17

18. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. 18

19. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
The Fargate Approach
19

20. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
$ AWS console
20

21. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
$ aws ecs describe-tasks —tasks taskarn
21

22. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
No Host.
No cgroups.
No procs.
No socket.
22

23. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. 23

24. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
$ curl http://169.254.170.2/v2/stats | jq ‘.'
* {
"277d764bde83262ea6eb32cde8cfe85e2378f4357d8c61137cf54b8621e9e922": {
"read": "2018-03-16T14:57:13.601159546Z",
"preread": "2018-03-16T14:57:12.613128625Z",
"num_procs": 0,
"pids_stats": {},
"network": {},
"memory_stats": {
"stats": {
"cache": 126029824,
"mapped_file": 53489664,
[…]
}
24

25. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
$ curl http://169.254.170.2/v2/metadata | jq ‘.'
* "Cluster": "charly-aws-summit",
* "TaskARN": "arn:aws:ecs:us-east-1:172597598159:task/3d00ef0b-9ff3-XXX",
* "DockerName": "ecs-charly-aws-summit-2-redis-b2",
* "Image": "redis:latest",
* "Labels": {
"com.amazonaws.ecs.cluster": "charly-aws-summit",
[...]
"com.amazonaws.ecs.task-definition-version": “2",
"com.datadoghq.ad.check_names": "["redisdb"]",
"com.datadoghq.ad.init_configs": "[{}]",
"com.datadoghq.ad.instances": "[{"host": "%%host%%", "port": "6379"}]"
}
25

26. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Demo
26

27. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Demo slide.
Not shown. For
dry run purposes
Exec in container
Curl /metadata
/stats

28. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Autodiscovery
Demo slide.
Not shown. For
dry run purposes

29. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Launching a Fargate
Task from the
console.
Demo slide.
Not shown. For
dry run purposes

30. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Demo slide.
Not shown. For
dry run purposes

31. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Changing a paradigm:
1 task is one workload
1 task is one mission
1 task is one set of actions and processes
31

32. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
There are no such things as challenges.
Just new approaches.
32

33. © 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved. 33