More Related Content Similar to 20190819 AWS におけるモニタリング 議論のための観点総ざらえ (20) More from Amazon Web Services Japan (20) 20190819 AWS におけるモニタリング 議論のための観点総ざらえ1. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
2019/8/19
JAWS-UG 2019/8/19
2. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
S W
S / Y/ /
/ D @ 0 A 2
D 1 @ J
A 90
• D S /
• 1 @
) (
3. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
•
• Amazon CloudWatch
•
•
4. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
5. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
CPU?
Memory?
6. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
“Monitoring is the action of observing and checking the behavior
and outputs of a system and its components over time.”
Mike Julian, Practical Monitoring: Effective Strategies for the Real World, O'Reilly Media
7. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
参考) Mike Julian, Practical Monitoring: Effective Strategies for the Real World, O'Reilly Media
AlerAng
AnalyAcs &
ReporAng
Visualization
Data Storage
Data Collection
/
CloudWatch
Amazon SNS
AWS Chatbot
Amazon Athena / QuickSight
Amazon Elasticsearch Service
CloudWatch Logs Insights
CloudWatch
Amazon Elasticsearch Service
AWS X-Ray
CloudWatch Metrics
CloudWatch Logs
Amazon S3
CloudWatch Metrics / Logs
CloudWatch
AWS X-Ray
AWS
8. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
参考) Mike Julian, Practical Monitoring: Effective Strategies for the Real World, O'Reilly Media
Alerting
Analytics &
Reporting
Visualization
Data Storage
Data Collection
/
CloudWatch
Amazon SNS
AWS Chatbot
Amazon Athena / QuickSight
Amazon Elasticsearch Service
CloudWatch Logs Insights
CloudWatch
Amazon Elasticsearch Service
AWS X-Ray
CloudWatch Metrics
CloudWatch Logs
Amazon S3
CloudWatch Metrics / Logs
CloudWatch
AWS X-Ray
AWS
/
Amazon Athena / QuickSight
Amazon Elasticsearch Service
CloudWatch Logs Insights
CloudWatch
Amazon Elasticsearch Service
AWS X-Ray
CloudWatch Metrics
CloudWatch Logs
Amazon S3
=
⊃
9. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
/
CloudWatch -> Amazon SNS ->
AWS Chatbot -> Slack/Chime
CloudWatch -> Lambda -> JIRA
Software
CloudWatch
CloudWatch Logs -> Amazon Elasticsearch
Service (Kibana)
CloudWatch Logs
CloudWatch Logs Insights
10. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
, Datadog
https://www.datadoghq.com/ja/blog/%E3%83%A2%E3%83%8B%E3%82%BF%E3%83%AA%E3%83%B3%E3%82%B0%E3%81%AE%E3%83%99%E3%82
%B9%E3%83%88%E3%83%97%E3%83%A9%E3%82%AF%E3%83%86%E3%82%A3%E3%82%B9/
/ …
CPU
…
( )
->
( )
->
11. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
•
•
•
•
•
•
•
“Mike Julian, Practical Monitoring: Effective Strategies for the Real World, O‘Reilly Media”の分類を元にメ
ンテナンスとコストの観点を追加
12. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon CloudWatch
[AWS Black Belt Online Seminar] Amazon CloudWatch
h9ps://www.slideshare.net/AmazonWebServicesJapan/20190326
-aws-black-belt-online-seminar-amazon-cloudwatch
13. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
CloudWatch Alarms
CloudWatch Events
CloudWatch Metrics
CloudWatch Logs
CloudWatch Logs Insights
AWS
CloudWatch Dashboardstime-baseevent-base
14. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• CloudWatch
•
•
•
• CloudWatch
•
• AWS AWS/<service> ( : AWS/EC2)
• / ( InstanceId=i-12345678)
15. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
C
16. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• % A8 , W _ e M dC W i
• W M s c W i
• FS % ur W FKlP dC aC Wo
• dC R 0, % D
0, %
ü I K W
LMEU
ü I K Wh
LMEU
ü C t KT
W N OE
ü % A8
17. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• / t ic} d{ himpeosncin pd
• himpeosndn f a [ gln]eosnd w
eosn i r r gln
/ TP
FD L
P TH
31
1 P G
RDL
LNH
EDSH
1 STPN
eosnv
HRSLP . LG .
D H,IHE E - FI D-I
EI , GHTDL
T QH . 31 7 STD FH TDTH
FKD H :PTLILFDTLP
SP RFH . DWS"HF
DFFP T .
TLNH .
,. . ,A RH LP .
S WHST RHSP RFHS .
B DR .DWS.HF . S WHST
. ,- .L STD FH
L ,- DEFGHI C
GHTDL . L STD FH LG . L
,- DEFGHI
STDTH . THRNL DTHG Z Z
:
0 L T L
9DNEGD
8L HSLS
THQ
4 FTLP
ybu
HTF HTF
QPT 31
/
6HD TK
5 DRG
2 T
31
/ 7 FD
1PGH
LQH L H
4PRHKPSH 7 SQHFTPR
18. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• l3 3 C n
• A a ) ( h
• c m t
• 1 C
CloudWatch Logs
C S W
C
AWS Lambda
Amazon Kinesis
Data Streams
Amazon Kinesis
Data Firehose
AWS
o
d
19. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
C
• g 5 W L ah
• 2 2 C L ah
• 1 0 / ld
8
c
C L
20. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• , 1 (0) C 5
• C 5 2 5
• W
21. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• 2 e kg 2 BA 8 p d z bmj f
ie kg ( (ouW l h
• Sc W 1 dws c zW L ay rtW b
• v e kg n
ü BA BG F
ü 2 08 F
ü 2 G :8F
ü 2 28 GD F 0G
ü BA /G D -GF
ü 2 BG .BD F BA
+ ( )
FFC ) BA B C B: 8HBC AFDB G A: F BF FBC 9BD
22. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
23. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
[ ]
•
•
•
•
•
•
•
“Mike Julian, Prac0cal Monitoring: Effec0ve Strategies for the Real World, O‘Reilly Media”の分類を元にメンテナンス
とコストの観点を追加
24. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
25. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• gS g o g U cg
• h ( ( ) 2 ( ) t
Real User Monitoring l M M
c Sc
R
Synthetic Monitoring g e/ i
M c S
e ag
Lambda ns r
R
26. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
CloudWatch
P
U
CloudWatch
Events
)
UE L
)
WT P P
A F
Ceh ) (
C
TL
F L
Alarms
F
ab cd C
Web
27. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
28. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• r M n M t
• ) ) ( A
M
e M
piAPI c g SQL c g
CloudWatch M
M
P
a c A o a CloudWatch Logs
P
f M l M f
M t m
AWS X-ray
P
29. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
D C
D
• D L S
• e D DL S
x
• a E d Es u
• 2 : ,2 D |
/ D
• ) W De W ih
/
• ( e c @ W
nt 2 : ,2
• lD (
• . . e c @ W
nt 2 :
• . . @
, . L S C
/ D
• wD e x
• , . L S o
###
30. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
A SX
A R -
W
[AWS Summit Tokyo 2019]
https://aws.amazon.com/jp/summits/tokyo-2019/
[AWS Summit Tokyo 2018] Application Performance Management (APM) on AWS
https://d1.awsstatic.com/events/jp/2018/summit/tokyo/aws/64.pdf
[AWS Black Belt Online Seminar] AWS X-Ray
https://www.slideshare.net/AmazonWebServicesJapan/aws-black-belt-online-seminar-2017-aws-xray
31. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
-
2
Elastic
Beanstalk
X-Ray
API
D A D
BD
X-Ray SDK/ D
C E
32. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
M
9
R 9
• T 9 9
• 55 / 2
• 55 / 2
• (// )
4 2 2
2
)
33. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
34. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
35. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( )
36. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
) (
•
( )
(EC2 )
EC2 /OS
(CPU/Memory/Network/Disk/Process…)
CloudWatch (Hypervisor )
CloudWatch / (OS )
(ELB) Health …
CloudWatch
ALB
(RDS)
CPU/
…
CloudWatch
RDS (OS )
RDS
(Kinesis Data Streams /
SQS / Amazon MQ…)
/
…
CloudWatch
(Elasticache)
/
Evictions…
CloudWatch
…
37. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
1
1 / 2
CPU CloudWatch CPUUtilization
CPU
CloudWatch CPUCreditBalance and/or
CloudWatch Agent mem_free
CloudWatch NetworkIn / NetworkOut
IO CloudWatch DiskReadOps / DiskWriteOps
CloudWatch Agent disk_free and/or
CloudWatch Agent pid_count and
38. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
2 /
CloudWatch StatusCheckFailed_Instance CloudWatch Events
->
CloudWatch StatusCheckFailed_System CloudWatch Events
->
(Auto
Recovery)
CloudWatch Agent ->
CloudWatch Logs
- CloudWatch Logs
Metric Filter
https://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/viewing_metrics_with_cloudwatch.html
CloudWatch
https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/monitoring/metrics-collected-by-
CloudWatch-agent.html
CloudWatch Agent
39. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
1/
1/ 2
DB eg eS
CPU 1
CloudWatch uly e CPUUtilization c r m
DB eg e
u y 2
CloudWatch uly e 22 2 2 c r m
p U a
el
d
(≠ egry t)
CloudWatch uly e 22 2 c r m and/or
ib l
egry t CloudWatch uly e 2 2,:21 c r m
y a CloudWatch uly e 2 c r m ans/or
ib l
A y e e l MC V ,
u y PF C RBL y c e l PR
M ce t S o s e
40. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
2
2 /
M CloudWatch M I and
L
M
( M T )
CloudWatch M I
D T CloudWatch M I and/or
CloudWatch Logs L
M
L S
https://docs.aws.amazon.com/ja_jp/AmazonRDS/latest/AuroraUserGuide/Aurora.Monitoring.html
CloudWatch M
41. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• OSGD A
• A AD
• M 1 R E
O E
:: :. :
. . : / :
43
42. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• c S_a e /A ) pG jinl
aR Da r G s - WD Nnl
44
US D D cd
/ jEh o
/A S_a e m
: / / /A /. : /A /
/ ( .
43. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
2 B
2 B /1
L P e CloudWatch
LP
5 R X
T gai CloudWatch
LP
R X
Hd A
EB _
CloudWatch
LP
5 4 R X and
chC e CloudWatch
LP
5 R X and
ELB 4XX / 5xx e
(ELB -> L )
CloudWatch
LP
5
5
R X and
4XX / 5xx
e ( -> ELB)
CloudWatch
LP
5
5
R X and
• 5 5
44. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
/
/ 2
C P CloudWatch
L
AB H F
N P CloudWatch
L
AB H F
CloudWatch
L
AB H F
B CloudWatch
L
AB H F and
•
ALB CloudWatch L
https://docs.aws.amazon.com/ja_jp/elasticloadbalancing/latest/application/load-balancer-cloudwatch-metrics.html
NLB CloudWatch L
https://docs.aws.amazon.com/ja_jp/elasticloadbalancing/latest/network/load-balancer-cloudwatch-metrics.html
45. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
u I l CloudWatch T
P
. R
PT g
t e
CloudWatch T
P
. . R and
T
EmEPa W T r CloudWatch T
P
. . R and
T
p mEPa W T r CloudWatch T
P
. . . R and
T
CloudWatch T Pd
https://docs.aws.amazon.com/ja_jp/streams/latest/dev/monitoring-with-cloudwatch.html
cM xi A G a T P nso
(hv PT a)
46. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
V CloudWatch
NA
S CloudWatch
NA
and
M
CloudWatch
NA
and
M
CloudWatch NA O
https://docs.aws.amazon.com/ja_jp/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-
cloudwatch-metrics.html
47. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
48. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• P P S A) D C S
• T P ET E A
S 2) 2 ( W c
e i
P
P S
P P S CloudWatchVP
(EC2 NetworkIn, NetworkOut)
P
S
VPCD C P S VPC S
VPC P S
49. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
eac C cjf a hg h
A / 1 0
eac C o f C
cjf a Er
ikb lCg eac C
lCl /113 3 3 31 R
cjf a
hg hW odW SR st
t Ag a o A
u
,( ,3 A) / 1 /1 3
/13 eac C o f C
cjf a P
p L / 1
p pD
VPC
50. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
52
• s ri G
• o VW c X le s
NVW s ri y y G
• krsVPT W
CD A 8: )8 8 9 A ) G
• s ri 7 ( 4 U
n T krsX G
• ta 3A8 B C 8C D8E PT
( 2a d sL X PT
NV G
• I S krsX G
•
• o L CA w c UI NV
• krs CA w c UI
W
VPC
Auto Scaling group
Instance Instance
krs )
krs
y y
EC2c
o
7 ( 4 U
n T
51. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
52. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
•
AWS Config / Config
CloudTrail + CloudWatch
AWS Systems Manager
AWS API OS AWS CloudTrail (API)
AWS Systems Manager Session Manager
(OS / )
RDS
Amazon GuardDuty
AWS WAF
53. W ( ) i A 1 1 ( )
AWS Config
IDS/IPS
AWS Config
Amazon GuardDuty
Service Control Policy
AWS CloudTrail AWS Config
AWS CloudTrail + Amazon CloudWatch
root
IP AWS Config Rule Security Group SSH
0.0.0.0/0
000
, 2 A S1 9 T1
o 9 k m
54. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• A
S
•
• i n 0
• i
• f W i
• 3 g
• i
• C N
N W A
https://www.slideshare.net/AmazonWebServicesJapan/20190618-aws-black-
belt-online-seminar-aws-config
o
[AWS Black Belt Online Seminar] AWS Config
55. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
m 2 B
W
A
B
• S
• C
L
• (
• bd E
• ( ) )B B
• a
56. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
( ) 3
• ( I C A
• K M
• S N T I
• C
• W L E
/ li Ccm
• IgA on
• / cm
• )
• 3 gdha
)
57. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• AWS CLI SSM Agent
Windows PowerShell
• SSM Agent→SSM API OK
• OS IAM SSMSessionRunAs
• AWS CLI
IAM Policy
• OS S3 / CloudWatch Logs
• "bash"
• PrivateLink
58. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
o n s tgC r F L CA w Ie Sd I
W
a
WS tv
Amazon GuardDuty I L W tg
o n ED N
VP W s uv
dTED
lrW
i
59. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Amazon
GuardDuty
VPC flow logs
DNS Logs
CloudTrail
HIGH
MEDIUM
LOW
60. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
61. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
•
AWS Budgets
/
Trusted Advosor
EC2
EC2 (RI)
62. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Ø
Ø A E
C E C E
A
Ø h a A
nb
Ø
Ø N A W
Ø
Ø A m o A S
Ø
) (
63. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
/
•
64. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
•
0 1P
•
P
• P 0 U 4
C U
U U %
65. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
RI ( )
https://techlife.cookpad.com/entry/cost-and-ri-optimization-in-cookpad
66. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
67. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• A A A
AWS AWS Personal Health Dashboard
A A A A Personal Health Dashboard
A A
API
68. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
_ _S Gcg EGcg
Ae C _U _ d a
_ G W cg
ih _ _ Ge _
- : : 2 2 .: :. -. / : .: : : . : .-
69. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
- #
2
70. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
ü 3 AWS
ü CloudWatch Event Lambda API rd Party
ü CloudTrail SNSTopic S3
Personal Health Dashboard 3
71. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
72. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
•
( OS )
• / “the right tool for the job”
1 ”One size doesn’t fit all.”
❌ ⭕ /
•
• ”Design For Failure”
(Multi-AZ Auto Scaling Auto Recovery )
•
• (Cloudwatch Agent )
73. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• Mike Julian, Practical Monitoring: Effective Strategies for the Real World,
O'Reilly Media
• [AWS Black Belt Online Seminar] Amazon Cloudwatch
https://www.slideshare.net/AmazonWebServicesJapan/20190326-aws-black-belt-online-
seminar-amazon-cloudwatch
• [AWS Summit Tokyo 2019]
https://aws.amazon.com/jp/summits/tokyo-2019/
• [AWS Summit Tokyo 2018] Application Performance Management (APM)
on AWS
https://d1.awsstatic.com/events/jp/2018/summit/tokyo/aws/64.pdf
74. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
• [AWS Black Belt Online Seminar] AWS Config
https://www.slideshare.net/AmazonWebServicesJapan/20190618-aws-black-belt-online-
seminar-aws-config
75. © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark
Thank you!