k8s monitoring with elk

1. k8s monitoring & alert
with elasticsearch
Kubernetes Korea Group Meetup
2018.11.23
윤종원 (sabper@gmail.com)

2. ...
• SI
•
• (?)
( ...)
• ( ??!!)
• infra ... ,,
• k8s ...

4. ...
• k8s ELK stack
• k8s
• k8s cpu, memory, disk resource
• alarm …
• ELK k8s log / monitoring

5. prometheus !!!
• ...
• node-exporter, kube-state-metrics, …
• !! - ?
• ..
• ... ( ... )
• [OpenInfra Days Korea 2018] OpenInfra monitoring with Prometheus

6. ELK
• ELK stack ..
• ES ELK stack
• - ...
• ...!!!
• ( )

7. for k8s pod

8. k8s application

9. ?
• k8s ,
• kubectl logs -f pod-name
• -
• ,
• ...
• ,,, (reponse time per sec, request per sec …)
-> -> ( ) -> ` ` !!

10. k8s filebeat to kafka
pod container log
system log
ingress-nginx log
with add_kubernetes_metadata
container_name, conatiner_label, node_name …

11. filebeat k8s deploy !!
https://github.com/elastic/examples/tree/master/MonitoringKubernetes
filebeat - k8s

12. ,,,
... ...

13. filebeat multiline
java exception 1 row multi row -> filebeat multi row
2018-08-30 09:44:22.847 [pool-2-thread-1] INFO com.barogo.dispatch.util.Util:46 APIPoint getCall0Riders 0 141
Aug 31, 2018 5:52:48 PM com.amazonaws.http.AmazonHttpClient executeHelper
INFO: Unable to execute HTTP request: Connection reset
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
ConfigMap - filebeat input
- input type docker - docker log
- multiline pattern
- java multiline example
- Test multiline pattern

14. filebeat drop_event
k8s pod healthcheck access log
2018-11-21T00:08:41.294Z - info: GET 200 /healthz HTTP/1.1 kube-probe/1.10 [time 0.318 ms]
"Request-Body" :{}
"Response-Body" : ""
2018-11-21T00:08:41.485Z - info: GET 200 /healthz HTTP/1.1 kube-probe/1.10 [time 0.317 ms]
"Request-Body" :{}
"Response-Body" : ""
2018-11-21T00:08:44.485Z - info: GET 200 /healthz HTTP/1.1 kube-probe/1.10 [time 0.338 ms]
"Request-Body" :{}
"Response-Body" : ""
2018-11-21T00:08:47.485Z - info: GET 200 /healthz HTTP/1.1 kube-probe/1.10 [time 0.371 ms]
"Request-Body" :{}
"Response-Body" : ""
ConfigMap - processors
- drop_event
- condition
-
- Test multiline pattern

15. Why? Logstash & Kafka
• metadata
• , id
• logstash order_no, uid filed
•
• filebeat ,
• filebeat output logstash , filebeat -> kafka -> logstash -> elasticsearch cloud
kibana

16. logstash - json parsing
flask_response {"timestamp": 1533106987681, "gps": [{"lat": 37.5208423071, "lon": 127.0370946609}, {"lat": 37.5168702957094, "lon": 127.038314337406}], "route": [{"lat":
37.5208423071, "lon": 127.0370946609}, {"lat": 37.52101092506974, "lon": 127.037041062907}, {"lat": 37.5211672, "lon": 127.0375327}, {"lat": 37.5204709, "lon": 127.0376081},
{"lat": 37.5197674, "lon": 127.0377019}, {"lat": 37.5194147, "lon": 127.0377445}, {"lat": 37.5192907, "lon": 127.0377578}, {"lat": 37.5194287, "lon": 127.0383417}, {"lat":
37.5188587, "lon": 127.0385572}, {"lat": 37.5183706, "lon": 127.0387465}, {"lat": 37.5178821, "lon": 127.0389347}, {"lat": 37.5173377, "lon": 127.0391453}, {"lat": 37.5172205,
"lon": 127.0387591}, {"lat": 37.51708223867784, "lon": 127.03825608200496}, {"lat": 37.5168702957094, "lon": 127.038314337406}], "second": 135, "distance": 718,
"call_status": "rest", "order_no": 1, "uid": "uid", "platform": "PostmanRuntime", "endpoint": "route", "duplicates": "remove", "nearest_node_within": 150,
"smoothing_node_within": 2.5, "st_ed": 0.1368551254272461, "st_sp1": -2.5033950805664062e-05, "sp2_sp3": -0.026725292205810547, "sp3_sp4": -1.430511474609375e-06,
"sp4_sp5": -0.0015358924865722656, "sp5_sp6": -0.019712209701538086, "sp6_sp7": -0.08764910697937012}
{
"timestamp":1533106987681,
"gps":[
{
"lat":37.5208423071,
"lon":127.0370946609
},
{
"lat":37.5168702957094,
"lon":127.038314337406
}
],
"route":[
{
"lat":37.5208423071,
"lon":127.0370946609
},
{
"lat":37.52101092506974,
"lon":127.037041062907
},
{
"lat":37.5168702957094,
"lon":127.038314337406
}
],
"second":135,
"distance":718,
"call_status":"rest",
"order_no":1,
"uid":"uid",
"platform":"PostmanRuntime",
"endpoint":"route",
"duplicates":"remove",
"nearest_node_within":150,
"sp5_sp6":-0.019712209701538086,
"sp6_sp7":-0.08764910697937012
}
json prefix flask_response
json parsing dissect message
`{` multiline pattern

17. logstash - custom field
, uid custom filed
2018-11-20T09:14:30.207Z - info: [1147] [WjZtOadDkhVsl0FpFnFaFJEMLAI3] [newcall-new-single] Dispatch Result From dispatch-Response SQS
{ order_no: '1147',
users:
[ { uid: 'WjZtOadDkhVsl0FpFnFaFJEMLAI3',
order_routes:
[ { order_no: '1147',
order_type: 'pickup',
order_status: 'pickup',
receiption_dt: '2018-11-20T09:14:28.448Z',
grok filter (grok test)
grok filter - order_no, uid, command
message2

18. Log Data - kibana
• kubernetes metadata
• cloud metadata
• custom data

19. filebeat nginx-ingress
• pod nginx-ingress
• response time per sec, request per sec
• nginx-ingress acceess log !?!?!
ConfigMap filebeat autodiscover
namespace ingress-nginx pod
nginx module

20. logstash - nginx
• filebeat nginx module field filed
• service container_name, response time
• metadata ingress
• pod, namespace,
• nginx
• service_name, namespace, response / request time

21. logstash nginx
121.135.235.252 - - [22/Nov/2018:08:15:28 +0000] 1542874528.663 "POST /api/v1/location/group HTTP/1.1" 200 3823
"https://dev-admin.mvmt.delivery/rider/control" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36" 930 0.348 [staging-dev-node-admin-api-svc-80]
100.118.136.176:3500 44425 0.348 200
1. grok pattern nginx
2. [staging-dev-node-admin-api-svc-80] namespace, service_name
3. service_name namespace filed filed

23. Resource Monitoring
for k8s

24. resource
• k8s
•
•
• k8s
- ,,
• k8s
• ,
!

25. metricbeat k8s to ES cloud
module - kubernetes metricset : fetch from kubelet, kube-stat-metrics

26. metricbeat - k8s
metricbeat k8s deploy !! - !!
https://github.com/elastic/examples/tree/master/MonitoringKubernetes

27. Monitoring k8s state_pod
metricset : state_deployment

28. Monitoring k8s node
metricset : node

29. Monitoring k8s container
metricset : container

30. Alert
for k8s

31. watcher - xpack
• alarm
• ES cloud xpack - watcher
• xpack -> elastalert
• watcher
• : trigger
• es query: input
• : condition
• slack, email noti : action

32. ?
error message log -
metricbeat metricset : event
type: Warning -

33. • app / k8s resource , alarm ,
• -
• aggregation ?
• ?
• logstash, filebeat, metribeat ?
• k8s ? - aws dns ? reigon ??
•
• ( ) ...
• infra