Lighting talk no DevOpsDays de Porto Alegre sobre monitoramento real-time com ELK

1. ELKMétricas e monitoramento "real-time"

2. Sobre nós
Waldemar Neto @waldemarnt
http://walde.co/
Jefferson Stachelski @jeffhsta
https://jeffhsta.github.io/
2

3. Elastic.co
https://www.elastic.
co/
3

4. Elastic.co
Produtos
● Elasticsearch
● Logstash
● Kibana
● Beats
● Shield
● Marvel
● Graph
● Watcher
4
Serviços
● Suporte
● Treinamento
● Cloud
● Consultoria

5. ELK
● Elasticsearch
● Logstash
● Kibana
5

6. Elasticsearch
https://www.elastic.co/products/elasticsearch
6

7. Logstash
https://www.elastic.co/products/logstash
7

8. Kibana
https://www.elastic.co/products/kibana
8

9. PARSEARINDEXARVISUALIZAR
9 ELK Workflow

10. Etapa de log parsing com Logstash
Entrada
● Beats
● Couchdb_changes
● Heroku
● File
● Kafka
● TCP
● sqs
● Redis
● Rabbitmq
● Elasticsearch
Filtro
● Aggregate
● Mutate
● Alter
● CSV
● environment
● TCP
● JSON
● range
● grok
● geoip
Saída
● CSV
● Datadog
● Email
● Elasticsearch
● Exec
● File
● Graphite
● InfluxDB
● MongoDB
● s3
10

11. Etapa de indexar no Elasticsearch
● Normalização de texto
○ Acentos
○ Sinônimos
○ Linguística...
11

12. Etapa de visualizar com Kibana
● Dashboard em tempo real
○ Com varias visualizations
● Vários tipos de gráficos
○ Pie chart
○ Mapas
○ Barras
○ Gráfico de linhas
● Criação de vizualizations dinâmica
○ Suporte a queries do Elasticsearch
○ Suporte a agregações
12

13. Criando Visualization13

14. Monitorando servidor14

15. Monitoramento e métricas de APIs15

16. Métricas de negócio, BigData e anomalias16

17. Shield
Watcher
Elastalert
Alertas e proteção17

18. Na produção
● ELK as a service
○ Logz.io
○ Qbox.io
● Cloud
○ AWS
○ DigitalOcean
○ Elastic
18

19. Valeu!
Perguntas?

20. Referências20
http://walde.co/tag/elasticsearch/
https://github.com/guardian/elk-stack
http://techblog.netflix.com/2016/02/evolution-of-netflix-data-pipeline.html
http://www.front2backdev.com/2016/05/03/geo-temporal-anomaly/
https://dzone.com/articles/which-elk-stack-solution-is-right-for-you?
utm_content=buffer29ebc&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer
http://blog.takipi.com/hosted-elasticsearch-the-future-of-your-elk-stack/
http://everythingshouldbevirtual.com/highly-available-elk-elasticsearch-logstash-kibana-setup