SlideShare a Scribd company logo
1 of 21
Download to read offline
Using Logstash, ElasticSearch and Kibana
Alejandro E Brito Monedero
@ae_bm
2013 / 05 / 23
Business as usual
Using Logstash, elasticsearch & kibana
So many hosts to check
Is there life out there?
http://upload.wikimedia.org/wikipedia/commons/a/aa/ESO-The_Milky_Way_above_La_Silla-phot-27-04-hires.jpg
Time to play whack a log
http://i102.photobucket.com/albums/m109/niceperson907/121331d1253497450-animated-gif-thre.gif
http://brotality.com/wp-content/uploads/2012/12/madness.jpg
I need a new toy
http://www.youtube.com/watch?v=8L6Dpq5kY_A
Video time
Logstash ✔ collects logs
✔ parses logs
✔ stores logs
✔ indexes logs
✔ searches logs
✔ and fixes timestamps
You only need
● JVM
● logstash.jar
$ log_producer | grep ... | sed … | awk … | tee output 
| sort | uniq -c | sort -n
Log source Logstash
Logstash
(optional)
ElasticSearch
DB
statsd
Pipes
Inputs Filters Outputs
File
Redis
Syslog
Lumberjack
Rabbitmq
SQS
…
Alter
Date
Grok
Multiline
Grep
...
AMQP
Cloudwatch
Elasticsearch
Mongodb
Redis
File
...
Plugins
Not here yet? JRuby to the rescue
ElasticSearch
Distributed RESTful search server
● Near real-time search
● RESTful API
● Easy to scale horizontally
● HA
● Full text search
● YAML config file / JSON format!!
● Document oriented JSON
Getting started: Logstash JAR includes it / download and set cluster.name
This is where it will be worth to spend some time tuning
Kibana
✔ Nice UI
✔ Better than the old frontend logstash
included
✔ Ruby / framework Sinatra
Web frontend to search / graph and more
Original plan
Apache
(ligthweight shipper)
Tomcat
(ligthweight shipper)
broker logstash
ElasticSearch
Kibana
After a few workarounds
Apache
(logstash shipper)
Tomcat
(logstash shipper)
Logstash ElasticSearch
Kibana
SSH tunnels
Example config 1/3
Logstash-httpd.conf
input {
file {
type => "httpd"
path => ["/var/log/httpd/*-logstash.log"]
exclude => ["*.gz"]
start_position => "beginning"
format => "json_event"
}
}
output {
tcp {
host => "0.0.0.0"
mode => "server"
port => 1666
}
}
Logstash-server.conf
input {
tcp {
type => "httpd"
format => "json_event"
host => "127.0.0.1"
mode => "client"
port => "1666"
}
tcp {
type => "app"
format => "json_event"
host => "127.0.0.1"
mode => "client"
port => "2666"
}
}
output {
elasticsearch {
cluster => "logstash"
}
}
Example config 2/3
Logstash-tomcat.conf
filter {
# Tomcat
# Remove blank lines
grep {
type => "tomcat"
match => [ "@message", "(.+)" ]
drop => true
add_tag => [ "no_blank_lines" ]
}
# make the multilines be treated like a single line
multiline {
type => "tomcat"
pattern => "^dddd"
negate => true
what => "previous"
}
Example config 3/3
Logstash-tomcat.conf
# mark the exceptions (multiline)
grep {
type => "tomcat"
tags => [ "multiline" ]
match => [ "@message", ".+Exception: .+" ]
drop => false
add_tag => [ "java_exception" ]
}
# get the log level, operation id, module and timestamp as separated fields
grok {
type => "tomcat"
pattern => "%{TIMESTAMP_ISO8601:timestamp} [%{OPERATION_ID:operation_id}]..."
add_tag => [ "groked" ]
}
# fix the timestamp
date {
type => "tomcat"
match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSSZZ" ]
add_tag => [ "timestamp_fix" ]
}
}
I need a new toy
Demo
Some remarks
● Don't forget about security
● The applications should be flexible enough for allowing to
publish their logs using brokers or other methods beyond
files and syslog
● Logging in JSON format is a nice to have
● Share the log visualization
● Use the brokers Luke
● If you develop internalize this
http://www.masterzen.fr/2013/01/13/the-10-
commandments-of-logging/
Extras
● http://logstash.net/
● http://www.logstashbook.com/code/ only $10.09
● https://github.com/logstash/logstash/blob/v1.1.12/patterns/grok-patterns
● http://grokdebug.herokuapp.com/
● http://www.infoq.com/articles/review-the-logstash-book (better diagrams)
●
http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/
● http://kibana.org/
●
https://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/queryparsersyntax.html
● http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/
●
http://blog.lusis.org/blog/2012/01/31/load-balancing-logstash-with-amqp/
Do you want to join the <some fancy words here> team?
I am not hiring, but I can tell you about some places where
it is better to stay away
Have a nice day
All the images, videos and stuff are property of their respective owners, look at the cat
and don't sue me
http://stuffpoint.com/cats/image/41633/cute-cat-picture/

More Related Content

What's hot

RocksDB compaction
RocksDB compactionRocksDB compaction
RocksDB compactionMIJIN AN
 
[2D1]Elasticsearch 성능 최적화
[2D1]Elasticsearch 성능 최적화[2D1]Elasticsearch 성능 최적화
[2D1]Elasticsearch 성능 최적화NAVER D2
 
ElasticSearch Basic Introduction
ElasticSearch Basic IntroductionElasticSearch Basic Introduction
ElasticSearch Basic IntroductionMayur Rathod
 
엘라스틱서치, 로그스태시, 키바나
엘라스틱서치, 로그스태시, 키바나엘라스틱서치, 로그스태시, 키바나
엘라스틱서치, 로그스태시, 키바나종민 김
 
M|18 Deep Dive: InnoDB Transactions and Write Paths
M|18 Deep Dive: InnoDB Transactions and Write PathsM|18 Deep Dive: InnoDB Transactions and Write Paths
M|18 Deep Dive: InnoDB Transactions and Write PathsMariaDB plc
 
Elastic Stack 을 이용한 게임 서비스 통합 로깅 플랫폼 - elastic{on} 2019 Seoul
Elastic Stack 을 이용한 게임 서비스 통합 로깅 플랫폼 - elastic{on} 2019 SeoulElastic Stack 을 이용한 게임 서비스 통합 로깅 플랫폼 - elastic{on} 2019 Seoul
Elastic Stack 을 이용한 게임 서비스 통합 로깅 플랫폼 - elastic{on} 2019 SeoulSeungYong Oh
 
Elasticsearch for beginners
Elasticsearch for beginnersElasticsearch for beginners
Elasticsearch for beginnersNeil Baker
 
MySQL Architecture and Engine
MySQL Architecture and EngineMySQL Architecture and Engine
MySQL Architecture and EngineAbdul Manaf
 
Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0Mydbops
 
Vacuum in PostgreSQL
Vacuum in PostgreSQLVacuum in PostgreSQL
Vacuum in PostgreSQLRafia Sabih
 
M|18 Architectural Overview: MariaDB MaxScale
M|18 Architectural Overview: MariaDB MaxScaleM|18 Architectural Overview: MariaDB MaxScale
M|18 Architectural Overview: MariaDB MaxScaleMariaDB plc
 
Nginx Reverse Proxy with Kafka.pptx
Nginx Reverse Proxy with Kafka.pptxNginx Reverse Proxy with Kafka.pptx
Nginx Reverse Proxy with Kafka.pptxwonyong hwang
 
Stability Patterns for Microservices
Stability Patterns for MicroservicesStability Patterns for Microservices
Stability Patterns for Microservicespflueras
 
ELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementEl Mahdi Benzekri
 
InnoDB MVCC Architecture (by 권건우)
InnoDB MVCC Architecture (by 권건우)InnoDB MVCC Architecture (by 권건우)
InnoDB MVCC Architecture (by 권건우)I Goo Lee.
 
Scaling your logging infrastructure using syslog-ng
Scaling your logging infrastructure using syslog-ngScaling your logging infrastructure using syslog-ng
Scaling your logging infrastructure using syslog-ngPeter Czanik
 
Presto At Treasure Data
Presto At Treasure DataPresto At Treasure Data
Presto At Treasure DataTaro L. Saito
 

What's hot (20)

RocksDB compaction
RocksDB compactionRocksDB compaction
RocksDB compaction
 
[2D1]Elasticsearch 성능 최적화
[2D1]Elasticsearch 성능 최적화[2D1]Elasticsearch 성능 최적화
[2D1]Elasticsearch 성능 최적화
 
ElasticSearch Basic Introduction
ElasticSearch Basic IntroductionElasticSearch Basic Introduction
ElasticSearch Basic Introduction
 
MongodB Internals
MongodB InternalsMongodB Internals
MongodB Internals
 
엘라스틱서치, 로그스태시, 키바나
엘라스틱서치, 로그스태시, 키바나엘라스틱서치, 로그스태시, 키바나
엘라스틱서치, 로그스태시, 키바나
 
Postgresql
PostgresqlPostgresql
Postgresql
 
M|18 Deep Dive: InnoDB Transactions and Write Paths
M|18 Deep Dive: InnoDB Transactions and Write PathsM|18 Deep Dive: InnoDB Transactions and Write Paths
M|18 Deep Dive: InnoDB Transactions and Write Paths
 
Elastic Stack 을 이용한 게임 서비스 통합 로깅 플랫폼 - elastic{on} 2019 Seoul
Elastic Stack 을 이용한 게임 서비스 통합 로깅 플랫폼 - elastic{on} 2019 SeoulElastic Stack 을 이용한 게임 서비스 통합 로깅 플랫폼 - elastic{on} 2019 Seoul
Elastic Stack 을 이용한 게임 서비스 통합 로깅 플랫폼 - elastic{on} 2019 Seoul
 
Elasticsearch
ElasticsearchElasticsearch
Elasticsearch
 
Elasticsearch for beginners
Elasticsearch for beginnersElasticsearch for beginners
Elasticsearch for beginners
 
MySQL Architecture and Engine
MySQL Architecture and EngineMySQL Architecture and Engine
MySQL Architecture and Engine
 
Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0Redo log improvements MYSQL 8.0
Redo log improvements MYSQL 8.0
 
Vacuum in PostgreSQL
Vacuum in PostgreSQLVacuum in PostgreSQL
Vacuum in PostgreSQL
 
M|18 Architectural Overview: MariaDB MaxScale
M|18 Architectural Overview: MariaDB MaxScaleM|18 Architectural Overview: MariaDB MaxScale
M|18 Architectural Overview: MariaDB MaxScale
 
Nginx Reverse Proxy with Kafka.pptx
Nginx Reverse Proxy with Kafka.pptxNginx Reverse Proxy with Kafka.pptx
Nginx Reverse Proxy with Kafka.pptx
 
Stability Patterns for Microservices
Stability Patterns for MicroservicesStability Patterns for Microservices
Stability Patterns for Microservices
 
ELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log Management
 
InnoDB MVCC Architecture (by 권건우)
InnoDB MVCC Architecture (by 권건우)InnoDB MVCC Architecture (by 권건우)
InnoDB MVCC Architecture (by 권건우)
 
Scaling your logging infrastructure using syslog-ng
Scaling your logging infrastructure using syslog-ngScaling your logging infrastructure using syslog-ng
Scaling your logging infrastructure using syslog-ng
 
Presto At Treasure Data
Presto At Treasure DataPresto At Treasure Data
Presto At Treasure Data
 

Similar to Using Logstash, elasticsearch & kibana

(Fios#02) 2. elk 포렌식 분석
(Fios#02) 2. elk 포렌식 분석(Fios#02) 2. elk 포렌식 분석
(Fios#02) 2. elk 포렌식 분석INSIGHT FORENSIC
 
Доклад Антона Поварова "Go in Badoo" с Golang Meetup
Доклад Антона Поварова "Go in Badoo" с Golang MeetupДоклад Антона Поварова "Go in Badoo" с Golang Meetup
Доклад Антона Поварова "Go in Badoo" с Golang MeetupBadoo Development
 
Logstash for SEO: come monitorare i Log del Web Server in realtime
Logstash for SEO: come monitorare i Log del Web Server in realtimeLogstash for SEO: come monitorare i Log del Web Server in realtime
Logstash for SEO: come monitorare i Log del Web Server in realtimeAndrea Cardinale
 
Shrimp: A Rather Practical Example Of Application Development With RESTinio a...
Shrimp: A Rather Practical Example Of Application Development With RESTinio a...Shrimp: A Rather Practical Example Of Application Development With RESTinio a...
Shrimp: A Rather Practical Example Of Application Development With RESTinio a...Yauheni Akhotnikau
 
Machine Learning Game Changer for IT - Maartens Lourens
Machine Learning Game Changer for IT - Maartens LourensMachine Learning Game Changer for IT - Maartens Lourens
Machine Learning Game Changer for IT - Maartens LourensOpenCredo
 
Troubleshooting real production problems
Troubleshooting real production problemsTroubleshooting real production problems
Troubleshooting real production problemsTier1 app
 
Elk devops
Elk devopsElk devops
Elk devopsIdeato
 
Why you should be using structured logs
Why you should be using structured logsWhy you should be using structured logs
Why you should be using structured logsStefan Krawczyk
 
#OOP_D_ITS - 9th - Template
#OOP_D_ITS - 9th - Template#OOP_D_ITS - 9th - Template
#OOP_D_ITS - 9th - TemplateHadziq Fabroyir
 
Troubleshooting performanceavailabilityproblems (1)
Troubleshooting performanceavailabilityproblems (1)Troubleshooting performanceavailabilityproblems (1)
Troubleshooting performanceavailabilityproblems (1)Tier1 app
 
Java/Spring과 Node.js의공존
Java/Spring과 Node.js의공존Java/Spring과 Node.js의공존
Java/Spring과 Node.js의공존동수 장
 
Undelete (and more) rows from the binary log
Undelete (and more) rows from the binary logUndelete (and more) rows from the binary log
Undelete (and more) rows from the binary logFrederic Descamps
 
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak   CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak PROIDEA
 
Fixing Web Data in Production
Fixing Web Data in ProductionFixing Web Data in Production
Fixing Web Data in ProductionAaron Knight
 
Hartwarming lightning talk in winter Sapporo
Hartwarming lightning talk in winter SapporoHartwarming lightning talk in winter Sapporo
Hartwarming lightning talk in winter SapporoJun OHWADA
 
Introduction of RiotJS
Introduction of RiotJSIntroduction of RiotJS
Introduction of RiotJSRyo Iinuma
 

Similar to Using Logstash, elasticsearch & kibana (20)

(Fios#02) 2. elk 포렌식 분석
(Fios#02) 2. elk 포렌식 분석(Fios#02) 2. elk 포렌식 분석
(Fios#02) 2. elk 포렌식 분석
 
Доклад Антона Поварова "Go in Badoo" с Golang Meetup
Доклад Антона Поварова "Go in Badoo" с Golang MeetupДоклад Антона Поварова "Go in Badoo" с Golang Meetup
Доклад Антона Поварова "Go in Badoo" с Golang Meetup
 
Logstash for SEO: come monitorare i Log del Web Server in realtime
Logstash for SEO: come monitorare i Log del Web Server in realtimeLogstash for SEO: come monitorare i Log del Web Server in realtime
Logstash for SEO: come monitorare i Log del Web Server in realtime
 
Shrimp: A Rather Practical Example Of Application Development With RESTinio a...
Shrimp: A Rather Practical Example Of Application Development With RESTinio a...Shrimp: A Rather Practical Example Of Application Development With RESTinio a...
Shrimp: A Rather Practical Example Of Application Development With RESTinio a...
 
Machine Learning Game Changer for IT - Maartens Lourens
Machine Learning Game Changer for IT - Maartens LourensMachine Learning Game Changer for IT - Maartens Lourens
Machine Learning Game Changer for IT - Maartens Lourens
 
Elk scilifelab
Elk scilifelabElk scilifelab
Elk scilifelab
 
Troubleshooting real production problems
Troubleshooting real production problemsTroubleshooting real production problems
Troubleshooting real production problems
 
Elk devops
Elk devopsElk devops
Elk devops
 
Mario
MarioMario
Mario
 
Why you should be using structured logs
Why you should be using structured logsWhy you should be using structured logs
Why you should be using structured logs
 
#OOP_D_ITS - 9th - Template
#OOP_D_ITS - 9th - Template#OOP_D_ITS - 9th - Template
#OOP_D_ITS - 9th - Template
 
Troubleshooting performanceavailabilityproblems (1)
Troubleshooting performanceavailabilityproblems (1)Troubleshooting performanceavailabilityproblems (1)
Troubleshooting performanceavailabilityproblems (1)
 
Java/Spring과 Node.js의공존
Java/Spring과 Node.js의공존Java/Spring과 Node.js의공존
Java/Spring과 Node.js의공존
 
Undelete (and more) rows from the binary log
Undelete (and more) rows from the binary logUndelete (and more) rows from the binary log
Undelete (and more) rows from the binary log
 
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak   CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
CONFidence 2015: DTrace + OSX = Fun - Andrzej Dyjak
 
Fixing Web Data in Production
Fixing Web Data in ProductionFixing Web Data in Production
Fixing Web Data in Production
 
Openstack 簡介
Openstack 簡介Openstack 簡介
Openstack 簡介
 
Kommons
KommonsKommons
Kommons
 
Hartwarming lightning talk in winter Sapporo
Hartwarming lightning talk in winter SapporoHartwarming lightning talk in winter Sapporo
Hartwarming lightning talk in winter Sapporo
 
Introduction of RiotJS
Introduction of RiotJSIntroduction of RiotJS
Introduction of RiotJS
 

More from Alejandro E Brito Monedero (14)

Mad scalability (perfomance debugging)
Mad scalability (perfomance debugging)Mad scalability (perfomance debugging)
Mad scalability (perfomance debugging)
 
Tres historias
Tres historiasTres historias
Tres historias
 
AMQP vs GRAPHITE
AMQP vs GRAPHITEAMQP vs GRAPHITE
AMQP vs GRAPHITE
 
Sysdig
SysdigSysdig
Sysdig
 
Sysdig SRECon 16 Europe
Sysdig SRECon 16 EuropeSysdig SRECon 16 Europe
Sysdig SRECon 16 Europe
 
Funcional para trollear
Funcional para trollearFuncional para trollear
Funcional para trollear
 
Top Bug
Top BugTop Bug
Top Bug
 
Fabric más allá de lo básico
Fabric más allá de lo básicoFabric más allá de lo básico
Fabric más allá de lo básico
 
Experiencias con PostgreSQL en AWS
Experiencias con PostgreSQL en AWSExperiencias con PostgreSQL en AWS
Experiencias con PostgreSQL en AWS
 
Fabric Fast & Furious edition
Fabric Fast & Furious editionFabric Fast & Furious edition
Fabric Fast & Furious edition
 
Así que pusiste MongoDB. Dime ¿cómo lo administras?
Así que pusiste MongoDB. Dime ¿cómo lo administras?Así que pusiste MongoDB. Dime ¿cómo lo administras?
Así que pusiste MongoDB. Dime ¿cómo lo administras?
 
AWS Baby steps circa 2008
AWS Baby steps circa 2008AWS Baby steps circa 2008
AWS Baby steps circa 2008
 
Wireshark tips
Wireshark tipsWireshark tips
Wireshark tips
 
Mi experiencia con Amazon AWS EC2 y S3
Mi experiencia con Amazon AWS EC2 y S3Mi experiencia con Amazon AWS EC2 y S3
Mi experiencia con Amazon AWS EC2 y S3
 

Recently uploaded

NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemAsko Soukka
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?IES VE
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAshyamraj55
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaborationbruanjhuli
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureEric D. Schabell
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 

Recently uploaded (20)

NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Bird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystemBird eye's view on Camunda open source ecosystem
Bird eye's view on Camunda open source ecosystem
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?How Accurate are Carbon Emissions Projections?
How Accurate are Carbon Emissions Projections?
 
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPAAnypoint Code Builder , Google Pub sub connector and MuleSoft RPA
Anypoint Code Builder , Google Pub sub connector and MuleSoft RPA
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online CollaborationCOMPUTER 10: Lesson 7 - File Storage and Online Collaboration
COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
OpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability AdventureOpenShift Commons Paris - Choose Your Own Observability Adventure
OpenShift Commons Paris - Choose Your Own Observability Adventure
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 

Using Logstash, elasticsearch & kibana

  • 1. Using Logstash, ElasticSearch and Kibana Alejandro E Brito Monedero @ae_bm 2013 / 05 / 23
  • 4. So many hosts to check Is there life out there? http://upload.wikimedia.org/wikipedia/commons/a/aa/ESO-The_Milky_Way_above_La_Silla-phot-27-04-hires.jpg
  • 5. Time to play whack a log http://i102.photobucket.com/albums/m109/niceperson907/121331d1253497450-animated-gif-thre.gif
  • 7. I need a new toy http://www.youtube.com/watch?v=8L6Dpq5kY_A Video time
  • 8. Logstash ✔ collects logs ✔ parses logs ✔ stores logs ✔ indexes logs ✔ searches logs ✔ and fixes timestamps You only need ● JVM ● logstash.jar
  • 9. $ log_producer | grep ... | sed … | awk … | tee output | sort | uniq -c | sort -n Log source Logstash Logstash (optional) ElasticSearch DB statsd Pipes
  • 11. ElasticSearch Distributed RESTful search server ● Near real-time search ● RESTful API ● Easy to scale horizontally ● HA ● Full text search ● YAML config file / JSON format!! ● Document oriented JSON Getting started: Logstash JAR includes it / download and set cluster.name This is where it will be worth to spend some time tuning
  • 12. Kibana ✔ Nice UI ✔ Better than the old frontend logstash included ✔ Ruby / framework Sinatra Web frontend to search / graph and more
  • 13. Original plan Apache (ligthweight shipper) Tomcat (ligthweight shipper) broker logstash ElasticSearch Kibana
  • 14. After a few workarounds Apache (logstash shipper) Tomcat (logstash shipper) Logstash ElasticSearch Kibana SSH tunnels
  • 15. Example config 1/3 Logstash-httpd.conf input { file { type => "httpd" path => ["/var/log/httpd/*-logstash.log"] exclude => ["*.gz"] start_position => "beginning" format => "json_event" } } output { tcp { host => "0.0.0.0" mode => "server" port => 1666 } } Logstash-server.conf input { tcp { type => "httpd" format => "json_event" host => "127.0.0.1" mode => "client" port => "1666" } tcp { type => "app" format => "json_event" host => "127.0.0.1" mode => "client" port => "2666" } } output { elasticsearch { cluster => "logstash" } }
  • 16. Example config 2/3 Logstash-tomcat.conf filter { # Tomcat # Remove blank lines grep { type => "tomcat" match => [ "@message", "(.+)" ] drop => true add_tag => [ "no_blank_lines" ] } # make the multilines be treated like a single line multiline { type => "tomcat" pattern => "^dddd" negate => true what => "previous" }
  • 17. Example config 3/3 Logstash-tomcat.conf # mark the exceptions (multiline) grep { type => "tomcat" tags => [ "multiline" ] match => [ "@message", ".+Exception: .+" ] drop => false add_tag => [ "java_exception" ] } # get the log level, operation id, module and timestamp as separated fields grok { type => "tomcat" pattern => "%{TIMESTAMP_ISO8601:timestamp} [%{OPERATION_ID:operation_id}]..." add_tag => [ "groked" ] } # fix the timestamp date { type => "tomcat" match => [ "timestamp", "YYYY-MM-dd HH:mm:ss,SSSZZ" ] add_tag => [ "timestamp_fix" ] } }
  • 18. I need a new toy Demo
  • 19. Some remarks ● Don't forget about security ● The applications should be flexible enough for allowing to publish their logs using brokers or other methods beyond files and syslog ● Logging in JSON format is a nice to have ● Share the log visualization ● Use the brokers Luke ● If you develop internalize this http://www.masterzen.fr/2013/01/13/the-10- commandments-of-logging/
  • 20. Extras ● http://logstash.net/ ● http://www.logstashbook.com/code/ only $10.09 ● https://github.com/logstash/logstash/blob/v1.1.12/patterns/grok-patterns ● http://grokdebug.herokuapp.com/ ● http://www.infoq.com/articles/review-the-logstash-book (better diagrams) ● http://www.elasticsearch.org/tutorials/using-elasticsearch-for-logs/ ● http://kibana.org/ ● https://lucene.apache.org/core/old_versioned_docs/versions/3_5_0/queryparsersyntax.html ● http://www.elasticsearch.org/tutorials/elasticsearch-on-ec2/ ● http://blog.lusis.org/blog/2012/01/31/load-balancing-logstash-with-amqp/
  • 21. Do you want to join the <some fancy words here> team? I am not hiring, but I can tell you about some places where it is better to stay away Have a nice day All the images, videos and stuff are property of their respective owners, look at the cat and don't sue me http://stuffpoint.com/cats/image/41633/cute-cat-picture/