SlideShare una empresa de Scribd logo

ELK Stack

•
•
Eberhard Wolff
Eberhard Wolff

Short presentation about the ELK stack (Elasticsearch, Logstash, Kibana) running on top of Docker / Vagrant.

Tecnología
1 de 19
Descargar para leer sin conexión
ELK
Eberhard Wolff - @ewolff Docker •  No true virtualization •  Linux Containers (lxc) •  i.e. shared kernel •  i.e. separate file systems
Eberhard Wolff - @ewolff Docker File Systems •  Read only base images •  +read/write image •  Can be stacked
Eberhard Wolff - @ewolff Docker •  Linux only •  So: Run it in VM •  Controlled by Vagrant
Docker Demo
Eberhard Wolff - @ewolff Communication Between Docker Container •  Via ports •  Via data volumes
Docker is a Component System
Eberhard Wolff - @ewolff Example •  Log file analysis •  ELK Stack •  Elasticsearch: Store •  Logstash: Parse •  Kibana: Web based GUI •  Install three servers •  Automate installation ... L
Eberhard Wolff - @ewolff Dockerized ELK Stack •  1 Vagrantfile (25 lines) •  6 Dockerfiles (total 28 lines) •  1 HTML file (11 lines) •  1 Config (Logstash) (29 lines) •  Total: 93 lines, 9 files •  Including one RegExp L
Eberhard Wolff - @ewolff Vagrant VM Container user-registration 8080 8081 Port 8081 Container elasticsearch Container logstash 9200 9200 Port 9200 9200 link Container kibana Port 8080 8080 8080 Browser Volume /log
Demo ELK Stack
Eberhard Wolff - @ewolff date time level 1988-10-12 19:42:07.350 INFO 683 [http-nio-8080-exec-6] --- c.e.u.service.RegistrationService : email=eberhard.wolff@gmail.com geloesc pid thread class logmessage
Eberhard Wolff - @ewolff Scaling Elasticsearch •  More servers •  Shards •  Easy to set up •  Index per day or week or month •  Smaller dataset
Eberhard Wolff - @ewolff Ideas about Logstash •  No parsing •  Send JSON
Eberhard Wolff - @ewolff Break Logstash apart •  Logstash does parsing and communication •  Not too sensible in a distributed sytem •  Separate process per responsibility
Eberhard Wolff - @ewolff Shipper •  Ship data from logging system •  Logstash •  Beaver https://github.com/josegonzalez/ beaver •  Woodchuck https://github.com/danryan/ woodchuck
Eberhard Wolff - @ewolff Broker •  Buffer data for later processing •  Redis •  In-Memory NoSQL database •  Small •  Fast •  ...and then Parse (Logstash)
Eberhard Wolff - @ewolff Links •  https://github.com/ewolff/user- registration
Eberhard Wolff - @ewolff Thank You!! eberhard.wolff@gmail.com @ewolff

Recomendados

What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...
What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...
What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...Edureka!
 
The Elastic ELK Stack
The Elastic ELK StackThe Elastic ELK Stack
The Elastic ELK Stackenterprisesearchmeetup
 
ELK Stack
ELK StackELK Stack
ELK StackPhuc Nguyen
 
Log analysis using elk
Log analysis using elkLog analysis using elk
Log analysis using elkRushika Shah
 
Elastic stack Presentation
Elastic stack PresentationElastic stack Presentation
Elastic stack PresentationAmr Alaa Yassen
 
Elastic Stack Introduction
Elastic Stack IntroductionElastic Stack Introduction
Elastic Stack IntroductionVikram Shinde
 
Elk - An introduction
Elk - An introductionElk - An introduction
Elk - An introductionHossein Shemshadi
 
Log analysis with the elk stack
Log analysis with the elk stackLog analysis with the elk stack
Log analysis with the elk stackVikrant Chauhan
 

Recomendados

What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...
What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...
What Is ELK Stack | ELK Tutorial For Beginners | Elasticsearch Kibana | ELK S...Edureka!
 
The Elastic ELK Stack
The Elastic ELK StackThe Elastic ELK Stack
The Elastic ELK Stackenterprisesearchmeetup
 
ELK Stack
ELK StackELK Stack
ELK StackPhuc Nguyen
 
Log analysis using elk
Log analysis using elkLog analysis using elk
Log analysis using elkRushika Shah
 
Elastic stack Presentation
Elastic stack PresentationElastic stack Presentation
Elastic stack PresentationAmr Alaa Yassen
 
Elastic Stack Introduction
Elastic Stack IntroductionElastic Stack Introduction
Elastic Stack IntroductionVikram Shinde
 
Elk - An introduction
Elk - An introductionElk - An introduction
Elk - An introductionHossein Shemshadi
 
Log analysis with the elk stack
Log analysis with the elk stackLog analysis with the elk stack
Log analysis with the elk stackVikrant Chauhan
 
Log analysis with elastic stack
Log analysis with elastic stackLog analysis with elastic stack
Log analysis with elastic stackBangladesh Network Operators Group
 
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and LogstashKeeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and LogstashAmazon Web Services
 
Centralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackCentralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackRich Lee
 
ELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementEl Mahdi Benzekri
 
Elasticsearch for beginners
Elasticsearch for beginnersElasticsearch for beginners
Elasticsearch for beginnersNeil Baker
 
Log analytics with ELK stack
Log analytics with ELK stackLog analytics with ELK stack
Log analytics with ELK stackAWS User Group Bengaluru
 
Log management with ELK
Log management with ELKLog management with ELK
Log management with ELKGeert Pante
 
Elk
Elk Elk
Elk Caleb Wang
 
Elastic - ELK, Logstash & Kibana
Elastic - ELK, Logstash & KibanaElastic - ELK, Logstash & Kibana
Elastic - ELK, Logstash & KibanaSpringPeople
 
ELK stack introduction
ELK stack introduction ELK stack introduction
ELK stack introduction abenyeung1
 
The Elastic Stack as a SIEM
The Elastic Stack as a SIEMThe Elastic Stack as a SIEM
The Elastic Stack as a SIEMJohn Hubbard
 
Introduction to Kibana
Introduction to KibanaIntroduction to Kibana
Introduction to KibanaVineet .
 
Centralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackRohit Sharma
 
Kibana overview
Kibana overviewKibana overview
Kibana overviewRinat Tainov
 
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...ForgeRock
 
Introduction to ELK
Introduction to ELKIntroduction to ELK
Introduction to ELKHarshakumar Ummerpillai
 
Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...
Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...
Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...Edureka!
 
Introduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of LuceneIntroduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of LuceneRahul Jain
 
My first 90 days with ClickHouse.pdf
My first 90 days with ClickHouse.pdfMy first 90 days with ClickHouse.pdf
My first 90 days with ClickHouse.pdfAlkin Tezuysal
 
ELK in Security Analytics
ELK in Security Analytics ELK in Security Analytics
ELK in Security Analytics nullowaspmumbai
 
Micro Services - Small is Beautiful
Micro Services - Small is BeautifulMicro Services - Small is Beautiful
Micro Services - Small is BeautifulEberhard Wolff
 
Java Architectures - a New Hope
Java Architectures - a New HopeJava Architectures - a New Hope
Java Architectures - a New HopeEberhard Wolff
 

Más contenido relacionado

La actualidad más candente

Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and LogstashKeeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and LogstashAmazon Web Services
 
Centralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackCentralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackRich Lee
 
ELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementEl Mahdi Benzekri
 
Elasticsearch for beginners
Elasticsearch for beginnersElasticsearch for beginners
Elasticsearch for beginnersNeil Baker
 
Log management with ELK
Log management with ELKLog management with ELK
Log management with ELKGeert Pante
 
Elastic - ELK, Logstash & Kibana
Elastic - ELK, Logstash & KibanaElastic - ELK, Logstash & Kibana
Elastic - ELK, Logstash & KibanaSpringPeople
 
ELK stack introduction
ELK stack introduction ELK stack introduction
ELK stack introduction abenyeung1
 
The Elastic Stack as a SIEM
The Elastic Stack as a SIEMThe Elastic Stack as a SIEM
The Elastic Stack as a SIEMJohn Hubbard
 
Introduction to Kibana
Introduction to KibanaIntroduction to Kibana
Introduction to KibanaVineet .
 
Centralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackRohit Sharma
 
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...ForgeRock
 
Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...
Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...
Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...Edureka!
 
Introduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of LuceneIntroduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of LuceneRahul Jain
 
My first 90 days with ClickHouse.pdf
My first 90 days with ClickHouse.pdfMy first 90 days with ClickHouse.pdf
My first 90 days with ClickHouse.pdfAlkin Tezuysal
 
ELK in Security Analytics
ELK in Security Analytics ELK in Security Analytics
ELK in Security Analytics nullowaspmumbai
 

La actualidad más candente (20)

Log analysis with elastic stack
Log analysis with elastic stackLog analysis with elastic stack
Log analysis with elastic stack
 
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and LogstashKeeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
Keeping Up with the ELK Stack: Elasticsearch, Kibana, Beats, and Logstash
 
Centralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stackCentralized log-management-with-elastic-stack
Centralized log-management-with-elastic-stack
 
ELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log ManagementELK Elasticsearch Logstash and Kibana Stack for Log Management
ELK Elasticsearch Logstash and Kibana Stack for Log Management
 
Elasticsearch for beginners
Elasticsearch for beginnersElasticsearch for beginners
Elasticsearch for beginners
 
Log analytics with ELK stack
Log analytics with ELK stackLog analytics with ELK stack
Log analytics with ELK stack
 
Log management with ELK
Log management with ELKLog management with ELK
Log management with ELK
 
Elk
Elk Elk
Elk
 
Elastic - ELK, Logstash & Kibana
Elastic - ELK, Logstash & KibanaElastic - ELK, Logstash & Kibana
Elastic - ELK, Logstash & Kibana
 
ELK stack introduction
ELK stack introduction ELK stack introduction
ELK stack introduction
 
The Elastic Stack as a SIEM
The Elastic Stack as a SIEMThe Elastic Stack as a SIEM
The Elastic Stack as a SIEM
 
Introduction to Kibana
Introduction to KibanaIntroduction to Kibana
Introduction to Kibana
 
Centralized Logging System Using ELK Stack
Centralized Logging System Using ELK StackCentralized Logging System Using ELK Stack
Centralized Logging System Using ELK Stack
 
Kibana overview
Kibana overviewKibana overview
Kibana overview
 
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
 
Introduction to ELK
Introduction to ELKIntroduction to ELK
Introduction to ELK
 
Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...
Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...
Elasticsearch Tutorial | Getting Started with Elasticsearch | ELK Stack Train...
 
Introduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of LuceneIntroduction to Elasticsearch with basics of Lucene
Introduction to Elasticsearch with basics of Lucene
 
My first 90 days with ClickHouse.pdf
My first 90 days with ClickHouse.pdfMy first 90 days with ClickHouse.pdf
My first 90 days with ClickHouse.pdf
 
ELK in Security Analytics
ELK in Security Analytics ELK in Security Analytics
ELK in Security Analytics
 

Similar a ELK Stack

Micro Services - Small is Beautiful
Micro Services - Small is BeautifulMicro Services - Small is Beautiful
Micro Services - Small is BeautifulEberhard Wolff
 
Java Architectures - a New Hope
Java Architectures - a New HopeJava Architectures - a New Hope
Java Architectures - a New HopeEberhard Wolff
 
NoSQL Riak MongoDB Elasticsearch - All The Same?
NoSQL Riak MongoDB Elasticsearch - All The Same?NoSQL Riak MongoDB Elasticsearch - All The Same?
NoSQL Riak MongoDB Elasticsearch - All The Same?Eberhard Wolff
 
Micro Service – The New Architecture Paradigm
Micro Service – The New Architecture ParadigmMicro Service – The New Architecture Paradigm
Micro Service – The New Architecture ParadigmEberhard Wolff
 
High Availability and Scalability: Too Expensive! Architectures for Future E...
High Availability and Scalability: Too Expensive! Architectures for Future E...High Availability and Scalability: Too Expensive! Architectures for Future E...
High Availability and Scalability: Too Expensive! Architectures for Future E...Eberhard Wolff
 
Microservice - All is Small, All is Well?
Microservice - All is Small, All is Well?Microservice - All is Small, All is Well?
Microservice - All is Small, All is Well?Eberhard Wolff
 
DockerCon 2016 - Windows Server and Docker
DockerCon 2016 - Windows Server and DockerDockerCon 2016 - Windows Server and Docker
DockerCon 2016 - Windows Server and DockerTaylor Brown
 
Windows Server and Docker - The Internals Behind Bringing Docker and Containe...
Windows Server and Docker - The Internals Behind Bringing Docker and Containe...Windows Server and Docker - The Internals Behind Bringing Docker and Containe...
Windows Server and Docker - The Internals Behind Bringing Docker and Containe...Docker, Inc.
 
Java Application Servers Are Dead! - Short Version
Java Application Servers Are Dead! - Short VersionJava Application Servers Are Dead! - Short Version
Java Application Servers Are Dead! - Short VersionEberhard Wolff
 
Java Application Servers Are Dead!
Java Application Servers Are Dead!Java Application Servers Are Dead!
Java Application Servers Are Dead!Eberhard Wolff
 
Microservice With Spring Boot and Spring Cloud
Microservice With Spring Boot and Spring CloudMicroservice With Spring Boot and Spring Cloud
Microservice With Spring Boot and Spring CloudEberhard Wolff
 
Openstack Cactus Survey
Openstack Cactus SurveyOpenstack Cactus Survey
Openstack Cactus SurveyPjack Chen
 
Introduction to Docker
Introduction to DockerIntroduction to Docker
Introduction to DockerAlan Forbes
 
Spring Boot
Spring BootSpring Boot
Spring Bootgedoplan
 
Docker Online Meetup #22: Docker Networking
Docker Online Meetup #22: Docker NetworkingDocker Online Meetup #22: Docker Networking
Docker Online Meetup #22: Docker NetworkingDocker, Inc.
 
Cloud Infrastructures Slide Set 7 - Docker - Neo4j | anynines
Cloud Infrastructures Slide Set 7 - Docker - Neo4j | anyninesCloud Infrastructures Slide Set 7 - Docker - Neo4j | anynines
Cloud Infrastructures Slide Set 7 - Docker - Neo4j | anyninesanynines GmbH
 
Magnum Networking Update
Magnum Networking UpdateMagnum Networking Update
Magnum Networking UpdateDaneyon Hansen
 
Getting started with Docker
Getting started with DockerGetting started with Docker
Getting started with DockerRavindu Fernando
 
Amazon Elastic Beanstalk
Amazon Elastic BeanstalkAmazon Elastic Beanstalk
Amazon Elastic BeanstalkEberhard Wolff
 

Similar a ELK Stack (20)

Micro Services - Small is Beautiful
Micro Services - Small is BeautifulMicro Services - Small is Beautiful
Micro Services - Small is Beautiful
 
Java Architectures - a New Hope
Java Architectures - a New HopeJava Architectures - a New Hope
Java Architectures - a New Hope
 
NoSQL Riak MongoDB Elasticsearch - All The Same?
NoSQL Riak MongoDB Elasticsearch - All The Same?NoSQL Riak MongoDB Elasticsearch - All The Same?
NoSQL Riak MongoDB Elasticsearch - All The Same?
 
Micro Service – The New Architecture Paradigm
Micro Service – The New Architecture ParadigmMicro Service – The New Architecture Paradigm
Micro Service – The New Architecture Paradigm
 
High Availability and Scalability: Too Expensive! Architectures for Future E...
High Availability and Scalability: Too Expensive! Architectures for Future E...High Availability and Scalability: Too Expensive! Architectures for Future E...
High Availability and Scalability: Too Expensive! Architectures for Future E...
 
Docker Dojo
Docker DojoDocker Dojo
Docker Dojo
 
Microservice - All is Small, All is Well?
Microservice - All is Small, All is Well?Microservice - All is Small, All is Well?
Microservice - All is Small, All is Well?
 
DockerCon 2016 - Windows Server and Docker
DockerCon 2016 - Windows Server and DockerDockerCon 2016 - Windows Server and Docker
DockerCon 2016 - Windows Server and Docker
 
Windows Server and Docker - The Internals Behind Bringing Docker and Containe...
Windows Server and Docker - The Internals Behind Bringing Docker and Containe...Windows Server and Docker - The Internals Behind Bringing Docker and Containe...
Windows Server and Docker - The Internals Behind Bringing Docker and Containe...
 
Java Application Servers Are Dead! - Short Version
Java Application Servers Are Dead! - Short VersionJava Application Servers Are Dead! - Short Version
Java Application Servers Are Dead! - Short Version
 
Java Application Servers Are Dead!
Java Application Servers Are Dead!Java Application Servers Are Dead!
Java Application Servers Are Dead!
 
Microservice With Spring Boot and Spring Cloud
Microservice With Spring Boot and Spring CloudMicroservice With Spring Boot and Spring Cloud
Microservice With Spring Boot and Spring Cloud
 
Openstack Cactus Survey
Openstack Cactus SurveyOpenstack Cactus Survey
Openstack Cactus Survey
 
Introduction to Docker
Introduction to DockerIntroduction to Docker
Introduction to Docker
 
Spring Boot
Spring BootSpring Boot
Spring Boot
 
Docker Online Meetup #22: Docker Networking
Docker Online Meetup #22: Docker NetworkingDocker Online Meetup #22: Docker Networking
Docker Online Meetup #22: Docker Networking
 
Cloud Infrastructures Slide Set 7 - Docker - Neo4j | anynines
Cloud Infrastructures Slide Set 7 - Docker - Neo4j | anyninesCloud Infrastructures Slide Set 7 - Docker - Neo4j | anynines
Cloud Infrastructures Slide Set 7 - Docker - Neo4j | anynines
 
Magnum Networking Update
Magnum Networking UpdateMagnum Networking Update
Magnum Networking Update
 
Getting started with Docker
Getting started with DockerGetting started with Docker
Getting started with Docker
 
Amazon Elastic Beanstalk
Amazon Elastic BeanstalkAmazon Elastic Beanstalk
Amazon Elastic Beanstalk
 

Más de Eberhard Wolff

Architectures and Alternatives
Architectures and AlternativesArchitectures and Alternatives
Architectures and AlternativesEberhard Wolff
 
The Frontiers of Continuous Delivery
The Frontiers of Continuous DeliveryThe Frontiers of Continuous Delivery
The Frontiers of Continuous DeliveryEberhard Wolff
 
Four Times Microservices - REST, Kubernetes, UI Integration, Async
Four Times Microservices - REST, Kubernetes, UI Integration, AsyncFour Times Microservices - REST, Kubernetes, UI Integration, Async
Four Times Microservices - REST, Kubernetes, UI Integration, AsyncEberhard Wolff
 
Microservices - not just with Java
Microservices - not just with JavaMicroservices - not just with Java
Microservices - not just with JavaEberhard Wolff
 
Deployment - Done Right!
Deployment - Done Right!Deployment - Done Right!
Deployment - Done Right!Eberhard Wolff
 
Data Architecture not Just for Microservices
Data Architecture not Just for MicroservicesData Architecture not Just for Microservices
Data Architecture not Just for MicroservicesEberhard Wolff
 
How to Split Your System into Microservices
How to Split Your System into MicroservicesHow to Split Your System into Microservices
How to Split Your System into MicroservicesEberhard Wolff
 
Microservices and Self-contained System to Scale Agile
Microservices and Self-contained System to Scale AgileMicroservices and Self-contained System to Scale Agile
Microservices and Self-contained System to Scale AgileEberhard Wolff
 
How Small Can Java Microservices Be?
How Small Can Java Microservices Be?How Small Can Java Microservices Be?
How Small Can Java Microservices Be?Eberhard Wolff
 
Data Architecturen Not Just for Microservices
Data Architecturen Not Just for MicroservicesData Architecturen Not Just for Microservices
Data Architecturen Not Just for MicroservicesEberhard Wolff
 
Microservices: Redundancy=Maintainability
Microservices: Redundancy=MaintainabilityMicroservices: Redundancy=Maintainability
Microservices: Redundancy=MaintainabilityEberhard Wolff
 
Self-contained Systems: A Different Approach to Microservices
Self-contained Systems: A Different Approach to MicroservicesSelf-contained Systems: A Different Approach to Microservices
Self-contained Systems: A Different Approach to MicroservicesEberhard Wolff
 
Microservices Technology Stack
Microservices Technology StackMicroservices Technology Stack
Microservices Technology StackEberhard Wolff
 
Software Architecture for Innovation
Software Architecture for InnovationSoftware Architecture for Innovation
Software Architecture for InnovationEberhard Wolff
 
Five (easy?) Steps Towards Continuous Delivery
Five (easy?) Steps Towards Continuous DeliveryFive (easy?) Steps Towards Continuous Delivery
Five (easy?) Steps Towards Continuous DeliveryEberhard Wolff
 
Nanoservices and Microservices with Java
Nanoservices and Microservices with JavaNanoservices and Microservices with Java
Nanoservices and Microservices with JavaEberhard Wolff
 
Microservices: Architecture to Support Agile
Microservices: Architecture to Support AgileMicroservices: Architecture to Support Agile
Microservices: Architecture to Support AgileEberhard Wolff
 
Microservices: Architecture to scale Agile
Microservices: Architecture to scale AgileMicroservices: Architecture to scale Agile
Microservices: Architecture to scale AgileEberhard Wolff
 
Microservices, DevOps, Continuous Delivery – More Than Three Buzzwords
Microservices, DevOps, Continuous Delivery – More Than Three BuzzwordsMicroservices, DevOps, Continuous Delivery – More Than Three Buzzwords
Microservices, DevOps, Continuous Delivery – More Than Three BuzzwordsEberhard Wolff
 

Más de Eberhard Wolff (20)

Architectures and Alternatives
Architectures and AlternativesArchitectures and Alternatives
Architectures and Alternatives
 
Beyond Microservices
Beyond MicroservicesBeyond Microservices
Beyond Microservices
 
The Frontiers of Continuous Delivery
The Frontiers of Continuous DeliveryThe Frontiers of Continuous Delivery
The Frontiers of Continuous Delivery
 
Four Times Microservices - REST, Kubernetes, UI Integration, Async
Four Times Microservices - REST, Kubernetes, UI Integration, AsyncFour Times Microservices - REST, Kubernetes, UI Integration, Async
Four Times Microservices - REST, Kubernetes, UI Integration, Async
 
Microservices - not just with Java
Microservices - not just with JavaMicroservices - not just with Java
Microservices - not just with Java
 
Deployment - Done Right!
Deployment - Done Right!Deployment - Done Right!
Deployment - Done Right!
 
Data Architecture not Just for Microservices
Data Architecture not Just for MicroservicesData Architecture not Just for Microservices
Data Architecture not Just for Microservices
 
How to Split Your System into Microservices
How to Split Your System into MicroservicesHow to Split Your System into Microservices
How to Split Your System into Microservices
 
Microservices and Self-contained System to Scale Agile
Microservices and Self-contained System to Scale AgileMicroservices and Self-contained System to Scale Agile
Microservices and Self-contained System to Scale Agile
 
How Small Can Java Microservices Be?
How Small Can Java Microservices Be?How Small Can Java Microservices Be?
How Small Can Java Microservices Be?
 
Data Architecturen Not Just for Microservices
Data Architecturen Not Just for MicroservicesData Architecturen Not Just for Microservices
Data Architecturen Not Just for Microservices
 
Microservices: Redundancy=Maintainability
Microservices: Redundancy=MaintainabilityMicroservices: Redundancy=Maintainability
Microservices: Redundancy=Maintainability
 
Self-contained Systems: A Different Approach to Microservices
Self-contained Systems: A Different Approach to MicroservicesSelf-contained Systems: A Different Approach to Microservices
Self-contained Systems: A Different Approach to Microservices
 
Microservices Technology Stack
Microservices Technology StackMicroservices Technology Stack
Microservices Technology Stack
 
Software Architecture for Innovation
Software Architecture for InnovationSoftware Architecture for Innovation
Software Architecture for Innovation
 
Five (easy?) Steps Towards Continuous Delivery
Five (easy?) Steps Towards Continuous DeliveryFive (easy?) Steps Towards Continuous Delivery
Five (easy?) Steps Towards Continuous Delivery
 
Nanoservices and Microservices with Java
Nanoservices and Microservices with JavaNanoservices and Microservices with Java
Nanoservices and Microservices with Java
 
Microservices: Architecture to Support Agile
Microservices: Architecture to Support AgileMicroservices: Architecture to Support Agile
Microservices: Architecture to Support Agile
 
Microservices: Architecture to scale Agile
Microservices: Architecture to scale AgileMicroservices: Architecture to scale Agile
Microservices: Architecture to scale Agile
 
Microservices, DevOps, Continuous Delivery – More Than Three Buzzwords
Microservices, DevOps, Continuous Delivery – More Than Three BuzzwordsMicroservices, DevOps, Continuous Delivery – More Than Three Buzzwords
Microservices, DevOps, Continuous Delivery – More Than Three Buzzwords
 

Último

Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Victor Rentea
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024The Digital Insurer
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistandanishmna97
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdfSandro Moreira
 
Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)Zilliz
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...apidays
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProduct Anonymous
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Zilliz
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWERMadyBayot
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...apidays
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyKhushali Kathiriya
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Angeliki Cooney
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...apidays
 
Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxRemote DBA Services
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfOrbitshub
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Jeffrey Haguewood
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...DianaGray10
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Orbitshub
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...apidays
 

Último (20)

Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024FWD Group - Insurer Innovation Award 2024
FWD Group - Insurer Innovation Award 2024
 
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In PakistanCNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
 
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
 
Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)Introduction to Multilingual Retrieval Augmented Generation (RAG)
Introduction to Multilingual Retrieval Augmented Generation (RAG)
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
Apidays New York 2024 - Passkeys: Developing APIs to enable passwordless auth...
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptx
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
Navigating the Deluge_ Dubai Floods and the Resilience of Dubai International...
 
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
Apidays New York 2024 - APIs in 2030: The Risk of Technological Sleepwalk by ...
 

ELK Stack

  • 1. ELK
  • 2. Eberhard Wolff - @ewolff Docker •  No true virtualization •  Linux Containers (lxc) •  i.e. shared kernel •  i.e. separate file systems
  • 3. Eberhard Wolff - @ewolff Docker File Systems •  Read only base images •  +read/write image •  Can be stacked
  • 4. Eberhard Wolff - @ewolff Docker •  Linux only •  So: Run it in VM •  Controlled by Vagrant
  • 6. Eberhard Wolff - @ewolff Communication Between Docker Container •  Via ports •  Via data volumes
  • 8. Eberhard Wolff - @ewolff Example •  Log file analysis •  ELK Stack •  Elasticsearch: Store •  Logstash: Parse •  Kibana: Web based GUI •  Install three servers •  Automate installation ... L
  • 9. Eberhard Wolff - @ewolff Dockerized ELK Stack •  1 Vagrantfile (25 lines) •  6 Dockerfiles (total 28 lines) •  1 HTML file (11 lines) •  1 Config (Logstash) (29 lines) •  Total: 93 lines, 9 files •  Including one RegExp L
  • 10. Eberhard Wolff - @ewolff Vagrant VM Container user-registration 8080 8081 Port 8081 Container elasticsearch Container logstash 9200 9200 Port 9200 9200 link Container kibana Port 8080 8080 8080 Browser Volume /log
  • 12. Eberhard Wolff - @ewolff date time level 1988-10-12 19:42:07.350 INFO 683 [http-nio-8080-exec-6] --- c.e.u.service.RegistrationService : email=eberhard.wolff@gmail.com geloesc pid thread class logmessage
  • 13. Eberhard Wolff - @ewolff Scaling Elasticsearch •  More servers •  Shards •  Easy to set up •  Index per day or week or month •  Smaller dataset
  • 14. Eberhard Wolff - @ewolff Ideas about Logstash •  No parsing •  Send JSON
  • 15. Eberhard Wolff - @ewolff Break Logstash apart •  Logstash does parsing and communication •  Not too sensible in a distributed sytem •  Separate process per responsibility
  • 16. Eberhard Wolff - @ewolff Shipper •  Ship data from logging system •  Logstash •  Beaver https://github.com/josegonzalez/ beaver •  Woodchuck https://github.com/danryan/ woodchuck
  • 17. Eberhard Wolff - @ewolff Broker •  Buffer data for later processing •  Redis •  In-Memory NoSQL database •  Small •  Fast •  ...and then Parse (Logstash)
  • 18. Eberhard Wolff - @ewolff Links •  https://github.com/ewolff/user- registration
  • 19. Eberhard Wolff - @ewolff Thank You!! eberhard.wolff@gmail.com @ewolff