SlideShare a Scribd company logo
1 of 38
Monitoring with syslog-ng, Riemann and Kibana
@algernoone @balabit
syslog-ng
syslog-ng
● Open source event processor and swiss army knife
syslog-ng
● Open source event processor and swiss army knife
● Developed since 1998, LGPL + GPL
syslog-ng
● Open source event processor and swiss army knife
● Developed since 1998, LGPL + GPL
– (Commercial offering since 2007)
syslog-ng
● Open source event processor and swiss army knife
● Developed since 1998, LGPL + GPL
– (Commercial offering since 2007)
● Collects, parses, filters, transforms, transfers events
syslog-ng
● Open source event processor and swiss army knife
● Developed since 1998, LGPL + GPL
– (Commercial offering since 2007)
● Collects, parses, filters, transforms, transfers events
● Wide variety of plugins
syslog-ng
● Open source event processor and swiss army knife
● Developed since 1998, LGPL + GPL
– (Commercial offering since 2007)
● Collects, parses, filters, transforms, transfers events
● Wide variety of plugins
● A sizable, helpful and very inclusive community
Riemann
Riemann
● Riemann monitors distributed systems
Riemann
● Riemann monitors distributed systems
● Event aggregator with a powerful stream processing
language
Riemann
● Riemann monitors distributed systems
● Event aggregator with a powerful stream processing
language
● Provides a low-latency, transient shared state
Kibana
Kibana
● Visualize logs and time-stamped data
Kibana
● Visualize logs and time-stamped data
● Powerful search syntax
Kibana
● Visualize logs and time-stamped data
● Powerful search syntax
● Flexible, powerful, yet intuitive interface
Kibana
Monitoring
Monitoring
What
Monitoring
● System state
What
Monitoring
● System state
● Application state
What
Monitoring
● System state
● Application state
● Exceptions
What
Monitoring
● System state
● Application state
● Exceptions
● Activity
What
Monitoring
Tools
Monitoring
● Nagios
● Collectd
● Munin
Tools
Monitoring
● Nagios
● Collectd
● Munin
● Riemann
Tools
Logging
Logging
Logging
● Persisting application state
Logging
● Persisting application state
● Format is usually application specific
Logging
● Persisting application state
● Format is usually application specific
● Structured vs unstructured
Logging
● Persisting application state
● Format is usually application specific
● Structured vs unstructured
● Great source for monitoring too!
Monitoring + Logging
Monitoring + Logging
● What do we already have?
Monitoring + Logging
● What do we already have?
● What can we add?
Monitoring + Logging
● What do we already have?
● What can we add?
● How will we benefit from all of this?
Q & A
Thanks!
● http://www.syslog-ng.org/
● https://github.com/balabit/syslog-ng-incubator
● https://talien.blogs.balabit.com/
● https://algernon.blogs.balabit.com/

More Related Content

What's hot

Realtime statistics using Java, Kafka and Graphite
Realtime statistics using Java, Kafka and GraphiteRealtime statistics using Java, Kafka and Graphite
Realtime statistics using Java, Kafka and Graphite
Hung Nguyen
 

What's hot (20)

AWS Mobile - Mohit Ajwani
AWS Mobile - Mohit AjwaniAWS Mobile - Mohit Ajwani
AWS Mobile - Mohit Ajwani
 
.NET Fest 2018. Андрей Тарарака. Как порулить грузовиком в Австралии, находяс...
.NET Fest 2018. Андрей Тарарака. Как порулить грузовиком в Австралии, находяс....NET Fest 2018. Андрей Тарарака. Как порулить грузовиком в Австралии, находяс...
.NET Fest 2018. Андрей Тарарака. Как порулить грузовиком в Австралии, находяс...
 
Observability at Expedia
Observability at ExpediaObservability at Expedia
Observability at Expedia
 
Pracital application logging and monitoring
Pracital application logging and monitoringPracital application logging and monitoring
Pracital application logging and monitoring
 
SAP Stammtisch Frankfurt 2017-10-11 - abapGit introduction
SAP Stammtisch Frankfurt 2017-10-11 - abapGit introductionSAP Stammtisch Frankfurt 2017-10-11 - abapGit introduction
SAP Stammtisch Frankfurt 2017-10-11 - abapGit introduction
 
Grails audit logging
Grails audit loggingGrails audit logging
Grails audit logging
 
Getting started with Yap core
Getting started with Yap coreGetting started with Yap core
Getting started with Yap core
 
OSMC 2015 | Current State of Icinga by Icinga Team
OSMC 2015 | Current State of Icinga by Icinga TeamOSMC 2015 | Current State of Icinga by Icinga Team
OSMC 2015 | Current State of Icinga by Icinga Team
 
Why we need open systems, and how to create them in the era of the Cloud (Ops...
Why we need open systems, and how to create them in the era of the Cloud (Ops...Why we need open systems, and how to create them in the era of the Cloud (Ops...
Why we need open systems, and how to create them in the era of the Cloud (Ops...
 
Introduction to Git
Introduction to GitIntroduction to Git
Introduction to Git
 
DevSecOps - Security in DevOps
DevSecOps - Security in DevOpsDevSecOps - Security in DevOps
DevSecOps - Security in DevOps
 
giip engine - open source revolution!
giip engine - open source revolution!giip engine - open source revolution!
giip engine - open source revolution!
 
Realtime statistics using Java, Kafka and Graphite
Realtime statistics using Java, Kafka and GraphiteRealtime statistics using Java, Kafka and Graphite
Realtime statistics using Java, Kafka and Graphite
 
Icinga Camp Kuala Lumpur 2015 Opening By Eric Lippmann
Icinga Camp Kuala Lumpur 2015 Opening By Eric LippmannIcinga Camp Kuala Lumpur 2015 Opening By Eric Lippmann
Icinga Camp Kuala Lumpur 2015 Opening By Eric Lippmann
 
Atlassian Latest Updates
Atlassian Latest UpdatesAtlassian Latest Updates
Atlassian Latest Updates
 
Tenants: A Look Behind the Scenes
Tenants: A Look Behind the ScenesTenants: A Look Behind the Scenes
Tenants: A Look Behind the Scenes
 
Spark industrialisation
Spark industrialisationSpark industrialisation
Spark industrialisation
 
Azure Functions Deep Dive
Azure Functions Deep DiveAzure Functions Deep Dive
Azure Functions Deep Dive
 
Piwik presentation 2011
Piwik presentation 2011Piwik presentation 2011
Piwik presentation 2011
 
Foreign Objects in RIPE IRR
Foreign Objects in RIPE IRRForeign Objects in RIPE IRR
Foreign Objects in RIPE IRR
 

Viewers also liked (6)

Simple Log Analysis and Trending
Simple Log Analysis and TrendingSimple Log Analysis and Trending
Simple Log Analysis and Trending
 
The Near Future of CSS
The Near Future of CSSThe Near Future of CSS
The Near Future of CSS
 
The Presentation Come-Back Kid
The Presentation Come-Back KidThe Presentation Come-Back Kid
The Presentation Come-Back Kid
 
Classroom Management Tips for Kids and Adolescents
Classroom Management Tips for Kids and AdolescentsClassroom Management Tips for Kids and Adolescents
Classroom Management Tips for Kids and Adolescents
 
The Buyer's Journey - by Chris Lema
The Buyer's Journey - by Chris LemaThe Buyer's Journey - by Chris Lema
The Buyer's Journey - by Chris Lema
 
Hype vs. Reality: The AI Explainer
Hype vs. Reality: The AI ExplainerHype vs. Reality: The AI Explainer
Hype vs. Reality: The AI Explainer
 

Similar to OSDC: Gergely Nagy: Monitoring with syslog-ng, Riemann and Kibana

Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControlWebinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Severalnines
 

Similar to OSDC: Gergely Nagy: Monitoring with syslog-ng, Riemann and Kibana (20)

OSMC 2023 | Large-scale logging made easy by Alexandr Valialkin
OSMC 2023 | Large-scale logging made easy by Alexandr ValialkinOSMC 2023 | Large-scale logging made easy by Alexandr Valialkin
OSMC 2023 | Large-scale logging made easy by Alexandr Valialkin
 
OSMC 2012 | Zabbix 2.0: Even Better by Rihards Olups
OSMC 2012 | Zabbix 2.0: Even Better by Rihards OlupsOSMC 2012 | Zabbix 2.0: Even Better by Rihards Olups
OSMC 2012 | Zabbix 2.0: Even Better by Rihards Olups
 
Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...
Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...
Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...
 
Open Source Monitoring in 2019
Open Source Monitoring in 2019 Open Source Monitoring in 2019
Open Source Monitoring in 2019
 
Dev.bg DevOps March 2024 Monitoring & Logging
Dev.bg DevOps March 2024 Monitoring & LoggingDev.bg DevOps March 2024 Monitoring & Logging
Dev.bg DevOps March 2024 Monitoring & Logging
 
Real-time GraphQL in Angular app
Real-time GraphQL in Angular appReal-time GraphQL in Angular app
Real-time GraphQL in Angular app
 
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org -  Dev...Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org -  Dev...
Real-Time Metrics and Distributed Monitoring - Jeff Pierce, Change.org - Dev...
 
Apache Flink 101 - the rise of stream processing and beyond
Apache Flink 101 - the rise of stream processing and beyondApache Flink 101 - the rise of stream processing and beyond
Apache Flink 101 - the rise of stream processing and beyond
 
Interactive Data Analysis in Spark Streaming
Interactive Data Analysis in Spark StreamingInteractive Data Analysis in Spark Streaming
Interactive Data Analysis in Spark Streaming
 
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControlWebinar slides: How to Automate & Manage PostgreSQL with ClusterControl
Webinar slides: How to Automate & Manage PostgreSQL with ClusterControl
 
State management with GraphQL [Angular Minsk, Online, 13.06.20]
State management with GraphQL [Angular Minsk, Online, 13.06.20]State management with GraphQL [Angular Minsk, Online, 13.06.20]
State management with GraphQL [Angular Minsk, Online, 13.06.20]
 
Lesson_08_Continuous_Monitoring.pdf
Lesson_08_Continuous_Monitoring.pdfLesson_08_Continuous_Monitoring.pdf
Lesson_08_Continuous_Monitoring.pdf
 
SIP2SIP - Direcciones SIP gratuitas para las masas
SIP2SIP - Direcciones SIP gratuitas para las masasSIP2SIP - Direcciones SIP gratuitas para las masas
SIP2SIP - Direcciones SIP gratuitas para las masas
 
CD using ArgoCD(KnolX).pdf
CD using ArgoCD(KnolX).pdfCD using ArgoCD(KnolX).pdf
CD using ArgoCD(KnolX).pdf
 
Data Science in the Cloud @StitchFix
Data Science in the Cloud @StitchFixData Science in the Cloud @StitchFix
Data Science in the Cloud @StitchFix
 
Intro to GitOps with Weave GitOps, Flagger and Linkerd
Intro to GitOps with Weave GitOps, Flagger and LinkerdIntro to GitOps with Weave GitOps, Flagger and Linkerd
Intro to GitOps with Weave GitOps, Flagger and Linkerd
 
Using Event Streams in Serverless Applications
Using Event Streams in Serverless ApplicationsUsing Event Streams in Serverless Applications
Using Event Streams in Serverless Applications
 
The Netflix Way to deal with Big Data Problems
The Netflix Way to deal with Big Data ProblemsThe Netflix Way to deal with Big Data Problems
The Netflix Way to deal with Big Data Problems
 
Beaming flink to the cloud @ netflix ff 2016-monal-daxini
Beaming flink to the cloud @ netflix   ff 2016-monal-daxiniBeaming flink to the cloud @ netflix   ff 2016-monal-daxini
Beaming flink to the cloud @ netflix ff 2016-monal-daxini
 
Monal Daxini - Beaming Flink to the Cloud @ Netflix
Monal Daxini - Beaming Flink to the Cloud @ NetflixMonal Daxini - Beaming Flink to the Cloud @ Netflix
Monal Daxini - Beaming Flink to the Cloud @ Netflix
 

Recently uploaded

The title is not connected to what is inside
The title is not connected to what is insideThe title is not connected to what is inside
The title is not connected to what is inside
shinachiaurasa2
 
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
masabamasaba
 
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
masabamasaba
 
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
chiefasafspells
 
Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...
Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...
Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...
Medical / Health Care (+971588192166) Mifepristone and Misoprostol tablets 200mg
 
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
masabamasaba
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
Health
 

Recently uploaded (20)

The title is not connected to what is inside
The title is not connected to what is insideThe title is not connected to what is inside
The title is not connected to what is inside
 
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Atlanta Psychic Readings, Attraction spells,Brin...
 
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
WSO2Con2024 - From Code To Cloud: Fast Track Your Cloud Native Journey with C...
 
%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand%in Midrand+277-882-255-28 abortion pills for sale in midrand
%in Midrand+277-882-255-28 abortion pills for sale in midrand
 
Microsoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdfMicrosoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdf
 
WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?WSO2CON 2024 - Does Open Source Still Matter?
WSO2CON 2024 - Does Open Source Still Matter?
 
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
%in Stilfontein+277-882-255-28 abortion pills for sale in Stilfontein
 
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
%+27788225528 love spells in Boston Psychic Readings, Attraction spells,Bring...
 
Announcing Codolex 2.0 from GDK Software
Announcing Codolex 2.0 from GDK SoftwareAnnouncing Codolex 2.0 from GDK Software
Announcing Codolex 2.0 from GDK Software
 
AI & Machine Learning Presentation Template
AI & Machine Learning Presentation TemplateAI & Machine Learning Presentation Template
AI & Machine Learning Presentation Template
 
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
Love witchcraft +27768521739 Binding love spell in Sandy Springs, GA |psychic...
 
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital TransformationWSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
WSO2Con2024 - WSO2's IAM Vision: Identity-Led Digital Transformation
 
Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...
Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...
Abortion Pills In Pretoria ](+27832195400*)[ 🏥 Women's Abortion Clinic In Pre...
 
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
%+27788225528 love spells in Toronto Psychic Readings, Attraction spells,Brin...
 
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
 
%in kempton park+277-882-255-28 abortion pills for sale in kempton park
%in kempton park+277-882-255-28 abortion pills for sale in kempton park %in kempton park+277-882-255-28 abortion pills for sale in kempton park
%in kempton park+277-882-255-28 abortion pills for sale in kempton park
 
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdfPayment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
Payment Gateway Testing Simplified_ A Step-by-Step Guide for Beginners.pdf
 
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
Devoxx UK 2024 - Going serverless with Quarkus, GraalVM native images and AWS...
 
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
WSO2CON 2024 - Building the API First Enterprise – Running an API Program, fr...
 
VTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learnVTU technical seminar 8Th Sem on Scikit-learn
VTU technical seminar 8Th Sem on Scikit-learn
 

OSDC: Gergely Nagy: Monitoring with syslog-ng, Riemann and Kibana