SlideShare a Scribd company logo

Making Logs Sexy Again: Can We Finally Lose The Regexes?

•
•
Anton Chuvakin
Anton Chuvakin

Making Logs Sexy Again: Can We Finally Lose The Regexes?, presented at DeepSec 2008 in Vienna, Austria

Technology
1 of 41
Making Logs Sexy Again: Can We Finally Lose The Regexes? Dr. Anton Chuvakin
Agenda ,[object Object],[object Object],[object Object],[object Object],[object Object],C O N F I D E N T I A L
Who is Anton? ,[object Object],[object Object]
First: WTH is Logs? ,[object Object],C O M P A N Y C O N F I D E N T I A L II. A huge messy pile of undocumented, unstructured or poorly structured “ stuff ” that might or might not mean anything – at all?
WTF?!! ,[object Object],[object Object],[object Object],[object Object]
Second: What Do We Do With Logs? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
[object Object],C O M P A N Y C O N F I D E N T I A L
Log Analysis Basics: Summary ,[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Manual ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Filtering ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Summary ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Search ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Log Analysis Basics: Correlation ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Log Management “Grand Challenges?” ,[object Object],[object Object],[object Object],[object Object]
GC1 – Log Chaos ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Example Log Chaos - Login? <122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from ::ffff:192.168.138.35 port 2895 ssh2 <13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Failure Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0     Logon account :  POWERUSER    <57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user:yellowdog] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 28 2006 <18> Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system-warning-00515: Admin User netscreen has logged on via Telnet from 10.14.98.55:39073 (2002-12-17 15:50:53)
GC2 – Secure and Reliable Log Collection ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GC3 - Log Parsing and Regexs ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GC4 – Automated Meaning Extraction ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
GC5 – “Fuzzy” Search ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
[object Object],C O M P A N Y C O N F I D E N T I A L
Handling The Challenges ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Log Mining ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Preliminary DATA Requirements ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
What Do We “Mine” for? ,[object Object],[object Object],[object Object],[object Object]
Simple Example ,[object Object],[object Object],[object Object],[object Object]
Deeper into interesting - I ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Can You Guess What Happened?!
Deeper into interesting - II ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Where Is The DATA? ,[object Object],[object Object],[object Object],[object Object]
“ Search+” ,[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Text Mining Logs? ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Keywords to Meaning? ,[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Log Visualization ,[object Object],[object Object],[object Object],[object Object]
Log Context Enrichment? C O M P A N Y C O N F I D E N T I A L
Finally, Can We Make Better Logs? CEE = Syntax + Vocabulary + Transport + Log Recommendations ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Conclusions and CALL TO ACTION! ,[object Object],[object Object],[object Object],[object Object],[object Object],C O M P A N Y C O N F I D E N T I A L
Thank You! ,[object Object],[object Object],[object Object],[object Object],[object Object]
Backup and Reference Slides C O M P A N Y C O N F I D E N T I A L
“ Top 11 Reasons People Hate Logs” ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]

Recommended

Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Anton Chuvakin
 
Anton's Log Management 'Worst Practices'
Anton's Log Management 'Worst Practices'Anton's Log Management 'Worst Practices'
Anton's Log Management 'Worst Practices'Anton Chuvakin
 
Log Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisLog Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisAnton Chuvakin
 
Application Logging Good Bad Ugly ... Beautiful?
Application Logging Good Bad Ugly ... Beautiful?Application Logging Good Bad Ugly ... Beautiful?
Application Logging Good Bad Ugly ... Beautiful?Anton Chuvakin
 
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...Anton Chuvakin
 
Log Management For e-Discovery, Database Monitoring and Other Unusual Uses
Log Management For e-Discovery, Database Monitoring and Other Unusual UsesLog Management For e-Discovery, Database Monitoring and Other Unusual Uses
Log Management For e-Discovery, Database Monitoring and Other Unusual UsesAnton Chuvakin
 
Choosing Your Log Management Approach: Buy, Build or Outsource
Choosing Your Log Management Approach: Buy, Build or OutsourceChoosing Your Log Management Approach: Buy, Build or Outsource
Choosing Your Log Management Approach: Buy, Build or OutsourceAnton Chuvakin
 
"Grand Challenges" of Log Management
"Grand Challenges" of Log Management"Grand Challenges" of Log Management
"Grand Challenges" of Log ManagementAnton Chuvakin
 

Recommended

Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Six Mistakes of Log Management 2008
Six Mistakes of Log Management 2008Anton Chuvakin
 
Anton's Log Management 'Worst Practices'
Anton's Log Management 'Worst Practices'Anton's Log Management 'Worst Practices'
Anton's Log Management 'Worst Practices'Anton Chuvakin
 
Log Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisLog Mining: Beyond Log Analysis
Log Mining: Beyond Log AnalysisAnton Chuvakin
 
Application Logging Good Bad Ugly ... Beautiful?
Application Logging Good Bad Ugly ... Beautiful?Application Logging Good Bad Ugly ... Beautiful?
Application Logging Good Bad Ugly ... Beautiful?Anton Chuvakin
 
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...
Logs: Can’t Hate Them, Won’t Love Them: Brief Log Management Class by Anton C...Anton Chuvakin
 
Log Management For e-Discovery, Database Monitoring and Other Unusual Uses
Log Management For e-Discovery, Database Monitoring and Other Unusual UsesLog Management For e-Discovery, Database Monitoring and Other Unusual Uses
Log Management For e-Discovery, Database Monitoring and Other Unusual UsesAnton Chuvakin
 
Choosing Your Log Management Approach: Buy, Build or Outsource
Choosing Your Log Management Approach: Buy, Build or OutsourceChoosing Your Log Management Approach: Buy, Build or Outsource
Choosing Your Log Management Approach: Buy, Build or OutsourceAnton Chuvakin
 
"Grand Challenges" of Log Management
"Grand Challenges" of Log Management"Grand Challenges" of Log Management
"Grand Challenges" of Log ManagementAnton Chuvakin
 
Enterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinEnterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinAnton Chuvakin
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Anton Chuvakin
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008guestc0c304
 
Baselining Logs
Baselining LogsBaselining Logs
Baselining LogsAnton Chuvakin
 
Logs = Accountability
Logs = AccountabilityLogs = Accountability
Logs = AccountabilityAnton Chuvakin
 
Pivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsPivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsKiyoto Tamura
 
Power of logs: practices for network security
Power of logs: practices for network securityPower of logs: practices for network security
Power of logs: practices for network securityInformation Technology Society Nepal
 
Log Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinLog Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinAnton Chuvakin
 
Logs vs Insiders
Logs vs InsidersLogs vs Insiders
Logs vs InsidersAnton Chuvakin
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)Brian Brazil
 
Python for Data Logistics
Python for Data LogisticsPython for Data Logistics
Python for Data LogisticsKen Farmer
 
LogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationLogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationAnton Chuvakin
 
CSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinCSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinAnton Chuvakin
 
Logs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMALogs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMAAnton Chuvakin
 
NIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldNIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldAnton Chuvakin
 
Functional and non functional application logging
Functional and non functional application loggingFunctional and non functional application logging
Functional and non functional application loggingSander De Vos
 
Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Andrew McNicol
 
What is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupWhat is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupMaarten Balliauw
 
The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!Michele Leroux Bustamante
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Demi Ben-Ari
 
Future of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsFuture of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsAnton Chuvakin
 
SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?Anton Chuvakin
 

More Related Content

Similar to Making Logs Sexy Again: Can We Finally Lose The Regexes?

Enterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinEnterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinAnton Chuvakin
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Anton Chuvakin
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008guestc0c304
 
Pivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsPivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsKiyoto Tamura
 
Log Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinLog Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinAnton Chuvakin
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)Brian Brazil
 
Python for Data Logistics
Python for Data LogisticsPython for Data Logistics
Python for Data LogisticsKen Farmer
 
LogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationLogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationAnton Chuvakin
 
CSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinCSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinAnton Chuvakin
 
Logs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMALogs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMAAnton Chuvakin
 
NIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldNIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldAnton Chuvakin
 
Functional and non functional application logging
Functional and non functional application loggingFunctional and non functional application logging
Functional and non functional application loggingSander De Vos
 
Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Andrew McNicol
 
What is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupWhat is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupMaarten Balliauw
 
The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!Michele Leroux Bustamante
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Demi Ben-Ari
 

Similar to Making Logs Sexy Again: Can We Finally Lose The Regexes? (20)

Enterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton ChuvakinEnterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
Enterprise Logging and Log Management: Hot Topics by Dr. Anton Chuvakin
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
 
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
Logs for Incident Response and Forensics: Key Issues for GOVCERT.NL 2008
 
Baselining Logs
Baselining LogsBaselining Logs
Baselining Logs
 
Logs = Accountability
Logs = AccountabilityLogs = Accountability
Logs = Accountability
 
Pivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logsPivotal Open Source: Using Fluentd to gain insights into your logs
Pivotal Open Source: Using Fluentd to gain insights into your logs
 
Power of logs: practices for network security
Power of logs: practices for network securityPower of logs: practices for network security
Power of logs: practices for network security
 
Log Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton ChuvakinLog Standards & Future Trends by Dr. Anton Chuvakin
Log Standards & Future Trends by Dr. Anton Chuvakin
 
Logs vs Insiders
Logs vs InsidersLogs vs Insiders
Logs vs Insiders
 
What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)What does "monitoring" mean? (FOSDEM 2017)
What does "monitoring" mean? (FOSDEM 2017)
 
Python for Data Logistics
Python for Data LogisticsPython for Data Logistics
Python for Data Logistics
 
LogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log StandardizationLogChaos: Challenges and Opportunities of Security Log Standardization
LogChaos: Challenges and Opportunities of Security Log Standardization
 
CSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton ChuvakinCSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
CSI NetSec 2007 Six MIstakes of Log Management by Anton Chuvakin
 
Logs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMALogs for Information Assurance and Forensics @ USMA
Logs for Information Assurance and Forensics @ USMA
 
NIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real WorldNIST 800-92 Log Management Guide in the Real World
NIST 800-92 Log Management Guide in the Real World
 
Functional and non functional application logging
Functional and non functional application loggingFunctional and non functional application logging
Functional and non functional application logging
 
Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016Beyond Automated Testing - RVAsec 2016
Beyond Automated Testing - RVAsec 2016
 
What is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User GroupWhat is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
What is going on? Application Diagnostics on Azure - Copenhagen .NET User Group
 
The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!The Ultimate Logging Architecture - You KNOW you want it!
The Ultimate Logging Architecture - You KNOW you want it!
 
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
 

More from Anton Chuvakin

Future of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsFuture of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsAnton Chuvakin
 
SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?Anton Chuvakin
 
Meet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton ChuvakinMeet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton ChuvakinAnton Chuvakin
 
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...Anton Chuvakin
 
SOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton ChuvakinSOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton ChuvakinAnton Chuvakin
 
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 BoothHey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 BoothAnton Chuvakin
 
20 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 202220 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 2022Anton Chuvakin
 
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton ChuvakinAnton Chuvakin
 
SOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC TrendsSOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC TrendsAnton Chuvakin
 
SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC Anton Chuvakin
 
Modern SOC Trends 2020
Modern SOC Trends 2020Modern SOC Trends 2020
Modern SOC Trends 2020Anton Chuvakin
 
Anton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in BriefAnton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in BriefAnton Chuvakin
 
Five SIEM Futures (2012)
Five SIEM Futures (2012)Five SIEM Futures (2012)
Five SIEM Futures (2012)Anton Chuvakin
 
RSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics PresentationRSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics PresentationAnton Chuvakin
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinAnton Chuvakin
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinAnton Chuvakin
 
Practical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton ChuvakinPractical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton ChuvakinAnton Chuvakin
 

More from Anton Chuvakin (20)

Future of SOC: More Security, Less Operations
Future of SOC: More Security, Less OperationsFuture of SOC: More Security, Less Operations
Future of SOC: More Security, Less Operations
 
SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?SOC Meets Cloud: What Breaks, What Changes, What to Do?
SOC Meets Cloud: What Breaks, What Changes, What to Do?
 
Meet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton ChuvakinMeet the Ghost of SecOps Future by Anton Chuvakin
Meet the Ghost of SecOps Future by Anton Chuvakin
 
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
SANS Webinar: The Future of Log Centralization for SIEMs and DFIR – Is the En...
 
SOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton ChuvakinSOC Lessons from DevOps and SRE by Anton Chuvakin
SOC Lessons from DevOps and SRE by Anton Chuvakin
 
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 BoothHey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
Hey SOC, Look LEFT! by Anton Chuvakin RSA 2023 Booth
 
20 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 202220 Years of SIEM - SANS Webinar 2022
20 Years of SIEM - SANS Webinar 2022
 
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
10X SOC - SANS Blue Summit Keynote 2021 - Anton Chuvakin
 
SOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC TrendsSOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
SOCstock 2020 Groovy SOC Tunes aka Modern SOC Trends
 
SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC SOCstock 2021 The Cloud-native SOC
SOCstock 2021 The Cloud-native SOC
 
Modern SOC Trends 2020
Modern SOC Trends 2020Modern SOC Trends 2020
Modern SOC Trends 2020
 
Anton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in BriefAnton's 2020 SIEM Best and Worst Practices - in Brief
Anton's 2020 SIEM Best and Worst Practices - in Brief
 
Generic siem how_2017
Generic siem how_2017Generic siem how_2017
Generic siem how_2017
 
Tips on SIEM Ops 2015
Tips on SIEM Ops 2015Tips on SIEM Ops 2015
Tips on SIEM Ops 2015
 
Five SIEM Futures (2012)
Five SIEM Futures (2012)Five SIEM Futures (2012)
Five SIEM Futures (2012)
 
RSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics PresentationRSA 2016 Security Analytics Presentation
RSA 2016 Security Analytics Presentation
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
 
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton ChuvakinFive Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
Five Best and Five Worst Practices for SIEM by Dr. Anton Chuvakin
 
Practical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton ChuvakinPractical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
Practical Strategies to Compliance and Security with SIEM by Dr. Anton Chuvakin
 
SIEM Primer:
SIEM Primer:SIEM Primer:
SIEM Primer:
 

Recently uploaded

Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsJoaquim Jorge
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Drew Madelung
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdflior mazor
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilV3cube
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityPrincipled Technologies
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024Rafal Los
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...DianaGray10
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Miguel Araújo
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...apidays
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonAnna Loughnan Colquhoun
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?Igalia
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoffsammart93
 

Recently uploaded (20)

Artificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and MythsArtificial Intelligence: Facts and Myths
Artificial Intelligence: Facts and Myths
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
 
Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...Driving Behavioral Change for Information Management through Data-Driven Gree...
Driving Behavioral Change for Information Management through Data-Driven Gree...
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUnderstanding Discord NSFW Servers A Guide for Responsible Users.pdf
Understanding Discord NSFW Servers A Guide for Responsible Users.pdf
 
Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?A Year of the Servo Reboot: Where Are We Now?
A Year of the Servo Reboot: Where Are We Now?
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 

Making Logs Sexy Again: Can We Finally Lose The Regexes?

  • 1. Making Logs Sexy Again: Can We Finally Lose The Regexes? Dr. Anton Chuvakin
  • 2.
  • 3.
  • 4.
  • 5.
  • 6.
  • 7.
  • 8.
  • 9.
  • 10.
  • 11.
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17. Example Log Chaos - Login? <122> Mar 4 09:23:15 localhost sshd[27577]: Accepted password for kyle from ::ffff:192.168.138.35 port 2895 ssh2 <13> Fri Mar 17 14:29:38 2006 680 Security SYSTEM User Failure Audit ENTERPRISE Account Logon Logon attempt by: MICROSOFT_AUTHENTICATION_PACKAGE_V1_0     Logon account :  POWERUSER    <57> Dec 25 00:04:32:%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user:yellowdog] [Source:10.4.2.11] [localport:23] at 20:55:40 UTC Fri Feb 28 2006 <18> Dec 17 15:45:57 10.14.93.7 ns5xp: NetScreen device_id=ns5xp system-warning-00515: Admin User netscreen has logged on via Telnet from 10.14.98.55:39073 (2002-12-17 15:50:53)
  • 18.
  • 19.
  • 20.
  • 21.
  • 22.
  • 23.
  • 24.
  • 25.
  • 26.
  • 27.
  • 28.
  • 29. Can You Guess What Happened?!
  • 30.
  • 31.
  • 32.
  • 33.
  • 34.
  • 35.
  • 36. Log Context Enrichment? C O M P A N Y C O N F I D E N T I A L
  • 37.
  • 38.
  • 39.
  • 40. Backup and Reference Slides C O M P A N Y C O N F I D E N T I A L
  • 41.

Editor's Notes

  1. “ Will be high-level, but the subject is definitely pretty DEEP… so DeepSec” Making Logs Sexy Again: Can We Finally Lose The Regexes?   As I was talking to one conference organizer a few years ago and he complained that there hasn’t been anything new in the field of automated log analysis for many years and that we have not progressed much since regular expressions (regexes) were used for extracting bits of useful information from logs. This talk is intended to change that! It will go through a set of approaches to “log problem”: some completely new (text mining of logs), some borrowed from other fields (Bayesian methods for log analysis) and show how people can use them to solve practical problems using these new methods. Only real-world-tested analysis methods and only those that can be used by the audience will be shown (and, of course, only the “0technically cool” and novel ones!) Example questions that will be answered: are logs data or text? Which data mining methods works really well on logs (which is still simple enough to be used by the non-Ph.Ds)? Can you make sense of all the keywords and key phrases in logs? Does Bayes works for logs? Can you predict when the system will crash based on logs?