This document discusses the importance of logging and log management. It explains that logs are critical for security, regulatory compliance, and IT operations. However, many organizations currently take an inefficient, ad hoc approach to log management that involves each team managing their own logs. The document argues that a centralized log management system is needed to efficiently collect, store, and analyze logs across an organization. Such a system improves security incident response and compliance efforts.

1. Logging: WTH?
Dr. Anton Chuvakin
WRITTEN: 2008
DISCLAIMER:
Security is a rapidly changing field of human endeavor. Threats we face literally
change every day; moreover, many security professionals consider the rate of
change to be accelerating. On top of that, to be able to stay in touch with such
ever-changing reality, one has to evolve with the space as well. Thus, even
though I hope that this document will be useful for to my readers, please keep in
mind that is was possibly written years ago. Also, keep in mind that some of the
URL might have gone 404, please Google around.
The what?
System logs, audit trails, network logs, IDS and IPS alerts – and all other data
that information systems are spewing at us at an ever-increasing rate. Soon
more log sources will be added to today’s mix of servers, applications, firewalls
and security appliances – all the way to mobile devices that log and then possibly
internet-connected appliance. A typical large company already has gigabytes of
logs produced each week.
Why is it important?
System logs are the source of three types of insight:
1. Security – logs are used to detect attacks; they are also indispensable for
incident investigations and forensics
2. Regulatory compliance – many regulations in USA, UK and other
countries mandate the collection and review of logs; auditors may ask
organizations for a proof of log collection and review
3. IT operational efficiency – sysadmins reach for logs when troubleshooting
the system problems; logs are also used to measure network and
application performance.
How does it work?
Many piece of IT infrastructure product heaps of logs by default (e.g. Unix
servers create syslog records and Windows systems log to Even Log); others
need to be configured to produce larger volumes of logs to make them useful for
security, compliance and operations. However, some technologies, such as
databases, come with almost no logging configured by default and those who

2. deploy them need to change configurations and enable logging settings so that
logs are created and can be used for the above purposes.
Who needs it?
IT administrators, IT managers, security analysts, incident responders, IT
directors and even CIOs and compliance officers will either look at logs or at
reports based on them. Given the diverse range of uses, it is not surprising that
both system admins and CIO will use logs to accomplish their goals.
Here are the recent poll results on log use that shows it:
How do they get it?
The best way to take control of the logs is to deploy a log management system.
More advanced organization use a log data warehouse approach. Such system
centralizes the collection, storage and access of log data across the enterprise,
freeing organizations from a device-by-device approach to log management,
which is inefficient and heavily relies on manual tasks
.
Providing a central repository for all log data, log data warehous enables you to
easily query and report on this data with unparalleled speed and efficiently
manage the massive amounts of log data generated through network devices,
security gear, operating systems, network servers, databases, and more. It also
goes beyond simple storage, allowing users to discover and act on relationships
between data from these heterogeneous data sources.
Additional: What’s wrong with what they have now?
Today many organizations employ an ad hoc approach to logs: a situation when
you have a security team "owning" network IDS logs, network team having
firewall and router logs (as well as all SNMP traps) and a system admins
possessing the logs from servers and desktop is not only sad,
counterproductive, inefficient and wasteful, but also dangerous.
Where does such approach to logs (when they are divided by both technical and
political chasms!) breaks down most painfully? In case of an incident response,

3. of course. This is where instead of one query across all logs and all log sources
(or whatever needed subset of logs or log sources), people end up with having
run around, beg for logs, connect, wait, wait, download logs, dig in many places
at once – overall, waste time and energy in massive amounts. All of the above
instead of connecting to a log management system and running a few reports,
drilldowns and searches across the relevant logs!
Where else does it break down? Compliance, of course! Most regulations and
mandates don't call out logs by the type of the log source, but apply to all logs
across. Thus one system to verify the compliance status is much more
productive compared to digging in many systems.
ABOUT THE AUTHOR:
This is an updated author bio, added to the paper at the time of reposting in
2009.
Dr. Anton Chuvakin (http://www.chuvakin.org) is a recognized security expert in
the field of log management and PCI DSS compliance. He is an author of books
"Security Warrior" and "PCI Compliance" and a contributor to "Know Your Enemy
II", "Information Security Management Handbook" and others. Anton has
published dozens of papers on log management, correlation, data analysis, PCI
DSS, security management (see list www.info-secure.org) . His blog
http://www.securitywarrior.org is one of the most popular in the industry.
In addition, Anton teaches classes and presents at many security conferences
across the world; he recently addressed audiences in United States, UK,
Singapore, Spain, Russia and other countries. He works on emerging security
standards and serves on the advisory boards of several security start-ups.
Currently, Anton is developing his security consulting practice
www.securitywarriorconsulting.com/ , focusing on logging and PCI DSS
compliance for security vendors and Fortune 500 organizations. Dr. Anton
Chuvakin was formerly a Director of PCI Compliance Solutions at Qualys.
Previously, Anton worked at LogLogic as a Chief Logging Evangelist, tasked with
educating the world about the importance of logging for security, compliance and
operations. Before LogLogic, Anton was employed by a security vendor in a
strategic product management role. Anton earned his Ph.D. degree from Stony
Brook University.