This document discusses log mining and structured logging. It provides examples of well-structured log entries that contain useful information like timestamps, log levels, and error messages. It also discusses how to use Logstash and Elasticsearch to collect, parse, and analyze logs at scale. Proper log filtering and parsing can provide insights into system performance and help debug issues.
3. 什么是LOG?
3
> tail -f /usr/local/log
INFO [2014-11-13 12:23:36,173]
com.thoughtworks.forcetalk.resources.ContactResource:
Updated Contact
{"FirstName":"Alper","LastName":"Mermer","Employee_ID__c
":"16906","Email":"amermer@thoughtworks.com","Grade__c":
"Senior Consultant”}
ERROR [2014-11-13 11:45:33,892]
com.thoughtworks.forcetalk.validators.ForceQueryResultsVali
dator: Unable to retrieve Project for Opportunity with id:
0065000000TE2evAAD
INFO [2014-11-13 12:23:36,505]
com.thoughtworks.tetalk.resources.UserResource: Contact
Update Response SObjectResponse{successful=true, id='null',
errorMessage='null', errorField='null', errorCode='null'}
INFO 2014-11-13 12:23:36,173
com.thoughtworks.forcetalk.resources.ContactResource
ERROR
4. 什么是好LOG?
4
▫ http://juliusdavies.ca/logging/llclc.html
Best Logs:
▫Tell you exactly what happened: when, where, and how.
▫Suitable for manual, semi-automated, or automated analysis.
▫Can be analysed without having the application that produced them at ha
▫Don't slow the system down.
▫Can be proven reliable (if used as evidence).
Avoid Logs:
▫Missing necessary information.
▫Unsuitable for grep because of redundant information.
▫Information split across more than one line (bad for grep).
▫Error reported to user, but not logged.
▫Never include any sensitive data.(for Security !).
12. SAVE OUR LIFE
12
WHAT TIME IS IT?
1304060505
29/Apr/2011:07:05:26 +0000
Fri, 21 Nov 1997 09:55:06 -0600
Oct 11 20:21:47
020805 13:51:24 110429.071055,118
@4000000037c219bf2ef02e94
DATE FILTER FIXES THIS BULLSHIT
filter {
date {
# Turn 020805 13:51:24
# Into 2002-08-05T13:51:24.000Z
mysqltimestamp => "YYMMdd HH:mm:ss"
}
}