filebeat, logstash

1. Integration
between Logstash and Filebeat
charsyam@naver.com

2. Integration between Logstash and Filebeat
Filebeat Logstash
Filebeat sends logs to logstash.

3. Common Config : Filebeat
filebeat.prospectors:
- type: log
enabled: true
paths:
- /data/logs/reallog/2018-12-27.log
output.logstash:
hosts: ["target.aggserver.com:5044"]

4. Common Config : Logstash
input {
beats {
port => 5044
}
}
output {
file {
path => "/data/logstash/2018-12-27.log"
codec => line { format => "%{message}" }
}
}

5. Case #1 : Simple, one file to one file
Just use common config

6. Case #1 : Simple, one file to one file
But we don’t need this case

7. Case #2 : Simple, multiple files to one file
filebeat.prospectors:
- type: log
enabled: true
paths:
- /data/logs/reallog/*.log
Just use *.

8. Case #3 : Advance, multiple files to multiple
files : Just move content by each file
filter {
grok {
match => {"source" => "data/logs/%{DATA:logdate}.log"}
}
}
output {
file {
path => "/data/logstash/%{logdate}.log"
codec => line { format => "%{message}" }
}
}
Filebeat sends original filename with source field

9. Case #4 : Advance, multiple files to multiple
files : with log timestamp
filter {
grok {
patterns_dir => ["/usr/local/logstash-5.4.1/patterns"]
match => { "message" => "^%{TIMESTAMP_ISO8601:timestamp}" }
}
date {
match => ["timestamp", "yyyy-MM-dd"]
}
}
output {
file {
path => "/data/logstash/%{+YYYY-MM-dd}.log"
codec => line { format => "%{message}" }
}
}
Filtering timestamp and using it as filename.

10. Case #4 : Advance, multiple files to multiple
files : with log timestamp
Logstash Parsing timestamp as UTC, so
If your log format is like below and your timezone
is UTC -8(PST),
2018-12-26T23:00:00-08:00, it will be handled by
2018-12-27 not 2018-12-26, because logstash uses
UTC as timestamp.

11. Case #4 : Advance, multiple files to multiple
files : with log timestamp
How to fix?

12. Case #4 : Advance, multiple files to multiple
files : with log timestamp
filter {
……
date {
match => ["timestamp", "yyyy-MM-dd'T'HH:mm:ss-08:00"]
Timezone => "UTC"
}
}
Parsing timezone part as string, and set other parts
as UTC