4. FileBeat config file. YML standard
FileBeat config file. YML
4
vim /etc/filebeat/filebeat.yml
-
paths:
- ${API_APP_LOG_PATH}/file.log
encoding: plain
input_type: log
fields:
level: apico_backend
review: 1
fields_under_root: false
ignore_older: 0
document_type: api_backend_requests
scan_frequency: 2s
multiline:
pattern: ^[[:digit:]]{4}
negate: true
match: after
max_lines: 500
timeout: 2s
tail_files: false
### Logstash as output
logstash:
# The Logstash hosts
hosts: ["cls.apico.net:9999"]
# Number of workers per Logstash host.
worker: 2
# Optional TLS. By default is off.
tls: certificate_authorities:["/etc/ssl/certs/rootCA.crt"]
insecure: false
5. Using Environment Variblaes in Configuration
Logstash config source
Link environment-variables @see /etc/sysconfig/filebeat
Filebeat_using_env_vars
Environment Logstash config result
input {
tcp {
port => "${TCP_PORT}"
}
}
export TCP_PORT=12345 input {
tcp {
port => 12345
}
}
input {
tcp {
port => "${TCP_PORT}"
}
}
No TCP_PORT defined Raise a logstash configuration error
input {
tcp {
port => "${TCP_PORT:54321}"
}
}
No TCP_PORT defined input {
tcp {
port => 54321
}
}
filter {
mutate {
add_field => {
"my_path" => "${HOME}/file.log"
}
}
export HOME="/path" filter {
mutate {
add_field => {
"my_path" => "/path/file.log"
}
}
5
6. SSL Certificate Authority
Certificate Authority — в 5 OpenSSL команд
https://habrahabr.ru/post/192446/
Using TLS between Beats and Logstash
https://gist.github.com/andrewkroh/fdc7e5f3f0f0ed63a11c
Validate crt key:
[root@MSG-predprod lostash_pp]# pwd
/etc/ssl/certs/lostash_pp
[root@MSG-predprod lostash_pp]# curl -v --cacert rootCA.crt https://cls.apico.net:9999
* Rebuilt URL to: https://cls.apico.net:9999/
* Connected to cls.apico.net (172.31.13.49) port 9999 (#0)
* CAfile: rootCA.crt
* Server certificate:
* subject: E=kh.airat14@gmail.com,CN=cls.apico.net,OU=ITY,O=Default Company Ltd,L=Moscow,ST=Moscow,C=MT
* start date: Apr 19 11:59:50 2016 GMT
* expire date: Sep 05 11:59:50 2043 GMT
* common name: cls.apico.net
> GET / HTTP/1.1
> Host: cls.apico.net:9999
> Accept: */*
* Empty reply from server
[root@MSG-predprod lostash_pp]#
6
Security:SSL Certificate AUthority.
7. Beats Platform: Collect, Parse, and Ship
Don't stop the Beat
Packetbeat - the best way to understand the value of a network packet analytics system like
Packetbeat is to try it on your own traffic.
Topbeat - helps you monitor the availability of your website or service by providing system-wide
and per-process statistics along with a disk usage overview.
Filebeat - read data from log files
7
9. Get data from Rabbit.
Get data from RabbitMq (Logstash plugin)
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-rabbitmq.html
9
Get data from RabbitMq.
10. RabbitMq configuration.
Output RabbitMq configuration.
10
Hostname IP cls.apico.net
127.0.0.1
Exchange logstash-exchange
Exchange_type direct
Routing Key logstash-routing-key
User user
11. LogStash Filter config file.
LogStash Filters. Filter for “log_status”.
WIKI: Добавление параметра message_sid в файла логирования для компонентов SmsController RadiusController.
Filter messages from log file by STATUS:
11
log file message
status
error warning info
log_status error warning info
12. LogStash Filter config file.
LogStash Filters. Filter for “account_sid”.
Filter API request by User Auth Status/Role:
12
user auth
status/role
User
unauthorized
User::
ROLE_SYSTEM
User::ROLE_ACCOUNT
account_sid unknown system acc01fe181e-741b-3693-88bb-3847abfb6e86
sac01fe181e-741b-3693-88bb-3847abfb6e55
22. Inspiration manual.
1.Собираем и анализируем логи с помощью Lumberjack+Logstash+Elasticsearch+RabbitMQ
(The Guardian )
2. Wiki MTT.Innovations. АPICO.CentralLogStorage.
3. Import to CvS Excel . Read Comments
https://habrahabr.ru/post/236341/
4. Logstash alert.
The throttle filter is for throttling the number of events received.
https://www.elastic.co/guide/en/logstash/current/plugins-filters-throttle.html
Plugins outputs email
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-email.html
22
Inspiration manual.
23. End. No just the beginning.
Elastic BackUp: Snapshot and restore.
Backing-up-your-cluster
File rotation linux BackUp & Restore(5).
Elasticsearch Monitoring Java
23
Just the beginning.