Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

PHP Log Tracking with ELK & Filebeat part#2

517 visualizaciones

Publicado el

PHP Log Tracking with ELK & Filebeat part#2

Publicado en: Tecnología
  • Sé el primero en comentar

  • Sé el primero en recomendar esto

PHP Log Tracking with ELK & Filebeat part#2

  1. 1. PHP Log Tracking with ELK & Filebeat part#2 appkr(김주원) 2018년 7월
  2. 2. Ends and Means 2 Log Tracking ELK & Filebeat
  3. 3. 지난 이야기 ● AWS Cloud Watch에서 꼭 필요할 때 로그 일부를 찾을 수 없었다. ● 회사에서는 MSA 전환을 위해 표준 로깅 플랫폼으로 ELK를 선정했다. ○ Beats 데이터 수집기 ○ Logstash 데이터 수집 및 가공을 위한 파이프 라인 ○ Elasticsearch JSON 문서 기반의 검색 및 분석 엔진 ○ Kibana 데이터 시각화용 UI 도구 ● 더 해야 할 일 ○ ① 개발 환경 구축 ○ ② filebeat.yml ○ ③ prime-main.conf (Logstash Filter) ○ ④ 서버 배치 스크립트 작성 ○ ⑤ 애플리케이션 로그 관련 코드 수정 3 지난 번에 ②~③ 에서 삽질하던 이야기까지 했어요~
  4. 4. ① 개발 환경 구축 ● elk.zip 내려 받은 후 $HOME 폴더에서 압축 해제 ● ELK Docker 구동 4 ~ $ unzip elk.zip ~ $ docker run -d --name elk -e TZ="Asia/Seoul" -p 9200:9200 -p 9300:9300 -p 5044:5044 -p 5601:5601 -v $HOME/elk/data:/var/lib/elasticsearch -v $HOME/elk/config/logstash:/etc/logstash/conf.d -v $HOME/elk/config/kibana:/opt/kibana/config -v $HOME/elk/logs/logstash:/var/log/logstash sebp/elk:latest
  5. 5. ① 개발 환경 구축 ● Logstash log tailing ● Filebeat 구동 ● Kibana에서 로그 확인 5 ~ $ brew install filebeat ~ $ filebeat --strict.perms=false -e -c $HOME/elk/config/filebeats/filebeat.yml ~ $ tail -f $HOME/elk/logs/logstash/logstash.stdout ~ $ open http://localhost:5601
  6. 6. ② filebeat.yml 6 filebeat.prospectors: - document_type: log paths: - /path/to/storage/logs/laravel.log fields_under_root: true fields: log_type: prime-main log_source: app multiline.pattern: '^[[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2}]' multiline.negate: true multiline.match: after multiline.max_lines: 100000 회사의 표준 로그 플랫폼(a.k.a. VoongELK) spec log_source 값에 따라 다른 Logstash 필터가 작동함
  7. 7. ② filebeat.yml(continue) 7 - document_type: log paths: - /path/to/httpd/prime_access_log fields_under_root: true fields: log_type: prime-main log_source: web instance_id: {INSTANCE_ID_TO_BE_REPLACED_DURING_PROVISIONING} channel: {CHANNEL_TO_BE_REPLACED_DURING_PROVISIONING} filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: true output.logstash: hosts: ["{HOST_TO_BE_REPLACED_DURING_PROVISIONING}"] 서버 프로비저닝할 때 교체되는 값 서버 프로비저닝할 때 교체되는 값
  8. 8. ③ prime-main.conf for Application Log 8 [2018-06-01 06:56:52] local.DEBUG: Request-Response log: { "request": { "fingerprint": "1206d351fd9ac5b3743334e4a13ed9a871a372a8", "...": "...", }, "response": { "code": 200, "content": { "result": "success", "...": "...", } }, "execution_time": 0.084259986877441 } { "app_version": "dev-build-180531", "instance_id": "i-07ae611c0b9e33a9d", "transaction_id": "WxBwlP9dgKQ8K7cfPxCgWAAAAA0", "trace_number": 0 } @timestamp channel level_name execution_time MONOLOG_HEADER PRIME_LOG_BODY PRIME_LOG_META
  9. 9. ③ prime-main.conf for Web Access Log 9 10.0.1.130 - - [31/May/2018:22:17:01 +0000] "WxB0XQ0YbL2OdR5CMbFkWQAAAAk" "-" "GET /pos/v1/stores/428/options/pickup HTTP/1.1" 200 109 "-" "RestSharp/105.2.3.0" @timestamp transaction_id request_id request_id는 클라이언트가 제출한 X-Mesh- Request-Id 헤더의 값이며, 값이 없으면 transaction_id의 값으로 폴백.Apache Web Server가 할당한 웹 트랜잭션 고유 식별자
  10. 10. ③ prime-main.conf 10 input { beats { port => 5044 } } filter { } output { elasticsearch { hosts => [ "localhost" ] index => "%{log_type}-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } } 개발해야 할 항목
  11. 11. ③ prime-main.conf(continue) 11 if [log_source] == "app" { grok { pattern_definitions => { "MONOLOG_HEADER" => "[%{TIMESTAMP_ISO8601:datetime}] %{GREEDYDATA:channel}.%{LOGLEVEL:level_name}:" "PRIME_LOG_BODY" => "[wWnt]+" "PRIME_LOG_META" => '(?<prime_log_meta>{ns{4}"app_version":.+ns{4}"instance_id":.+ns{4}"transaction_id":.+ns{4}"trace _number":.+n})' } match => { "message" => "%{MONOLOG_HEADER} %{PRIME_LOG_BODY}s{1,2}%{PRIME_LOG_META}" } } # continued
  12. 12. ③ prime-main.conf(continue) 12 if [level_name] == "DEBUG" and [message] =~ /"execution_time":s?[0-9]+.[0-9]*/ { grok { match => { "message" => '"execution_time":s?(?<execution_time>[0-9]+.[0-9]+)' } } } json { source => "prime_log_meta" } date { match => [ "datetime", "yyyy-MM-dd HH:mm:ss" ] timezone => "Asia/Seoul" } } # continued
  13. 13. ③ prime-main.conf(continue) 13 if [log_source] == "web" { if [message] =~ /.+(internal dummy connection|ELB-HealthChecker).+/ { drop { } } grok { match => { "message" => ".+[%{HTTPDATE:timestamp}] "%{NOTSPACE:transaction_id}" "%{NOTSPACE:request_id}".+" } } if [request_id] =~ /[S]{2,}/ { mutate { replace => { "transaction_id" => "%{request_id}" } } } date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] } }
  14. 14. ④ Apache uniqueid Module deployment script 14 # .ebextensions/70mod-uniqueid.config files: /etc/httpd/conf.modules.d/10-uniqueid.conf: # ... content: | LoadModule unique_id_module modules/mod_unique_id.so /etc/httpd/conf.d/mod_unique_id.conf: # ... content: | RequestHeader set X-Unique-Id %{UNIQUE_ID}e /etc/httpd/conf.d/prime_access_log.conf: # ... content: | <IfModule log_config_module> LogFormat "%h %l %u %t "%{X-Unique-Id}i" "%{X-Mesh-Request-Id}i" "%r" %>s %b "%{Referer}i" "%{User-Agent}i"" prime_combined CustomLog "logs/prime_access_log" prime_combined </IfModule>
  15. 15. ④ filebeat binary/config deployment script 15 # .ebextensions/30filebeat.config Resources: AWSEBAutoScalingGroup: Metadata: AWS::CloudFormation::Authentication: S3Access: type: S3 roleName: aws-elasticbeanstalk-ec2-role buckets: vroong-server-config files: /tmp/filebeat.zip: # ... source: https://vroong-server-config.s3.amazonaws.com/filebeat.zip commands: 50copy-filebeat: command: /bin/cp -f /opt/elasticbeanstalk/hooks/appdeploy/post/300-install_filebeat.sh /opt/elasticbeanstalk/hooks/configdeploy/post/300-install_filebeat.sh
  16. 16. 16 # .ebextensions/30filebeat.config /opt/elasticbeanstalk/hooks/appdeploy/post/300-install_filebeat.sh: mode: "000755" owner: root group: root content: | #!/usr/bin/env bash set -xe echo "Unzipping filebeat resources." /usr/bin/unzip -o /tmp/filebeat.zip -d /tmp echo "Preparing filebeat configuration." INSTANCE_ID=$(/usr/bin/curl http://169.254.169.254/latest/meta-data/instance-id 2> /dev/null) /bin/sed -i "s/instance_id: .*/instance_id: ${INSTANCE_ID}/g" /tmp/filebeat/filebeat.yml source /opt/elasticbeanstalk/support/envvars APP_ENV=$(printenv APP_ENV) /bin/sed -i "s/channel: .*/channel: ${APP_ENV}/g" /tmp/filebeat/filebeat.yml # continued ④ filebeat binary/config deployment script (continue)
  17. 17. 17 # .ebextensions/30filebeat.config { LOGSTASH_HOST=$(printenv LOGSTASH_HOST) } || { echo "LOGSTASH_HOST variable not found. Falling back to default value." LOGSTASH_HOST="DEFAULT_LOGSTASH_HOST:5043" } /bin/sed -i "s/hosts: .*/hosts: ["${LOGSTASH_HOST}"]/g" /tmp/filebeat/filebeat.yml if /usr/bin/pgrep filebeat; then echo "Filebeat Agent already running. Skipping installation." else echo "Installing filebeat binary." /bin/rpm -vi --replacepkgs /tmp/filebeat/filebeat.rpm fi echo "Placing filebeat configuration." /bin/cp -bv /tmp/filebeat/filebeat.yml /etc/filebeat/filebeat.yml /bin/chmod 600 /etc/filebeat/filebeat.yml ④ filebeat binary/config deployment script (continue)
  18. 18. 18 # .ebextensions/30filebeat.config echo "Starting filebeat service." { /sbin/service filebeat restart } || { /sbin/service filebeat start } ④ filebeat binary/config deployment script (continue)
  19. 19. ⑤ CustomizedLoggingProvider application code 19 // app/Providers/CustomizedLoggingProvider.php class CustomizedLoggingProvider extends ServiceProvider { public function boot() { $logger = $this->app->make(LoggerInterface::class); $formatter = new SnakeContextKeyFormatter(null, null, true, true); $streamHandler = new StreamHandler( $this->app->storagePath().'/logs/laravel.log', $this->app->make('config')->get('app.log_level', Logger::DEBUG) ); $streamHandler->setFormatter($formatter); $monolog = $logger->getMonolog(); $monolog->setHandlers([$streamHandler]); $extraLogContextProcessor = $this->app->make(ExtraLogContextProcessor::class); $monolog->pushProcessor($extraLogContextProcessor); } }
  20. 20. ⑤ SnakeContextKeyFormatter application code 20 // app/Support/Logging/SnakeContextKeyFormatter.php class SnakeContextKeyFormatter extends LineFormatter { public function format(array $record) { $record['context'] = ToSnakeCaseArray::run($record['context']); return parent::format($record); } protected function toJson($data, $ignoreErrors = false) { $json = json_encode($data, JSON_PRETTY_PRINT|JSON_UNESCAPED_UNICODE|JSON_UNESCAPED_SLASHES); if ($json === false) { $json = parent::toJson($data, $ignoreErrors); } return $json; } } 로그 컨텍스트 키를 snake_case로 일괄 변경 가독성을 위해 인코딩하지 않고 Pretty Print
  21. 21. ⑤ ExtraLogContextProcessor application code 21 // app/Support/Logging/ExtraLogContextProcessor class ExtraLogContextProcessor { private $appContext; public function __construct(ApplicationContext $appContext) { $this->appContext = $appContext; } public function __invoke(array $record) { $record['extra']['app_version'] = $this->appContext->getAppVersion(); $record['extra']['instance_id'] = $this->appContext->getInstanceId(); $record['extra']['transaction_id'] = $this->appContext->getTransactionId(); $record['extra']['trace_number'] = $this->appContext->getTraceNumber(); $this->appContext->increaseTraceNumber(); return $record; } } PRIME_LOG_META에 해당하는 ExtraContext
  22. 22. ⑤ LogRequestResponse application code 22 // app/Http/Middleware/LogRequestResponse.php class LogRequestResponse { private $exractor; public function __construct(ExtractFilteredData $extractor){ $this->exractor = $extractor; } public function handle($request, Closure $next) { return $next($request); } public function terminate($request, $response) { $requestData = $this->exractor->fromIlluminateRequest($request); $responseData = $this->exractor->fromSymfonyResponse($response); $data = [ 'request' => $requestData, 'response' => $responseData, 'execution_time' => microtime(true) - LARAVEL_START, ]; Log::debug('Request-Response log:', $data); } } PRIME_LOG_BODY에 해당하는 로그 본문
  23. 23. 미션 완료 ● 해결해야 할 문제점 ○ 로그 유실 최소화 ■ Filebeat log streaming to VroongELK ■ Log rotate ■ Log publish to S3 ○ 로그 검색 성능 향상으로 개발자 피로도 최소화 ○ awslogs 데몬이 일으키는 서버 부하 감소 ● 상위 조직에서 받은 미션 ○ MSA로 진화하기 위한 선행 과제 ○ Cloud Watch 로그 요금 절약 23
  24. 24. 24 DEMO ● 부릉 프라임 서비스는 하루에 100GB+, 25백만+ 로그 인스턴스를 생산하고 있어요. ● 특정 상점의 배송 신청 찾기 log_source:"app" AND message:"POST /pos/v1/stores/16023/deliveries" ● transaction_id로 찾기 "5ab1988a-17df-48f3-b6a6-1bdd505c67ee" ● execution_time > 1 이상인 로그만 찾기 tags:"_exetimeparsed" AND execution_time:>=1 SSH tunneling 해야 접속할 수 있는 비 공개 호스트에요~
  25. 25. 25 필터 조건에 해당하는 로그 카운트 ➁ 조회 기간 선택 ➀ 조회할 인덱스 선택 ➂ 테이블에 노출할 필드 또는 필터 선택 ➂ 쿼리 표현식 입력
  26. 26. Application Log Why? 26 Code Data Log A large part of software developers' lives are monitoring, troubleshooting and debugging.”
  27. 27. Application Log Why? ● 로그는… ● Blackbox에 대한 Visibility 확보 ● "어제까지 작동했는데, 오늘 왜 갑자기 작동하지 않는지 모르겠다"라는 개발자들의 전형적인 질문에 대한 힌트 ● 복잡도가 낮은, 투명한 애플리케이션에서는 로깅을 할 필요 없다. ● 로그가 개발자에게 애플리케이션의 상태를 말하도록 하라. 27 If Dog is a man’s best friend, Log is a developer’s best friend. ” source: https://www.quora.com/Why-is-Logging-an-important-part-of-Software-Development
  28. 28. Logging Best Practice ● 애플리케이션에서 발생한 예외 트레이스, "RFC5424 The Syslog Protocol"에 따라 레벨 적용 권장 (사례). ● 의심스러운 애플리케이션 이벤트 ● 추적하고 싶은 애플리케이션 상태 ● 풀리지 않는 버그를 잡기 위한 디버그 로그 ● SQL statement ● 클라이언트의 HTTP 요청 ● 클라이언트에게 돌려주는 HTTP 응답 ● 프로세스/쓰레드 정보 ● 클라이언트(자바스크립트, 닷넷, ..) 측에서 발생하는 예외 28 source: https://dzone.com/articles/application-logging-what-when , https://geshan.com.np/blog/2015/08/importance-of-logging-in- your-applications/
  29. 29. Logging Best Practice ● Essential Components ○ Who UserName ○ When Timestamp ○ Where Context ServletOrPage,Database ○ What Command ○ Result Exception ● Things to Consider ○ Under clustered application environment → Logging as a Service ○ Trade off between logging and performance → Find optimum 29 source: https://dzone.com/articles/application-logging-what-when

×