DevOps Day Jakarta Day 2 Logging in The World of DevOps by Cecep Mahbub Raqibbudin

1. Cecep Mahbub
cecep.mahbub@bukalapak.com
Logging In The World of DevOps

2. About Me
3.5 years with KMK (Liputan6, BBM,
Vidio). In the last 2 years working on
Logging Infrastructure for BBM.
On the last Google Cloud Summit Jakarta
(Oct, 2018), I gave a presentation entitled
"Logging at Petabytes scale on GCP"
Starting this month (Apr 2019), I have
been joining at Bukalapak. I am working
on migration Bukalapak logging
infrastructure to GCP

3. Searching on your logs, is usually just a simple
steps like this
ssh admin@192.168.1.101
cd /var/log/
grep "something interesting" app.log

4. Then your user base is growth significantly...
And now you have more servers.
ssh admin@192.168.1.101
cd /var/log/
grep "something interesting" app.log
ssh admin@192.168.1.102
cd /var/log/
grep "something interesting" app.log

6. and bigger logs size

9. Common Usage of Logs
● Debugging
● Source of Metrics/Monitoring
● Audit tracking

10. Centralized Logs Server
● Logs is indexed, searching on your logs should be fast
● Easy to aggregate search result
● Easy to create data visualization (trending over time, spot anomali traffic,
comparing current data with previous time windows)
● Role based access control
● Scalable (horizontal/vertical)

11. Logs Servers
● ELK (Elasticsearch, Logstash, Kibana)
● Graylog
● Splunk

12. Logs Servers (Cloud)
● CloudWatch (AWS)
● Stackdriver (GCP)
● Azure?
● Alibaba Cloud?

13. Logs Servers (SaaS)
● Datadog
● Elastic Cloud
● Loggly
● Scalyr

14. How to integrate your logs into
DevOps process

15. Everyone should have an access to your logging tools
● Everyone should know how to search their logs (i.e learn query DSL)
● Minimize Silo
● Team is not too dependent on other teams
● Security and personal data protection still applied

16. Examples Case: Monitoring Dashboard

17. Examples Case: Canary Release
Source: https://www.infoq.com/articles/gilt-deploying-microservices-aws

18. Logging Infrastructure at Bukalapak

19. ~7 TB logs/days
● Net Appliance
● Microservices running under Docker / Kubernetes
● Application Logs
● Server App Logs (Nginx, MySQL, MongoDB etc)

20. Bukalapak Logging Infrastructure

21. Current Issues
● Too many index (1 microservices ~ 1 index)
● Field mapping conflict
● Unparsed logs

22. Bukalapak Logging Infrastructure (GCP)

23. Thank You!