This talk will cover the need of a centralized logging system, showcasing the architecture of the system. Also, I talk about how we ended up building this centralized logging system, What was the need for such a system, what problems we faced, how MongoDB fits into this and what others can learn from this. I also covered some how can we use mongoDB to make our logging system for realtime analytics and alerting system . The major use case of this system it to keeping track of meaningful events. This could be -: 1. How many users registered ? 2. How many registrations fails ? 3. Most occurred errors while doing something. 4. Realtime Analytics and Alerts 5. Identifying the possible threats.

1. Centralized Logging System Using MongoDB
@vparihar
AVP Engineering,Webonise Lab
Vivek Parihar

2. Who Am I?
● A Weboniser and Rubyist
● Blogger(vparihar01.github.com)
● MongoDb user
● Geek
● DevOps
● Mainly write Ruby, but have great passion for Javascript and Cloud
Platforms...

3. ● What is Logging?
● Why we need Logging?
● Logging DO’s and Don’t
● Logs are Streams, Not FIles
● Problems managing Logs for huge INFRA
● What Central Logging System can do for us?
● Central Logging System Architecture
● What and why Fluentd?
● Why MongoDB is good fit.
Agenda

4. What is Logging?
Mmmm Logging: It is the most important part of any
application.
In General, Logging refers to keeping track of
something.

5. Why we need Logging?

6. Logging: Helps me finding and fixing bugs

7. Logging: Extensively used for Debugging

8. Logging: Helps us diagnose & understand the
behaviour of application.

9. Logging: Tells us exactly what happened
when, where and why?
Who did it ?
At what time ?
What did he steal ?

10. Logging: Do’s and Don’t
#1 It should be FAST

11. Logging: Do’s and Don’t
#2 Should not affect user
Prevent DISK BLOAT

12. It should not be like-:
{
● "#########its working#########"
● "!!!!!coming here in to get secondary users!!!!!"
● "#########I am Here#########"
● "#########Task completed#######"
}
Logging: Do’s and Don’t
#3 Do Log only useful INFO

13. Logging: Do’s and Don’t
4. Differentiate Log Levels

14. Logs Are Streams, Not Files
Logs are a stream, and it behooves everyone to treat them as such. Your
programs should log to stdout and/or stderr and omit any attempt to handle
log paths, log rotation, or sending logs over the syslog protocol.
Directing where the program’s log stream goes can be left up to the runtime
container: a local terminal or IDE (in development environments), an Upstart
/ Systemd launch script (in traditional hosting environments), or a system
like Logplex/Heroku (in a platform environment).
By: Adam Wiggins, Heroku co-founder.

15. Problems managing Logs for huge Infra

16. What about infra like these ?
Problems managing Logs for huge Infra

17. Expression like:

18. How can we solve huge Infra problem ?

19. Solution: Centralized Logging System

20. What Centralized Logging System can do
for us?

21. What Centralized Logging System can do for
us?
All of the logs are in one place, this makes things like searching
through logs and analysis across multiple servers easier than
bouncing around between boxes. Greatly simplifying log analysis
and correlation tasks.
#1 Log Collections

22. #2 Aggregation
Scaled-out servers behind load balancers each produce their
own log files, making it impossible to debug a single action flow
that distributed between servers, unless the logs converge into
a single article.
What Centralized Logging System can do for
us?

23. #3 High Availability
Suppose your system is down or overloaded and unable to tell
you what happened.
What Centralized Logging System can do for
us?

24. Local logs from the server may be lost in the event of an
intrusion or system failure. But by having the logs elsewhere
you at least have a chance of finding something useful about
what happened.
#4 Security
What Centralized Logging System can do for
us?

25. It reduces disk space usage and disk I/O on core servers
that should be busy doing something else.
#5 Prevent Disk BLOAT
What Centralized Logging System can do for
us?

26. #6 Visual Indicators
Abnormal behaviors can be detected faster when we see
them in a visual instrument such as a graph, where peak
points are easily noticed.
What Centralized Logging System can do for
us?

27. Centralized Logging System Architecture

28. What and Why ?

29. What’s Fluentd?
It’s like syslogd, but uses JSON for log messages

30. What’s Fluentd?

31. What’s Fluentd?
time
tag
record

32. What’s Fluentd?

33. What’s Fluentd?
Plug-in Plug-in Plug-in

34. So Fluentd is a:
Buffer
Router
Collector
Converter
Aggregator
…….
What’s Fluentd?

35. It’s written in RUBY :)
Why Fluentd?

36. Extensibility - Plugin Architecture
Why Fluentd?

37. Unified log format - JSON format
Why Fluentd?

38. Reliable - HA configuration
Why Fluentd?

39. Easy to install - RPM/deb packages
> sudo fluentd --setup && fluentd
Very small footprint
> small engine (3,000 line) + plugins
Why Fluentd?

40. Why is good fit ?

41. 1. It’s Schemaless
Document-oriented / JSON is
a great format for log
information. Very flexible and
“schemaless” in the sense we
can throw in an extra field
any time we want.
Why ?

42. 2. Fire and Forget
MongoDB inserts can be done asynchronously.
Why ?

43. 3. Scalable and easy to replicate.
Built in ReplicaSet and Sharding provides high availability.
Why ?

44. 4. Centralized and easy remote access
Why ?

45. 5. Capped Collection
● They "remember" the insertion order of their documents
● They store inserted documents in the insertion order on disk
● They remove the oldest documents in the collection automatically as new
documents are inserted
However, you give up some things with capped collections:
● They have a fixed maximum size
● You cannot shared a capped collection
● Any updates to documents in a capped collection must not cause a document to
grow. (i.e. not all$set operations will work, and no $push or $pushAll will)
● You may not explicitly .remove() documents from a capped collection
Why ?

46. 6. Tailing Logs
● You’ll really miss ability to tail logfiles
● Or, .. will you?
● MongoDB offers tailable cursors
Why ?

47. Tailable Cursors
What with Tailable Cursors ?
We can implement the pub/sub using
Node.js and MongoDB
https://github.com/scttnlsn/mubsub
Why ?

48. Thanks
Would Love to answer your queries...
Vivek Parihar
@vparihar