6. Used AWS products
• RDS
• Store service data
• Queue / Scheduler
• S3
• Columnar storage
• EC2
• Clusters: Hadoop,Workers,APIs, etc…
6
Separate
Storage and Processor!
16. In short
• Open sourced log collector written in Ruby
• Customization is essential
small core + many plugins
16
Fluentd is a robust log collector
designed for processing data streams
18. M x N → M + N
18
Nagios
MongoDB
Hadoop
Alerting
Amazon S3
Analysis
Archiving
MySQL
Apache
Frontend
Access logs
syslogd
App logs
System logs
Backend
Databases
buffer / buffer / routing
22. Collecting instance logs
22
• A sign of Immutable Infrastructure
• Hard to manage state-full instance
• Almost instance should be disposable
• Excluding DB, Master, etc...
• How to manage such instance logs?
• Common problem on Cloud environment
23. • Start Fluentd at launch phase
• It is also useful for Docker / other containers
• Including metadata or host to identify
Collecting using Fluentd
23
Collector Aggregator