The KELK Stack - Kinesis, Elasticsearch, Lambda, Kibana. An example of a low maintenance logging pattern using AWS services.

2. SteamhausKELK ON AWS
Who am I?
Sean Clerkin
Senior Site Reliability Engineer

3. Logging is difficult

4. No centralised logging
User needs OS
knowledge
Distribution
Of keys
Enemy of
autoscaling
Log
rotation
Users download
logs unnecessarily
Doesn’t scale
To many servers
Slow to
find issues
Alerting
is hard
Sshing to
servers :(

5. SteamhausKELK ON AWS
ELK is awesomE

6. SteamhausKELK ON AWS
ELK on ec2

7. SteamhausKELK ON AWS
KELK on AWS
• Low maintenance - No ec2, Uses entirely AWS serverless technologies and services
• ALB, Cloudfront and Cloudtrail logs are ingested as well as EC2 logs
• Logs are archived in S3 for long term storage, and indexed in Elasticsearch for short
term analytics
• Automated with Terraform
• Open source
Kinesis: buffering and delivering instance logs
Elasticsearch: Indexing and log storage
Lambda: processing and delivering S3 logs
Kibana: Search and analytics

8. SteamhausKELK ON AWS
How does it work?

9. SteamhausKELK ON AWS

10. SteamhausKELK ON AWS

11. SteamhausKELK ON AWS

12. SteamhausKELK ON AWS

13. SteamhausKELK ON AWS

14. SteamhausKELK ON AWS

15. SteamhausKELK ON AWS

16. SteamhausKELK ON AWS

17. SteamhausKELK ON AWS

18. SteamhausKELK ON AWS

19. SteamhausKELK ON AWS

20. SteamhausKELK ON AWS

21. SteamhausKELK ON AWS
Automation
code
Sample
Web Stack
VPC
ALB
EC2
Logging
Stack
Kinesis
Elasticsearch
Service
Lambda
S3
CloudfrontPython
Terraform
Do try this at home!
github.com/steamhaus/kelk-example

22. SteamhausKELK ON AWS
Callouts from the build
• It’s not production ready, built for readability
• Nailing iam and bucket policies can take a while!
• Testing lambda - create a test event in the UI
• Use Terraform, rinse and repeat

23. SteamhausKELK ON AWS
Any Questions..?

24. Thank you :)
Contact us
hello@steamhaus.co.uk
0161 820 2020
@steamhausmcr
Locate us
Fourways House
57 Hilton Street
Manchester M1 2EJ