Reducing Latency and Increasing Performance while Cutting Infrastructure Costs
•
2 recomendaciones•4,472 vistas
Discussion on Datadog’s experiences, both successes and challenges, as they built our monitoring solutions on top AWS Lambda and Amazon API gateway with the goal of reducing latency and increasing performance while cutting infrastructure costs.
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Destacado (20)
Similar a Reducing Latency and Increasing Performance while Cutting Infrastructure Costs
Similar a Reducing Latency and Increasing Performance while Cutting Infrastructure Costs (20)
Más de Amazon Web Services
Más de Amazon Web Services (20)
Último
Último (20)
Reducing Latency and Increasing Performance while Cutting Infrastructure Costs
- 1. From Pull to Push Using Lambda to minimize latency and reduce operational costs tristan@datadoghq.com
- 2. Datadog gathers performance data from all your application components
- 5. What is Datadog? ELB Web Servers
- 6. What is Datadog? ELB Web Servers Databases
- 7. What is Datadog? ELB Web Servers Databases Custom
- 8. What is Datadog? Datadog gathers performance data from all your application components. Detail metrics about one of the components
- 9. Agent based data collection Crawler based data collection
- 10. Agenda ● Pulling data via crawlers generates latency and operational cost ● Using Lambda to minimize latency and reduce operational costs
- 11. Agenda ● Pulling data via crawlers generates latency and operational cost ● Using Lambda to minimize latency and reduce operational costs
- 12. AWS is one of our most used integrations
- 13. What is Datadog?
- 14. What is Datadog?
- 15. Latency as seen by users Upstream Latency Data points are available after 2 to 11 minutes Scheduling Latency We schedule crawlers every 1 to 10 minutes Crawler Latency How much time it takes to fetch the data Crawler based metrics can’t be real time
- 16. Latency as seen by users Upstream Latency Data points are available after 2 to 11 minutes Scheduling Latency We schedule crawlers every 1 to 10 minutes Crawler Latency How much time it takes to fetch the data Crawler based metrics can’t be real time Throttling API call charges Infrastructure cost
- 17. How can we minimize metric latency?
- 18. Agenda ● Pulling data via crawlers generates latency and operational cost ● Using Lambda to minimize latency and reduce operational costs
- 20. AWS Lambda - Serverless / Operation-less
- 21. - Serverless / Operation-less - Event driven architecture with many integrations AWS Lambda
- 22. AWS Lambda - Serverless / Operation-less - Event driven architecture with many integrations - Api Gateway for custom integrations
- 23. Agenda ● Pulling data via crawlers generates latency and operational cost ● Using Lambda to minimize latency and reduce operational costs ○ RDS enhanced monitoring with Lambda
- 24. Amazon RDS enhanced monitoring CloudWatch Metrics
- 25. Amazon RDS enhanced monitoring CloudWatch Metrics CloudWatch Logs
- 26. Amazon RDS enhanced monitoring
- 27. How to sync data between CWLogs and Datadog? Datadog CloudWatch Logs ?
- 28. How to sync data between CWLogs and Datadog? Datadog CloudWatch Logs Crawler Pull Data Submit Data
- 29. How to sync data between CWLogs and Datadog? Datadog CloudWatch Logs Push with Lambda Crawler Pull Data Submit Data
- 30. Lambda function that subscribes to RDSOS Metrics
- 31. Lambda function to process to RDSOS Metrics
- 32. Lambda function to process to RDSOS Metrics
- 33. Lambda allows sub minute latency for those metrics Crawler based Lambda based
- 34. Lambda allows sub minute latency for those metrics Crawler based Lambda based
- 35. Amazon RDS enhanced monitoring + Sub minute latency + No crawler to run and maintain + No internal state to remember which points to process + No Ops
- 36. Amazon RDS enhanced monitoring + Sub minute latency + No crawler to run and maintain + No internal state to remember which points to process + No Ops - Not as easy to setup and troubleshoot - Not easy to update - No ad hoc replay
- 37. Agenda ● Pulling data via crawlers generates latency and operational cost ● Using Lambda to minimize latency and reduce operational costs ○ Agent Release Process with Lambda
- 39. Datadog’s Agent releases: Invalidate the cache Amazon CloudFrontAmazon LambdaAmazon S3Jenkins Push package Trigger Lambda Invalidate cache
- 40. Datadog’s Agent releases: Notify the security team Amazon CloudFrontAmazon LambdaAmazon S3Jenkins Push package Trigger Lambda Notify Security Invalidate cache
- 41. Stateless, simple, event based tasks Stateful, complex tasks AWS Lambda +++ ???
- 42. Agenda ● Pulling data via crawlers generates latency and operational cost ● Using Lambda to minimize latency and reduce operational costs ○ Using Lambda to extract custom metrics from Lambda
- 45. Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Agent Aggregation Server Application Application Application Application Push over UDP Datadog increment(metric, value) host
- 46. Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Datadog ? Agent Aggregation Server Application Application Application Application Push over UDP Datadog increment(metric, value) host
- 47. Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda Lambda CloudWatch Logs Datadog ? log(MONITORING|user.submit|count|14733433|1|#demo) MONITORING|name|type|timestamp|value|#tags) Agent Aggregation Server Application Application Application Application Push over UDP Datadog increment(metric, value) host
- 48. Datadog Intake CloudWatch Logs How to submit data from CloudWatch Logs to Datadog?
- 49. Datadog Intake CloudWatch Logs How to submit data from CloudWatch Logs to Datadog? user.submit|1|timestamp1 user.submit|1|timestamp2 user.submit|1|timestamp1 Contains user.submit|1|timestamp2 user.submit|2|timestamp1 Expects
- 50. Datadog Intake CloudWatch Logs We need to aggregate the data points user.submit|1|timestamp1 user.submit|1|timestamp2 user.submit|1|timestamp1 Contains user.submit|1|timestamp2 user.submit|2|timestamp1 Expects Aggregation user.submit|timestamp1:[1, 1] user.submit|timestamp2:[1]
- 51. Cloud Watch Logs DatadogAggregate Data user.submit|1|timestamp2 user.submit|1|timestamp2 user.submit|1|timestamp1 user.submit|1|timestamp2 user.submit|2|timestamp1 Pull Data CRAWLER Submit Data user.submit|timestamp1:[1, 1] user.submit|timestamp2:[1] A crawler from CloudWatch Logs to Datadog
- 52. Datadog Intake CloudWatch Logs How can we build this with Lambda? user.submit|1|timestamp1 user.submit|1|timestamp2 user.submit|1|timestamp1 Contains user.submit|1|timestamp2 user.submit|2|timestamp1 Expects Aggregation user.submit|timestamp1:[1, 1] user.submit|timestamp2:[1] Push Push
- 53. Datadog Intake CloudWatch Logs How can we build this with Lambda? user.submit|1|timestamp1 user.submit|1|timestamp2 user.submit|1|timestamp1 Contains user.submit|1|timestamp2 user.submit|2|timestamp1 Expects Aggregation Service user.submit|timestamp1:[1, 1] user.submit|timestamp2:[1] Push Push
- 54. Datadog Intake CloudWatch Logs We need a stateful Lambda pipeline to aggregate metrics user.submit|1|timestamp1 user.submit|1|timestamp2 user.submit|1|timestamp1 Contains user.submit|1|timestamp2 user.submit|2|timestamp1 Expects Aggregation Service user.submit|timestamp1:[1, 1] user.submit|timestamp2:[1] Push Push
- 55. Building a stateful lambda pipeline Aggregation Service user.submit|timestamp2:[1] user.submit|timestamp1:[1, 1] user.submit|1|ts2 user.submit|2|ts1 user.submit|1|ts1 user.submit|1|ts2 user.submit|1|ts1
- 56. A Database stores the state Aggregation Service DynamoDB Kinesis ElastiCache user.submit|timestamp1:[1, 1] user.submit|1|ts2 user.submit|2|ts1 user.submit|timestamp2:[1] user.submit|1|ts1 user.submit|1|ts2 user.submit|1|ts1
- 57. user.submit|1|ts1 user.submit|1|ts2 user.submit|1|ts1 user.submit|1|ts2 user.submit|2|ts1 Aggregation Service API Gateway forwards the Data Points to the store Lambda Lambda DynamoDB Kinesis ElastiCache user.submit|timestamp1:[1, 1] user.submit|timestamp2:[1]Lambda API Gate way
- 58. user.submit|1|ts1 user.submit|1|ts2 user.submit|1|ts1 user.submit|1|ts2 user.submit|2|ts1 Aggregation Service Lambda subscribes to state updates and forwards to Datadog Lambda Lambda Lambda Lambda DynamoDB Kinesis ElastiCache user.submit|timestamp1:[1, 1] user.submit|timestamp2:[1]Lambda API Gate way
- 59. user.submit|1|ts1 user.submit|1|ts2 user.submit|1|ts1 user.submit|1|ts2 user.submit|2|ts1 Aggregation Service A Serverless stateful pipeline to aggregate Data Points Lambda Lambda Lambda Lambda DynamoDB Kinesis ElastiCache user.submit|timestamp1:[1, 1] user.submit|timestamp2:[1]Lambda API Gate way
- 60. Takeaways
- 61. Takeaways - Lambda allows us move towards a push system
- 62. Takeaways - Lambda allows us move towards a push system - Lambda is great for small stateless event based tasks
- 63. Takeaways - Lambda allows us move towards a push system - Lambda is great for small stateless event based tasks - We’re seeing adoption amongst our users