Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

AWS Summit Singapore - Lambda, Step Functions and Datadog: A Symphony

254 visualizaciones

Publicado el

Alex Poe, Solutions Engineer Team Lead, Datadog

OK, you have built your first Lambda function. It may even be useful (and not just a thumbnail generator). But an application is more than just a single function; you need a lot of them. How do you get your functions working together? How do you know each function is performing well and how do you ensure that they improve over time. In this session I’ll make sure you are on the right track with AWS Lambda, get you started with Step Functions, and help you get the most out of your functions with a little monitoring.

  • Sé el primero en comentar

AWS Summit Singapore - Lambda, Step Functions and Datadog: A Symphony

  1. 1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Alex Poe Solutions Engineer Team Lead, Datadog Lambda, Step Functions and Datadog: A Symphony
  2. 2. Who am I Alex Poe Lead Solutions Engineer, APAC Datadog @alextpoe alex.poe@datadoghq.com
  3. 3. Who is Datadog SaaS-based Monitoring & Analytics Infrastructure, APM, Logs Open Source Agent Trillions of data points per day
  4. 4. What is Lambda • Serverless • As far as you are concerned • Don’t pay for idle • Scale with ease • FaaS • Node 6.10 (java, go, python, c#)
  5. 5. …but AWS Lambda != ec2
  6. 6. …but AWS Lambda != application
  7. 7. …applications are complicated
  8. 8. …lambdas should respect the single responsibility principle…
  9. 9. we can create many serverless functions
  10. 10. we can create many serverless functions but how do we orchestrate them?
  11. 11. Amazon Kinesis Amazon SQS Amazon DynamoDB Event/Message/Database as Traffic Cop
  12. 12. https://github.com/Nordstrom/hello-retail
  13. 13. Enter AWS Step Functions
  14. 14. a step function is
  15. 15. a step function is a state machine
  16. 16. but what about SWF? (Amazon Simple Workflow Service)
  17. 17. AWS Step Function Concepts • State Machine • Transitions • Executions • States • Task • Choice • Succeed/Fail • Pass • Wait • Parallel • You define the State Machine with the Amazon States Language • A diagram will be generated based on that structure
  18. 18. State Example: Task
  19. 19. State Example: Choice
  20. 20. State Example: Succeed / Fail
  21. 21. State Example: Pass
  22. 22. State Example: Wait
  23. 23. State Example: Parallel
  24. 24. Many ways to build Step Functions Serverless Framework
  25. 25. Benefits of Serverless Framework • Define everything in a single file: • IAM Role Statements • Lambda Function Handlers • Lambda Function Triggers • Step Function States • CloudFormation for other things
  26. 26. serverless.yml service: name: mattw-reinvent2017-build-website plugins: - serverless-pseudo-parameters - serverless-step-functions - serverless-webpack custom: repoName: reinvent2017-website
  27. 27. serverless.yml cont’d provider: name: aws runtime: nodejs6.10 stage: ${opt:stage, 'dev'} region: ${opt:region, 'us-east-1'} iamRoleStatements: - Effect: "Allow" Action: - "states:StartExecution" Resource: - ${self:resources.Outputs.MyStateMachine.Value} - Effect: "Allow" Action: - "codecommit:GetCommit" - "codecommit:GetDifferences" Resource: - arn:aws:codecommit:${self:provider.region}:#{AWS::AccountId}:${…custom.repoName}
  28. 28. serverless.yml cont’d functions: buildSite: handler: handler.build whatChanged: handler: handler.whatChanged startStepFunction: handler: handler.startStepFunction events: - sns: arn:aws:sns:${…provider.region}:#{AWS::AccountId}:mattw-reinvent-websitebuild environment: statemachine_arn: ${self:resources.Outputs.MyStateMachine.Value}
  29. 29. serverless.yml cont’d stepFunctions: stateMachines: FirstStateMachine: name: myStateMachine definition: StartAt: WhatChanged States: WhatChanged: Type: Task Resource: arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId} :function:${self:service}-${self:provider.stage}- whatChanged Next: BuildDockerOrNot
  30. 30. AWS CodeCommit Amazon SNS AWS Lambda
  31. 31. startStepFunction: handler: handler.startStepFunction events: - sns: arn:aws:sns…mattw-reinvent-websitebuild environment: statemachine_arn: ${self:reso…puts.MyStateMachine.Value}
  32. 32. StartStepFunction Get Step Function ARN Get SNS details Start SF Execution
  33. 33. 1. What changed in the repo? 2. If docker-related, build docker 1. Tell CodeBuild to rebuild docker image 2. Wait for it 3. Push to ECS repo 3. Use the docker image to build the website 4. Wait for it 5. Send Matt an SMS
  34. 34. WhatChanged
  35. 35. WhatChanged: Type: Task Resource: arn:aws:lambda:... Next: BuildDockerOrNot Retry : - ErrorEquals: - Lambda.Unknown States.ALL IntervalSeconds: 3 MaxAttempts: 4
  36. 36. WhatChanged Get the commit that triggered the sf Get the previous commit Whats different If it was docker related return hasDocker = true else return hasDocker = false
  37. 37. BuildDockerOrNot
  38. 38. BuildDockerOrNot: Type: Choice Choices: - Variable: "$.hasDocker" BooleanEquals: true Next: BuildDocker - Variable: "$.hasDocker" BooleanEquals: false Next: BuildSite
  39. 39. BuildDocker: Type: Task Resource: arn:aws:lambda…buildDocker Next: WaitForDockerBuild
  40. 40. BuildDocker Start the CodeBuild for docker image
  41. 41. WaitForDockerBuild: Type: Wait Seconds: 20 Next: IsTheContainerBuilt
  42. 42. IsTheContainerBuilt: Type: Task Resource: arn:aws:lambda... Next: ContainerBuiltOrNot
  43. 43. IsTheContainerBuilt Get build info from CodeBuild Check if its done
  44. 44. IsTheContainerBuild
  45. 45. ContainerBuiltOrNot
  46. 46. ContainerBuiltOrNot: Type: Choice Choices: - Variable: "$.containerBuildDone" BooleanEquals: true Next: BuildSite - Variable: "$.containerBuildDone" BooleanEquals: false Next: WaitForDockerBuild
  47. 47. 1. What changed in the repo? 2. If docker-related, build docker 1. Tell CodeBuild to rebuild docker image 2. Wait for it 3. Push to ECS repo 3. Use the docker image to build the website 4. Wait for it 5. Send Matt an SMS
  48. 48. How about monitoring? We monitor Lambda via Logs MONITORING|unix_epoch_timestamp|value|count| my.metric.name|#tag1:value,tag2 console.log(monitoringstring);
  49. 49. What are the Lambda Metrics aws.lambda.duration, min, max, sum aws.lambda.errors aws.lambda.invocations aws.lambda.throttles aws.lambda.iterator_age
  50. 50. What are the Step Function Metrics aws.states.execution_time (.maximum, .minimum) aws.states.executions_aborted, failed, started, succeeded, timed_out aws.states.lambda_functions_scheduled, started, succeeded, timed_out aws.states.execution_throttled aws.states.lambda_function_time (.maximum, .minimum) aws.states.lambda_function_run_time (.maximum, .minimum) aws.states.lambda_function_schedule_time (.maximum, .minimum) aws.states.lambda_functions_started
  51. 51. Conclusion • Lambda Basics • It’s Serverless! • Step-Functions • Orchestrate • Demo • Monitoring
  52. 52. Questions? Alex Poe Lead Solutions Engineer, APAC Datadog @alextpoe alex.poe@datadoghq.com

×