Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

利用 AWS Step Functions 建構穩定的資料處理流程.pdf

127 visualizaciones

Publicado el

當企業開始普遍使用 Machine Learning/AI 相關技術時,隨之而來的,便是如何管理眾多的資料處理流程。
面對大量且多樣的資料,確保資料正確與一致,便成為一件複雜但又不得不做好的事情。
本技術分享將介紹趨勢科技如何利用 AWS Step Functions 服務,大幅簡化資料處理架構,
在短時間內建構出繁雜的資料驗證流程,並介紹如何利用 AWS Step Functions 實作出實務上必須
面對的 timeout、retry 與 error handling 等機制,以確保資料處理流程能夠穩健運作。

  • DOWNLOAD FULL eBOOK INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc eBook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookeBOOK Crime, eeBOOK Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Responder 
    ¿Estás seguro?    No
    Tu mensaje aparecerá aquí
  • Sé el primero en recomendar esto

利用 AWS Step Functions 建構穩定的資料處理流程.pdf

  1. 1. S U M M I T Taipei
  2. 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T AWS Step Functions Kwunhok Chan Solutions Architect Amazon Web Services S e s s i o n I D CoCo Lin Engineer Trend Micro Bruce WY Lin Senior Engineer Trend Micro
  3. 3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T No servers to provision or manage Scales with usage Never pay for idle Availability and fault-tolerance built in Serverless means…
  4. 4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T SERVICES (ANYTHING) Changes in data state Requests to endpoints Changes in resource state EVENT SOURCE FUNCTION Node.js Python Java C# Go Ruby and more… Serverless applications
  5. 5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T DBMSQueue Modern App
  6. 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Modern App
  7. 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Turning functions into apps “I want to select functions based on data” “I want to retry functions” “I want try/catch/finally” “I have code that runs for hours” “I want to run functions in parallel” “I want to sequence functions”
  8. 8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Example Use Case Suppose I want to book my vacation to the Grand Canyon, …but unwind my reservations if I can’t rent a car.
  9. 9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Coordination by method call
  10. 10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Coordination by function chaining
  11. 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Coordination by database
  12. 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Coordination by queues
  13. 13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T AWS Step Functions
  14. 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Benefits of AWS Step Functions Manages the logic of your application. This removes extra code that may be repeated in your microservices and functions. Easy to connect and coordinate distributed components and microservices to quickly create apps AWS Step Functions manages state, checkpoints and restarts to make sure your application executes in order and as expected. Build and update Apps Quickly Write Less Code Improve Resiliency
  15. 15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Run each step in sequence Vendor A Vendor B Vendor C
  16. 16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T …and unwind if my plans fail at any point Vendor A Vendor B Vendor C
  17. 17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Coordination must-haves • Scales out • Doesn’t lose state • Deals with errors/timeouts • Easy to build & operate • Auditable
  18. 18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Application lifecycle in AWS Step Functions Visualize in the Console Define in JSON Monitor Executions
  19. 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Seven state types Task A single unit of work Choice Adds branching logic Parallel Fork and join the data across tasks Wait Delay for a specified time Fail Stops an execution and marks it as a failure Succeed Stops an execution successfully Pass Passes its input to its output
  20. 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Task states for any compute Long poll Traditional server Request Worker requests tasks from Step Functions Step Functions invokes the Task
  21. 21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Build visual workflows from state types Task Choice Fail Parallel
  22. 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Common Use Cases • Build modern applications • Data processing • Automate tasks • Break-up monoliths • Automate Machine Learning (ML) jobs
  23. 23. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  24. 24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Big Data AI/ML Data
  25. 25. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  26. 26. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Raw Data Behavior Features Static Features Verify Data Analyze
  27. 27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Succeed Choice Fail Verification on EMR Task Wait [ "static_features_1.log", "static_features_2.log" ] {"schema": "static_features", "result": "passed"}
  28. 28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T { "static_features": [ "static_features_1.log", "static_features_2.log"], … … … ”behavior_features": [ "behavior_features_1.log", "behavior_features_2.log"], … … … } Get value from key “static_features” : ["static_features_1.log", "static_features_2.log"] Succeed Choice Fail Verification on EMR Task Wait Get value from key “static_features” : ["static_features_1.log", "static_features_2.log"] Verification on EMR Get value from key “static_features” : ["static_features_1.log", "static_features_2.log"] Verification on EMR Get value from key “behavior_features” : [" behavior_features _1.log", " behavior_features _2.log"] Verification on EMR
  29. 29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T ["static_features_1.log", "static_features_2.log"] Parallel InputPath=$.static_features Succeed Choice Fail Verification on EMR Task Wait Succeed Choice Fail Verification on EMR Task Wait { "static_features": [ "static_features_1.log", "static_features_2.log"], … … … ”behavior_features": [ "behavior_features_1.log", "behavior_features_2.log"], … … … }
  30. 30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T ["static_features_1.log", "static_features_2.log"] Parallel InputPath=$.static_features Succeed Choice Fail Verification on EMR Task Wait Succeed Choice Fail Verification on EMR Task Wait ["behavior_features _1.log", "behavior_features _2.log"] InputPath=$.static_features
  31. 31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T ["static_features_1.log", "static_features_2.log"] Parallel InputPath=$.static_features Succeed Choice Fail Verification on EMR Task Wait Succeed Choice Fail Verification on EMR Task Wait ["behavior_features _1.log", "behavior_features _2.log"] InputPath=$.static_features Fail
  32. 32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T ["static_features_1.log", "static_features_2.log"] Parallel InputPath=$.static_features Succeed Choice Fail Verification on EMR Task Wait Succeed Choice Fail Verification on EMR Task Wait ["behavior_features _1.log", "behavior_features _2.log"] InputPath=$.static_features Fail
  33. 33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T ["static_features_1.log", "static_features_2.log"] Parallel InputPath=$.static_features Succeed Choice Fail Verification on EMR Task Wait Succeed Choice Fail Verification on EMR Task Wait ["behavior_features _1.log", "behavior_features _2.log"] InputPath=$.static_features Fail
  34. 34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Succeed Choice Pass Task Wait
  35. 35. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Succeed Choice Pass Task Wait
  36. 36. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  37. 37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T 3 Problems Infinite Loop Dead and… End Only One Timeout 3 Solutions Retry Has Max Custom Errors Save Life Retry Makes Timeout Anywhere
  38. 38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T 3 Problems Infinite Loop Dead and… End Only One Timeout 3 Solutions Retry Has Max Custom Errors Save Life Retry Makes Timeout Anywhere
  39. 39. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Infinite Loop • Your bill may be infinite, too SucceedPass
  40. 40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T 3 Problems Infinite Loop Dead and… End Only One Timeout 3 Solutions Retry Has Max Custom Errors Save Life Retry Makes Timeout Anywhere
  41. 41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Dead and… End • Often happened while waiting • Do nothing • Get nothing SucceedPass
  42. 42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T 3 Problems Infinite Loop Dead and… End Only One Timeout 3 Solutions Retry Has Max Custom Errors Save Life Retry Makes Timeout Anywhere
  43. 43. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Only One Timeout • No timeout for sub- workflow SucceedPass SucceedPass Parallel 8 mins 24 mins
  44. 44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Risk Summary Run It Infinite Loop Add Timeout Dead and… End Adjust Timeout Only One Timeout
  45. 45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T 3 Problems Infinite Loop Dead and… End Only One Timeout 3 Solutions Retry Has Max Custom Errors Save Life Retry Makes Timeout Anywhere
  46. 46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Simplified Job Status Poller Poller Submitter Error Handler Succeed SucceedPass
  47. 47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T 3 Problems Infinite Loop Dead and… End Only One Timeout 3 Solutions Retry Has Max Custom Errors Save Life Retry Makes Timeout Anywhere
  48. 48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T If emr_status == ‘COMPLETED’: pass elif emr_status in [‘PENDING’, ‘RUNNING’]: raise AsyncTaskUnfinished(‘go to retry’) "Poller":{ "Type": "Task", "Next": "Succeed", "ResultPath": "$.output.poller", "Retry":[ { "ErrorEquals": ["AsyncTaskUnfinished"], "IntervalSeconds": 10, "MaxAttempts": 3, } ] }
  49. 49. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Poller Submitter Error Handler Succeed "Poller":{ "Type": "Task", "Next": "Succeed", "ResultPath": "$.output.poller", "Retry":[ { "ErrorEquals": ["AsyncTaskUnfinished"], "IntervalSeconds": 10, "MaxAttempts": 3, "BackoffRate": 2.0 } ] }
  50. 50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T 3 Problems Infinite Loop Dead and… End Only One Timeout 3 Solutions Retry Has Max Custom Errors Save Life Retry Makes Timeout Anywhere
  51. 51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Custom Errors Save Life "Poller":{ "Type": "Task", "Next": "Succeed", "ResultPath": "$.output.poller", "Retry":[ { "ErrorEquals": ["AsyncTaskUnfinished"], "IntervalSeconds": 10, "MaxAttempts": 3, "BackoffRate": 2.0 } ], "Catch":[ { "ErrorEquals": [" AsyncTaskFailed"], "ResultPath": "$.error.poller", "Next": "ErrorHandler" } ] }, "ErrorHandler": { "Type": "Task", "ResultPath": "$.error", "End": true } if emr_status in [‘CANCELLED’, ‘FAILED’, ‘INTERRUPTED’]: raise AsyncTaskFailed(‘go to error handler ’)
  52. 52. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T 3 Problems Infinite Loop Dead and… End Only One Timeout 3 Solutions Retry Has Max Custom Errors Save Life Retry Makes Timeout Anywhere
  53. 53. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T "PollerA":{ "Type": "Task", "Next": "Succeed", "ResultPath": "$.output.poller", "Retry":[ { "ErrorEquals": ["AsyncTaskUnfinished"], "IntervalSeconds": 10, "MaxAttempts": 3, } ], }, "PollerB":{ "Type": "Task", "Next": "Succeed", "ResultPath": "$.output.poller", "Retry":[ { "ErrorEquals": ["AsyncTaskUnfinished"], "IntervalSeconds": 5, "MaxAttempts": 100, } ], }, Parallel Start PollerA Submitter Error Handler Succeed 30s PollerB Submitter Error Handler Succeed 500s
  54. 54. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  55. 55. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Step Functions
  56. 56. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T 3 Problems Infinite Loop Dead and… End Only One Timeout 3 Solutions Retry Has Max Custom Errors Save Life Retry Makes Timeout Anywhere
  57. 57. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Better Step Functions Step Functions Better
  58. 58. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Thank you! S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. Kwunhok Chan Solutions Architect Amazon Web Services CoCo Lin Engineer Trend Micro Bruce WY Lin Senior Engineer Trend Micro
  59. 59. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

×