Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

I Love APIs 2015: Building Predictive Apps with Lamda and MicroServices

2.853 visualizaciones

Publicado el

I Love APIs 2015

Machine learning, big data, and API technologies have drastically reduced the complexity of building predictive apps. But all these advances also mean that these apps require a new approach to system architecture. This talk discusses the lamda architecture and microservices, and best practices on decomposing your app into batch, near-realtime, an real-time services. Learn how Apigee uses both new architectures to implement predictive apps using Hadoop, Node.js, Cassandra, and ElasticSearch.

Publicado en: Software
  • Sé el primero en comentar

I Love APIs 2015: Building Predictive Apps with Lamda and MicroServices

  1. 1. 1 Building predictive apps with λ & μ-services architecture# Alan Ho @karlunho Director of Product
  2. 2. Context is King 2
  3. 3. Technology Choices Today 3 Datawarehouse RDBMS J2EE AppServer
  4. 4. Lambda Architecture 4
  5. 5. What is Context ? 5 Historical # Events# Recent# Events# Now#
  6. 6. Lambda Architecture 6 Batch Storage Near RT Storage Batch Processing (MR) Near RealTime Processing Query DB Batch Layer# Serving Layer# Speed Layer# APIs Direct Mail Email Web Mobile Outreach Req / Res Processing Historical Events Recent Events
  7. 7. Netflix 7
  8. 8. •  Offline = Batch Layer •  Nearline = Speed Layer •  Online = Serving Layer
  9. 9. Batch Layer (Offline) •  Singular Value Decomposition •  Batch like operations •  High algorithm complexity •  Results Flow into Cassandra
  10. 10. Speed Layer (Near Line) 10 •  Medium request/response •  Medium data •  Medium algorithm complexity •  Intermediate caching, incremental learning algos
  11. 11. Serving layer (Online) • Fast request/response, performance is key – high availability/SLA • Fallback mechanism to Nearline/Offline operations • Low algorithm complexity
  12. 12. Final Thoughts: •  Very complete system •  Very custom and high performance •  Lots of specialized systems for specialized tasks •  Not practical for most enterprises
  13. 13. Microservices & Lambda Architecture 13
  14. 14. App Explosion 14 How can I embed context-awareness in all customer experiences?
  15. 15. Micro-services 15
  16. 16. Micro-services Best Practice 16 Isolation REST API HTTP://<Resource>/
  17. 17. Isolation at all tiers 17 Batch Storage Near RT Storage Batch Processing (MR) Near RealTime Processing Query DB APIs Direct Mail Email Web Mobile Outreach Req / Res Processing Historical Events Recent Events Batch Processing (MR) Near RealTime Processing Req / Res Processing Batch Processing (MR) Near RealTime Processing Req / Res Processing Batch Processing (MR) Near RealTime Processing Req / Res Processing Batch Storage Near RT Storage Query DB Batch Storage Near RT Storage Query DB YARN Batch Layer# Serving Layer#
  18. 18. API at all tiers 18 Batch Layer# Serving Layer# Speed Layer# Direct Mail Email Web Mobile Outreach /Scores# /Counts# /Recommendations# Historical Events Recent Events
  19. 19. Real-time Scoring for Recommendations 19
  20. 20. Recommendations Example 20
  21. 21. Real-time Scoring Architecture 21 Hadoop In-Memory Insights (GRASP) Node.js Node.js API BaaS (APIs) Direct Mail Email Web Mobile Outreach (Batch Scores) (Activities / Counters) (Query) Historical Events Realtime Events (Context) Backend (Mashup) 3 4 5 1 2 6 Node.js
  22. 22. Batch Layer •  Uses R for building predictive models •  Algorithms based on Event Sequences •  Lots and lots of tooling 22 Hadoop Insights (GRASP) API BaaS (Batch Scores) 1 Apigee UI
  23. 23. 23 R Code#
  24. 24. 24 Scored Export to Serving Layer#
  25. 25. Speed Layer •  Easy to program via node.js •  Low algorithm complexity – Storm is overkill for most Apigee customers •  Simple aggregates via C* counters good enough •  Good for capturing locations, activity stream, etc. 25 Node.js API BaaS Apigee UI
  26. 26. BaaS Node.js Counter Example var event = new Usergrid.Counter(event_options, function (error, result) { … }); // Then we POST the event to increment the counter var counter_options = { name: productName + ’.add_to_cart', value: 1 } event.increment (counter_options, function (error, result) 26 curl -X GET https://api.usergrid.com/my-org/my-app/counters? counter=appleIpad3.add_to_cart Input code: Output code:#
  27. 27. Microservices Layer •  API Layer (Swagger) •  Context Injection •  Service Orchestration •  Fallback logic 27 Node.js Node.js API BaaS Direct Mail Email Web Mobile Outreach (Context) Backend
  28. 28. Swagger Based API Design 28
  29. 29. Node.js Code 29
  30. 30. APIs are Everywhere 30 Hadoop In-Memory Insights Node.js Node.js API BaaS Direct Mail Email Web Mobile Outreach Historical Events Realtime Events Backend Hive RHadoop Node.js Storm Spark
  31. 31. Conclusion •  Lambda architecture is a great way to combine realtime & batch processing •  Microservices complements Lambda architecture for loosely-coupled systems 31 ©2015 Apigee. All Rights Reserved.

×