I Love APIs Europe 2015: Developer Sessions
•
2 likes•1,727 views
The document discusses building predictive applications using a lambda architecture with batch, speed, and serving layers to handle both historical and real-time data. It provides examples of how Netflix uses this architecture with offline, nearline, and online layers. Finally, it advocates for building applications as microservices with APIs at each tier for isolation, scalability, and independent development.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to I Love APIs Europe 2015: Developer Sessions
Similar to I Love APIs Europe 2015: Developer Sessions (20)
More from Apigee | Google Cloud
More from Apigee | Google Cloud (20)
Recently uploaded
Recently uploaded (20)
I Love APIs Europe 2015: Developer Sessions
- 1. Vibrant Ecosystems: Adoption in Private or Partner Developer Programs Kay Lummitsch, API Technology Evangelist - Swisscom! Steve Rowland, SVP WW Sales & Operations - Apigee!
- 2. Building predictive apps with λ & μ-services architecture Alan Ho!
- 4. Context is King Apps 4
- 5. Technology Choices Today 5 Datawarehouse RDBMS J2EE AppServer
- 6. What is Context ? 6 Historical ! Events! Recent! Events! Now!
- 7. Lambda Architecture 7 Batch Storage Near RT Storage Batch Processing (MR) Near RealTime Processing Query DB Batch Layer! Serving Layer! Speed Layer! APIs Direct Mail Email Web Mobile Outreach Req / Res Processing Historical Events Recent Events
- 9. • Offline = Batch Layer • Nearline = Speed Layer • Online = Serving Layer
- 10. Batch Layer (Offline) • Singular Value Decomposition • Batch like operations • High algorithm complexity • Results Flow into Cassandra
- 11. Speed Layer (Near Line) 11 • Medium request/response! • Medium data! • Medium algorithm complexity! • Intermediate caching, incremental learning algos!
- 12. Serving layer (Online) • Fast request/response, performance is key – high availability/SLA • Fallback mechanism to Nearline/Offline operations • Low algorithm complexity
- 13. Final Thoughts:! ! • Very complete system! • Very custom and high performance! • Lots of specialized systems for specialized tasks! • Not practical for most enterprises!
- 14. Data Scientists & Developers 14 Direct Mail Email Web Mobile Outreach How can they work independently ?
- 15. App Explosion 15 How can I embed predictive in all apps ?
- 17. Micro-services Best Practice 17 Isolation REST API HTTP://<Resource>/
- 18. Isolation at all tiers 18 Batch Storage Near RT Storage Batch Processing (MR) Near RealTime Processing Query DB Serving Layer! APIs Direct Mail Email Web Mobile Outreach Req / Res Processing Historical Events Recent Events Batch Processing (MR) Near RealTime Processing Req / Res Processing Batch Processing (MR) Near RealTime Processing Req / Res Processing Batch Processing (MR) Near RealTime Processing Req / Res Processing Batch Storage Near RT Storage Query DB Batch Storage Near RT Storage Query DB
- 19. API at all tiers 19 Batch Layer! Serving Layer! Speed Layer! Direct Mail Email Web Mobile Outreach /Scores! /Counts! /Recommendations! Historical Events Recent Events
- 22. Recommendation Architecture 22 Hadoop In-Memory Insights (GRASP) Node.js Node.js API BaaS (APIs) Direct Mail Email Web Mobile Outreach (Batch Scores) (Counters) (Query) Historical Events Realtime Events (Context) Backend (Mashup) 3 4 5 1 2 6 Node.js
- 23. Batch Layer • Uses R for building predictive models • Algorithms based on Event Sequences • Lots and lots of tooling 23 Hadoop Insights (GRASP) API BaaS (Batch Scores) 1 Apigee UI
- 24. 24 R Code!
- 25. 25 Scored Export to Serving Layer!
- 26. Speed Layer • Easy to program via node.js • Low algorithm complexity – Storm is overkill for most Apigee customers • Simple aggregates via C* counters good enough • Good for capturing locations, activity stream, etc. 26 Node.js API BaaS Apigee UI
- 27. BaaS Node.js Counter Example var event = new Usergrid.Counter(event_options, function (error, result) { … }); // Then we POST the event to increment the counter var counter_options = { name: productName + ’.add_to_cart', value: 1 } event.increment (counter_options, function (error, result) 27 curl -X GET https://api.usergrid.com/my-org/my-app/counters? counter=appleIpad3.add_to_cart Input code: Output code:!
- 28. Microservices Layer • API Layer (Swagger) • Context Injection • Service Orchestration • Fallback logic 28 Node.js Node.js API BaaS Direct Mail Email Web Mobile Outreach (Context) Backend
- 29. Swagger Based API Design 29
- 30. Node.js Code 30
- 31. APIs are Everywhere 31 Hadoop In-Memory Insights Node.js Node.jsAPI BaaS Direct Mail Email Web Mobile Outreach Historical Events Realtime Events Backend Hive RHadoop Node.js Storm Spark
- 32. Thank you 32
- 33. LUNCH General Hall / Main Concourse (Sessions resume at 14:00)
- 34. Model-Driven APIs with 127 Greg Brail!
- 35. Overview • A bit about API design • Some ways to turn APIs into code • A different approach • Introducing Apigee 127 35
- 36. API Design 36
- 37. REST 37
- 38. API Design is Important • We have been talking API design for a long time • We have an approach that we like and often recommend • We believe that it is important to design APIs carefully, up front
- 39. My Perspective on API Design • URIs are important • Verbs should be used in a consistent way • Follow common usage • Links can be helpful 39
- 40. 40
- 41. Turning Designs into APIs 41
- 42. An API Design Story • I designed a lovely, consistent API – In a Google Doc • The team implemented it – In Java • I verified that the implementation matched the doc – Using my eyeballs • We wrote docs – Using information from both places
- 43. There must be a better way… 43
- 44. Modern Web / API Frameworks • Write code • Get an API up quickly • API design and code are the same • Examples: – Express for Node.js – Ruby on Rails – Lots of others 44
- 45. Another approach: Code First 45 @Path("/my-resource") @Api(value="/my-resource”, description="Rest api for do operations on admin", produces=MediaType.APPLICATION_JSON) @Produces({ MediaType.APPLICATION_JSON }) class MyResource{ @ApiOperation(value = "Get specific element", httpMethod = "GET", notes = "Fetch the selement of the collection", response = Response.class) @ApiResponses(value = { @ApiResponse(code = 200, message = ”Element found"), @ApiResponse(code = 404, message = “Element not found"), @ApiResponse(code = 500, message = “Server error due to encoding"), @ApiResponse(code = 400, message = "Bad request: decoding error"), @ApiResponse(code = 412, message = ”Prereq: Required data not found") }) public Response get( @ApiParam(value = "UUID of the element", required = true) @PathParam("uuid") String uuid) { Generate
- 47. Why Can’t We Do This? 1. Design the API 2. Annotate it with policies 3. Wire it to some back end 4. Run it
- 48. Swagger 2.0 • Swagger 2.0 • Describes the shape of the API – URIs – Verbs – Query parameters – Headers – Request and response formats 48
- 49. Building a Model in Swagger 2.0 • Specify API design using YAML – URI Paths – Verbs – Parameters – Request format (optional) – Response format (optional) – Error responses (optional) 49
- 50. Plugging in Code • Point each API operation to some code – In our project it is a Node.js module 50
- 51. Annotating the Model • The API definition can have “policies” – OAuth – Quota – Spike arresting – Caching • Each operation can have parameters – Quota – OAuth scope – Etc. 51
- 53. Introduction to 127 $ npm install -g apigee-127 http://github.com/apigee-127
- 54. Enterprise APIs, Fast We Handle These: • Swagger Tools – Message validation & routing – Authorization (OAuth 2.0) • Volos.js – Caching (Apigee Swagger 2.0 Extension) – Quota (Apigee Swagger 2.0 Extension) – Analytics (Apigee Swagger 2.0 Extension) Developers Handle This: • Business logic 54 Controller Controller.js
- 55. Design-First Approach 55 Add Annotations Implement Controllers Add Annotations • Business Logic • Policies: • Authorization (OAuth) • Caching • Quota • Spike Arrest • Analytics Implement Controllers • Controllers in JavaScript via Node.js • Future ideas: • Declarative pipelines • Other languages
- 56. How it Works 56 • No code generation • Model is built at start-up time • A127 middleware applied as defined in Swagger: – Caching, Quota, OAuth, Spike Arrest, Analytics, etc. • Paths and parameters • Model Definitions • Vendor Extensions (middleware) Interprets Swagger
- 57. Execution Options 57 Anywhere / PaaS 127 On-Premises 127 On Edge 127 Anywhere / PaaS 127 On-Premises + Edge 127 127
- 58. Thank you
- 59. APIs and Internet of Things Ed Anuff VP Product Strategy Apigee John Calagaz, Chief Technology Officer Centralite
- 60. IoT is more than sensors, beacons, Arduinos, and Raspberry Pi’s… 60
- 61. App to Device Connection - Smartphone & Tire Sensor
- 62. Device To Device Connection - Tire Sensor & Air Compressor 62 AIR $ 1
- 63. The “I” in IoT is for Internet 63 AIR $ 1
- 64. Why does this matter? 64
- 65. IoT Seeks Network Value Metcalfe’s Law Value = O(n2)
- 66. Why is this hard? 66
- 67. Building The Network Via The Ecosystem 67
- 68. The number of connected devices is set to dramatically increase 68
- 69. Too Many Devices, Too Few Developers • Connected Devices • Software Developers • 2013: 6 Billion • 2018: 20 Billion • 333% Increase Source: Business Insider Intelligence - The Internet Of Everything: 2014 • 2013: 18 Million • 2019: 26 Million • 45% Increase Source: Evans Data Corporation - Global Developer Population and Demographic Study 2013
- 70. MISSING: 34 Million Software Developers 70
- 71. This is a scale problem 71
- 72. IoT systems are a complex combination of web, app and embedded systems.
- 74. What happens when development is so fragmented? 74
- 75. The Promise: Everything is Connected
- 78. Today: 78 • Multiple Experiences • Multiple Apps • Multiple Developers • Multiple Ecosystems & Platforms • Multiple Devices
- 79. This is a scale problem 79
- 80. Things Go from Bad app app app app
- 81. To Worse app app app appapp app app 72 7272 app cloud
- 82. How are we going to fix this? 82
- 84. The usual suspects… (And countless others, I just got lazy building this slide…)
- 85. NO! 85
- 86. The platforms can’t save us… (Again, but your own favorite IoT, OS, or mobile platform here, still wont save you…)
- 87. If only there were something like the web, but for apps and devices… 87
- 88. …so that developers could work together to make apps, devices, and systems interoperate… 88
- 90. Opportunity: Internet of Things ecosystems 90 app app app app app app app Powered by APIs Digital Experiences Business Ecosystems
- 91. A Day in the Life of an Embedded IoT Architect
- 92. Centralite: Who are we? 92
- 93. The Problem
- 102. 5-24 months
- 103. 5-24 months As fast as I can!
- 105. Q & A
- 106. Thank you
- 107. Afternoon Break General Hall / Main Concourse (Sessions resume at 16:00)
- 108. Management API’s Michel Burger Apigee Herve Poupon Orange Peter Dunmore TMF
- 109. 1. From Best Effort to Operation Excellence 109 2015 Apigee Confidential 2. How to quickly build or easily participate to a Digital Service Ecosystem
- 110. So you having fun developing solutions using many API’s 110 2015 Apigee Confidential
- 111. And this happens…. 111 2015 Apigee Confidential
- 112. Or this…. 112 2015 Apigee Confidential
- 113. Or this…. 113 2015 Apigee Confidential
- 114. And your solution starts to behave badly… 114 2015 Apigee Confidential
- 115. Will it be nice to be informed that something is going wrong your the solution before using it? 115 2015 Apigee Confidential
- 116. No need to reinvent the wheel 116 2015 Apigee Confidential • Fault • Configuration • Accounting • Performance • Security • Simple Management • Service Level Agreement • Performance Management
- 117. Going from Best Effort to Operational Excellence 117 2015 Apigee Confidential
- 118. Many service providers are building digital service ecosystems leveraging their core assets 118 2015 Apigee Confidential Identity Powered by
- 119. Your journey in an Digital Service Ecosystem 119 2015 Apigee Confidential Product Definition Product Offering Product Ordering/ Activation Product Usage and Mediation Billing Invoicing
- 120. • Trouble ticketing • Catalog Management • Product Ordering • DSRA No need to reinvent the wheel 120 2015 Apigee Confidential • Common Service API’s • Based on Open Standards UnifiedProfileMgmt UnifiedIdentityMgmt ConsistentOperation Mgmt EventuallyConsistent OperationMgmt Analytics Commerce API Services
- 121. Creating Digital Service Ecosystem by quickly on-boarding independent services 121 2015 Apigee Confidential
- 122. Management infrastructure needs to be simple and easy to use 122 2015 Apigee Confidential Focusing on API
- 123. Management API’s: are they .. 123 2015 Apigee Confidential Pain killers Vitamins vs
- 124. Q & A
- 125. Thank you
- 126. All about that BaaS To build enterprise apps! !
- 127. @karlunho Alan Ho
- 128. Evolving Digital Architecture 128 ERP Product CatalogsCRM Data Lake Data Warehouse Systems of Engagement Partner Apps Employee Apps Consumer Apps Insights Systems of Record
- 129. Trend 1 : Application Explosion 129
- 130. Data De-normalization per interaction 130 ERP Product CatalogsCRM Data Lake Data Warehouse
- 131. Trend 2 : Micro-services & Decentralization 131
- 132. Micro-services Types 132 Business Logic Services Data Services
- 133. Microservices Best Practice 133 Isolation REST API HTTP://<Resource>/
- 134. Building Business Logic Services 134 Virtualization Light-weight Scripting +
- 135. Building Data Services Approach 135 Virtualization Light-weight Scripting + + NoSQL
- 136. Scaling becomes very hard 136 X X
- 137. BaaS : Rethinking of Multi-tenancy 137 Shared Schema Shared KeyspaceSeparate Schema Separate Keyspace + Isolation - Scalability - Scalability + Isolation - Management + Isolation + Management + Scalability + Isolation - Management + Scalability + Management
- 141. Scalable Persistence 141 SQL JOINS “These are not the indexes you are looking for…” -Obi-Wan K.
- 143. Graph Data Persistence: Entities & Connections 143 User A Product A User B User C Likes>> <<Liking
- 144. API-Driven : REST Semantics GET PUT POST DELETE 144
- 145. Filters / Limits / Cursors GET http://api.usergrid.com/MyOrg/MyApp/ users?ql=select * where name = ’John*'&limit=50&cursor=LTIxNDg0NDUxNDpn R2tBQVFFQWdITUFDWFJ2YlM1emJXbDBhQUNBZFFB UUQyMVZneExfRWVLRlV3TG9Hc1doZXdDQWRRQVFI YVdjb0JwREVlS1VCd0xvR3NWT0JRQQ ’ 145
- 147. API BaaS – Users, Roles & Security 147
- 149. Location Queries GET /stores?ql=location within 500 of 40.042016, -86.900749
- 151. Push Notifications
- 153. Integrated Application Performance Management 153 Errors & Crashes API Performance
- 154. Remote Application Configuration 154 Monitoring Config App Config A/B Testing
- 155. Demo 155
- 156. Thank you 156