True Reusable Code - DevSum2016
•Descargar como PPTX, PDF•
1 recomendación•329 vistas
This presentation covers architectural principles for Software defined "Everything", Microservices - their impact on Azure, a Geo-Spatial Fleet analysis using Spark and HDFS on ESRI and FlowBasedProgramming
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a True Reusable Code - DevSum2016
Similar a True Reusable Code - DevSum2016 (20)
Último
Último (20)
True Reusable Code - DevSum2016
- 2. Who am I ? Sr. Solution Consultant with LastMileLink Technologies, a CitySprint UK Innovation Lab, the leading Same Day UK network, often cited as amongst top 5 in Europe. : #Lazaredd :https://uk.linkedin.com/in/lazared
- 3. Process Flow. Topics • Software Defined “Everything” • Geo-Spatial Fleet Analysis using Big-Data • Truly Reusable Code – FlowBaseProgramming
- 5. Community Network Marketplace Technology Infrastructure Data Source: Sangeet Paul Chaudry , http://platformed.info/platform-stack/ Community, Network, Marketplace Technology Infrastructure Data Community Network Marketplace Technology Infrastructure Data Defunct or Trailing Hanging On Or Declining Thriving & Growing Leveraging Data produced by Community, Network & Marketplace is Key to Platform Success. From Pipes to Platforms & API Economy
- 6. Platform Security – Emphasis on Data
- 7. Platform Security – Data Component
- 9. Process Flow. “Software is Eating the World” Image : Pivotal Marc Andreessen , WSJ 2011
- 10. Cloud Natives $6B $50B $41B $25B $33.5B
- 11. 11 SPEED UBIQUITY* SCALE SAFETY (MOBILE and APIs Four key patterns
- 12. 12 Day One Day Two and Beyond Deliver Continuously DevSecOps and Rugged How we do it
- 13. 13 Build Reliable Systems from Unreliable Components Invite Chaos
- 14. 14 What a Cloud Native is NOT: •Just Configuring Infrastructure •Just Orchestrating Containers •Composing Distributed Systems •Supporting Ad-Hoc General Purpose Automation
- 15. Conway’s Law. Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization's communication structure. Melvyn Conway, 1967
- 16. Invoke the Inverse Conway Maneuver
- 17. Breaking the Monolith & Scale
- 18. How deep is deep enough?
- 19. An example - Mantl.io Image Source: Mantle.io
- 20. Infrastructure Metrics, Monitoring and Microservices Simulation Image Source: Instana, Image: Spigo and SIMIANVIZ https://github.com/adrianco/spigo
- 22. Omakase omakase əʊməˈkaseɪ,əʊˈmakəseɪ/ noun 1.(in a Japanese restaurant) a type of meal consisting of dishes selected by the chef. 2."we had the five-course omakase" Source:Wikipedia “ Rails is omakase (DHH) - David Heinemeier Hansson “
- 24. Zeta Architecture - Technologies That Work • Dynamic compute resources • Common storage platform • Real-time application support • Flexible programming models • Deployment management • Solution based approach • Applications to operate a business * This is a pluggable architecture Image Source: courtesy of Jim Scott @ MapR
- 27. Process Flow. Blue signals a pick-up Red signals a drop-off Sample of how one driver’s journey looks like Used for: • Viewing the base unit of analysis
- 28. Process Flow. Demand heat map
- 29. Demand heat map Heat map of pickup locations density Used for: • Optimising resource allocation • Identifying areas for potential expansion
- 30. Process Flow.K-means clustering analysis – 40 centres
- 31. K-means clustering analysis – 40 centres Employed the K-means algorithm to identify clusters of pickup points Used for: • Validating against current service centres map • Identifying areas for potential expansion
- 32. Process Flow. K-means 100 centres Can we be better placed ? Where can we plan the next M&A?
- 33. K-means 100 centres Higher granularity clustering Used for: • Assessing the frequency of pickups for micro- clusters (e.g. villages, neighbourhoods) • Directing drivers to hotter waiting areas
- 34. Process Flow.Geographical supply & demand
- 35. Geographical supply & demand Pickup locations shown vs to routes Used for: • Improving likelihood of parcel pickup while on-route
- 36. Demand variation across time
- 37. 0.0 4.5 9.0 13.5 18.0 0 3 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Expectedparcels Time of day Expected parcels allocated to cluster 41 (Stevenage) Demand variation across time Used for: • Positioning couriers in the right place at the right time For each demand cluster we calculated the frequency of pickups per hour
- 38. Solution Summary
- 39. The solution outline • Data science capabilities of Spark, easy to use with SQL knowledge • Map plotting on Esri– heat mapping, zoom in/out capabilities, real-time • High-performance due to in-memory processing capabilities of Spark • Can work with large data sets due to high performance disk-based data access in Hadoop File System (HDFS) • Can import data from EDW • All delivered with Bigstep on their high performance Full Metal Cloud
- 40. IaaS-Paas Shout out to Bigstep
- 41. Process Flow. Heavy Lifting done by: Shout out to Bigstep
- 42. Process Flow. Shout out to Bigstep 84 Lines of Scala code in Spark
- 44. DevSum 2016 Stockholm Please vote for this session on DevSum App
Notas del editor
- Paraphrasing Commodus in “Gladiator”, who am I ?The general who became a slave. The slave who became a “Gladiator”. The gladiator who defied an emperor. Striking story!
- I would like to touch on the following topics as I feel what happens these days is fundamentaly influencing the way we think , operate and deploy.
- Simon Sinek - Ted talk.
- Big shout out to Sangeet Paul- Platform Revolution for encoding the Platform Blueprint What I want to emphasize here is that all 3 examples have the same technology and KNOWING the “WHY” they support and enble INTERACTION and VALUE creation and Exchange.
- GDPR
- Data In Motion , Data at Rest and all other aspects considering all it’s relationships with the other platform components.
- We identified four key patterns. Speed, safety, scale, and what I’m now calling “ubiquity.” I used to call this mobile, but what I’m really trying to highlight is the idea that anybody, anywhere, can at anytime interact with your services. Speed is obvious - we can innovate, experiment, deliver value quickly. Safety balances speed with the simultaneous ability to maintain stability, availability, and durability. And scale refers to our ability to elastically respond to changes in demand.
- Deliver Continuously using CI and CD. You see, it’s not enough to deploy something, that’s just Day One. We also have to deploy again, and again, and again. And when you do that, you can’t break stuff. That’s day two…and beyond.
- And so we have to figure out how we can build reliable systems from unreliable components. Meaning that we MUST invite CHAOS into our daily grind. There are many way we can introduce chaos , Simian Army , Gauntl…, the main benefit being: operating at scale without disruption in a distributed manner. Throught the young history of our industry, we have formulated and followed various methodologies like Waterfall ( represented by Monoliths, Paralysis), Agile (N-tier , Sprints ) and today Continuous Delivery (Micro-Sprints , daily deployments – enabled by Microservices, CI+CD, Automation/Orchestration, attributes of a Cloud native Company)
- Cloud Native does not mean build a massive DevOps team and tell them to go configure all of the infrastructure, orchestrate all of the containers, compose the distributed systems, and generally support any kind of ad-hoc, general purpose automation you might dream up, like configuring load balancers or creating NFS mounts. Cloud Native definition – reword !!!
- Conway said you can’t help but reproduce in your architecture the structure of your organization. So now you’ve freed up brainpower to think about where you are and where you need to go.
- Define your architecture and then morph your organization. You then have the ability to invoke the “Inverse Conway Maneuver,” taking steps like Netflix did. Once you decide the architecture that your system needs to have, you can restructure your organization to look like that architecture, and according to Conway, that architecture will emerge. Quote alistair cockburn Hexangonal Architecture which paved the way towards Microservices and Continuous Delivery
- The Scale Cube Define your architecture and then morph your organization. You have the ability to invoke the “Inverse Conway Maneuver,” taking steps like Netflix did. Once you decide the architecture that your system needs to have, you can restructure your organization to look like that architecture, and according to Conway, that architecture will emerge.
- Why do we do it this particular way, where should we focus our development effort-is it IP-able , ROI ? , any silver bulltes to watch out for when we consider COTS solutions for parts of our platform.
- Mantl.io from Cisco DevOps, It’s all about agility, so we’re told; Get your application up fast, be the first, pivot quickly, improve often. But when you're deploying microservices, you'll soon run into problems. How to describe a deployment? Where should you run which service? How do you connect them together? Where should you store secrets? Where should the logs go? These questions call for a new kind of stack and methology, aka “Continuos Delivery”.
- Modern Microservice and Process/Container based applications have attributes that introduce challenges for operating and monitoring these environments: Scale Modern applications can be constructed out of hundreds/thousands µServices and containers which are loosely coupled but communicate with each other. These applications can dynamically scale up and down depending on the load of the system and can spread all over the world in cloud datacenters and availability zones. Speed µServices can be deployed independently from each other and are build for Continuous Delivery processes, so that changes are happening at a high frequency. Steadiness µServices are not steady anymore. They can be started and stopped just for a functional call and containers are moved around by tools like Kubernetes or Mesos to utilize the hardware in a better way. This requires that monitoring tools identify these services and understand their role in the overall context – without relying on manual tagging.
- Golden Copy , Notary for example – Fingerprinted Images , Redeploy, forensics, learn, fix , Rinse and Repeat
- So enter Omakase - This is not A La Carte - Rails for exmaple is Omakase because every component was carefully selected by the team who built it however this is not stopping people using their own.
- <RIFF ON SPRING CLOUD> About opinionated expression of distributed systems patterns. Drawing from existing implementations.
- These technology concepts are now mature enough that they will stick around and the specific implementation is flexible. Hexagonal – Alistair Cockborn
- But what about polyglot programming? Isn’t one of the promises of microservices that everyone can write their services in whatever languages they choose? Sure. But polyglot is a RED HERRING. Polyglot is not what makes you productive. Polyglot actually makes it harder to enforce the simple rules and explicit contracts that make the platform so powerful. Is it impossible? No. But there’s definitely a cost to providing the same omakase experience to every taste in programming language. Polyglot Using the right tool for the problem is one of the paradigms of modern applications. This leads to multiple languages, frameworks and even persistence models for an application. In particular, new distributed caching and database models are getting adopted quickly.
- Shows the hot points of pickup points along the uk. A good overview of the overall dataset.
- Compared against our service center locations it shows a few differences. A clustering algorithm identifies ‘clusters’ of elements by it’s own. K-means needs to be told how many clusters to look for.
- This is what happens if we tell k-means to split the dataset into 100 hot locations.
- The blue dots are actual gps information of en-route drivers. Shows typical routes but only some routes go through hot areas.
- A ‘cluster’ timetable is used to predict demand at a particular cluster on a particular time. Useful to instruct the driver if he is to stay or to go to it’s destination. This can help uberize the business.
- Used a combination of technologies, mostly Spark on Hadoop on Bigstep. Imported data from production Postgres DB via Sqoop into avro and from there via spark into varous CSV files rendered by the ESRI (ARCGIS). Postgres concentrates information from mobile devices.