Scaling infrastructure beyond containers

•

1 recomendación•794 vistas

These are the slides from talk given by Mateusz Gajewski at AWS UG meeting in Warsaw. Content: Apache Mesos, Cluster Scheduling, Hybrid Environment, Scalability, Fault Tolerance.

Tecnología

@wendigo
Mateusz „Serafin” Gajewski • AWS UG Meetup
Scaling infrastructure beyond containers

@wendigo
Agenda
• Evolution of infrastructure at Allegro,
• Why Apache Mesos™?
• Apache Mesos key concepts,
• Future of datacenter and cloud computing?

@wendigo
History of scaling
infrastructure @ Allegro

@wendigo
Web Scale Resource management
100s dots 100s dots

@wendigo
Infrastructure 2.0
1000s dots another 1000s dots

@wendigo
Infrastructure 2.1
1000s dots another 1000s dots

@wendigo
Challenges
• cloud not used as cloud ;),
• high cost of virtualization,
• effective resource utilization,
• microservice architecture,
• spread of new technologies,
• heterogenous resources,
• scalability, fault tolerance & HA,
• performance isolation,
• data processing at scale

@wendigo
Infrastructure 3.0
A Platform for Fine-Grained
Resource Sharing in the Data Center

@wendigo
Mesos HA
• master election/failover with ZooKeeper,
• master maintains soft-state,
• framework state reconciliation,
• slave checkpointing,
• slave recovery,
• framework checkpointing

@wendigo
Beyond offers
• offer filters (constraints),
• static (pre-startup) reservations,
• dynamic (post-startup) reservations,
• oversubscription,
• persistent volumes,
• pluggable allocator scheduling policy (fair, priority based)

@wendigo
Mesos recap
• „programming against the datacenter",
• distributed datacenter kernel,
• two-level multi-resource scheduler,
• scalable, highly-available & fault-tolerant,
• performance isolation with containers,
• exposes homogeneous resources,
• elastic, dynamic partitioning,
• high resource utilization

@wendigo
Google’s Omega
source: Omega: flexible, scalable schedulers for large compute clusters

http://meetup.com/allegro.tech
http://allegro.tech
@AllegroTechBlog
Work with us

Más contenido relacionado

La actualidad más candente

Improving CloudStack for operatorsShapeBlue

Leveraging OpenStack at Scale: How the Elastic Cloud Drives Innovation VelocityTesora

OpenStack in the EnterpriseTesora

CloudStack EU user group - fast SAP provisioningShapeBlue

OpenStack Management at HyperscaleTesora

Fast SAP system provisioning based on CloudStack ShapeBlue

Brendon Foxen (Channel 4) - Speeding up Software Delivery at Channel 4Outlyer

Securing your Cloud Environment v2ShapeBlue

56k.cloud intro and pitch deckBrian Christner

Datadog- Monitoring In Motion Cloud Native Apps SF

Microservices with Spring Cloud, Netflix OSS and Kubernetes Christian Posta

Cloudstack: the best kept secret in the cloudShapeBlue

Running OpenStack in ProductionTesora

Owain Perry (Just Giving) - Continuous Delivery of Windows Micro-Services in ...Outlyer

Microservices, DevOps, and Containers with OpenShift and Fabric8Christian Posta

Netflix Cloud Platform and Open Sourceaspyker

容器革命的「利」與「必」 inwin stack

Spring Cloud Netflix OSSSteve Hall

Atlanta Microservices Day: Istio Service MeshChristian Posta

Microservices with Spring CloudDaniel Eichten

La actualidad más candente (20)

Improving CloudStack for operators

Leveraging OpenStack at Scale: How the Elastic Cloud Drives Innovation Velocity

OpenStack in the Enterprise

CloudStack EU user group - fast SAP provisioning

OpenStack Management at Hyperscale

Fast SAP system provisioning based on CloudStack

Brendon Foxen (Channel 4) - Speeding up Software Delivery at Channel 4

Securing your Cloud Environment v2

56k.cloud intro and pitch deck

Datadog- Monitoring In Motion

Microservices with Spring Cloud, Netflix OSS and Kubernetes

Cloudstack: the best kept secret in the cloud

Running OpenStack in Production

Owain Perry (Just Giving) - Continuous Delivery of Windows Micro-Services in ...

Microservices, DevOps, and Containers with OpenShift and Fabric8

Netflix Cloud Platform and Open Source

容器革命的「利」與「必」

Spring Cloud Netflix OSS

Atlanta Microservices Day: Istio Service Mesh

Microservices with Spring Cloud

Destacado

allegro.tech Data Science Meetup #2: Elasticsearch w praktyceallegro.tech

JDD 2014: Adam Dubiel - Import allegro.tech.internal.*allegro.tech

Fighting with scaleallegro.tech

RxJava & Hystrix - Perfect match for distributed applicationsallegro.tech

RxJava - introduction & designallegro.tech

Confitura 2015 - Mikrousługi nie lubią być samotneallegro.tech

[WHUG] Wielki brat patrzy - czyli jak zbieramy dane o użytkownikach allegroallegro.tech

Destacado (7)

allegro.tech Data Science Meetup #2: Elasticsearch w praktyce

JDD 2014: Adam Dubiel - Import allegro.tech.internal.*

Fighting with scale

RxJava & Hystrix - Perfect match for distributed applications

RxJava - introduction & design

Confitura 2015 - Mikrousługi nie lubią być samotne

[WHUG] Wielki brat patrzy - czyli jak zbieramy dane o użytkownikach allegro

Similar a Scaling infrastructure beyond containers

[DSC DACH 23] The Modern Data Stack - Bogdan PirvuDataScienceConferenc1

Accelerate DevOps/Microservices and KubernetesRick Hightower

Microservices in the Enterprise Jesus Rodriguez

Clould Computing and its application in LibrariesAmit Shaw

Webcast: DevOps in AWS is different! How can containers help? Applatix

Tackling complexity in giant systems: approaches from several cloud providersPatrick Chanezon

On CloudStack, Docker, Kubernetes, and Big Data…Oh my ! By Sebastien Goasguen...Radhika Puthiyetath

Meetup #3: Migrate a fast scale system to AWSAWS Vietnam Community

Microservices and serverless for MegaStartups - DLD TLV 2017Boaz Ziniman

John Willis Cc Use CasesGovCloud Network

Harness the Power of Hybrid Cloud with AWS and AvereAmazon Web Services

2014.10.22 Building Azure Solutions with Office 365Marco Parenzan

Introducing to serverless computing and AWS lambda - Israel Clouds MeetupBoaz Ziniman

Amazon AWS vs Azure Cloud vs KubernetesStridely Solutions

CHAPTER 2 cloud computing technology in csTSha7

Impact of cloud services on the work of oracle technology expertsAndreas Chatziantoniou

Spca2014 7 tenets of highly scalable applications kapicNCCOMMS

Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...SnapLogic

Yow Conference Dec 2013 Netflix Workshop Slides with NotesAdrian Cockcroft

Similar a Scaling infrastructure beyond containers (20)

[DSC DACH 23] The Modern Data Stack - Bogdan Pirvu

Accelerate DevOps/Microservices and Kubernetes

Microservices in the Enterprise

Clould Computing and its application in Libraries

Webcast: DevOps in AWS is different! How can containers help?

Tackling complexity in giant systems: approaches from several cloud providers

On CloudStack, Docker, Kubernetes, and Big Data…Oh my ! By Sebastien Goasguen...

Meetup #3: Migrate a fast scale system to AWS

Microservices and serverless for MegaStartups - DLD TLV 2017

John Willis Cc Use Cases

Harness the Power of Hybrid Cloud with AWS and Avere

2014.10.22 Building Azure Solutions with Office 365

Introducing to serverless computing and AWS lambda - Israel Clouds Meetup

Amazon AWS vs Azure Cloud vs Kubernetes

CHAPTER 2 cloud computing technology in cs

Impact of cloud services on the work of oracle technology experts

Spca2014 7 tenets of highly scalable applications kapic

Weathering the Data Storm – How SnapLogic and AWS Deliver Analytics in the Cl...

Yow Conference Dec 2013 Netflix Workshop Slides with Notes

Último

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...apidays

2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong

Presentation on how to chat with PDF using ChatGPT code interpreternaman860154

Finology Group – Insurtech Innovation Award 2024The Digital Insurer

Factors to Consider When Choosing Accounts Payable Services Providers.pptxKatpro Technologies

Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko

Driving Behavioral Change for Information Management through Data-Driven Gree...Enterprise Knowledge

Understanding Discord NSFW Servers A Guide for Responsible Users.pdfUK Journal

What Are The Drone Anti-jamming Systems Technology?Antenna Manufacturer Coco

Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia

Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun

TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc

04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG

08448380779 Call Girls In Greater Kailash - I Women Seeking MenDelhi Call girls

Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC

Real Time Object Detection Using Open CVKhem

Slack Application Development 101 Slidespraypatel2

Scaling API-first – The story of a global engineering organizationRadu Cotescu

CNv6 Instructor Chapter 6 Quality of Servicegiselly40

GenCyber Cyber Security Day PresentationMichael W. Hawkins

Scaling infrastructure beyond containers

1. @wendigo Mateusz „Serafin” Gajewski • AWS UG Meetup Scaling infrastructure beyond containers

2. @wendigo Agenda • Evolution of infrastructure at Allegro, • Why Apache Mesos™? • Apache Mesos key concepts, • Future of datacenter and cloud computing?

3. @wendigo History of scaling infrastructure @ Allegro

4. @wendigo Infrastructure 1.0

5. @wendigo Job allocation problem

6. @wendigo Web Scale Resource management 100s dots 100s dots

7. @wendigo Infrastructure 2.0 1000s dots another 1000s dots

8. @wendigo Infrastructure 2.1 1000s dots another 1000s dots

9. @wendigo Challenges • cloud not used as cloud ;), • high cost of virtualization, • effective resource utilization, • microservice architecture, • spread of new technologies, • heterogenous resources, • scalability, fault tolerance & HA, • performance isolation, • data processing at scale

10. @wendigo Beyond cloud computing

11. @wendigo Holly Grail of TCO

12. @wendigo Infrastructure 3.0 A Platform for Fine-Grained Resource Sharing in the Data Center

13. @wendigo Scheduling

14. @wendigo Cluster scheduling

15. @wendigo Mesos architecture

16. @wendigo Mesos frameworks

17. @wendigo Offers

18. @wendigo Execution isolation

19. @wendigo External Containerizers

20. @wendigo Mesos HA • master election/failover with ZooKeeper, • master maintains soft-state, • framework state reconciliation, • slave checkpointing, • slave recovery, • framework checkpointing

21. @wendigo Beyond offers • offer filters (constraints), • static (pre-startup) reservations, • dynamic (post-startup) reservations, • oversubscription, • persistent volumes, • pluggable allocator scheduling policy (fair, priority based)

22. @wendigo Mesos frameworks

23.

24. @wendigo Mesos recap • „programming against the datacenter", • distributed datacenter kernel, • two-level multi-resource scheduler, • scalable, highly-available & fault-tolerant, • performance isolation with containers, • exposes homogeneous resources, • elastic, dynamic partitioning, • high resource utilization

25. @wendigo Future Datacenter

26. @wendigo IaaC

27. @wendigo Efficient utilization

28. @wendigo Google’s Omega source: Omega: flexible, scalable schedulers for large compute clusters

29.

30. Questions?

31. http://meetup.com/allegro.tech http://allegro.tech @AllegroTechBlog Work with us