Don't Repeat Our Mistakes! Lessons Learned from Running Go Daddy's Private Cloud (OpenStack Queens Summit)
•
0 recomendaciones•627 vistas
After years of running one of the largest OpenStack clouds, we've learned a thing or two. Early architectural decisions about networking, storage, and scaling have real and lasting consequences. We'll walk through some of these early decisions, some which turned out to be good, but many turned out to be bad. Also included are some strategies for thinking through the long term impacts, to help you avoid similar pitfalls for your cloud. (Video at https://www.youtube.com/watch?v=LzIkTqfb1nI )
Recomendados
Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Similar a Don't Repeat Our Mistakes! Lessons Learned from Running Go Daddy's Private Cloud (OpenStack Queens Summit)
Similar a Don't Repeat Our Mistakes! Lessons Learned from Running Go Daddy's Private Cloud (OpenStack Queens Summit) (20)
Último
Último (20)
Don't Repeat Our Mistakes! Lessons Learned from Running Go Daddy's Private Cloud (OpenStack Queens Summit)
- 1. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Don’t Repeat Our Mistakes! Lessons Learned from Running Go Daddy’s Private Cloud Kris Lindgren klindgren@godaddy.com Mike Dorman mike.dorman@sendgrid.com OpenStack Queens Summit, November 2017, Sydney
- 2. Copyright© 2016 GoDaddy Inc. All Rights Reserved. OpenStack at Go Daddy ● 2013: POC cloud (Havana) ● 2014: First production apps (Icehouse) ● 2014: Nova cells v1 (Kilo) ● 2015: “OpenStack everywhere” (Liberty) ● 2017: Working toward containerized services
- 3. Copyright© 2016 GoDaddy Inc. All Rights Reserved. OpenStack at Go Daddy ● What we built: ○ Shared nothing regions ○ Ephemeral disk on local storage ○ Simple networking ○ No live migration ○ Multiple AZ’s ● Scale ○ 1000’s Computes, >100,000 Cores ○ 10,000’s VM’s
- 4. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Avoiding “Accidental Architecture” Product Infrastructure & Scaling Management
- 5. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Private Cloud = Free Compute High Demand = Overconsumption Product - Need for Chargeback/Showback Free Compute = High Demand
- 6. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Product - Have a Cohesive Vision • Which OpenStack Services/features • User onboard/off-boarding • Patching cadences/methodology • Legacy integrations • Adding capacity • SLAs • How do end users “consume” OpenStack? • Procedure for changing the vision • Helps with cloud paradigm shift • Expect and tolerate failure
- 7. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Product Issues - How to Avoid • Manage expectations (for yourself and for users) • Showback and controls around quota • Education and evangelism • Docs and sample code • “Cloud ready” early adopters • Ongoing guidance 1.Cloud 2.?????? 3.Profit!X
- 8. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Scaling - Nova Cells (v1) Justification • Assumed we would grow fast • Challenges with scaling Nova/RMQ • Easier earlier than later • Ongoing debt to manage patches • Cells v2 was coming soon http://www.dorm.org/blog/converting-to-openstack-nova-cells-without-destroying-the-world/
- 9. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Scaling - Nova Cells (v1) Retrospective Good • Helped us to scale • Gained expertise with Nova • Community street cred Bad • No scaling for Neutron • Patches get more difficult • Non-standard config • Delays on v2 • Migration to v2 is unknown 20/20 Hindsight • Scale/shard RMQ instead • Aspirations about scale • Porting patches is top blocker
- 10. Copyright© 2017 GoDaddy Inc. All Rights Reserved. • Colocated API services and RMQ • (Except Glance) • Dedicated hardware overkill • Local python packages • Made sense for POC • Nova separated later with Cells v1 Scaling - Collapsed Architecture Justification
- 11. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Scaling - Collapsed Architecture Retrospective Good • Simple architecture • Minimal hardware • Easy network ACLs • Up and running fast Bad • Large failure impacts • Resource contention • Single API endpoints 20/20 Hindsight • OK for POC • Ignored it too long • Easy to scale out • (Implementing now)
- 12. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Infrastructure - Special Neutron Architecture Justification • Neutron L2 assumptions • L3 folded clos network • L2 stops at leafs • Uncomfortable with overlays • Provider network per rack • Routed floating IPs • Overload AZ to pick a network • Local patches for network scheduling
- 13. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Infrastructure - Special Neutron Architecture Retrospective Good • Same for VMs and metal • Simple infrastructure • Easy on users • Network IP usages API • Segmented networks spec Bad • Snowflake setup • L2 adjacency expectations • Added features difficult (LBaaS) • Migration to Neutron segmented networks? 20/20 Hindsight • Works pretty well • Patches are limited • IP usages API extension • Segmented networks in Neutron • Many others with same problem
- 14. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Management - Puppet Single Source of Truth Justification • Big Puppet shop • Single source of config • Good for server bootstrapping • OpenStack-Puppet modules • API providers • Code pipeline already in place • Ansible kicks off puppet apply
- 15. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Management - Puppet Single Source of Truth Retrospective Good • Single source of config (in theory) • Efficient bootstrapping • NOOP mode for sanity Bad • State in Puppet, Hiera, APIs • Some managed manually • Duplicate API objects • Omnibus deployments • NOOP report not always accurate! • Orphaned/forgotten servers • Orchestration difficult 20/20 Hindsight • Many unintended problems • Not really a single source • Need for targeted deployments • Other tools for orchestration • Use for bootstrapping
- 16. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Strategies for Avoiding Accidental Architecture • Think of your future selves •Quantify tech debt interest • Almost nothing will be temporary •Make a specific plan and timeline • Carefully consider scale •Overestimating can be as bad as underestimating • Automate first •At least make it capable
- 17. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Strategies for Avoiding Accidental Architecture • KISS! http://stella.report
- 18. Copyright© 2017 GoDaddy Inc. All Rights Reserved. Strategies for Avoiding Accidental Architecture • Spread the knowledge wealth http://stella.report * The Coming Software Apocalypse: https://www.theatlantic.com/technology/archive/2017/09/saving-the-world-from-code/540393/ “The problem, [...] is that we are attempting to build systems that are beyond our ability to intellectually manage.” *
- 19. Copyright© 2016 GoDaddy Inc. All Rights Reserved. Recap: How to Live with No Regrets Questions? Other Ideas? klindgren@godaddy.com mike.dorman@sendgrid.com ● Manage expectations ● Education and evangelism ● Helpful early adopters ● Ongoing guidance ● Remember your future self ● Account and plan for tech debt ● Sane scale expectations ● Automate, automate, automate ● Simplicity ● Knowledge sharing
Notas del editor
- KRIS
- KRIS Late 2013, Havana, POC/”dev pilot” cloud Morphed into production cloud by 2014 Liberty since 2015, blocked on containerization
- KRIS So what did we end up building? Totally separate clouds at multiple location (no shared keystone) VM’s boot to local storage, may have cinder volume to store “persistent data” Network centric approach to networking. Letting the networking gear take care of the packets. No live migration - meaning we didn’t want to have pets. Teams were advised to be able rebuild servers. Anything of state should not go on openstack (Databases)
- MIKE Decisions have long lasting impacts Tough to change later Talk about general categories of things Infrastructure & Scaling Management (Config Management) Product We’re going to work backwards, and Kris is going to kick us off with some issues around product
- MIKE Free for everybody, then it gets all used up and isn’t there when needed Tragedy of the commons (find some good images for this) Had a lot of trouble keeping up with capacity consumption, so we would run out of space We had ridiculously high quotas, intending on reporting back to corporate finance (but that never happened)
- KRIS I know there is a lot of text on this slide, but I am not going to go through everything here DOCUMENT WHAT YOUR ARE PROVIDING SLA’s Patching policy How do you want end users to use what you built Example deployment’s/architectures Integrations with legacy applications If you are running a cloud and you don’t have this documented. Please, Please, Please do the work to get this documented and agreed upon. product vision drives your technical requirements Small changes to the vision/requirements can have a fundamental shift in what you need to provide. After getting this documented, also get the process for how to change the vision documented.
- MIKE Be clear on what you’re providing and what you’re not (before you build it) Know where you are going! Even if you don’t plan to actually charge others real money for using your cloud, you need to show them what they’re using and translate that to value somehow Definitely enforce some quota control Talk about how we opened up our quotas with intentions to report back to finance department against budgets (which never happened) Unless you can actually scale hardware super fast (you can’t) then it can’t just be a free-for-all Education and evangelism Good docs, getting started guides, sample code Give them something to copy and paste Start with teams that are already “cloud ready” as early adopters Provide ongoing architectural advice and constructive advice Don’t be arrogant or treat people who aren’t to your level yet poorly This should go without saying, but if we’re honest, we all have condescending attitudes toward some Help when things go wrong Describe SendGrid ProdOps team Now, moving on to the more technical architecture decisions
- MIKE We knew we would grow fast (see earlier graph) Known challenges with scaling Nova/RMQ Easier to move to cells v1 early, rather than a fire drill scaling exercise later Knew we would take on some ongoing debt to forward port v1 patches for each new version Cells v2 was coming “real soon now” Details about how we did it in Link to my YVR talk about moving to cells
- MIKE Good Helped us to scale and segment our infrastructure (failure boundaries) Gained a lot of expertise with Nova Street cred in community (LDT group, etc.) Bad Neutron doesn’t scale the same way, which ended up being our main bottleneck (not Nova) Forward porting patches becomes more and more difficult over time (eternal thanks to Sam from NeCTAR) Unknown how/if we can online migrate to cells v2 Cells v2 still coming “real soon now” (mostly there now)
- KRIS Run all API/server services, plus RMQ all on one set of servers Glance separate to stay network adjacent to computes Most nova services moved later as part of cells v1 Symptom of starting small with POC environment and then growing larger
- KRIS Good Less hardware to deal with Simpler architecture Easier network/firewall ACLs It helped us get started quickly Bad Any problems are very impactful, it takes out a wide swath of services Resource contention (RMQ and Neutron fighting over RAM and oom killing each other) No admin vs. public endpoints, more difficult to do maintenance that doesn’t expose errors to users
- KRIS Neutron assumes (or used to assume) L2 everywhere and its available anywhere In our datacenter network L2 stops at TOR So getting to a server in another rack goes though the gateway of the local switch to the spine and in to the other rack Persistent IP’s can be routed to any vm within the network Overlays viewed as unnecessarily complex, difficult to troubleshoot Provider network per rack (L2 domain), we pick a network for you based on AZ selection Local patches to do network scheduling
- KRIS Good Able to provide the same networking paradigm to VMs as to metal Simple infrastructure, VMs just get an IP and they’re good to go Network IP usages API implemented and committed upstream Kicked off segmented networks spec as collaboration between LDT and Neutron This remains the thing I’m most proud of accomplishing/helping with in OpenStack Bad Our Neutron doesn’t work like everybody else’s People love their L2 adjacency Unable to support more complex networking features out of the box (e.g. LBaaS) Unsure how we will go about migrating to real Neutron segmented networks
- MIKE Big puppet shop Pretty good for server bootstrapping and config management Wanted one stop shop for all config OpenStack API providers for managing users, groups, roles, AZs, networks, etc. Code review/pipeline already in place Config mostly in Puppet and Hiera repos State of OS resources inside APIs Physical hosts in manually curated Ansible hosts file
- MIKE Good Single place for all config (in theory) Helpful for new server bootstrapping and initial config Noop mode helpful to see what will happen Bad Config and current state was actually split across Puppet and Hiera repos, as well as the service APIs Difficulties with API providers led to duplicate objects (networks, AZs) Difficult to do non-omnibus targeted deployments (Puppet upgraded RabbitMQ, woops!) Roles and grants still managed manually ad-hocly Noop report not always accurate! Sometimes servers are forgotten about because we forget to put them in the list Difficult to do more intelligent orchestration of things when the data is all over the place
- MIKE Think of your future self Almost nothing will be temporary Unless have you have a specific plan and timeline for moving away from it, and you can trust yourself to follow through Try to quantify the interest you will pay on the tech debt Consider your expected scale (more than seat-of-your-pants) Just as bad to overestimate and overbuild than to underestimate Automate first (or at least make sure the capability is there)
- MIKE Keep it simple The perfect design is not when nothing else can be added, but when nothing else can be removed. As we were working on this, the Stella Report came out which articulates pretty well a lot of the ideas we were thinking about. Particularly around the idea of complexity, and a term they coined “dark debt” Dark debt/unknown unknowns that come from complexity (link to Velocity talk/Stella paper) Best thing you can do to minimize is to keep things as simple as possible
- MIKE Do as much as you can to simplify, but it’s still complex. Spread the knowledge wealth Try to keep everybody up to speed with what’s going on Keep the “mental models” of the system accurate and up to date (above the line/below the line) Avoid individual/tribal knowledge
- KRIS