Se ha denunciado esta presentación.
Utilizamos tu perfil de LinkedIn y tus datos de actividad para personalizar los anuncios y mostrarte publicidad más relevante. Puedes cambiar tus preferencias de publicidad en cualquier momento.

Denver Cloud Foundry Meetup - February 2016

468 visualizaciones

Publicado el

DigitalGlobe's talk about how they're using Cloud Foundry at the Cloud Foundry meetup on February 18, 2016.

Publicado en: Software
  • Inicia sesión para ver los comentarios

  • Sé el primero en recomendar esto

Denver Cloud Foundry Meetup - February 2016

  1. 1. CopyrightDigitalGlobe2016,All RightsReserved Cloud Foundry Meetup - DigitalGlobe 02/18/2016 MountFuji
  2. 2. CopyrightDigitalGlobe2016,All RightsReserved What does DigitalGlobe do? DigitalGlobe 2 • We have a constellation of satellites that take images of the earth. • We collect about 3,500,000 square kilometers of imagery everyday - Slightly more than the landmass of India - 21,875 times the landmass of Lichtenstein • We downlink ~5-6 TB of new “raw” imagery per day (~2 PB year) • We then create products from that imagery with various forms of image processing and analytics • We create between 40 TB - 100 TB of new products per day
  3. 3. CopyrightDigitalGlobe2016,All RightsReserved And we get to do fun things like this. Go Broncos! DigitalGlobe 3
  4. 4. DigitalGlobe sets the standard for quality 4 Accuracy. Currency.Completeness.Consistency.We lead the industry. DigitalGlobe
  5. 5. CopyrightDigitalGlobe2016,All RightsReserved 24 Hours of Collections DigitalGlobe 5
  6. 6. CopyrightDigitalGlobe2016,All RightsReserved 7 Days of Collections DigitalGlobe 6
  7. 7. CopyrightDigitalGlobe2016,All RightsReserved 1 Month of Collections DigitalGlobe 7
  8. 8. CopyrightDigitalGlobe2016,All RightsReserved 6 Months of Collections DigitalGlobe 8
  9. 9. CopyrightDigitalGlobe2016,All RightsReserved 1 Year of Collections DigitalGlobe 9
  10. 10. CopyrightDigitalGlobe2016,All RightsReserved DigitalGlobe Needed A New Architecture DigitalGlobe 10 • ‘Old Way’ at DigitalGlobe - ~3-6 months to get a new applications with new VMs deployed. All snowflakes, all F5 configuration was snowflakes, etc. Some puppet used in production. - Lead us to do unnatural things. - Bolt on inconsistent functionality onto existing software to avoid a new deployment. - Integration through a centralized Database. - Fragile, slow to change, hard to use our data in new and different ways. - Very projectfocused. - Twist the monolith to meet the needs of a new project
  11. 11. CopyrightDigitalGlobe2016,All RightsReserved DigitalGlobe Needed A New Architecture DigitalGlobe 11 • New Opportunity – WorldView-4 - Upper management gave us the green light to change. - Tight schedule. Need to move at ‘rapid’ pace. - Designed and put in a microservice based architecturewith flexible/extensible workflows. - Scheduled satellite launch Q3 2016
  12. 12. CopyrightDigitalGlobe2016,All RightsReserved 12 • Survey of the available PaaS alternatives • Created “knockout” criteria • Down-Select – CF chosen as leading candidate • Performed tasks to verify knock out criteria met - e.g. upgraded PCF release while measuring uptime of platform services • Migrated DG sample Java, Ruby and Python services & verify portability across multiple environments • Built pricing and staffing models Technical Decision Process DigitalGlobe
  13. 13. CopyrightDigitalGlobe2016,All RightsReserved Path to Adoption 1. Develop One Service 2. Small Team – Pioneers 3. Initial Build Out
  14. 14. CopyrightDigitalGlobe2016,All RightsReserved Architecture Platform as a Service (PaaS) Cloud Foundry - Runs DG Custom Services Infrastructure as a Service (IaaS) OpenStack -Runs Vendor Supplied Programs -Runs some Custom DG Services Bare Metal Hardware -Sits underneath OpenStack -Runs High Performance Compute Clusters -Runs some GFE and specialized software GFE Crypto HPC Clusters CSSSAP JBoss A-MQ 1 2 3 Application Patterns Pattern Identification# DG Custom Services Service Wrapper Service Wrapper Postgres
  15. 15. CopyrightDigitalGlobe2016,All RightsReserved Cloud Foundry @ DigitalGlobe DigitalGlobe 15 • CurrentState - Open Source Cloud Foundry Running on OpenStack - Kudosto ECS Team. Great work helpingus here. - Custom deployment – best-effort at High-Availability config: - 47 VMsin the foundation. 20 are DEAs. - Need some tweakingto fullyutilize 2 AZ’s correctly. - This environment supports: Development, Test, CI/CD Pipeline, and ‘production’ until we get our PCF OpenStack deployment finished. - Total of ~500 services running in the current environment - Not all unique. Developersmayhaveto deploya service (or mock of the service) to theirspace to do developmentof test witha common service. - ~330 are in developersspaces - ~170 are in variousstages of test - DEA VMs are 2 CPU, 16 GB RAM with Cloud Foundry configured for a 3x overcommit on memory. - All logs being shipped into an ELK stack via log-drains bound to apps. Looking at firehose integration.
  16. 16. CopyrightDigitalGlobe2016,All RightsReserved Cloud Foundry Wins DigitalGlobe 16 • Development Speed - Once we had ‘patterns’ down, it is now easy to develop a new microservice - Template-based bootstrap of new team members and services • Ease of Deployment - Self-service Jenkins portal to create a new service - Merge to master in github to initiate the CI/CD pipeline (still a work in progress) - Managed deployment to several stops in the pipeline (still a work in progress) • Visibility - A few cf curl commands reveals all that is deployed • Auditability - When new services are found, Nimsoft alerts are sent - Monitoring automatically creates a new dashboard in Kibana for the discovered service (still a work in progress) • Control - Now that we have Visibility, and Auditability, we have control over our services.
  17. 17. CopyrightDigitalGlobe2016,All RightsReserved Cloud Foundry Wins DigitalGlobe 17 • Testing - The test groups are ecstatic. Generally very easy to test micro services. Test coverage and ability to isolate and troubleshootvastly superior to legacy. • Resiliency - OpenStack compute node failures - Bosh realized its was missing bits and rebuilt the environment - No users knew - We lost a few VMs outside of CF that affected people, but nobody knew the CF environment was even affected. - Has happened twice, with the same results. - We’vecaptured stats on these failures and they have been highly useful in discussion with Program and Executive management
  18. 18. CopyrightDigitalGlobe2016,All RightsReserved Cloud Foundry Challenges DigitalGlobe 18 • Open Source - No out-of-the-boxsupportfor an HA environment. - Manually keeping up to date. Editing deploymentsfor new features, etc. - Fall behind quickly - Lack of Graphical Management tools, even just for visibility • Other infrastructure that you are going to need - Eureka/consulor some other runtime service discovery. - Configuration isn’t going to cut it in a world of microservices. - Log Aggregation - Impossible to manage without something bringing all of your logs together. • Integration with Enterprise SSO - We have not found a lot of guidance. Maybe we are not looking in the right places. Anyone?
  19. 19. CopyrightDigitalGlobe2016,All RightsReserved Cloud Foundry Challenges DigitalGlobe 19 • Synchronization across Foundations - E.g. UAA, Client IDs, Secrets, Scopes. - Best practices around load-balancing across foundations. - Best practices around naming of domains across foundations… - Still want each foundation individually addressable for some functions. - E.g. we want to be able to target a foundation to do a deploy. - We want applications load-balanced across foundations though. • SSL & Domains - Since CF only supports a single cert, we have to use a ton of SANs (subject alternative names). - Impossible to manage with 100+ developers/testers. - (HTTPS required in our environment) - Need to get this rectified. We should be able to get a cert for a developers space and just add it.
  20. 20. CopyrightDigitalGlobe2016,All RightsReserved Cloud Foundry Challenges DigitalGlobe 20 • Developer/DevOps access to spaces - We have a lot of spaces, and some ephemeral spaces. - It is tough to add all the developersas space auditors so they can see the logs from all spaces. - Sometimes this is needed with certain types of failures that cant be diagnosed from the ELK stack logs. • Tension between microservices and licensing models - The more you do the right thing, the more it costs.
  21. 21. CopyrightDigitalGlobe2016,All RightsReserved Adoption Cultural Challenges Don’t change it now - I just got my head wrapped around the old one Can’t we put off this ‘PaaS thingy’ – seems bleeding edge Let’s do this on a ‘toy’ project It’s ‘open’, but is it really ‘open’?”
  22. 22. CopyrightDigitalGlobe2016,All RightsReservedDigitalGlobe 22 • This is a significant change of: - Culture - Tools - Process - Dependency management • The gains are significant – but you need someone in Executive Management that’s willing to go to bat for you • As an Architecture Team – you need to be willing to: - Put it on the line (reputation / job) - Abandon the ivory tower – Dig in, implement, provideguidance, “Keep pumping gas into the plane” - We’vedone all of this (and to a significant degree asked our EA colleagues to “keep the lights on” in the rest of the enterprise) And then – there is the politics
  23. 23. CopyrightDigitalGlobe2016,All RightsReservedDigitalGlobe 23 • A small footprint, OEM type installation of CF • Manage multiple foundations as a single environment - E.g. Cross Region type of deploymentsand management. • Persistent storage access – NFS mounts available in a container specifically Future needs
  24. 24. CopyrightDigitalGlobe2016,All RightsReserved 24 • Ward Maddux – Pivotal • Mark Carlson – ECS Team • Josh Ghiloni – ECS Team • Steve Wall – ECS Team • Mike Minges – ECS Team • Joe Fitzgerald - Pivotal • James Watters - Pivotal Thank You! DigitalGlobe
  25. 25. CopyrightDigitalGlobe2016,All RightsReserved 25 • We are hiring! DigitalGlobe
  26. 26. CopyrightDigitalGlobe2016,All RightsReserved 26 • Mike Waters Enterprise Software Architect • Mike Wierzbinski Enterprise Cloud Architect Contact Information (MW2) DigitalGlobe