SpringOne Platform 2019 Highly Available and Resilient Multi-Site Deployments Using Spinnaker Speaker(s): Koundinya Srinivasarao, Senior Platform Architect, Pivotal; Dodd Pfeffer, Platform Architect, Pivotal YouTube: https://youtu.be/-gkt_Ch-qb0

1. Highly Available and Resilient
Multi-Site Deployments Using
Spinnaker
October 7–10, 2019
Austin Convention Center

2. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Crazy Hat Wearers/Senior Platform Architects
Koundinya (KD) Srinivasarao Dodd Pfeffer

3. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Safe Harbor Statement
The following is intended to outline the general direction of Pivotal's offerings. It is intended for information
purposes only and may not be incorporated into any contract. Any information regarding pre-release of
Pivotal offerings, future updates or other planned modifications is subject to ongoing evaluation by Pivotal
and is subject to change. This information is provided without warranty or any kind, express or implied, and
is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making
purchasing decisions regarding Pivotal's offerings. These purchasing decisions should only be based on
features currently available. The development, release, and timing of any features or functionality described
for Pivotal's offerings in this presentation remain at the sole discretion of Pivotal. Pivotal has no obligation to
update forward looking information in this presentation.

4. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Agenda
● Rethink How You Achieve High Availability
● Application, Data & Platform Design Factors
● Active-Passive Deployments
● Active-Active Deployments
● Tools / Best Practices / Recommendations

5. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
A Framework
● SRE
● Deployment Topologies
○ Active-Passive / Active-Active
● Design Factors: Application / Data / Platform / Other
● Platform/IaaS Prerequisites

6. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
“Site Reliability Engineering (SRE) is
what you get when you treat operations
as if it’s a software problem”1
and aims at
developing software systems to solve
complex problems.
1
“What is SRE?”: https://landing.google.com/sre/

7. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Reliability is a Product Feature
Users expect new features
We need to respond to users
needs
Users also expect reliability We need a system that works
There is an inherent tension
between change and
stability
We need a way of balancing
these needs

8. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Failure is Normal
Site Reliability Engineering
Error Budget (Time)
Embrace Risk
Eliminate Toil
Reliability is
Fundamental
Service Level Indicator
Service Level
Objective

9. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Balancing Change
and Reliability

10. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Credit: Corey Innis

11. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

12. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

13. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

14. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Managing and Embracing Risk
with Error Budgets

15. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Your Error
Budget is the
inverse of your
SLO
SLO Error Budget
99% < 7 Hours
99.5% < 3.5 Hours
99.9% ~40 Minutes
99.95% ~20 Minutes
99.99% ~4 Minutes
99.999% ~24 Seconds

16. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

17. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

18. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

19. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

20. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

21. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/

22. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Sources of Risk
● A new release breaks a small set of requests; not detected for a day; quick rollback when
detected.
● A configuration mishap reduces capacity; causing overload and dropped requests
● Operator is slow to debug and root cause bug due to noisy alerting
● Other hardware failure; restore from backup required on other site
● Internal Network outage ( between servers )
● Power failure
● A new release breaks a sizeable subset of requests; unfamiliar rollback procedure
extends outage
● Storage overloaded
● Hardware disk failure
● Operator accidentally deletes database; restore from backup is required
● Unnoticed growth in usage triggers overload; service collapses.
● External Internet outage ( to DC )

23. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Deployment Topologies

24. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Active-Passive Deployment
West East

25. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Active-Passive Deployment
West East
The Right Choice
for Most Apps
Default to this … or
at least start from
here

26. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Active-Active Deployment
West East

27. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Analysis Tools
Active-Passive Worksheet
Active-Active Worksheet

28. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Design Factors
Application Factors
What is the impact on the end
users?
Can users access the app?
Can developers successfully cf
push?
Data Factors
What kind of replication do my
of apps need?
Does this architecture support
that?
Platform Factors
What is the operational
experience with this
architecture?
How are backups and patching
affected?
What about major upgrades?
How is failover handled?
What is the ongoing operational
cost?
Other Factors: Iaas Resources/Capabilities,
Network etc...

29. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Spring Pet Clinic - Demo Application
● Application Considerations
○ Observability
○ 12 factors: Multiple instances
● Data Considerations
○ Clustered Datastore
○ WAN Replication
● Extended Application Deployment Considerations
○ Separate CI/CD
○ Stateful Inventory
○ Sophisticated Deployment w/ Canary

30. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Spring Pet Clinic - Demo Application

31. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
PCC Topology
2
Cache
Server
Cache
Server
Cache
Server
Locator
1
1 2 33 1
PCC
Client
Connection Pool
54.209.251.110 54.209.251.111 54.209.251.112
Locator
Cache
Server
2
54.209.251.113
Pivotal
Platform
AZ 1
Pivotal
Platform
AZ 2
Cache
Server
3 1
54.209.251.112
Locator
Cache
Server
2
54.209.251.113
Pivotal
Platform
AZ 3
Cluster Spanning Multiple Pivotal Platform AZs
PCC Native Redundancy prevent data loss

32. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Design Pattern: Foundation Replication
Pivotal
Platform
Pivotal
Platform

33. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Pivotal Cloud Cache HA and Fault Tolerance
Failing data copies are
replaced transparently
Data is replicated to other
clusters and sites (WAN)
Split Brain Detection is enabled Client and cluster disconnections
are handled gracefully
Data is persisted on local
disk for ultimate durability
“split brain”

34. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Continuous Delivery Pipeline
Automated builds
Unit tests
Blue/Green deploys
Canary analysis
A/B testing
Monitoring
Security scans
Chaos engineering
Test-driven dev
Iterative coding/fixing
Frequent integration
Commit code change
Store binaries &
build artifacts
Continuous Integration
Continuous
Delivery
Compliance checks
Service tickets
Performance tests
Security validation

35. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Application Centric Control Plan
=
Turn monitoring
into action
Deploy safely
with speed

36. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Automated Canary Analysis
Ensures your applications meet your
SLAs
https://medium.com/netflix-techblog/automated-canary-analysis-at-netflix-with-kayenta-3260bc7acc69

37. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Platform/IaaS Prerequisites and
Considerations

38. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Prerequisites
DNS Global Load Balancer
● Configured to point to the two Pivotal Platform foundations
● Each with their own local load balancer
● Global DNS is configured to point to one of the two local load balancers’ VIPs
Connectivity/Peering
● Global VNET / Inter-region VPC Peering
Identical Release Versions
● Foundations should have the same Pivotal Platform release versions,
configurations and capacity, to the extent possible.

39. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Active-Active Deployment
West East
Azure Front
Door
Global VNET
Peering

40. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Platform Automation Simplifies Updates &
Upgrades
● Works with Concourse and leverages
Concourse tasks
● Contains and extends om commands
(the Ops Manager API)
● Provides an example pipeline to show
how to string together tasks
● Provides a way to extract configuration
files, so tasks can support
multi-foundation environments.
● IaaS agnostic. Tasks are consistent
across IaaS targets.
● Reusable tasks. Run tasks over and
over, instead of having to make
manual changes.
Building blocks for teams to
create a repeatable and
reusable automated pipeline
for upgrading and installing
Pivotal Platform foundations.

41. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Platform Automation Engine
Opinionated Deployment of
Concourse, UAA & Credhub
Used for Platform Ops Team to run
Platform Automation based pipelines

42. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Unified Observability

43. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Key Takeaways
● Understand and agree on SLO/Error Budget
○ > 99.9% ⇒ Active-Passive or Active Active
○ > 99% ⇒ Single Foundation in a Datacenter/Region
○ Plan for Cluster/AZ capacity to support failed instances from the other
Cluster/AZ
● Start with Active-Passive before moving to Active-Active
● Mix of Active-Passive vs Active-Active Based on Workload SLO
● Availability of Data
○ Pivotal data services support cross-site replication (PCC excels at this).
Non-Pivotal options exist as well.
● Automate Everything
● Measure Everything, Learn, Adapt

44. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
References
● Whitepaper: Multi-Site Pivotal Cloud Foundry Reference Architecture
○ Active-Passive / Active-Active Worksheets
● Petclinic PCC Github Repo
● GitHub Repo: Platform Automation Example
● GitHub Repo: cf-mgmt
● MySQL for Pivotal Platform 2.7, Featuring High Availability

45. Stay Connected.
Concourse, and Spinnaker - Today 4:20
Square Pegs, Square Holes: CI/CD That Fits - Yesterday
Scalable, Cloud-Native Data Applications by Example -
Today 2:00 [Geode/Pivotal Cloud Cache]
#springone@s1p

46. Unless otherwise indicated, these slides are © 2013-2019 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Credit
Corey Innis for SRE Illustrations
Customer [0] Team whose work this framework is based on