Multi-Cloud Management of Micro-Services using CloudFoundry APIs

1. Multi-Cloud Management &
Application Customized Scaling
with CloudFoundry
Edison Ting
Solutions Architect / Field Engineering
October 23, 2014
eting@pivotal.io

2. Motivation:
Change Capture:
• On Average = 1,500 – 5,500
ops/sec (inserts/updates/deletes)
captured
• Highly variable and bursty
• Real time data flow problem
A few months ago, we built a distributed replication solution for a customer, using
distributed processes that read from queues populated with change capture data
Each Replicator would read from a set of queues, then write data into a in-memory
data grid, while periodically retiring batches of changes into a distributed database
Replicator:
On Average can
handle 1,000+ ops/sec

3. On average each Replicator can process 1,000+ operations per second and
so to keep up with the change capture process, we would add as many
Replicator processes as needed to support the highest rate observed
Adding Replicators
allows the overall
task to scale
linearly
Distributed Replication

4. We created a performance dashboard for monitoring of the overall
throughput of the system, as replicators are manually added/removed to
process the changing demands of the system.
Distributed Replication Monitoring

5. We could not dynamically scale our replication processes
to react fast enough to the demands of the incoming
transactions
We couldn’t take advantage of resources that might be
available from other clusters to scale out the replication
processes
We needed an automated way to monitor whether the
replication processes had reach peak loads so that we can
begin to decide how to add or remove replication
processes that are not needed
We needed a way to allocate resources properly to
processes so as to not to take away resources that can be
used by other processes replicating other parts of the
system
Challenges:

6. Goals
 Define multi-cloud management and application
customized scaling and how it begins to solve some of
the challenges
 What is CloudF0undry: a PaaS that provides a polyglot
and cloud agnostic environment with application
lifecycle recovery and resource management capabilities
 Demo of Multi-Cloud Management and Custom App
Scaling Prototype
 Discuss the architecture of the MCC and Custom App
Scaling solution using CloudFoundry capabilities and
APIs.

7. Pivotal - Our Mission

8. mccController can analyze resources globally, across multiple
clusters managed by a PaaS, to determine where applications
can run or where resource can be adjusted for provide optimal
use of the system
Application #1
PaaS (CloudFoundry)
Multi-Cloud Controller Architecture
mccMonitor
mccController
Application #2 Application #N
Application #1
PaaS (CloudFoundry)
mccMonitor
Application #2 Application #N

9. mccController tracks the resource utilization of each PaaS that it
manages, and based on the data can provide recommendations
on where to deploy new applications and scale resources
Application #1
PaaS (CloudFoundry)
Multi-Cloud Controller Architecture
mccMonitor
mccController
Application #2 Application #N
Application #1
PaaS (CloudFoundry)
mccMonitor
Application #2 Application #N

10. Applications periodically reports resource utilization to mccMonitor as
well as changing workload requirements. mccMonitor will manage the
information and report to mccController for workload analysis
Application #1
PaaS (CloudFoundry)
Application #2 Application #N
mccMonitor
mccController
Multi-Cloud Controller Architecture

11. mccController communicates with mccMonitors periodically while
mccMonitors runs locally on each PaaS to allows application to
communicate workload status as well as scaling requirements.
mccController then communicates with the PaaS (CloudFoundry) to
manage resources of applications and scale the number of
application instances.
Application #1
PaaS (CloudFoundry)
Application #2 Application #N
mccMonitor
mccController
Multi-Cloud Controller Architecture

12. mccController will communicate with multiple PaaS instances with
different load characteristics, so that it can scale applications and add
or drop app instances as necessary to achieve desired throughput and
balance resource utilization between different PaaS instances.
Application #1
PaaS (CloudFoundry)
Multi-Cloud Controller Architecture
mccMonitor
mccController
Application
#2
Application
#N
Application
#1
PaaS (CloudFoundry)
mccMonitor
Application #2 Application
#N
Application #1
Application #1
Application #2

13. CloudFoundry: An Operating System for the Cloud

14. CloudFoundry and Bosh provides an
IaaS Agnostic Infrastructure
+

15. Bosh manages other distributed processes
besides CloudFoundry, such as Hadoop

16. CloudFoundry (PaaS) Architecture
Containers
Containers
Containers
Containers

17. What does CloudFoundry do with an Application ?
Containers
Containers Containers Containers Containers
+
+
Kernel
Containers

18. CloudFoundry manages the applications placement for
effective scaling , high availability and recovery
Containers

19. CloudFoundry promotes loose coupling of
applications and services
Containers
Containers
Environment
variables
Provision
Services
CloudFoundry provision
services and make them
available for use to
applications. Applications
access services using
environment variables
configured thru
CloudFoundry

20. How CloudFoundry enables
the Multi-Cloud Controller Architecture
mccMonitor
mccController
Environment
variables
mccController can (1) deploy
mccMonitor’s to each CloudFoundry
instance, and have applications bind to
the mccMonitor, enabling the
application to communicate
performance metrics, then (2) enable
mccController to dynamically scale
their resources
mccMonitor
Environment
variables

21. Our Tools: CloudFoundry Client APIs
How can we use these to accomplish what we need ?
 Package: org.cloudfoundry.client.lib
 CloudFoundryClient
 CloudCredentials
 Package: org.cloudfoundry.client.lib.domain
 CloudInfo
 CloudApplication
 CloudService
 ApplicationStats
 InstanceStats
 InstanceInfo
 InstanceState

22. Demo

23. Pushing (deploying) Application(s)
Containers
Containers
+
+
CloudFoundryClient client =
new CloudFoundryClient(new CloudCredentials(user, pw),
cfURL, org, space,..); client.login();
List<CloudDomain> domains = client.getDomains();
String appURL = “myApp” + domains(0).getName(); //
myApp.cfapps.io
client.createApplication( “myApp”, new Staging(), memSize,
appURLs, services );
client.uploadApplication( “myApp”, file.getCanonicalPath() );

24. Pushing (deploying) Application(s)
Containers
Containers
+
+
Create CloudFoundry client object
Assign application name and URL
Upload (push) and start application

25. Creating Service(s)
Containers
CloudFoundryClient client = new CloudFoundryClient( … );
String appURL = “myApp” + client.getDomains(0).getName(); //
myApp.cfapps.io
CloudService service = new CloudService(metaData, “myService”);
Map<String, Object> credentials = new HashMap<String, Object>();
credentials.put( “myServiceAPI”, appURL );
client.createUserProvidedService( service, credentials );
myApp.cfapps.io

26. Creating Service(s)
Containers
myApp.cfapps.io
Assign application name and URL
Create Service object
Provide credentials information to allow applications to
use the service

27. Binding Service(s) to Application(s)
Containers
CloudApplication yourApp = client.getApplication( “yourApp” );
client.bindService( yourApp, “myService” );
client.stopApplication( “yourApp” );
client.startApplication( “yourApp” );
Environment
variables
Containers

28. Binding Service(s) to Application(s)
Containers
Environment
variables
Containers
Get application object related of the app to be bound
Bind application to service
Stop and Start application

29. How can Applications find ‘bound’ Services ?
Containers
String myEnv = System.getenv( “VCAP_SERVICES” );
JSONObject obj = JSONValue.parse( myEnv );
JSONArray arrySrv = obj.get( “yourService” );
for( JSONObject srvObj: arrySrv ) {
JSONObject credentials = srvObj.get( “credentials” );
String serviceUrl = credentials.get( “yourServiceAPI” );
Environment
variables
Containers

30. How can Applications find ‘bound’ Services ?
Containers
Environment
variables
Containers
Get environment variable VCAP_SERVICES
Find the service’s entry and get credentials information
to be used to communicate with the service

31. How do we gather application statistics ?
CloudFoundryClient client = new
CloudFoundryClient(..);
Map<> env =
client.getApplication().getEnvAsMap();
int instCnt =
client.getApplication().getInstances();
ApplicationStats stats =
client.getApplicationStats(“yourApp”);
for( InstanceStats is : stats.getRecords()) {
InstanceStats.Usage usage =
is.getUsage();
long memuse = usage.getMem();
long cpuuse = usage.getCpu();
long diskuse = usage.getDisk();
long memQuota = is.getMemQuota();
long diskQuota = is.getDiskQuota();
…
}
mccController

32. How do we gather application statistics ?
mccController
Create CloudFoundry Client Object
Get Application Stats Object
Iterate Applications Instances
Stats object to get memory,
disk, and instance count
information

33. How do we scale application resources ?
CloudFoundryClient clnt = new
CloudFoundryClient();
clnt.stopApplication(“yourApp”);
clnt.updateApplicationEnv(“yourApp”,
Map<> env);
clnt.updateApplicationMemory(“yourApp”,
newVal);
clnt.updateApplicationDiskQuota(“yourApp”
, nDisk);
clnt.updateApplicationInstances(“yourApp”,
nInstns);
clnt.startApplication(“yourApp”);
mccController

34. How do we scale application resources ?
mccController
Create CloudFoundry Client Object
Update application memory,
disks, and instances
Stop and Start application

35. How can we apply these APIs to the
Multi-Cloud Controller Architecture ?
mccMonitor
mccController
Environment
variables
mccMonitor
Environment
variables

36. Let us deploy the monitor and bind it to applications
running on each CloudFoundry instance
mccMonitor
mccController
Environment
variables
/* Push mccMonitor to CloudFoundry Instances */
pushMccMonitor( cloudFoundryURL );
/* Bind CloudFoundry Applications to MccMonitor */
bindApplicationsToMccMonitor( cloudFoundryURL, MccMonitorUrl );
/* Next Applications will provide workload specfiic statistics to
MccMonitor… */

37. How do we enable the application to customize its scaling
characteristics ?
mccMonitor
Environment
variables
/* Applications get mccMonitor URL via VCAP environment variable */
mccURL = getMccURL( System.getenv( “VCAP_SERVICES” ) );
/* Application provides known workload thresholds for each instance
to mccURL */
URL( mccURL + “&invokeThreshold=1000&updateThreshold=500” );
/* Application periodically reports accumulated statistics about
specific workloads */
URL( mccURL + “&binc=1&invokeCount=27&updateCount=4” );
App1 :{ invokeCount: 34
invokeThreshold: 1000
updateThreshold: 500 }

38. Let us follow the protocol between the applications and
mccMonitor
mccMonitor
Environment
variables
/* Applications get mccMonitor URL via VCAP environment variable */
mccURL = getMccURL( System.getenv( “VCAP_SERVICES” ) );
/* Application provides known workload thresholds for each instance
to mccURL */
URL( mccURL + “&invokeThreshold=1000&updateThreshold=500” );
/* Application periodically reports accumulated statistics about
specific workloads */
URL( mccURL + “&binc=1&invokeCount=27&updateCount=4” );
App1 :{ invokeCount: 34
invokeThreshold: 1000
updateThreshold: 500 }

39. mccMonitor
mccController
Environment
variables
Let us follow the protocol between mccMonitor and mccController
/* mccController loops thru all
applications and checks last
pollDateTime environment variable that
it sets on every application to compute
elapsed time since last poll */
CloudApplication appl =
client.getApplication(..);
Map<String,String> env =
appl.getEnvAsMap();
prevPoll = env.get(“lastpoll”);
prevPollDateTime = new
Date(prevPoll);
elapsedTime = currDate -
prevPollDateTime
/* mccController get applStats from
monitor */
String mccMonitorUrl =
App1 :{ invokeCount: 34
invokeThreshold: 1000
updateThreshold: 500 }

40. mccMonitor
mccController
Environment
variables
Protocol between mccMonitor and mccController (contd…)
/* mccController takes variables ending
in Count and Threshold and computes
overall rate */
String jsonStruct =
post2Monitor(mccMonitorUrl);
JSONObject obj =
JSONValue.parse(jsonStruct);
Iterator it = obj.entrySet().iterator();
while( it.hasNext() )
if(
it.next().getKey().endsWith(“Count”)) {
thres = thresKey(key); …
// compute rate achieved per
instance
ratePerInst =
getRate(elapsed,numIns);
// Add instance if threshold
exceeded
App1 :{ invokeCount: 34
invokeThreshold: 1000
updateThreshold: 500 }

41. mccMonitor
mccController
Environment
variables
Protocol between mccMonitor and mccController (contd…)
/* mccController also reads in
pctGrowTrigger
and pctShrinkTrigger variables to scale
memory */
String jsonStruct =
post2Monitor(mccMonitorUrl);
JSONObject obj =
JSONValue.parse(jsonStruct);
pctGrowTrigger =
obj.get(“pctGrowTrigger”)
pctGrowAmount =
obj.get(“pctGrowAmount”);
pctShrinkTrigger =
obj.get(“pctShrinkTrigger”);
pctShrinkAmount =
obj.get(“pctShrinkAmount”);
App1 :{ invokeCount: 34
invokeThreshold: 1000
updateThreshold: 500 }

42. Multi-Cloud Controller gathers resource utilization data
directly from multiple CloudFoundry Instances
for( cloudFoundryUrl : cfUrls ) {
CloudFoundryClient client =
new
CloudFoundryClient(cloudFoundryUrL);
List<CloudApplication> apps =
client.getApplications();
for( CloudApplication app: apps ) {
ApplicationStats stats =
client.getApplicationStats(
app.getName() );
for( InstanceStats is :
stats.getRecords()) {
InstanceStats.Usage usage =
is.getUsage();
long memuse = usage.getMem();
long memQuota =
mccController

43. Multi-Cloud Controller provides global view of how each
CloudFoundry Instance utilize their resources
mccController

44. mccController
This CloudFoundry Instance
shows a large portion of
overall memory not being
used, possible candidate for
scaling down some instances ?

45. mccController
This CloudFoundry Instance
shows hosts only a few low
footprint applications, should
these be moved to the other
instance that can host more
instances ?

46. How do we adapt these tools to the
distributed replication solution ?
mccMonitor
mccControllermccMonitor

47. How do we adapt these tools to the
distributed replication solution ?
mccMonitor
mccController
mccMonitor

48. How do we deploy
the distributed apps shown below ?

49. Summary
 Multi-cloud management and App customized scaling
architecture makes applications active participants in the
way they are scaled and how they consume resources
within a collection of PaaS instances
 Applications automatically scale up and down based on
specific workloads they need to process
 Distributed processes and Micro-services can self deploy
and self bind to one another achieving loose coupling
and independent scaling
 Applications have full control of their destiny

50. Where can you find more information ?
 CloudFoundry Client Java APIs
 https://github.com/cloudfoundry/cf-java-client
 CloudFoundry Documentation
 http://docs.cloudfoundry.org/
 https://github.com/cloudfoundry
 Bosh Documentation
 http://docs.cloudfoundry.org/bosh/
 http://www.think-foundry.com/cloud-foundry-bosh-introduction/
 https://github.com/cloudfoundry/bosh-lite
 MicroServices Architectures
 http://www.activestate.com/blog/2014/09/microservices-resources

51. Thank you!