Business Track presented by Michael Maximilien, Chief Architect PaaS Innovation at IBM & James Bayer, Director of Product Management, Cloud Foundry at Pivotal.

1. PaaS Comparison 2014
comparing and contrasting leading PaaSes on multiple dimensions
and feature sets to see where CF leads and lags
dr.max @maximilien
maxim@us.ibm.com
IBM Cloud Labs
!
and
!
James Bayer
jbayer@gopivotal.com
Pivotal
v0.8.0
10 June 2014
credit: http://www.teacherspayteachers.com/

2. Agenda
• Background
• Limitations and threat to validity
• Methodology - briefly
• Results - along comparison main axes
‣ Workloads, Tooling, Integration services
‣ SLAs, Datastores, Programming Models,
Management, and Misc.
• Insights
• QoE - quality of experience test
• Potential next steps

3. Background
• Based IBM Research study (circa 2010)
• Comparing & contrasting 6 leading PaaSes
‣ Microsoft’s Azure
‣ Google App Engine (GAE), GAE4Business
‣ Amazon AWS+, i.e, EC2, Beanstalk, + services
‣ Heroku from SalesForce.com
‣ OpenShift OSS PaaS from RedHat
‣ CloudFoundry OSS PaaS
• Started at IBM but revised with Pivotal in ‘14

4. Limitations and validity threat
• Challenges in comparing PaaSes
‣ what is claimed may not equal day-to-day reality!
‣ no standard measuring stick for most features!
‣ “signal vs. noise” problem measuring some features
• Limitation of approach
‣ non-scientific, i.e., subjective rather objective data!
‣ potentially biased (limited numbers of evaluators)!
• Value is providing a framework for own analysis
• Goal is measure PaaS QoE (quality of experience)

5. Overview of comparison
• Found 50 features of comparisons
• Eight groups
‣ (1) workloads, (2) tooling!
‣ (3) integration services, (4) SLAs!
‣ (5) datastores, (6) programming models!
‣ (7) management, and (8) miscellaneous
• Question: to what extent does the PaaS support
each of the features in group?
0: none, 1: basic, 2: strong, and 3: leading

6. workloads & tooling

7. integration services & SLAs

8. datastores & programming models

9. management

10. misc.

11. Quality of Experience
Scenario: Run (deploy, scale, update, troubleshoot)
a java web application with a database
1. OpenShift (OpenShift Online)
2. Heroku
3. Amazon Beanstalk
4. Google AppEngine
5. Microsoft Azure
6. Cloud Foundry (Pivotal Web Services)

12. AWS Beanstalk
Does well:
• Super well integrated into all AWS Services and feels like eases
IaaS friction
• OOTB Monitoring is best-in-class
!
Not so well:
• Responsiveness - VM orchestration takes a really long time
• Treats frameworks differently
• Must understand many AWS concepts to use effectively
• Extensibility – only fixed number of options

13. Google AppEngine
Does well:
• Super well integrated into many Google Services like Cloud
SQL and other APIs
• Events and runtime info well displayed
!
Not so well:
• Java support limited (limitations on what you can do with
threads, requests, SQL driver, services integration)
• Forces you to learn too much about Google Cloud
• Limited ability to customize container
• No debug/shell access in standard AppEngine

14. Azure Websites
Does well:
• Many options for frameworks
• Well integrated web UX
• Impressive list of SCM integration options
!
Not so well:
• Windows default option instead of Linux
• Very much an IaaS / VM feel
!
Did not finish yet

15. OpenShift
Does well:
• Excellent container ssh and port forwarding UX
• SCM and CI directly integrated as an option (did not try
Jenkins cartridge but it is available)
Not so well:
• No auto-detection, must explicitly choose a cartridge
• Out of date dependencies like JDK, Tomcat version, etc
• No Status, Health Management
• Scaling and managing many app instances limited
• Host network connections visible with $netstat -a
• WAR file support requires SCP

16. Heroku
Does well:
• Best-in-class polish all around on UX
• Run command, pipelines, stats in logs, marketplace
• Holds request at router when updating, unpausing apps
!
Not so well:
• Java seems to be a 2nd or 3rd class citizen
• Must use git, does not support war file directly
• Teams and managing multiple apps

17. Cloud Foundry
Does well:
• Java build tools best in class with maven, gradle, Eclipse and
many buildpacks to choose from
• Scales and responds quickly
• Logs (both app and audit events)
• Super easy Services integration
Not so well:
• Shell access without workarounds
• SCM / CI integrations without workarounds
• Provide app metrics
• Build in zero downtime updates

18. Insights and lessons learned
1. Difficult to extract comparison axes across PaaSes
2. Many comparison points are supported in all
3. AWS+ seem to be most mature PaaS (advertised?)
4. Might need to define a benchmark for PaaS
a. should contain common apps
b. should allow PaaS apps to be tested openly
5. Make spreadsheet and full presentation OSS
6. Crowdsource ratings to remove some subjectivity?
7. Involve members of each other PaaS community
8. Iterate, iterate, iterate

19. Potential next steps
• Socialize, update comparison matrix
• QoE test expanded to thorough scenario
• Could we develop a PaaS benchmark?
‣ Sample apps that can be ported for each PaaS
‣ Develop data collection tools and spreadsheets
‣ Collect data for each PaaS for benchmarking
‣ Making sense of the results data
• Iterate benchmark results for new PaaS releases

20. Thank you
(Q&As)

21. workloads & tooling
• Workload classes representing cloud apps
• Five types of workloads
1. Analytics - Hadoop-style workload
2. Transaction - DB-driven (with TX) workload
3. Media - workloads with large static files (read)
4. Web2 - workloads with read/write intensive
characteristics but no transactions and not sensitive to
read-coherency
5. Mobile - workloads for mobile apps and backend
• Tooling support includes IDE, CLI, APIs, git

22. integration services & SLAs
• Integration services
‣ includes BSS-types of services
‣ also includes other types of cloud services
• Microsoft seems to dominate this space
• Service level agreements
‣ most IaaS and PaaS providers advertise SLAs
‣ most advertised SLAs are coarse grained
‣ no monetary guarantees advertised
‣ most give more time as compensation
‣ no aggregate stats yet

23. datastores & programming models
• Datastores
‣ relational models support
‣ so-called No-SQL databases, e.g.,
Cassandra
‣ Column-store vs. Row-store
• Programming models
‣ support for various scripting languages
‣ support for different frameworks
‣ customized frameworks for cloud
‣ shared-runtimes?

24. misc.
• Misc. (other)
‣ pattern (or template) -based deployment
‣ support for old-style (mature enterprise) platforms
‣ support for multiple target IaaS clouds