The talk mostly focuses on private cloud computing to support Science and High Performance Computing environments, the different architectures to federate cloud infrastructures, the existing challenges for cloud interoperability, and the OpenNebula's vision for the future of existing Grid infrastructures.

1. Workshop GARR - Calcolo e Storage Distribuito
Rome, Italy, November 30th, 2012
Federated Cloud Computing
The OpenNebula Experience
Ignacio M. Llorente
Project Director
© OpenNebula Project. Creative Commons Attribution-NonCommercial-ShareAlike License

2. Contents
Federated Cloud Computing - The OpenNebula Experience!
This presentation is about:
• Infrastructure as a Service Cloud Computing
• Private Clouds in Science and HPC
• Federated Cloud Computing
• Cloud federation in Grid infrastructures
Federated Cloud Computing - The OpenNebula Experience! 2/32

3. What is Cloud Computing?
Provision of IT Capabilities as a Service!
What Who
On-demand End-user
Software as a Service access to any (does not care about hw or sw)
application
Platform for Developer
Platform as a Service building and (no managing of the underlying hw
delivering web & swlayers)
applications
Infrastructure as a Raw computer System Administrator
ﾺ
Service
infrastructure (complete management of the
computer infrastructure)
Physical Infrastructure
Federated Cloud Computing - The OpenNebula Experience! 3/32

4. What is Cloud Computing?
IaaS Clouds for Provision of Virtualized Resources as a Service!
• Raw infrastructure
• Management Console resources
• Simple REST API’s
Manage
Instances
• Pay-as-you-go & elastic capacity
Federated Cloud Computing - The OpenNebula Experience! 4/32

5. What is Cloud Computing?
Different Models of Deployment!
Model Infrastructure Cloud Cases
Owned by a single • Optimize and simplify internal operation
Private
organization and made • SaaS/PaaS support
available only to the • IT consolidation within large organizations
organization (Goverment Clouds, University Clouds…)
Owned by a single • Commercial cloud providers, mostly
organization and made hosting providers, with limited control/
Public
available to other security
organizations over the • Science public clouds by ICT service
Internet centers to enable scientific projects or
experiment with cloud computing
Owned by a single • Telecom cloud providers with premium
Private
Virtual
organization and made solutions with additional control/security
available to other
organization over a
dedicated private network
Composition of two or • Cloudbursting to address peak demands
• Cloud Federation to share infrastructure
Hybrid
more clouds
with partners
• Cloud Aggregation to provide a larger
resource infrastructure
Federated Cloud Computing - The OpenNebula Experience! 5/32

6. What is OpenNebula?
Open Cloud Management Solution for Building and Managing Virtualized Data Centers!
Public
Private
Hybrid
Federated Cloud Computing - The OpenNebula Experience! 6/32

7. What is OpenNebula?
Most Advanced and Flexible, Enterprise-grade IaaS Cloud Manager!
Adaptable
● Customizable and Extensible
Proven
● Many Massive Scale Production Deployments
Powerful
● Most Advanced Enterprise-class
Functionality
No Lock-in
● Platform Independent and Interoperable
Interoperable
● Most popular cloud APIs and standard based
Openness
● Fully open-source, Apache license
Federated Cloud Computing - The OpenNebula Experience! 7/32

8. What is OpenNebula?
User-driven Open Platform for Innovation!
Adopt as innovation
Standards Contribute to
platform or Projects
interoperability tool
standards
Contribute innovative
Adopt
functionality
standards
Distribution
channel
Linux Distributions
Requirements
Feedback
Contributions Adopt
open-source
Industry and Research
Federated Cloud Computing - The OpenNebula Experience! 8/32

9. Private Cloud in Science and HPC
The Pre-cloud Era!
Access
Grid Middleware
LRMS (LSF, PBS, SGE…)
Provision
Federated Cloud Computing - The OpenNebula Experience! 9/32

10. Private Cloud in Science and HPC
OpenNebula as an Infrastructure Tool!
Access Service/Provisioning Decoupling!
• Common interfaces
• Grid integration
Grid Middleware
LRMS (LSF, PBS, SGE…)
• Custom environments
Service
• Dynamic elasticity
Virtual Worker Nodes
• Consolidation of WNs
• Simplified management
Provision
• Physical – Virtual WNs
• Dynamic capacity partitioning
• Faster upgrades
Federated Cloud Computing - The OpenNebula Experience! 10/32

11. Private Cloud in Science and HPC
OpenNebula as an Provisioning Tool!
• Simple Provisioning Interface
Access
• Raw/Appliance VMs
IaaS Interface
Pilot Jobs, SSH…
Service
• Dynamic scalable computing
• Custom access to capacity
• Not only batch workloads
• Not only scientific workloads
Provision
• Improve utilization
• Reduced service management
• Cost efficiency
Federated Cloud Computing - The OpenNebula Experience! 11/32

12. Private Cloud in Science and HPC
Examples: CERN’s lxcloud! http://blog.opennebula.org/?p=620
Goal
• Virtualized HTC Batch Nodes: limited life-time, dynamic capacity
• EC2 Query: CernVM
Deployment Notes
• Custom network integration MAC/IP pinning
• Fast image distribution & boot: BitTorrent + LVM snapshots
• OpenNebula tests up to 20,000 VMs (great feedback!)
Federated Cloud Computing - The OpenNebula Experience! 12/32

13. Private Cloud in Science and HPC
Examples: FermiCloud! http://www-fermicloud.fnal.gov/
Goal
• Scientific stakeholders get access to on-demand VMs
• Developers & integrators of new Grid applications
• MPI and legacy applications
Deployment Notes
• VMs access Fermilab Networking and Storage Services
• OpenNebula + X509 support (contributed back!)
• Other areas: HA, Batch queues look-ahead, cluster on-demand…
Federated Cloud Computing - The OpenNebula Experience! 13/32

14. Private Cloud in Science and HPC
Examples: SARA! https://www.cloud.sara.nl/
Goal
• HPC clusters on-demand: Self service, elastic and fully
configurable HPC systems
• Offer production infrastructure for several engineering and
scientific communities: Bioinformatics, ecology…
Deployment Notes
• Low latency network for HPC
• Extensions in accounting, GUI, firewalling… (contributed back!)
Federated Cloud Computing - The OpenNebula Experience! 14/32

15. Private Cloud in Science and HPC
One of Our Main User Communities!
Supercomputing Centers
Research Centers
Distributed Computing Infrastructures
Federated Cloud Computing - The OpenNebula Experience! 15/32

16. Federated Cloud Computing
Next Step in the Evolution of an Utility!
Utility Utility
Generation Distribution Utility Grid
Federated Cloud Computing - The OpenNebula Experience! 16/32

17. Federated Cloud Computing
Benefits of Federation!
Scalability
● Cloudbursting to address peak demands
Collaboration
● Sharing of infrastructure between partners
Multi-site Deployments
● Infrastructure aggregation across distributed data centers
Reliability
● Fault tolerance architectures across sites
Performance
● Deployment of services closer to end users
Cost
● Dynamic placement to reduce the overall infrastructure cost
Energy Consumption
● Minimize energy consumption
Federated Cloud Computing - The OpenNebula Experience! 17/32

18. Federated Cloud Computing
The Cluster Computing Use Case!
Scalability
Performance
Cost
Fault tolerance
Federated Cloud Computing - The OpenNebula Experience! 18/32

19. Federated Cloud Computing
Different Levels of Control, Monitoring, Cross-site Functionality and Security!
Private Cloud
Partner Company
Cloud Cloud
Cloud
OS
Public (Broker)
Cloud
Public
Public
Public
Cloud
1
Cloud
2
Cloud
3
Federated Cloud Computing - The OpenNebula Experience! 19/32

20. Federated Cloud Computing
Loosely Coupled Federation - Cloudbursting!
Federation with a Cloud without Interoperation Support
Corporate
cloud
Cloud
OS
Public
Public
Cloud
1
Cloud
2
Local data center
Aim • Meet peak demands
• Basic operations over VMs (start, shutdown, restart…)
Control
• Different instance types
Monitoring & • Basic virtual resource monitoring (resource consumption…)
Accounting
Cross-site • None
Security • Single account representing the organization
Federated Cloud Computing - The OpenNebula Experience! 20/32

21. Federated Cloud Computing
Dynamic Combination of Local with Remote Cloud Resources!
Service Level Agreement
• SLO 1: Average_CPU_Utilization > 75%
• Action: Execute remote WN
Grid Services
Federated Cloud Computing - The OpenNebula Experience! 21/32

22. Federated Cloud Computing
Partially Coupled Federation - Aggregated Cloud Architecture!
!
Federation with a Cloud with Partial Interoperation Support
Corporate Corporate
cloud 1 cloud 2
Cloud
OS
Cloud
OS
Local data center Local data center
Aim • Sharing of resources to meet peak demands
• Advanced operations over VMs (live migration…)
Control
• VM location and affinity constraints
Monitoring & • Advanced virtual resource monitoring (energy consumption,
Accounting VM placement…)
• Virtual networks
Cross-site
• Virtual storage
Security • Framework agreement
Federated Cloud Computing - The OpenNebula Experience! 22/32

23. Federated Cloud Computing
Cloud Broker Architecture!
Building Service Testbeds on FIRE
Agreement 257386 (2010-2013) Design, build and operate a multi-site cloud-based facility to
New Infrastructure Paradigms support research across applications, services and systems
and Experimental Facilities targeting services research community on Future Internet
www.BonFIRE-Project.eu
Source: BonFIRE Project
Federated Cloud Computing - The OpenNebula Experience! 23/32

24. Federated Cloud Computing
Tightly Coupled Federation - Multi-tier Architecture!
Federation with a Cloud with Advanced Interoperation Support
Corporate
cloud
Cloud
OS
Cloud Cloud
site 1 site 2
Cloud
OS
Cloud
OS
Local data center Local data center
Aim • Very large scale or geographically distributed data centers
• Placement on specific physical resources
Control
• Same instance types
Monitoring & • Physical resource consumption
Accounting
• Live migration
Cross-site
• High availability
Security • User space sharing
Federated Cloud Computing - The OpenNebula Experience! 24/32

25. Federated Cloud Computing
Multi-tier Cloud Architecture!
CloudFront
• Portal
• Cloud API (EC2, OCCI)
User • Global AuthN
OpenNebula Zone OpenNebula Zone
• Multi-tenancy • Multi-tier arch.
• Runs Instances • Scalable
• Stores Images • Isolation
• Networking • Multiple-site
• Authorization support
Federated Cloud Computing - The OpenNebula Experience! 25/32

26. Federated Cloud Computing
Challenges for Interoperability!
Transparent Combination of Local Resources with Cloud Resources
1. Management Interfaces 2. Management Interfaces
for Virtual Workloads for Data Elements
VM
VM VM VM VM
4. Contextualization
3. VM Image Description
5. Cross-site Networking and Instance Type
and Latency 6. Common framework for
security and QoS
Federated Cloud Computing - The OpenNebula Experience! 26/32

27. Federated Cloud Computing
Leveraging Existing Standards and Implementing Interoperation!
Standardization
● Implement standards
● Integrate with standards
Which Standard?
● Different de jure standards
● Several de facto standards
Interoperation
● Implement adaptors
● Use transformers
Federated Cloud Computing - The OpenNebula Experience! 27/32

28. Cloud Federation in Grid Infrastructures
Grid and Cloud as Complementary Computing Models!
Usage Grids Usage Clouds
 Job Processing  Raw infrastructure
 Big Batch System  Elasticity & Pay-per-use
 File Sharing Services  Simple Web Interface
Achievements Achievements
 Federation of Resources
 Agile Infrastructures
 VO Concept
 IT is another Utility
But…
 User experience
But…
 Interoperability
 Complexity
 Federation
Resource Sharing Resource Management
Scientific Applications Customize Environments
Uniform Security Flexibility & Simplicity
Federated Cloud Computing - The OpenNebula Experience! 28/32

29. Cloud Federation in Grid Infrastructures
The OpenNebula Vision for Grid Sites!
• Batch Job Processing • Industry Applications
• Custom Execution Environments • Other WMS (pilots)
Access
• Grid Service Integration • Complete Services (cluster)
Grid Middleware IaaS Interface
LRMS (LSF, PBS…)
Service
Virtual CE, WN… Other (web, mail...) Raw machines
Provision
Grid Site
Federated Cloud Computing - The OpenNebula Experience!
External Providers
29/32

30. Cloud Federation in Grid Infrastructures
The OpenNebula Vision for Grid Infrastructures !
Grid Services
MarketPlace
• Federation facilities
• Security • Sharing existing VM images
• Grid specific services • Registry of metadata
• Image are kept elsewhere
• Supports trust
Appliance Repo
• Storage VM images
• Distributed
• Multi-protocol
Grid Services Cloud API Grid Services Cloud API
Cloud/Grid Site Cloud/Grid Site
Federated Cloud Computing - The OpenNebula Experience! 30/32

31. Join our Growing Community!
OpenNebula is Driven by User Needs !
How to contribute
● Join our mailing list
● Follow the development at dev.opennebula.org
● Contributions of by users: RIM, Akamai, Logica, FermiLab, SARA, Terradue,
China Mobile… (approx. 100 listed at http://www.opennebula.org/
about:contributors)
● 500 validated users at dev.opennebula
Ecosystem projects
● OpenNebula hosts an ecosystem catalog
● Promote and discuss ecosystem projects in our ecosystem mailing list
IRC Channel
● #opennebula on irc.freenode.net
Federated Cloud Computing - The OpenNebula Experience! 31/32

32. Questions?
We Will Be Happy to Answer Any Question !
CloudPlan.org @imllorente
Federated Cloud Computing - The OpenNebula Experience! 32/32